Physical AI Developer Stack: A Practical Map from ROS 2 to Robot Learning

The short answer

A physical AI stack is not one model or one robotics framework. It is the complete path from sensors and task definitions to commands, physical action, measured outcomes, and recovery. The right developer stack is the smallest set of tools that can close that loop on one declared robot and task.

For many projects, the layers look like this:

Robot and sensor interfaces expose cameras, joint state, force, gripper state, and actuator commands.
Middleware and lifecycle tools move data between components and manage startup, health, and shutdown.
Control turns desired motion into commands the hardware can execute.
Planning and perception estimate the scene and select feasible motion or task steps.
Simulation tests software, generates experience, or trains policies under controlled conditions.
Data tooling records synchronized observations, actions, task context, outcomes, and human involvement.
Learning trains a perception model, policy, reward model, world model, or other adaptive component.
Evaluation measures all attempts, failures, latency, interventions, recovery, and accepted output.
Deployment and operations package the validated system, monitor it, control changes, and preserve rollback.

The popular tools in this guide occupy different layers. ROS 2 is integration infrastructure, not a learned policy. ros2_control is a hardware and controller framework, not a task planner. MoveIt 2 plans manipulation, but does not prove the perception input or grasp target is correct. Gazebo, MuJoCo, and Isaac Lab support different simulation and learning workflows. LeRobot connects datasets, policies, supported hardware, training, evaluation, and rollouts, but it is not a complete safety system. The embodied AI guide explains how body, sensors, actions, and environment define the agent that this stack must support.

That separation matters. Teams lose time when they ask one framework to solve a problem that belongs to another layer, or when they adopt a large learning stack before they can measure a simple physical baseline.

After choosing the layers, use the open-source robotics project health audit to check releases, support, governance, artifacts, tests, compatibility, and rollback before adopting a dependency.

The stack at a glance

Layer	Main question	Common tools in this map	What the tool does not prove
Hardware interface	Can software read sensors and command the robot?	Vendor drivers, ROS 2 packages, ros2_control hardware interfaces	Correct calibration, safe motion, or task success
Middleware	Can components communicate and be operated coherently?	ROS 2	Real-time deadlines, autonomy, or reliability of every node
Low-level control	Can desired state become stable physical motion?	ros2_control, vendor controllers	Good task selection or collision-free behavior
Motion planning	Can the robot find a feasible manipulation path?	MoveIt 2	Correct scene understanding or successful contact
System simulation	Can the integrated robot software be exercised virtually?	Gazebo	Physical equivalence or sim-to-real transfer
Physics and learning experiments	Can dynamics, control, or policies be studied at scale?	MuJoCo, Isaac Lab	Production safety or physical performance
Data and policy workflow	Can episodes be recorded, trained, evaluated, and replayed?	LeRobot	Fair benchmarks, model validity, or unattended operation
Cross-robot research data	Can models learn from multiple embodiments?	Open X-Embodiment and related datasets	Plug-and-play transfer to a new robot
Evaluation	Did the complete system succeed under a declared protocol?	Project-specific tests, logs, hardware trials	General capability outside the tested distribution
Operations	Can a validated version run, be monitored, and be rolled back?	Containers, service managers, telemetry, release controls	That the underlying policy is correct or safe

Start with a task contract, not a tool list

Before choosing software, define one physical task tightly enough to evaluate.

A useful task contract states:

robot, end effector, sensors, and compute;
starting state and permitted variation;
object set, workspace, and environmental limits;
required output and quality tolerance;
time limit and throughput target;
collision, drop, damage, and safety failure rules;
human demonstration, teleoperation, intervention, and reset policy;
number of development and held-out evaluation trials;
evidence needed before moving from simulation to hardware.

Without this contract, a team can integrate cameras, planners, language models, and dashboards while remaining unable to answer whether the robot works.

A narrow example might be: pick one of ten known cartons from three randomized poses and place it inside a marked tote, with no collision, drop, manual adjustment, or hidden reset, within 20 seconds. That definition is more useful for architecture decisions than “build an intelligent warehouse robot.”

The task contract also reveals whether learning is necessary. If object geometry, poses, and fixtures are stable, conventional perception and planning may establish a stronger baseline than a general policy. If the task varies across objects, language instructions, scenes, or contact conditions, a learned component may add value. The comparison should still hold the robot, test states, success rule, and trial count fixed.

Layer 1: hardware interfaces and timing

The first layer is not AI. It is an accurate, timed connection to the machine.

A hardware interface should expose the state needed for control and evidence:

joint position, velocity, and effort where available;
commanded and measured end-effector state;
camera frames with stable timestamps and calibration;
force, torque, tactile, range, and inertial data where relevant;
controller mode, protective stops, faults, and limit states;
command acceptance, execution timestamps, and dropped messages.

The distinction between commanded action and measured motion is essential. A policy can issue the correct command while the actuator saturates, a safety controller clips the motion, a packet arrives late, or contact prevents execution. Record both sides of that boundary.

ROS 2 as the integration spine

The ROS 2 documentation organizes a broad ecosystem around nodes, topics, services, actions, parameters, packages, launch, security, testing, tracing, and distribution-specific guidance. That makes ROS 2 a common integration spine for robot software.

ROS 2 is useful when a project needs multiple processes or devices to exchange typed information, discover one another, expose long-running actions, and start in a controlled configuration. It also supplies conventions that reduce custom transport code.

It does not automatically guarantee:

a real-time deadline for every callback and network path;
deterministic startup or recovery for an application;
synchronized sensors without an explicit clock and buffering policy;
safe authority over actuators;
compatibility among arbitrary community packages;
production support for an unpinned dependency graph.

Treat message definitions and node boundaries as system contracts. State the units, coordinate frame, timestamp source, valid range, timeout, stale-data behavior, and authority of every command that can move hardware.

ros2_control at the control boundary

The ros2_control documentation covers hardware interfaces, controller management, controller chaining, and related packages for connecting ROS 2 software to robot control. It is the layer where abstract command and state interfaces meet a specific device or simulator.

Use it to make hardware access explicit and to separate controllers from vendor-specific device handling where the supported interfaces fit. Still verify the complete timing path. A nominal control frequency does not prove that observations, policy inference, middleware, controller updates, and actuators meet the same deadline under load.

For every control loop, record target frequency, achieved frequency, missed deadlines, tail latency, saturation, limit handling, watchdog behavior, and the safe state after stale commands.

Layer 2: motion planning and task execution

A robot that can move joints still needs to decide where and how to move.

MoveIt 2 provides manipulation-oriented tooling around motion planning, collision checking, kinematics, planning scenes, trajectories, and execution within the ROS 2 ecosystem. It is a strong fit when the problem includes robot arms, known geometry, collision-aware motion, and structured manipulation.

A successful plan is only one part of the evidence. Planning can fail or mislead when:

the scene model is stale or geometrically wrong;
camera-to-robot calibration is inaccurate;
an object pose has high uncertainty;
the grasp point is physically poor;
contact, cable routing, compliance, or payload differs from the model;
the trajectory is feasible in geometry but too slow or brittle for the process;
execution diverges from the planned path.

Log planning and execution separately. Useful fields include planning time, number of attempts, selected planner, path length, clearance, execution duration, tracking error, replans, controller aborts, contact failures, and recovery outcome.

A learned model can propose a grasp, goal pose, subtask, or full action sequence. It should not silently bypass geometric, dynamic, or safety checks. Define which component has authority to command motion and which checks can veto it.

Layer 3: simulation, with three different goals

“Use simulation” is incomplete advice. Teams use simulation for at least three different purposes:

system integration, such as exercising ROS nodes, sensors, frames, controllers, and workflows;
physics and control experiments, such as testing dynamics or model-based control;
large-scale policy learning, such as running many environments for reinforcement learning or domain randomization.

The best simulator depends on which purpose dominates. Our hardware-transfer review explains what domain randomization can address and what still needs calibration, adaptation, and physical evaluation.

Gazebo for ROS-integrated system simulation

The Gazebo Harmonic documentation covers a general robotics simulator with worlds, models, sensors, plugins, and physics. Its ROS 2 integration documentation describes bridges and interfaces between the two ecosystems.

Gazebo is a natural choice when the main question is whether an integrated ROS 2 system behaves coherently in a simulated world. It can support sensor streams, robot descriptions, controllers, navigation, and multi-component testing before hardware is available.

Its value depends on the fidelity of the robot, sensors, contacts, materials, delays, and environment assets. A simulated driver responding correctly does not prove that the physical driver handles disconnects, encoder noise, thermal limits, or protective stops.

MuJoCo for compact physics and control research

MuJoCo is a physics engine designed for model-based optimization, control, and robotics research. It is often useful when the experiment needs fast dynamics, contact, a concise model description, and direct programmatic control of the simulation loop.

Choosing MuJoCo does not make a benchmark automatically comparable with a Gazebo, Isaac, or physical result. Assets, integrators, contact parameters, observation pipelines, action rates, reset states, and success functions can all change the outcome.

Isaac Lab for large-scale robot learning

Isaac Lab is a robot-learning framework built on NVIDIA simulation infrastructure. Its documentation covers environment design, assets, sensors, controllers, reinforcement-learning workflows, imitation learning, and related learning integrations.

It is a stronger fit when GPU-parallel simulation and robot-learning experiments are central requirements. That power adds operational complexity. Teams should pin the simulator, drivers, learning library, asset versions, and hardware assumptions, then record throughput and reproducibility rather than reporting only a final policy score.

A simulator selection rule

Choose the simulator that minimizes uncertainty in the next decision.

If the immediate risk is ROS integration, frames, messages, and system behavior, start with Gazebo.
If the risk is dynamics, contact, or control experimentation in a compact loop, consider MuJoCo.
If the risk is learning throughput and large randomized training, consider Isaac Lab.
If two simulators are required, define the source of truth for models, units, frames, controller settings, and evaluation.

Do not infer physical task success from any of them. Move to hardware with staged limits, low energy, safe fixtures, explicit abort conditions, and a physical evaluation set that was not used to tune the simulator.

Layer 4: data and robot-learning workflows

A learned policy requires more than a folder of videos. A useful episode connects observations, robot state, actions, task context, timing, outcomes, and human involvement.

LeRobot brings several of these workflows into one open ecosystem, including supported robot interfaces, teleoperation, recording, datasets, policy training, evaluation integrations, and real-robot rollouts. It can reduce the amount of custom glue needed for a learning experiment.

That integration should not be confused with proof that every model, robot, and dataset combination is valid. Before adopting a policy, verify:

the exact robot and camera configuration it expects;
observation names, shapes, normalization, and timing;
action representation, limits, horizon, and command rate;
dataset provenance, license, task distribution, and failures;
checkpoint source and training recipe;
memory, latency, and accelerator requirements;
physical evaluation protocol and intervention rules.

Our LeRobot practical guide covers the framework’s current model, reward, benchmark, rollout, intervention, and release boundaries in more detail. The robot training data guide explains why teleoperation, human video, simulation, interventions, and online interaction are different evidence classes.

Cross-embodiment data does not erase the target robot

The Open X-Embodiment project assembled datasets across many robot embodiments and studied models intended to learn from that diversity. It is important evidence that common structure can be useful across robots.

It does not establish that a policy can be placed on a new robot without adaptation. The target system can differ in kinematics, cameras, gripper, action space, control frequency, payload, latency, calibration, and safety limits. Preserve the source data, alignment data, fine-tuning, modified components, and target-hardware evaluation as separate records.

Layer 5: decide what learning is allowed to control

A physical AI system can use learning at many points:

object detection or segmentation;
depth, pose, contact, or state estimation;
grasp or goal proposal;
task planning from language;
action policy or trajectory generation;
reward or success estimation;
anomaly detection and intervention prediction;
recovery selection.

These positions carry different risks. A learned detector that proposes a bounding box can be checked by geometry and confidence. A policy that outputs high-rate joint commands has much more direct authority. A learned reward model can corrupt training or evaluation even if it never moves the robot.

For each learned component, document:

its inputs and outputs;
the action or decision it can influence;
independent checks before that decision reaches hardware;
latency and stale-output handling;
confidence, abstention, and fallback behavior;
training and evaluation distribution;
failure logging and rollback.

Do not begin with a VLA because it is fashionable. Begin with the simplest measurable baseline. Add language or a general policy when the requirement genuinely includes instruction variation, semantic reasoning, or broad object and scene diversity. Then compare the learned system against the baseline under the same physical trials.

For the model boundary itself, our vision-language-action model guide explains how visual observations and instructions become discrete tokens, continuous commands, or action chunks, and what a physical evaluation must report.

Layer 6: evaluation is part of the architecture

Evaluation cannot be added after the stack is assembled. The system must expose enough state to reconstruct every attempt.

At minimum, preserve:

task and initial-state identifier;
robot, tool, calibration, software, firmware, dataset, and checkpoint versions;
observation and command timestamps;
planned and executed motion;
success, partial success, failure, abort, and exclusion reason;
collision, drop, damage, limit, and safety-stop events;
teleoperation, intervention, reset, and recovery;
latency by stage and deadline misses;
wall-clock and active execution time;
output quality and process acceptance.

Our robot demo evidence checklist provides a compact protocol for continuity, attempt counts, speed, variation, control modes, failures, and operating evidence. The benchmark success-rate guide explains why a percentage needs trials, task distribution, uncertainty, checkpoint selection, and all-run accounting.

A sound progression is:

unit tests for transforms, limits, parsing, and fallback logic;
deterministic replay on recorded observations;
simulation tests with declared variation;
hardware tests at reduced energy and bounded workspace;
repeated task trials across held-out conditions;
recovery and fault-injection tests;
supervised pilot operation;
recurring operational measurement.

Passing one stage does not imply the next. Keep simulation and physical results separate. Keep assisted completion and autonomous completion separate. Keep a public software release and a production deployment separate.

Layer 7: deployment, monitoring, and change control

A model that succeeded during development can fail after a camera moves, firmware changes, an object supplier changes packaging, or a dependency modifies preprocessing.

Production packaging should preserve the complete system, not only the model weights:

operating-system and dependency versions;
container or image identifier where used;
robot and sensor firmware;
device permissions and network assumptions;
calibration and robot description;
controller and safety parameters;
model, normalization, tokenizer, and preprocessing;
task configuration and allowed operating domain;
startup order, health checks, watchdogs, and safe shutdown;
telemetry schema and retention;
rollback artifact and acceptance test.

Monitor both software health and physical outcomes. CPU, GPU, memory, temperature, queue delay, dropped frames, inference latency, and node restarts matter. So do intervention, recovery, accepted output, damage, safety stops, maintenance, and drift by object or scene.

Every change that can affect action needs a declared validation scope. A camera driver update may alter timestamps or color conversion. A model update may change behavior on familiar objects. A new end effector changes kinematics, contact, payload, and training-data alignment. Run the appropriate regression set before deployment and retain the previous known-good version.

Three practical starter stacks

These are architectural starting points, not universal prescriptions.

1. Structured manipulation prototype

Use when the robot arm, workspace, object family, and task are bounded.

ROS 2 for component integration and operations.
ros2_control or a supported vendor interface for hardware control.
MoveIt 2 for kinematics, collision checking, and motion planning.
Gazebo for early system integration where suitable models exist.
Conventional or learned perception for object and pose estimation.
A project-specific trial harness for success, timing, and recovery.

Start without an end-to-end learned policy. Establish the geometric and control baseline first. Add learning at the failure point that needs adaptation.

2. Real-robot imitation-learning project

Use when demonstrations and repeated physical rollouts are central.

A supported robot and teleoperation interface.
LeRobot or an equivalent data and policy workflow.
Synchronized cameras, robot state, actions, task labels, outcomes, and interventions.
A narrow policy and a held-out physical test set.
External safety limits and supervised rollout controls.
Versioned datasets, checkpoints, calibration, and evaluation logs.

The key cost is not only training compute. It is robot availability, demonstrations, resets, maintenance, supervision, and safe failure collection.

3. Simulation-heavy reinforcement-learning project

Use when policy learning requires large numbers of interactions.

Isaac Lab or MuJoCo, selected by the required learning and dynamics workflow.
Versioned assets, randomization ranges, sensors, rewards, termination rules, and seeds.
A training system that preserves checkpoints and evaluation-only seeds.
A separate physical interface and staged transfer plan.
Matched simulation and hardware metrics, reported separately.

Do not tune indefinitely against the same physical evaluation set. Preserve held-out hardware conditions and count every adaptation cycle.

What not to combine into one claim

A credible developer report keeps these distinctions visible:

Evidence	Supports	Does not support by itself
ROS 2 nodes communicate in a lab	Software integration	Deadline compliance or task success
MoveIt finds a collision-free path	Geometric planning under the current scene model	Correct perception, grasp success, or safe contact
A policy scores well in simulation	Performance under the declared simulator and protocol	Physical reliability or production readiness
A framework supports a robot	An integration path exists	Every policy works on that robot
A dataset includes many embodiments	Broad source diversity	Zero-shot transfer to a new embodiment
One hardware demonstration succeeds	The behavior occurred once	Repeatability, autonomy, or useful throughput
A supervised pilot completes work	The complete team can deliver under pilot conditions	Unattended operation or scalable economics
A release is containerized	The artifact can be packaged reproducibly	That the behavior is correct, safe, or approved

A selection checklist

Choose each layer by answering a concrete question.

Hardware and middleware

Are maintained drivers available for the exact robot, sensors, and operating-system distribution?
What clock, units, frames, rates, and timeout behavior do interfaces use?
Can commands be traced from policy output to measured motion?
What happens after a disconnect, stale command, node crash, or partial restart?

Planning and control

Does the task need collision-aware motion planning, force control, compliance, or only a fixed trajectory?
Which component owns actuator authority?
Are limits enforced outside the learned component?
Can planning and execution failures be classified separately?

Simulation

Is the goal system integration, dynamics research, or parallel learning?
Which physical effects must be credible for the decision being made?
How are assets, randomization, rewards, resets, and seeds versioned?
What physical experiment will test transfer?

Data and learning

Are observation and action schemas compatible with the target robot?
Are provenance, licenses, failures, interventions, and resets preserved?
What simple baseline must the learned system beat?
Is the evaluation set separated from training and online adaptation?

Deployment

Can the complete artifact be reproduced and rolled back?
Which telemetry detects timing, data, model, controller, and physical drift?
What change requires revalidation?
Who can stop, recover, approve, and restart the system?

Verdict

The physical AI developer stack is best understood as a set of explicit boundaries.

ROS 2 connects components. ros2_control connects control software to devices. MoveIt 2 supplies manipulation planning. Gazebo, MuJoCo, and Isaac Lab serve different simulation and learning goals. LeRobot can connect real-robot data, policies, evaluation, and rollout workflows. Cross-embodiment datasets can improve reusable learning, while target hardware still requires adaptation and validation.

The winning stack is not the one with the most tools. It is the one that makes one physical task measurable, keeps authority and timing explicit, records every failure and human intervention, and can reproduce the exact system that passed the physical test.

Frequently asked questions

What software do I need to start a physical AI project?

Start with the smallest stack that closes one real perception-to-action loop. A common path uses ROS 2 for communication and lifecycle management, ros2_control for hardware-facing control, Gazebo or MuJoCo for simulation, MoveIt 2 for manipulation planning, and a separate robot-learning framework such as LeRobot or Isaac Lab when learned policies are actually needed.

Is ROS 2 a robot AI framework?

ROS 2 is primarily middleware and a software integration ecosystem. It connects nodes, messages, services, actions, devices, visualization, and operational tooling. It does not by itself train a policy, define task success, guarantee real-time behavior, or make a robot autonomous.

Should I use Gazebo, MuJoCo, or Isaac Lab?

Choose by the experiment. Gazebo is a strong fit for ROS-integrated system simulation. MuJoCo is a compact physics engine widely used for model-based control and learning experiments. Isaac Lab is built around large-scale robot-learning workflows on NVIDIA simulation infrastructure. None removes the need for physical validation.

Do I need a vision-language-action model?

Usually not for the first working prototype. A fixed task can often be solved and measured with conventional perception, planning, control, and a narrow learned component. Add a VLA only when language, scene variation, or cross-task behavior is central to the requirement and you can evaluate its added value against a simpler baseline.

Can a simulation success rate predict real robot performance?

Not by itself. Simulation can test logic, train policies, and expose failures under controlled variation, but the physical system adds sensor artifacts, calibration error, latency, contact, wear, safety controls, and human intervention. Keep simulation and physical results in separate tables.

What should I version in a robot-learning project?

Version the robot and end effector, calibration, sensor configuration, task definition, software and firmware, dataset, model checkpoint, simulator and assets, controller settings, safety limits, evaluation protocol, and deployment configuration. A model hash alone is not enough to reproduce the system.