Can Robot Policies Learn from Failure? What Q-Planning Actually Proves

The short answer

Q-Planning shows that a robot can improve action selection from both successful and failed deployment attempts without changing its large behavior-cloning policy. A smaller value model learns which policy proposals are more likely to succeed, then guides the next attempt.

The August 2026 preprint Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning reports gains across LIBERO, RoboTwin, and two physical bimanual tasks. Its strongest physical results are concrete:

• stack-cups rose from a 40% behavior-cloning baseline to 90%;
• insert-wallet rose from 25% to 80%;
• each task began with 100 demonstrations;
• each self-improvement iteration collected 20 new episodes per task;
• five iterations produced 100 online episodes per task;
• the behavior-cloning policy stayed frozen throughout.

This is meaningful evidence that failed attempts can improve a separately trained value model. It is not evidence that a robot can discover arbitrary new behaviors, improve without supervision, or fail safely in an open environment. People reset the scenes and labeled success after every physical episode. The study reports task outcomes, not unsafe-event or damage rates.

What actually learns

Q-Planning separates the system into an actor and a critic.

The actor is the large FastWAM behavior-cloning policy. It observes camera images and a language instruction, then proposes chunks of future joint commands. It is trained to imitate successful demonstrations and remains frozen during the Q-Planning experiments.

The critic is a separate Q-function with about one billion parameters. It has its own DINOv2 visual encoder, T5 language encoder, transformer decoder, and categorical value head. It estimates how promising a proposed action chunk is from the current observation and instruction.

That separation creates the central asymmetry:

Component	Learns from successful demonstrations	Learns from failed rollouts	Updated online
Frozen behavior-cloning policy	Yes	No	No
Q-function	Yes	Yes	Yes

A failed episode is not imitated. It becomes evidence that the observed state and executed action chunks did not reach the terminal reward. The Q-function can use that zero-return trajectory to lower its estimate of similar choices.

The policy’s weights do not improve. The combined system improves because the critic changes which policy proposals receive the most weight.

The inference loop

At each planning step, the frozen policy samples several candidate action chunks. The Q-function scores every candidate. The system then executes a softmax-weighted average, giving more influence to candidates with higher predicted value.

The reported configurations use:

• 64 candidates per planning step on LIBERO;
• 32 candidates on RoboTwin;
• three flow-matching denoising steps for candidate generation;
• a 32-step action horizon in simulation;
• repeated replanning after only part of each chunk has executed.

This is not a long tree search. It is one value-guided selection step over actions the existing policy can already propose.

That boundary matters. The authors state that Q-Planning cannot learn a behavior that the frozen policy never produces with meaningful probability. It can amplify a weak but available behavior. It cannot bootstrap a missing skill from zero.

Our long-horizon robot control guide covers the wider stack of planning, memory, low-level execution, and adaptation. Q-Planning addresses one layer in that stack: value-guided choice among short policy-generated action chunks. The separate VLA control interfaces guide maps how prompt authority, spatial grounding, behavior intent, and memory shape the information available before that choice.

How failure enters the update

The self-improvement loop has two phases:

1. Run the current Q-guided system and collect complete successful and failed episodes.
2. Add those episodes to a replay buffer and update only the Q-function.

Each update minibatch is split between the original demonstration data and accumulated online rollouts. The paper uses categorical value regression and action chunking to make sparse terminal rewards more stable. A successful episode receives a terminal reward of one. A failed episode receives no terminal reward.

The mechanism needs a task-level success detector. Simulation provides an environment success bit. On the physical robot, a person labels each episode. Open-ended operation would need another trustworthy reward or success system.

Failure reuse is therefore real but bounded. The method does not infer rich failure causes from language. It learns a value distinction from trajectories and binary outcomes inside predefined tasks.

What the simulation results establish

The authors evaluate the frozen FastWAM policy, Q-guided action selection without online updates, and Q-Planning after ten self-improvement iterations.

Benchmark	Frozen FastWAM	Q-Planning before online updates	Q-Planning after online updates
LIBERO-Spatial	90.5%	91.5%	98.5%
LIBERO-Object	100.0%	99.5%	100.0%
LIBERO-Goal	97.0%	99.0%	99.0%
LIBERO-10	90.0%	93.0%	99.0%
RoboTwin, 47 tasks	83.2%	83.8%	91.4%

The reported evaluation uses 20 episodes per task. The RoboTwin row covers 47 of the benchmark’s tasks for which the authors had working evaluations, not the full suite. LIBERO uses a 540-step episode cap, which the paper says is shorter than the protocol used for FastWAM’s original published result.

The online loop collects 100 episodes per task in each simulation iteration, followed by 200 Q-only gradient steps. These are substantial controlled rollout budgets. The result should not be described as learning from a few failures.

The comparison against Best-of-N, filtered supervised fine-tuning, IBRL, DSRL, and DAWR is useful because the methods receive the same online episode budget in the reported LIBERO-10 study. Q-Planning rose from 93% to 99%, while the alternatives plateaued, oscillated, or degraded under the authors’ implementation.

It remains an author-run comparison on one frozen policy family. Code availability, independent reproduction, and sensitivity to alternative policy backbones are still open evidence questions.

What the physical study establishes

The physical setup uses two table-mounted six-degree-of-freedom YAM arms, three RGB cameras, joint-space commands, and one RTX 5090 for inference. The two tasks are:

• stack-cups, stacking plastic cups;
• insert-wallet, placing a credit card into a wallet slot.

Initial object positions vary within the demonstration distribution. The wallet stays fixed while the card’s starting slot changes. Each task starts from 100 demonstrations.

The first Q-guided iteration already improves over the unguided behavior-cloning baseline:

Task	Frozen behavior-cloning baseline	Q-guided starting point	After five Q-only update iterations	Success-only fine-tuning after five iterations
Stack-cups	40%	65%	90%	55%
Insert-wallet	25%	40%	80%	30%

Each point is measured on the 20 collection episodes for that iteration. That makes a five-percentage-point change equal to one episode. The curves do not report a separate, larger final holdout evaluation after learning ends.

The study’s phrase “no human intervention” needs careful interpretation. The robot was not teleoperated during action execution. Human input was still required for:

• scene resets between episodes;
• one success or failure label per episode;
• experimental setup and supervision.

This is autonomous execution inside a human-maintained training loop. It is not an unattended deployment.

Why the safety budget is missing

Online robot learning intentionally creates attempts that may fail. A complete deployment claim therefore needs more than a success curve.

The Q-Planning paper does not report:

• collisions by severity;
• emergency stops;
• unsafe motion or near misses;
• object, gripper, or arm damage;
• human exposure during failed attempts;
• reset time and physical recovery work;
• a stopping rule for risky exploration;
• whether any attempts were excluded from the plotted curves.

The tasks use plastic cups, a card, and a wallet in a controlled lab. That limits consequence, but it does not turn success-rate improvement into a general safety result.

Our adaptive robot safety case guide explains why task completion, safe completion, monitor engagement, recovery, and change control need separate evidence. A self-improving controller also needs revalidation after every update, even when the base policy is frozen, because the executed action distribution has changed.

What freezing the policy does and does not protect

Freezing the behavior-cloning policy has two practical advantages.

First, failed Q-function updates cannot overwrite the policy’s parameters. Second, training a smaller critic can cost less than applying online reinforcement learning to the complete multi-billion-parameter policy.

It does not guarantee safe behavior. The Q-function still changes action selection. A poorly calibrated critic can overvalue an unsafe candidate, and averaging several plausible chunks can produce an action that was not itself sampled. Parameter separation protects the policy weights, not the physical environment.

The same distinction appears in robot intervention and handoff evidence. A system needs a rule for detecting unsupported execution and transferring control. Q-Planning reports no runtime handoff monitor, operator takeover path, or intervention-rate measurement during its physical rollouts.

Latency is system-specific

The authors report a 400-millisecond planning step for the RoboTwin configuration on an NVIDIA L40S, within a 960-millisecond replanning budget. The real robot runs inference on an RTX 5090 and executes a one-second action chunk before replanning.

The LIBERO configuration with 64 candidates takes 640 milliseconds under strict evaluation settings, while the paper lists a 333-millisecond execution budget for ten actions at 30 Hz. The authors describe that configuration as latency-neutral relative to the slower ten-step FastWAM baseline and report that a 16-candidate version fits the budget with comparable success.

This is a useful engineering disclosure. It also shows why policy latency, planning latency, action horizon, and physical control rate must be reported together. A method can improve benchmark success while missing a nominal real-time deadline in one configuration.

Evidence boundaries

Claim	Evidence class	Confidence	Boundary
A separate Q-function can guide a frozen VLA policy	Author-reported simulation and physical study	High for the tested implementation	One FastWAM-based stack and selected tasks
Failed rollouts improved later action selection	Matched online experiments	Medium-high	Binary terminal labels, controlled tasks, and no independent replication
The physical system improved from 40% to 90% and 25% to 80%	Twenty-episode iteration points on two lab tasks	Medium-high	One trial changes the rate by five points, with no separate final holdout
No human intervention was required	Supported only for teleoperation during rollout execution	High with qualification	People reset scenes and supplied terminal labels
Freezing the policy makes online learning safe	Not supported	High	No unsafe-event, collision, stop, or damage evaluation
Q-Planning can create new skills outside the policy’s support	Not supported	High	The authors explicitly identify the frozen policy’s proposal distribution as the exploration boundary
The approach is ready for production self-improvement	Not supported	High	No customer deployment, unattended duration, safety case, or change-control evidence

A disclosure checklist for self-improving robot policies

Future reports should publish:

1. Base policy state: architecture, checkpoint, training data, and which parameters remain frozen.
2. Proposal diversity: number of candidates and evidence that useful actions appear in the proposal distribution.
3. Value training: reward definition, replay composition, bootstrap target, calibration, and update compute.
4. Online budget: episodes, attempts, resets, gradient steps, and elapsed robot time per iteration.
5. Human support: demonstrations, labels, monitoring, takeovers, resets, repairs, and exclusions.
6. Physical denominator: task-level attempts for every plotted point and a final held-out evaluation.
7. Failure accounting: failure classes, repeated failures, recovery path, and whether failed data remains useful after the scene changes.
8. Safety accounting: contacts, near misses, emergency stops, damage, operator exposure, and stopping criteria.
9. Latency: complete observation-to-action distribution on the deployed computer and control loop.
10. Change control: validation before and after every update, rollback conditions, and retained model versions.
11. Generality: multiple policies, robots, sensors, tasks, and out-of-distribution conditions.
12. Artifacts: code, weights, logs, splits, and evaluation harnesses sufficient for independent reproduction.

For success-rate interpretation, use our robot benchmark evidence checklist. Percentages should remain attached to their attempts, task distribution, checkpoint, human-support rules, and failure protocol.

Verdict

Q-Planning presents a clear alternative to updating an entire robot foundation model online. It keeps the expensive imitation policy fixed, learns a separate value function from every outcome, and uses that value function to choose among policy proposals.

The physical result is promising because it discloses the tasks, demonstrations, online episode budget, human-label requirement, and per-iteration denominators. The evidence supports a precise conclusion:

A frozen behavior-cloning policy can become more effective when a separately updated value model learns from successful and failed rollouts, provided the useful behavior already exists in the policy’s proposal distribution.

The next step is not a larger success claim. It is a complete operating record: held-out trials, failure and safety events, intervention paths, update costs, artifact release, and independent reproduction across other robot policies and embodiments.

Frequently asked questions

What is Q-Planning for robot policies?

Q-Planning adds a separately trained value model to a frozen behavior-cloning policy. The policy proposes several action chunks, the value model scores them, and a value-weighted combination is executed. New successful and failed rollouts update the value model, not the base policy.

Does Q-Planning change the robot policy weights?

No. In the reported experiments, the FastWAM behavior-cloning policy remains frozen during offline value training, inference, and online self-improvement. Only the separate Q-function is updated after deployment rollouts.

Can Q-Planning learn from failed robot attempts?

Under the authors' protocol, yes. Failed rollouts receive zero terminal reward and are retained in the value-model replay buffer. The reported real-robot tasks improved over five iterations while the base policy stayed frozen. This shows learning from bounded author-run failures, not unrestricted autonomous learning.

Did Q-Planning improve a real robot?

The authors report two bimanual lab tasks. Stack-cups improved from a 40% behavior-cloning baseline to 90%, and insert-wallet improved from 25% to 80%, after five iterations. Each iteration used 20 episodes per task, so the final points follow 100 online episodes per task.

Was Q-Planning autonomous during the real-robot study?

The robot was not teleoperated during rollouts, but people still reset the scene between episodes and supplied a success label after each attempt. The study therefore supports autonomous action execution inside a supervised experimental loop, not unattended operation.

Does Q-Planning make failure safe?

No. The paper evaluates task success and planning latency, not a safety case. It does not report unsafe-event counts, collision severity, emergency stops, hardware damage, operator exposure, or a rule for ending exploration before a hazardous attempt.