Why One Robot Success Rate Hides the Failure: A Diagnostic Benchmark Guide

The short answer

A robot benchmark needs to show both whether the system finished and why it succeeded or failed. One final success rate can hide two opposite problems:

1. the benchmark may reward a shortcut that avoids the capability it is meant to test;
2. an early failure may prevent later capabilities from being tested at all.

Two new preprints make those problems concrete from different directions.

Driving on Memory removes current camera input from an end-to-end driving planner and replaces it with latent memories from earlier drives near the same location. The memory-only system reaches 91.1 PDMS on NAVSIMv1, compared with 93.7 for its camera-based counterpart. On NAVSIMv2, it reaches 45.0 EPDMS, compared with 46.7. It cannot observe the evaluated traffic scene.

That result does not show that perception is unnecessary for driving. It shows that these scores can remain high when a planner receives initial ego status and persistent location context instead of current camera evidence. The same intervention falls much further on longer or more interactive closed-loop tests.

Behavior-Skill begins with the other failure. A long household task may stop at the first bad manipulation, leaving every later skill unobserved. Its authors segment 10,000 BEHAVIOR-1K demonstrations into 235,492 skill instances across 50 tasks and 34 semantic skill categories. Each evaluated skill starts from a reconstructed intermediate simulator state with a local symbolic success condition.

Together, the papers suggest a practical benchmark design:

aggregate outcome + intended-input ablation + stage reach + independent skill execution + sequential recovery + failure taxonomy

Our broader robot success-rate evidence audit explains how to reconstruct the task, denominator, embodiment, and protocol behind a percentage. This guide focuses on the next question: did the benchmark require the intended evidence, and did it expose the actual failure location?

Two ways an aggregate can mislead

A final task metric compresses a trajectory into one number. Compression is useful for comparison, but it destroys information.

Hidden shortcut

Suppose a driving benchmark is intended to test reaction to current traffic. A model receives camera images, vehicle state, and a navigation command. A high score is often read as evidence that it saw current actors and signals, interpreted them, and reacted safely.

That reading is not guaranteed. Initial speed or acceleration may reveal that braking has already begun. A known route may predict a turn. A location prior may indicate that a crosswalk or busy merge is nearby. Conservative motion may avoid interactions during a short horizon.

The final score says the trajectory satisfied the metric. It does not say which evidence produced it.

Hidden untested capability

Now suppose a household task contains eight skills. The robot fails while opening a door at step three. Skills four through eight are never attempted.

A zero full-task score correctly reports that the task did not finish. It does not establish whether the robot could execute the five unreached skills from valid states. Repeating complete rollouts can keep testing the same early bottleneck while collecting little evidence about the rest of the capability chain.

The two cases require different diagnostics. Observation ablation tests whether success survives after intended evidence is removed. Intermediate-state restoration tests capabilities that sequential execution did not reach.

When the hidden bottleneck is physical contact, a diagnostic also needs clearance, force quality, control-layer boundaries, and recovery. Our contact-rich robot manipulation audit applies that evidence stack to precision electronics assembly and modular insertion tests.

What Driving on Memory actually tests

The authors start with DrivoR, an end-to-end planner that normally encodes multi-camera images into scene tokens. They build a memory bank from training-set traversals. Each stored item contains latent scene tokens and a global pose.

At evaluation time, MemoryDrivoR:

1. receives current ego status and absolute pose;
2. retrieves up to ten memories within 20 meters, with at most one from each prior traversal;
3. encodes each memory’s relative pose;
4. compresses the remembered scene tokens;
5. supplies those tokens to the planner instead of current camera tokens.

The model can receive persistent information such as road geometry, crosswalk placement, and location-conditioned traffic regularities. It cannot see the current actors, current signal phase, or current weather through the evaluated camera frame.

The resampler is trained to filter the stored representation. A probe reported in the paper retained map information more strongly than agent information, supporting the authors’ intended static-context interpretation. It does not prove that every dynamic trace was removed.

The headline comparison

Protocol	Camera DrivoR	MemoryDrivoR without current cameras	Memory result relative to camera model
NAVSIMv1 PDMS	93.7	91.1	Near the camera score
NAVSIMv2 EPDMS	46.7	45.0	Near the camera score
Bench2Drive success rate	30.5%	9.5%	Large loss
Bench2Drive driving score	61.0	34.7	Large loss
RealEngine base closed-loop PDMS	88.2	72.3	Material loss
RealEngine counterfactual safety test	53.8	28.1	Larger loss with inserted actors

These are author-run results from different protocols, not one unified leaderboard. The rows use different metrics and scenarios.

The important comparison is within each protocol. Memory nearly preserves the NAVSIM aggregate but does not preserve camera-model performance on Bench2Drive or RealEngine safety tests.

Why the NAVSIM score survives

NAVSIMv1 queries the agent once from a recorded sensor frame, asks for a four-second trajectory, and evaluates that trajectory while background actors replay recorded futures. NAVSIMv2 adds a second pseudo-simulation stage and reactive traffic, but still begins from rule-compliant states with informative ego status.

The authors identify three mechanisms:

• initial ego status: velocity and acceleration can reveal that a critical maneuver such as braking is already underway;
• persistent context: retrieved memories reveal road structure and likely interaction locations;
• conservative planning: slowing down can avoid collisions and violations over a short horizon, with the main penalty appearing in progress.

On NAVSIMv1, MemoryDrivoR scores 98.3 for no-at-fault collision, 98.5 for drivable-area compliance, and 95.3 for time to collision. Its largest component loss against camera DrivoR is progress, 86.0 versus 90.0.

This is precisely why component scores alone do not solve the diagnosis. Collision-related subscores remain high even though the planner cannot see current vehicles. A benchmark must create situations where current observation changes the required action.

Why the closed-loop tests separate the systems

Bench2Drive evaluates 220 routes in CARLA. The agent receives updated observations after each planning step. Successful MemoryDrivoR routes averaged about 110 meters and 79 seconds, much longer than NAVSIMv1’s four-second prediction horizon.

MemoryDrivoR completed 21 of 220 routes without infractions. Camera DrivoR reached a 30.5% success rate, while MemoryDrivoR reached 9.5%. In the paper’s failure grouping, larger gaps appear around pedestrians and cyclists, cut-ins and sudden obstacles, and traffic-controlled junctions.

RealEngine provides another useful intervention. Its counterfactual safety tests insert actors. MemoryDrivoR’s score falls from 72.3 in base closed loop to 28.1 in the safety test. Camera DrivoR falls from 88.2 to 53.8. Adding current interactions widens the diagnostic gap.

A long horizon alone is not sufficient. On 87 HUGSIM-nuScenes scenarios built from mostly straightforward routes, MemoryDrivoR reaches 32.2 HD-Score versus 36.4 for camera DrivoR. The tested scenario distribution must force observation-dependent choices, not merely run for more seconds.

What the driving study does not establish

The evidence is a benchmark audit, not an autonomous-driving deployment result.

It does not show that:

• cameras or current perception are unnecessary on roads;
• MemoryDrivoR is safe for a vehicle;
• NAVSIM has no value;
• location memory is inherently an invalid model input;
• camera-based models use the same shortcut as MemoryDrivoR;
• performance on one benchmark predicts field collision risk.

The method also needs repeated traversals and sufficiently accurate relative localization. The main memory bank is built from benchmark training data. MemoryDrivoR is fine-tuned to use the replacement input, so this is not a zero-shot test where cameras are simply blanked at inference.

A geographical split gives one important boundary. Removing evaluation locations from camera-model training did not reduce the tested DrivoR scores. That makes direct location memorization in model weights an unlikely primary explanation for those camera results. It does not repair the separate observation-demand weakness exposed by the memory baseline.

What Behavior-Skill actually tests

Behavior-Skill builds on simulated BEHAVIOR-1K demonstrations collected through human teleoperation. The authors identify constituent skill intervals and annotate action, objects, spatial relation, context, and end-effector use.

The resulting dataset contains:

• 235,492 skill instances;
• 10,000 demonstrations;
• 50 long-horizon household tasks;
• 34 semantic skill categories;
• an average recorded skill duration of 16.6 seconds;
• head, left-wrist, and right-wrist visual context;
• frame-level 23-degree-of-freedom action sequences.

Annotations are partly model-assisted. Qwen3-VL-235B-A22B-Instruct generates annotations from structured, visual, and action context. The authors manually inspect a representative demonstration per task, use ChatGPT-5 to flag inconsistencies against that reference, and manually review flagged cases. This process is documented, but the paper does not report a complete independent annotation-accuracy study over all 235,492 instances.

Restoring valid intermediate states

An independent skill trial needs more than a clipped video. It needs the state that existed immediately before the skill.

The underlying recordings use an active-object filter, so one serialized frame may omit sleeping objects. The authors reconstruct state by replaying transitions from the demonstration’s initial state to the target skill boundary. They then serialize the complete simulator state, including articulated object states, poses, robot configuration, interaction context, and assisted-grasp constraints.

Each evaluation unit contains:

• the restored state before the skill;
• a skill instruction;
• a local success condition;
• a maximum evaluation horizon.

The local goal uses the Behavior Domain Definition Language. Existing predicates cover relations such as placement or object activation. The authors add predicates for behaviors such as proximity, grasping, and handover when the task-level language is insufficient. A trial ends on goal satisfaction or after twice the recorded demonstration duration.

This design answers a clean question: can the policy execute this constituent skill when its preconditions are already valid?

It does not answer whether the policy can plan the task, reach that state through its own earlier actions, recover from drift, or preserve conditions for later skills.

The Behavior-Skill results

The main experiment evaluates two VLA families on all 50 tasks. Ten demonstrations per task are reserved for evaluation, producing 500 evaluated demonstrations with every contained skill included.

The paper compares task-prompt and skill-prompt variants under matched model architecture, observations, actions, training data, and schedule.

Policy	Task-prompt TSCR	Skill-prompt TSCR	Difference
π0.5	42.4%	48.4%	+6.0 percentage points
GR00T N1.7	36.9%	42.5%	+5.6 percentage points

Task Skill Completion Rate, or TSCR, first calculates the share of independently completed skills within each demonstration, then averages across demonstrations. It is not full-task success. The independently tested skills do not execute as one continuous rollout.

Skill-specific instructions improve the aggregate under this protocol, but even the strongest result remains below 50%. The paper reports relatively high performance for skills such as Move To, Release, Hold, Sweep Surface, and Spray. Contact-rich or articulated-object skills such as Open Lid, Close Door, Close Lid, Pick Up From, and Pour remain bottlenecks.

The two model families show similar capability profiles. That is useful comparative evidence under the shared benchmark. It does not prove a universal hierarchy of robot difficulty.

Why stage reach matters

In a conventional full-task rollout, an early failure can censor later evidence. The paper examines three examples with π0.5. Wash a Baseball Cap and Cook Hotdogs each receive 0% full-task QScore while their independently evaluated TSCR values are 80.0% and 69.0%. Make Microwave Popcorn receives 10.0% QScore and 75.0% TSCR.

These pairs are not contradictory. QScore measures sequential task outcome. TSCR measures constituent skills from restored valid states. The gap identifies unreached or sequence-sensitive capability, not a better final-task result.

A diagnostic report should preserve both:

• stage reach: how often the sequential policy arrived at each skill;
• conditional success after reach: how often it succeeded when it arrived;
• independent skill success: how often it succeeded from a standardized valid state;
• downstream preservation: whether successful execution left the right state for the next skill.

Without stage reach, a benchmark cannot distinguish “the policy cannot do this skill” from “the policy never reached this skill.” Without independent trials, later capabilities remain censored. Without sequential trials, standardized skill tests can hide accumulated drift and planning failure.

Why a semantic skill label is not enough

Behavior-Skill also shows that the same named skill can vary sharply by context. In a 12-task subset, the paper reports Open Door success ranging from 9.0% in Make Microwave Popcorn to 78.0% in Cook Hot Dogs and Wash a Baseball Cap under the task-prompt setting. Place In ranges from 15.2% in Putting Shoes on Rack to 88.0% in Cook Hot Dogs.

A skill benchmark therefore needs more than a category average. Preserve:

• manipulated object and target object;
• object geometry, state, articulation, and clutter;
• starting robot pose and grasp state;
• one-arm, two-arm, base, and gripper requirements;
• spatial relation and tolerance;
• collision, force, and timeout rules;
• task context and preceding state;
• trial count and uncertainty for each context.

“Open Door” is a family of conditions, not one atomic capability.

A practical diagnostic benchmark stack

A strong robot evaluation can use six layers. Each answers a different question.

1. Aggregate task outcome

Did the complete system finish under the declared protocol? Report success, partial progress, safety outcome, intervention, time, and resource cost.

Do not discard this layer. It captures planning, state transitions, compounding error, and recovery that isolated tests miss.

2. Intended-input dependence

Did the system need the information the benchmark claims to test?

Run controlled ablations such as:

• current images replaced with historical images;
• image order shuffled within defensible boundaries;
• language removed or replaced with task IDs;
• robot state frozen, delayed, or quantized;
• privileged map, pose, or simulator state removed;
• history truncated;
• force or tactile channels removed from contact tasks;
• current actors replaced with counterfactual actors;
• initial state changed while route or task identity remains fixed.

An ablation must preserve enough of the system to remain executable. A catastrophic out-of-distribution input proves little. Retraining or matched adaptation may be necessary, but then the result must be labeled as a benchmark-capacity audit rather than a drop-in model ablation.

3. Stage reach and transition coverage

For each stage, report how many trials:

• reached the stage;
• entered with valid preconditions;
• completed it;
• left valid postconditions;
• required intervention or reset;
• failed before it could be observed.

This makes censoring visible.

4. Independent constituent tests

Restore a valid state and test one capability with an executable local goal. Keep multiple object and context variants. Publish the restoration procedure and verify that the snapshot does not inject privileged information unavailable to the policy.

5. Sequential stress and recovery

Rejoin the skills. Perturb intermediate states, delay observations, change object placement, and test whether the system notices a failed postcondition before continuing. Measure retries, replans, regressions, human handoffs, and safe termination.

Our dynamic robot benchmark guide explains how progress and recovery can be measured without turning one final frame into the whole result. The robot handoff evidence guide keeps autonomous recovery separate from operator rescue.

6. Failure taxonomy and trace

Assign each failed trial to the earliest supported cause and preserve downstream effects. Useful categories include:

• observation missing or stale;
• object or state not detected;
• instruction or reference misgrounded;
• invalid task decomposition;
• wrong skill selected;
• motion planning or control failure;
• grasp or contact failure;
• timeout or excessive latency;
• violated precondition or postcondition;
• unsafe action rejected;
• protective stop;
• human intervention;
• simulator, sensor, or hardware fault.

Do not force one cause when evidence is ambiguous. Use multi-label or “unresolved” categories and retain traces.

Metrics that should appear beside success

Metric	What it reveals
Observation-dependence gap	Score difference between intended input and a controlled replacement
Counterfactual sensitivity	Whether behavior changes when a relevant current-state factor changes
Stage reach rate	How often a sequential rollout exposes each later capability
Conditional stage success	Success among trials that reached the stage with valid preconditions
Independent skill success	Capability from a standardized restorable state
Postcondition validity	Whether a completed skill leaves a valid state for the next one
Recovery rate	Successful return after a defined disturbance or failed step
Intervention rate	Human takeovers or corrections per attempt, hour, or transition
Failure distribution	Where and how the system fails, not only how often
Exposure-normalized safety rate	Stops, near misses, and violations per relevant operating exposure

Do not combine these into one universal score without a declared decision model. A weighted aggregate can hide the same structure the diagnostics were added to expose.

Evidence boundaries

Claim	Evidence class	Confidence	Boundary
NAVSIM scores can remain high without current camera input	Author-run benchmark intervention	High for the reported models and protocols	MemoryDrivoR is fine-tuned, receives ego status and pose, and uses prior traversals
Current-scene information matters more on the tested closed-loop interactive protocols	Matched author-run comparison	Medium-high	Bench2Drive, RealEngine, and HUGSIM differ in scenario distribution and metrics
Behavior-Skill exposes capability hidden after early full-task failures	Restored-state simulator benchmark	High for the protocol design	Independent skills omit planning and sequential dependencies
π0.5 and GR00T N1.7 complete fewer than half of constituent skills on average	Author-run simulation evaluation	Medium-high	Two model families, selected checkpoints, model-assisted annotations, no physical robot
Contact-rich skills are universal VLA bottlenecks	Not established	Low	Similar profiles in two tested policies do not define every robot, object, or task
Independent skill rates predict deployed task reliability	Not established	High confidence in the negative boundary	Skills interact, contexts vary, and field hardware adds untested failure modes

Both studies are preprints released on August 31, 2026. Neither is independent replication. Behavior-Skill is simulation-only. Driving on Memory audits simulation-based driving protocols. Neither reports road deployment, household robot operation, hardware safety, or commercial reliability.

The MemoryDrivoR repository was publicly accessible when this guide was prepared. The Behavior-Skill paper names a repository, but that repository returned not found during our September 1 verification, so artifact availability should be rechecked rather than assumed.

A publication checklist for benchmark authors

Before publishing one robot success rate, provide:

1. the exact system, checkpoint, observations, actions, and compute;
2. task and environment sampling weights;
3. complete attempt, seed, exclusion, reset, and intervention counts;
4. executable task and stage success definitions;
5. observation and control ablations tied to the intended construct;
6. counterfactual scenes where current information changes the correct action;
7. stage reach and conditional stage success;
8. restorable intermediate states or another method for testing unreached capabilities;
9. independent skill results across object and context variants;
10. sequential postcondition, compounding-error, and recovery tests;
11. latency, failure, safety, and human-support distributions;
12. code, task manifests, state reconstruction, annotations, and versioned traces;
13. a clear separation between simulation, lab hardware, pilot, and recurring operation;
14. an explicit list of capabilities the benchmark does not measure.

The benchmark redundancy audit adds suite-level checks for repeated signal, model ranking sensitivity, and broad vision-language factors. The two problems are complementary. A suite can double-count an ability, while each individual benchmark can still hide shortcuts and untested stages.

Bottom line

Driving on Memory shows why a high score must be challenged with the absence of intended evidence. Behavior-Skill shows why a low final score must be opened into reached, unreached, and independently testable capabilities.

Neither result makes aggregate success obsolete. Full-task execution remains the outcome that a deployed system ultimately needs. The correct response is to stop asking one number to serve as outcome, explanation, capability map, and safety claim at the same time.

Keep the final success rate. Then publish the tests that reveal what the system observed, which stages it reached, which skills it can execute from valid states, how failures compound, and how it recovers.

Frequently asked questions

Why is one robot success rate not enough?

A final rate reports whether a protocol reached its endpoint. It may not show whether the system used the intended observation, which constituent skill failed, which later skills were never attempted, how often a human intervened, or whether the same policy succeeds from valid intermediate states. Keep the aggregate, but publish diagnostic tests beside it.

What is an observation ablation?

It removes, replaces, delays, corrupts, or freezes an input that the benchmark is intended to require. If performance remains high without current-scene information, the score may be supported by static context, initial state, route regularity, or another shortcut rather than the intended perception capability.

Does Driving on Memory show that cameras are unnecessary for autonomous driving?

No. It shows that a memory-based planner without current camera input scored surprisingly well on the tested NAVSIM protocols. The same intervention lost much more performance on Bench2Drive and RealEngine. The result audits benchmark demand, not road-going safety or the value of perception.

What does independent skill evaluation add?

It restores the valid state before a constituent skill, gives the policy the corresponding instruction, and tests a local success condition. This reveals capabilities that a full-task rollout may never reach after an early failure. It complements, but does not replace, sequential evaluation.

Can skill success rates be multiplied to predict full-task success?

Only under strong independence and distribution assumptions that usually do not hold. Skills change the scene for later skills, errors compound, planning matters, and the same semantic skill can vary greatly across objects and task contexts. Use skill rates to diagnose bottlenecks, not to manufacture a synthetic full-task claim.

What should a diagnostic robot benchmark publish?

Publish the aggregate task result, observation and control ablations, stage reach rates, independent skill results, local success rules, failure categories, reset and intervention rules, trial counts, exclusions, timing, repeated seeds where relevant, and representative traces. State which results come from simulation, lab hardware, or field operation.