Robot Benchmark Success Rates: A Practical Evidence Audit

The short answer

A robot success rate is interpretable only with its task, denominator, embodiment, execution stack, and failure rules. A result of 97% can describe thousands of simulator rollouts across a fixed suite, 97 successes in 100 physical attempts, or a hand-picked subset after excluded runs. Those are different claims.

HumanCLAW makes this problem unusually visible. Its authors built a simulated humanoid benchmark that separates high-level action choice from low-level motion execution. Nine frozen vision-language models receive egocentric images and choose atomic skills such as walking, turning, climbing, and sitting. A separate motion system executes those commands. Across 1,218 episodes in 41 indoor scenes, the strongest tested model reached 16.8% interaction success (paper, project page).

That low percentage does not contradict TurboVLA’s reported 97.7% average LIBERO success. HumanCLAW tests long-horizon finding, navigation, body placement, and interaction through a simulated humanoid. TurboVLA’s headline result covers tabletop manipulation in a different simulator, with a trained action policy, different observations, different actions, and a different success definition (TurboVLA paper).

The correct conclusion is not that one model is better. It is that the two percentages measure different systems under different protocols. For the broader model landscape, start with our overview of robot foundation models.

The same discipline applies when a model predicts future scenes. Our world action model evidence guide explains why action-to-motion fidelity, grounded contact, dynamics, and execution timing must be tested separately from visual quality and final task success.

Protocol completeness is only one part of benchmark quality. Our physical AI benchmark redundancy audit shows how correlated tests can give one ability several votes, materially changing a model ranking even when every source score is reported correctly.

Static success also misses whether an action arrived too late, how far a failure progressed, and whether a robot recovered after losing progress. Our dynamic robot benchmark guide uses ReflexBench and PRM-as-a-Judge 1.5 to connect latency-aware execution with process-level evaluation.

One complete-task rate can hide two additional problems: a model may preserve its score without the intended current observation, and an early failure may prevent later skills from being tested. Our guide to observation ablations and skill-level diagnostics shows how to report counterfactual sensitivity, stage reach, restorable intermediate states, local success conditions, and sequential recovery beside the aggregate.

A reward evaluator can also reverse its judgment while the trajectory stays fixed. Our robot reward-language robustness audit separates Score Crossing Rate and Flip Rate from average error, then checks whether wording instability changes offline trajectory selection.

Online improvement curves need the same discipline. The Q-Planning physical evidence audit shows how 20-attempt iteration points, a frozen base policy, human-provided labels, and the absence of a separate final holdout constrain claims that a robot learned from failure.

Four results, four different questions

Result	What was tested	Denominator	Embodiment and setting	Defensible reading
HumanCLAW, 16.8%	Best tested VLM on find, navigate, and sit progression	1,218 episodes overall, interaction evaluated on the 597-episode sit subset	Simulated humanoid in 41 HSSD indoor scenes	High-level embodied decisions remain weak even when low-level skill execution is stabilized
TurboVLA, 97.7%	Average across 40 LIBERO tasks	50 simulator rollouts per task, 2,000 total	Simulated single-arm tabletop manipulation	A compact policy performed strongly under the authors’ LIBERO protocol
TurboVLA, 80% to 92.5%	Four language-conditioned manipulation tasks	40 physical trials per task, 160 total	AgileX Piper arm with wrist and third-view RGB-D cameras	The policy transferred to four bounded physical tasks after 65 demonstrations per task
SymmGrid, task-specific curves	On-robot reinforcement learning for peg insertion, cable routing, and object relocation	Five seeds for two tasks, three for relocation, with 50-attempt evaluations	Franka arms on contact-rich tasks	Symmetry augmentation improved learning speed and success under the authors’ matched setup

These results can all be useful. They cannot be sorted into one universal ranking.

HumanCLAW isolates one source of failure

End-to-end robot evaluation normally combines at least two layers:

1. decision quality, choosing what the body should do next;
2. motor execution, converting that choice into stable physical motion.

If a humanoid falls while following a sensible plan, an end-to-end failure rate does not say which layer failed. HumanCLAW addresses that ambiguity by giving every tested VLM the same atomic skill interface and the same motion generator. The VLM remains frozen and chooses one parameterized skill at each sub-second step. The body still experiences gravity, collisions, stairs, and displaced objects inside simulation.

The paper validates the execution layer separately. Reported achievement ratios are near the commanded magnitude for walking, turning, side-stepping, stepping back, and sitting. Stair skills show a lower but consistent gain. This does not make execution perfect, but it reduces one major confound before the VLM comparison begins.

The benchmark then asks the model to find an object category, move its body to the target, and, for suitable targets, sit on it. The 1,218 episodes span six target categories. All episodes contribute to finding and navigation metrics. The 597 episodes involving beds, couches, and toilets also contribute to interaction metrics.

This split matters. The strongest tested model reported:

• 64.9% FindSR;
• 42.4% NavSR;
• 16.8% InteractSR.

The paper’s main finding is therefore not simply that success was low. Performance fell at each stage after recognition. Models often saw the target but failed to place and track the body well enough to complete the physical sequence.

What HumanCLAW does not prove

HumanCLAW is a controlled simulation study, not a physical humanoid deployment. It does not test:

• camera noise, calibration drift, or missing observations on real hardware;
• actuator limits, thermal behavior, battery life, wear, or maintenance;
• safety-rated speed and force constraints;
• real people, pets, clutter, fabrics, or moving obstacles;
• teleoperation, remote rescue, or field support requirements;
• production throughput, uptime, or commercial value.

The benchmark also uses a structured harness. Prompts define the available skills, elicit spatial reasoning, and verify proposed actions. The measured result belongs to the complete VLM-plus-harness decision system, not an unprompted model in isolation.

A low HumanCLAW score supports a narrow conclusion: current tested VLM systems struggled with closed-loop body-aware decisions in this simulator. It does not establish a universal ceiling for every robot foundation model.

Why TurboVLA’s 97.7% is a different claim

TurboVLA removes a large language model from the central action-generation path. It combines visual features, a compact language encoder, robot state, and a non-autoregressive action decoder. The authors report a 0.2-billion-parameter LIBERO policy, 0.9 GB of inference VRAM, and 31.2 millisecond policy latency on one RTX 4090.

The headline 97.7% result has a useful denominator. LIBERO contains four suites with ten tasks each. The authors report 50 rollouts per task, or 2,000 simulator trials. TurboVLA predicts 12-step chunks of seven-degree-of-freedom actions.

That percentage is meaningful inside the disclosed protocol. Its boundary is equally important:

• it is a simulation aggregate, not physical-robot uptime;
• suite averages can conceal weaker tasks, including the reported 94.2% LIBERO-Long result;
• the comparison depends on dataset versions, checkpoint choices, preprocessing, framework details, and baseline implementations;
• latency and VRAM were measured on specified hardware with batch size one;
• one training run does not show variation across training seeds unless those seeds are reported.

TurboVLA also includes real-robot evidence, but it is separate. The team fine-tuned on 65 teleoperated demonstrations for each of four tasks, then ran 40 physical trials per task. It reports 92.5% for grabbing a roller, 80% for moving a playing card away, 90% for pressing a stapler, and 87.5% for stacking three bowls.

Those 160 trials are more relevant to physical transfer than the LIBERO number. They still cover one arm, one sensing setup, four tasks, and a company-run evaluation. Neither the simulation result nor the real-robot result establishes sustained operation.

SymmGrid asks how quickly a physical robot can learn

SymmGrid studies another question: can geometric symmetry generate useful replay-buffer variations and reduce physical learning time? The authors train directly on Franka robot arms for peg insertion, cable routing, and object relocation.

Its reporting unit is not one final leaderboard percentage. The paper compares learning curves, wall-clock convergence, success at matched checkpoints, and normalized area under the curve. It reports five training seeds for peg insertion and cable routing, three for object relocation, and 50 task attempts at evaluation points.

For peg insertion, the paper reports average convergence to 100% training accuracy in 18.5 minutes for SymmGrid versus 26 minutes for the SERL baseline. For cable routing, it reports 98% evaluation success as early as 10.9 minutes and 99.2% at 18.5 minutes. The authors also note that hardware differed from the original SERL report and explicitly call for more seeds or further analysis where results surprised them.

That disclosure is a reason to keep the conclusion bounded. SymmGrid supports an author-reported improvement under matched experiments in this study. It does not show that every symmetry augmentation will accelerate every robot, task, or controller.

A success rate needs at least twelve fields

Before comparing robot results, reconstruct this evidence card.

1. System under test

Is the tested object a vision-language model, a VLA policy, a complete robot stack, a controller, or a human-assisted service? HumanCLAW tests a frozen VLM inside a fixed harness. TurboVLA tests a trained action policy. Their units are different.

2. Task distribution

List the actual tasks and how they are weighted. An average across 40 tabletop tasks differs from 597 long-horizon interaction episodes or repeated trials of one contact task.

3. Environment

Record simulation, lab hardware, customer pilot, or recurring field operation. Simulation can improve repeatability and scale. Physical tests add sensing, control, and hardware failure modes. Customer operations add organizational and economic constraints.

4. Embodiment and sensors

A humanoid with an egocentric camera, a fixed arm with two RGB-D views, and a dual-arm system do not face the same action space or observability problem.

5. Training and test separation

Ask whether test tasks, homes, objects, scenes, and robot data appeared in training. A held-out environment test and a fine-tuned task test answer different questions. Our dataset split audit explains how episode, site, object, and pretraining boundaries constrain the word “unseen”.

6. Denominator

Convert every percentage into attempts where possible. Fifty rollouts per task makes a two-point change equal to one trial on that task. A percentage without a denominator cannot express evidence volume.

7. Repeated training seeds

Robot policies can vary with initialization and data order. Rollout count measures one checkpoint’s behavior. Multiple training seeds measure how reliably the method produces a good checkpoint.

8. Success definition

Define completion geometrically or operationally. HumanCLAW requires both an objective condition and model acknowledgement. Other benchmarks may use simulator predicates, human grading, milestone progress, or task-specific sensors.

9. Intervention, reset, and exclusion rules

State whether a human can rescue the robot, reset an object, stop unsafe motion, or discard invalid runs. Report aborted and excluded attempts separately. A nominally autonomous trial can still depend on substantial support between actions.

Our robot intervention and handoff guide expands this field into cut-in and cut-out quality, operator-control time, switching latency, recovery outcomes, and the downstream value of corrective data.

10. Compute and latency protocol

Record the complete hardware, precision, batch size, observation-to-action boundary, and latency distribution. A 31.2 millisecond policy measurement on an RTX 4090 is not automatically a 32 Hz closed-loop robot system on another computer.

11. Failure distribution

A single average can hide whether failures come from perception, language grounding, navigation, grasping, collision, timeout, or human support. HumanCLAW’s staged rates are more diagnostic than its final score alone.

12. Independence and artifacts

Check whether code, weights, data, task definitions, evaluation scripts, and trial logs are public. Then ask whether another team has reproduced the result. A public repository improves inspectability, but it is not independent replication.

Confidence intervals: what 90% success leaves unknown

A denominator makes a percentage readable. An uncertainty interval makes its precision visible. Eighteen successes in twenty attempts and ninety successes in one hundred attempts both report 90%, but they do not provide equally precise evidence.

The NIST/SEMATECH statistical handbook describes the Wilson interval for a proportion and gives its calculation. The examples below apply that formula to hypothetical binary task outcomes, using a two-sided 95% confidence level. They are calculations, not reported results from any robot company or benchmark.

Hypothetical successes / attempts	Observed success	95% Wilson interval
18 / 20	90%	69.9% to 97.2%
45 / 50	90%	78.6% to 95.7%
90 / 100	90%	82.6% to 94.5%
20 / 20	100%	83.9% to 100.0%

A 95% confidence level describes the interval procedure’s coverage across repeated samples under its assumptions. It does not mean that 95% of future robot attempts will succeed or that the fixed unknown success probability has a 95% chance of lying inside this particular interval.

The last row is particularly important when reading a perfect demo result. Twenty successes with no observed failure do not establish a failure-free robot. Under this interval method and its sampling assumptions, the lower confidence limit remains well below 100%.

NIST also discusses exact binomial intervals for small samples and one-sided limits. Report the method and whether the interval is one-sided or two-sided. Do not switch methods after seeing which produces the most attractive number.

What the interval assumes

For these illustrations, the attempts are modeled as independent binary outcomes with one fixed success probability under a declared protocol. That is an analytical assumption, not a fact established by counting runs.

In robot evaluations, ask whether attempts share an object, scene, training checkpoint, day, operator, or reset procedure. Shared conditions can create dependence or narrow the tested distribution. A pooled count does not automatically provide the same evidence as independent draws from the intended operating conditions.

Keep distinct uncertainties separate:

• Within-checkpoint trial uncertainty: how a frozen policy performs under the declared attempt distribution.
• Training variation: whether another training seed produces a similarly capable policy.
• Task and environment coverage: whether the tested objects, layouts, and disturbances represent the intended use.
• Measurement uncertainty: whether the success rubric, grading, and exclusions are consistent.

A single binomial interval does not resolve all four. These are reporting recommendations for robot evaluation, not a claim that NIST has validated any particular robot protocol.

Do not turn an average into imaginary trials

If a paper reports an average across task percentages, do not multiply that average by a total trial count and assume the result is the number of successes. Unequal denominators, task weights, rounding, and repeated checkpoints can change the relationship. Ask for integer successes and attempts per task, plus the aggregation rule.

For a training method tested across several seeds, keep the seed-level results visible. Hundreds of rollouts from one selected checkpoint do not show how reliably training produces that checkpoint. For customer operations, retain the day, site, shift, and operating window rather than treating every repeated motion as unrelated evidence. Our operating-hours guide explains why task attempts and field runtime answer different questions.

A practical reporting rule

Publish successes and attempts, a declared interval method and confidence level, task-level results, training seeds, exclusions, and the sampling procedure. Freeze the tested checkpoint and success rule before the final evaluation. Keep exploratory improvement runs separate from a held-out assessment.

Do not infer superiority merely because one point estimate is higher. Interval overlap alone is also not a complete comparison test. A defensible comparison needs an analysis matched to the design, including whether systems faced paired initial states and how tasks or seeds were grouped. If those details are missing, describe the observed difference and the missing evidence instead of declaring a winner.

Use uncertainty to bound the experiment, not to upgrade it into a safety, autonomy, uptime, or commercial-readiness claim. A narrow interval on one easy task can still leave the important operating conditions untested.

More data does not repair a missing objective

Benchmark interpretation also reaches into representation learning. A separate July 2026 study created controlled environments with hidden mass, drag, and contact stiffness, then checked whether those properties were recoverable from observations and retained in learned world-model representations (paper).

The authors report that inputs limit what can be known, while prediction targets influence what is retained. Contact stiffness entered the tested latent representation when touch was forecast, but not merely because touch was supplied as an input. In RH20T experiments across 4,258 episodes, only the full multimodal objective beat a persistence baseline for force prediction, with gains increasing as data scaled.

The practical implication is narrow and important: a larger dataset cannot be assumed to fix a benchmark if the sensors omit the required signal or the objective does not pressure the model to retain it. Benchmark scope must include what the system can observe and what training asks it to predict.

A worked comparison rule

Two percentages can be compared directly only if a reader can answer yes to most of these questions:

• Are the tasks and task weights the same?
• Are the environments and object distributions aligned?
• Is the embodiment identical or intentionally controlled?
• Are training and test splits equivalent?
• Are checkpoints selected under the same rule?
• Are trial counts and repeated seeds comparable?
• Is success defined identically?
• Are intervention, reset, timeout, and exclusion rules the same?
• Are compute and latency measured on the same boundary and hardware?
• Are results independently reproduced or at least run in one matched harness?

If not, present the results side by side as different evidence, not as a ranking.

Proxy scores can also hide the downstream task. The latent action model evidence guide shows why reconstruction and action-probe metrics can screen representations without reliably ranking final robot policies.

What better robot benchmark reporting looks like

A strong report should publish:

1. a task manifest and sampling weights;
2. robot, sensor, controller, and compute specifications;
3. training data and test-separation rules;
4. checkpoint selection and the number of training seeds;
5. every trial denominator, timeout, reset, intervention, and exclusion;
6. an executable success definition;
7. task-level results with uncertainty, not only one aggregate;
8. failure categories and representative failure runs;
9. latency percentiles measured across the complete decision path;
10. code, weights, configs, evaluation harness, and versioned results where possible;
11. an explicit statement of what simulation or lab evidence does not establish;
12. independent reproduction before making broad comparative claims.

For a concrete example of unusually detailed task reporting, our ACT-2 reliability audit examines 785 laundry attempts, garment categories, fold quality, speed, adaptation cost, and missing field evidence. Our Xiaomi-Robotics-1 audit shows why simulation suites, bounded real-robot trials, and a 100,000-hour pretraining corpus must remain separate claims. The FLUX-mimic evidence audit applies the same discipline to a 20-trial company benchmark and factory partner evidence.

The Gemini Robotics 2 audit applies the same evidence card to whole-body Apollo 2 demonstrations, ER 2 progress metrics, multi-robot orchestration, and on-device adaptation claims.

The Ego2Robot synthetic-data audit shows why a fixed-frame comparison is stronger than ranking raw dataset hours. It also keeps 50-episode simulation tasks separate from 20-trial physical evaluations that use target-robot demonstrations.

Verdict

HumanCLAW’s most useful contribution is methodological. It does not merely report a low number. It constructs a benchmark where high-level decisions can fail while low-level skill execution remains comparatively controlled, then reports stage-by-stage outcomes, collisions, scene disturbance, motion quality, and computational cost.

TurboVLA and SymmGrid demonstrate why the same reading discipline must work in both directions. High percentages can be meaningful when the protocol and denominator are clear. They still belong to their declared task, embodiment, compute, and evaluation setting.

For public videos and live events that do not provide a benchmark paper, use the robot demo evidence checklist to recover continuity, control mode, resets, speed, variation, failure, and operating-status boundaries.

Read every robot success rate as a compact label for an experiment, not as a property of the robot or model. Reconstruct the experiment first. Compare the percentages second.

Frequently asked questions

Does a perfect robot success rate prove reliability?

No. Twenty successes in twenty hypothetical independent attempts yield a two-sided 95% Wilson interval of about 83.9% to 100%. A perfect observed result leaves uncertainty, and the interval does not establish safety, field uptime, or coverage of untested tasks.

Can two robot success rates be compared directly?

Only when the task, environment, embodiment, training data, checkpoint selection, trial protocol, success rule, intervention policy, and reporting unit are sufficiently aligned. Otherwise, the percentages answer different questions and should not be placed on one leaderboard.

What does HumanCLAW measure?

HumanCLAW measures high-level embodied decisions through a simulated humanoid whose low-level motion skills are supplied by a separate controller. It tests whether a frozen vision-language model can find, navigate to, and interact with targets while motor execution is held comparatively stable.

Is HumanCLAW a physical robot benchmark?

No. HumanCLAW runs in simulated indoor scenes with gravity, collisions, continuous whole-body motion, and movable objects. It is useful for isolating decision errors, but it does not measure sensing noise, actuator wear, hardware safety, or field reliability on a physical robot.

What does TurboVLA's 97.7% result mean?

The authors report 97.7% average success across four LIBERO suites, using 50 simulated rollouts for each of 40 tasks. The same paper separately reports 160 real-robot trials across four tasks. The simulation percentage should not be presented as the real-robot result.

How many trials are enough for a robot benchmark?

There is no universal minimum. More trials narrow uncertainty, but task diversity, repeated training seeds, independent replication, declared exclusions, and complete failure reporting also matter. A large denominator on one narrow task does not establish broad capability.

What should a complete robot benchmark report?

At minimum, report the task and environment distribution, robot and sensors, training and test separation, checkpoint selection, number of training seeds, trial count, success definition, intervention and reset rules, excluded attempts, confidence intervals, latency hardware, failure categories, and artifact availability.