Robot Benchmark Success Rates: A Practical Evidence Audit

The short answer

A robot success rate is interpretable only with its task, denominator, embodiment, execution stack, and failure rules. A result of 97% can describe thousands of simulator rollouts across a fixed suite, 97 successes in 100 physical attempts, or a hand-picked subset after excluded runs. Those are different claims.

HumanCLAW makes this problem unusually visible. Its authors built a simulated humanoid benchmark that separates high-level action choice from low-level motion execution. Nine frozen vision-language models receive egocentric images and choose atomic skills such as walking, turning, climbing, and sitting. A separate motion system executes those commands. Across 1,218 episodes in 41 indoor scenes, the strongest tested model reached 16.8% interaction success (paper, project page).

That low percentage does not contradict TurboVLA’s reported 97.7% average LIBERO success. HumanCLAW tests long-horizon finding, navigation, body placement, and interaction through a simulated humanoid. TurboVLA’s headline result covers tabletop manipulation in a different simulator, with a trained action policy, different observations, different actions, and a different success definition (TurboVLA paper).

The correct conclusion is not that one model is better. It is that the two percentages measure different systems under different protocols. For the broader model landscape, start with our overview of robot foundation models.

The same discipline applies when a model predicts future scenes. Our world action model evidence guide explains why action-to-motion fidelity, grounded contact, dynamics, and execution timing must be tested separately from visual quality and final task success.

Protocol completeness is only one part of benchmark quality. Our physical AI benchmark redundancy audit shows how correlated tests can give one ability several votes, materially changing a model ranking even when every source score is reported correctly.

Static success also misses whether an action arrived too late, how far a failure progressed, and whether a robot recovered after losing progress. Our dynamic robot benchmark guide uses ReflexBench and PRM-as-a-Judge 1.5 to connect latency-aware execution with process-level evaluation.

Online improvement curves need the same discipline. The Q-Planning physical evidence audit shows how 20-attempt iteration points, a frozen base policy, human-provided labels, and the absence of a separate final holdout constrain claims that a robot learned from failure.

Four results, four different questions

Result	What was tested	Denominator	Embodiment and setting	Defensible reading
HumanCLAW, 16.8%	Best tested VLM on find, navigate, and sit progression	1,218 episodes overall, interaction evaluated on the 597-episode sit subset	Simulated humanoid in 41 HSSD indoor scenes	High-level embodied decisions remain weak even when low-level skill execution is stabilized
TurboVLA, 97.7%	Average across 40 LIBERO tasks	50 simulator rollouts per task, 2,000 total	Simulated single-arm tabletop manipulation	A compact policy performed strongly under the authors’ LIBERO protocol
TurboVLA, 80% to 92.5%	Four language-conditioned manipulation tasks	40 physical trials per task, 160 total	AgileX Piper arm with wrist and third-view RGB-D cameras	The policy transferred to four bounded physical tasks after 65 demonstrations per task
SymmGrid, task-specific curves	On-robot reinforcement learning for peg insertion, cable routing, and object relocation	Five seeds for two tasks, three for relocation, with 50-attempt evaluations	Franka arms on contact-rich tasks	Symmetry augmentation improved learning speed and success under the authors’ matched setup

These results can all be useful. They cannot be sorted into one universal ranking.

HumanCLAW isolates one source of failure

End-to-end robot evaluation normally combines at least two layers:

1. decision quality, choosing what the body should do next;
2. motor execution, converting that choice into stable physical motion.

If a humanoid falls while following a sensible plan, an end-to-end failure rate does not say which layer failed. HumanCLAW addresses that ambiguity by giving every tested VLM the same atomic skill interface and the same motion generator. The VLM remains frozen and chooses one parameterized skill at each sub-second step. The body still experiences gravity, collisions, stairs, and displaced objects inside simulation.

The paper validates the execution layer separately. Reported achievement ratios are near the commanded magnitude for walking, turning, side-stepping, stepping back, and sitting. Stair skills show a lower but consistent gain. This does not make execution perfect, but it reduces one major confound before the VLM comparison begins.

The benchmark then asks the model to find an object category, move its body to the target, and, for suitable targets, sit on it. The 1,218 episodes span six target categories. All episodes contribute to finding and navigation metrics. The 597 episodes involving beds, couches, and toilets also contribute to interaction metrics.

This split matters. The strongest tested model reported:

• 64.9% FindSR;
• 42.4% NavSR;
• 16.8% InteractSR.

The paper’s main finding is therefore not simply that success was low. Performance fell at each stage after recognition. Models often saw the target but failed to place and track the body well enough to complete the physical sequence.

What HumanCLAW does not prove

HumanCLAW is a controlled simulation study, not a physical humanoid deployment. It does not test:

• camera noise, calibration drift, or missing observations on real hardware;
• actuator limits, thermal behavior, battery life, wear, or maintenance;
• safety-rated speed and force constraints;
• real people, pets, clutter, fabrics, or moving obstacles;
• teleoperation, remote rescue, or field support requirements;
• production throughput, uptime, or commercial value.

The benchmark also uses a structured harness. Prompts define the available skills, elicit spatial reasoning, and verify proposed actions. The measured result belongs to the complete VLM-plus-harness decision system, not an unprompted model in isolation.

A low HumanCLAW score supports a narrow conclusion: current tested VLM systems struggled with closed-loop body-aware decisions in this simulator. It does not establish a universal ceiling for every robot foundation model.

Why TurboVLA’s 97.7% is a different claim

TurboVLA removes a large language model from the central action-generation path. It combines visual features, a compact language encoder, robot state, and a non-autoregressive action decoder. The authors report a 0.2-billion-parameter LIBERO policy, 0.9 GB of inference VRAM, and 31.2 millisecond policy latency on one RTX 4090.

The headline 97.7% result has a useful denominator. LIBERO contains four suites with ten tasks each. The authors report 50 rollouts per task, or 2,000 simulator trials. TurboVLA predicts 12-step chunks of seven-degree-of-freedom actions.

That percentage is meaningful inside the disclosed protocol. Its boundary is equally important:

• it is a simulation aggregate, not physical-robot uptime;
• suite averages can conceal weaker tasks, including the reported 94.2% LIBERO-Long result;
• the comparison depends on dataset versions, checkpoint choices, preprocessing, framework details, and baseline implementations;
• latency and VRAM were measured on specified hardware with batch size one;
• one training run does not show variation across training seeds unless those seeds are reported.

TurboVLA also includes real-robot evidence, but it is separate. The team fine-tuned on 65 teleoperated demonstrations for each of four tasks, then ran 40 physical trials per task. It reports 92.5% for grabbing a roller, 80% for moving a playing card away, 90% for pressing a stapler, and 87.5% for stacking three bowls.

Those 160 trials are more relevant to physical transfer than the LIBERO number. They still cover one arm, one sensing setup, four tasks, and a company-run evaluation. Neither the simulation result nor the real-robot result establishes sustained operation.

SymmGrid asks how quickly a physical robot can learn

SymmGrid studies another question: can geometric symmetry generate useful replay-buffer variations and reduce physical learning time? The authors train directly on Franka robot arms for peg insertion, cable routing, and object relocation.

Its reporting unit is not one final leaderboard percentage. The paper compares learning curves, wall-clock convergence, success at matched checkpoints, and normalized area under the curve. It reports five training seeds for peg insertion and cable routing, three for object relocation, and 50 task attempts at evaluation points.

For peg insertion, the paper reports average convergence to 100% training accuracy in 18.5 minutes for SymmGrid versus 26 minutes for the SERL baseline. For cable routing, it reports 98% evaluation success as early as 10.9 minutes and 99.2% at 18.5 minutes. The authors also note that hardware differed from the original SERL report and explicitly call for more seeds or further analysis where results surprised them.

That disclosure is a reason to keep the conclusion bounded. SymmGrid supports an author-reported improvement under matched experiments in this study. It does not show that every symmetry augmentation will accelerate every robot, task, or controller.

A success rate needs at least twelve fields

Before comparing robot results, reconstruct this evidence card.

1. System under test

Is the tested object a vision-language model, a VLA policy, a complete robot stack, a controller, or a human-assisted service? HumanCLAW tests a frozen VLM inside a fixed harness. TurboVLA tests a trained action policy. Their units are different.

2. Task distribution

List the actual tasks and how they are weighted. An average across 40 tabletop tasks differs from 597 long-horizon interaction episodes or repeated trials of one contact task.

3. Environment

Record simulation, lab hardware, customer pilot, or recurring field operation. Simulation can improve repeatability and scale. Physical tests add sensing, control, and hardware failure modes. Customer operations add organizational and economic constraints.

4. Embodiment and sensors

A humanoid with an egocentric camera, a fixed arm with two RGB-D views, and a dual-arm system do not face the same action space or observability problem.

5. Training and test separation

Ask whether test tasks, homes, objects, scenes, and robot data appeared in training. A held-out environment test and a fine-tuned task test answer different questions.

6. Denominator

Convert every percentage into attempts where possible. Fifty rollouts per task makes a two-point change equal to one trial on that task. A percentage without a denominator cannot express evidence volume.

7. Repeated training seeds

Robot policies can vary with initialization and data order. Rollout count measures one checkpoint’s behavior. Multiple training seeds measure how reliably the method produces a good checkpoint.

8. Success definition

Define completion geometrically or operationally. HumanCLAW requires both an objective condition and model acknowledgement. Other benchmarks may use simulator predicates, human grading, milestone progress, or task-specific sensors.

9. Intervention, reset, and exclusion rules

State whether a human can rescue the robot, reset an object, stop unsafe motion, or discard invalid runs. Report aborted and excluded attempts separately. A nominally autonomous trial can still depend on substantial support between actions.

Our robot intervention and handoff guide expands this field into cut-in and cut-out quality, operator-control time, switching latency, recovery outcomes, and the downstream value of corrective data.

10. Compute and latency protocol

Record the complete hardware, precision, batch size, observation-to-action boundary, and latency distribution. A 31.2 millisecond policy measurement on an RTX 4090 is not automatically a 32 Hz closed-loop robot system on another computer.

11. Failure distribution

A single average can hide whether failures come from perception, language grounding, navigation, grasping, collision, timeout, or human support. HumanCLAW’s staged rates are more diagnostic than its final score alone.

12. Independence and artifacts

Check whether code, weights, data, task definitions, evaluation scripts, and trial logs are public. Then ask whether another team has reproduced the result. A public repository improves inspectability, but it is not independent replication.

More data does not repair a missing objective

Benchmark interpretation also reaches into representation learning. A separate July 2026 study created controlled environments with hidden mass, drag, and contact stiffness, then checked whether those properties were recoverable from observations and retained in learned world-model representations (paper).

The authors report that inputs limit what can be known, while prediction targets influence what is retained. Contact stiffness entered the tested latent representation when touch was forecast, but not merely because touch was supplied as an input. In RH20T experiments across 4,258 episodes, only the full multimodal objective beat a persistence baseline for force prediction, with gains increasing as data scaled.

The practical implication is narrow and important: a larger dataset cannot be assumed to fix a benchmark if the sensors omit the required signal or the objective does not pressure the model to retain it. Benchmark scope must include what the system can observe and what training asks it to predict.

A worked comparison rule

Two percentages can be compared directly only if a reader can answer yes to most of these questions:

• Are the tasks and task weights the same?
• Are the environments and object distributions aligned?
• Is the embodiment identical or intentionally controlled?
• Are training and test splits equivalent?
• Are checkpoints selected under the same rule?
• Are trial counts and repeated seeds comparable?
• Is success defined identically?
• Are intervention, reset, timeout, and exclusion rules the same?
• Are compute and latency measured on the same boundary and hardware?
• Are results independently reproduced or at least run in one matched harness?

If not, present the results side by side as different evidence, not as a ranking.

Proxy scores can also hide the downstream task. The latent action model evidence guide shows why reconstruction and action-probe metrics can screen representations without reliably ranking final robot policies.

What better robot benchmark reporting looks like

A strong report should publish:

1. a task manifest and sampling weights;
2. robot, sensor, controller, and compute specifications;
3. training data and test-separation rules;
4. checkpoint selection and the number of training seeds;
5. every trial denominator, timeout, reset, intervention, and exclusion;
6. an executable success definition;
7. task-level results with uncertainty, not only one aggregate;
8. failure categories and representative failure runs;
9. latency percentiles measured across the complete decision path;
10. code, weights, configs, evaluation harness, and versioned results where possible;
11. an explicit statement of what simulation or lab evidence does not establish;
12. independent reproduction before making broad comparative claims.

For a concrete example of unusually detailed task reporting, our ACT-2 reliability audit examines 785 laundry attempts, garment categories, fold quality, speed, adaptation cost, and missing field evidence. Our Xiaomi-Robotics-1 audit shows why simulation suites, bounded real-robot trials, and a 100,000-hour pretraining corpus must remain separate claims. The FLUX-mimic evidence audit applies the same discipline to a 20-trial company benchmark and factory partner evidence.

The Gemini Robotics 2 audit applies the same evidence card to whole-body Apollo 2 demonstrations, ER 2 progress metrics, multi-robot orchestration, and on-device adaptation claims.

The Ego2Robot synthetic-data audit shows why a fixed-frame comparison is stronger than ranking raw dataset hours. It also keeps 50-episode simulation tasks separate from 20-trial physical evaluations that use target-robot demonstrations.

Verdict

HumanCLAW’s most useful contribution is methodological. It does not merely report a low number. It constructs a benchmark where high-level decisions can fail while low-level skill execution remains comparatively controlled, then reports stage-by-stage outcomes, collisions, scene disturbance, motion quality, and computational cost.

TurboVLA and SymmGrid demonstrate why the same reading discipline must work in both directions. High percentages can be meaningful when the protocol and denominator are clear. They still belong to their declared task, embodiment, compute, and evaluation setting.

Read every robot success rate as a compact label for an experiment, not as a property of the robot or model. Reconstruct the experiment first. Compare the percentages second.

Frequently asked questions

Can two robot success rates be compared directly?

Only when the task, environment, embodiment, training data, checkpoint selection, trial protocol, success rule, intervention policy, and reporting unit are sufficiently aligned. Otherwise, the percentages answer different questions and should not be placed on one leaderboard.

What does HumanCLAW measure?

HumanCLAW measures high-level embodied decisions through a simulated humanoid whose low-level motion skills are supplied by a separate controller. It tests whether a frozen vision-language model can find, navigate to, and interact with targets while motor execution is held comparatively stable.

Is HumanCLAW a physical robot benchmark?

No. HumanCLAW runs in simulated indoor scenes with gravity, collisions, continuous whole-body motion, and movable objects. It is useful for isolating decision errors, but it does not measure sensing noise, actuator wear, hardware safety, or field reliability on a physical robot.

What does TurboVLA's 97.7% result mean?

The authors report 97.7% average success across four LIBERO suites, using 50 simulated rollouts for each of 40 tasks. The same paper separately reports 160 real-robot trials across four tasks. The simulation percentage should not be presented as the real-robot result.

How many trials are enough for a robot benchmark?

There is no universal minimum. More trials narrow uncertainty, but task diversity, repeated training seeds, independent replication, declared exclusions, and complete failure reporting also matter. A large denominator on one narrow task does not establish broad capability.

What should a complete robot benchmark report?

At minimum, report the task and environment distribution, robot and sensors, training and test separation, checkpoint selection, number of training seeds, trial count, success definition, intervention and reset rules, excluded attempts, confidence intervals, latency hardware, failure categories, and artifact availability.