Dynamic Robot Benchmarks: How to Measure Latency, Recovery, and Process Quality

The short answer

A dynamic robot benchmark must measure when an action becomes useful, not only whether a task eventually ends in success. A policy can identify the correct target but react after it has moved. It can reach 90% of a task and lose the object. It can recover from a setback or finish only after repeated corrections. A single success bit hides all four cases.

Two August 2026 research releases make this gap concrete. ReflexBench tests six simulated manipulation tasks with moving targets and configurable policy delay. Its evaluation loop supports synchronous and asynchronous inference instead of assuming the environment freezes while the policy thinks (paper, project page). PRM-as-a-Judge 1.5 takes rollout videos, estimates a progress curve, and derives metrics for partial progress, regression, recovery, and the quality of successful execution (paper, assessment suite).

These tools answer different questions:

Evaluation layer	Main question	Useful outputs	Important limitation
Latency-aware task benchmark	Did the policy react within the task’s timing window?	Success by task, latency, frequency, schedule, action horizon	Simulator timing still needs physical validation
Outcome metric	Did the rollout complete the declared task?	Success rate and trial count	Hides timing, path, regression, and recovery
Progress assessment	How did execution evolve before the outcome?	Milestones, maximum progress, stagnation, drawdown, recovery	Depends on the progress judge’s validity
Field evaluation	Can the full system perform recurring work safely and economically?	Uptime, throughput, interventions, recovery time, quality, incidents	Expensive and specific to the deployment

The releases are complementary, not a universal leaderboard. Both report author-run results. Neither establishes independent replication, certified safety, long-duration reliability, or production value.

For the broader rules behind denominators and protocol matching, read our practical audit of robot benchmark success rates. This guide focuses on what changes when the environment moves and the execution path matters.

Why a correct action can still be a failed action

A robot policy operates in a time pipeline:

sensors expose an observation;
the system captures and preprocesses it;
the policy computes an action or action chunk;
middleware schedules the output;
the controller and actuators execute it;
the environment changes throughout the process.

If an object moves during steps 2 through 5, the action can be logically correct for an old state and physically wrong for the current one. Reporting model-forward latency alone does not capture camera buffering, preprocessing, synchronization, network transfer, queueing, action decoding, controller delay, or actuator response.

Static simulation benchmarks often avoid this problem by pausing the simulator during inference. That makes experiments repeatable, but it removes the observation-to-execution gap that a physical system must survive. ReflexBench instead decouples simulator stepping from robot control and injects a delay based either on a configured value or a real policy latency adjusted by the simulator’s real-time factor (ReflexBench paper, Section III).

This is not merely a speed test. Dynamic control depends on at least three linked properties:

responsiveness, how quickly a usable action arrives;
anticipation, whether the action accounts for where the scene is heading;
scheduling, how perception, policy computation, and action execution overlap.

A fast policy can still chase the current image without anticipating motion. A predictive policy can still miss if its output arrives too late. An asynchronous policy can keep moving while computing, but may execute actions based on stale observations.

What ReflexBench adds

ReflexBench defines six reaction-critical tasks:

Conveyor Belt Pick-and-Place, grasp a moving cube and place it in a bin;
Ball Catching, intercept an incoming ball;
Whack-a-Mole, reach targets before they disappear;
Rolling Ball Interception, stop a ball before it leaves a slope;
Ball Throwing, throw a ball into a bin;
Rotating Peg Insertion, align a peg with a continuously rotating socket.

The set includes interception, timed contact, moving-goal alignment, and target disappearance. ReflexBench uses 200 demonstration episodes per task for training. The reported benchmark result for each policy is based on 150 evaluation episodes per task and is presented as the mean and standard deviation across three runs with different random seeds (paper, experimental setup).

Those denominators are useful. They let a reader separate one rollout from repeated simulator evaluation and one checkpoint from variation across runs. They do not establish that the same policy will retain its ranking on different hardware, sensors, objects, motion distributions, or safety constraints.

Synchronous and asynchronous execution are different experiments

Under synchronous inference, the policy receives an observation, computes, then begins the selected action after the modeled delay. Under asynchronous inference, the robot continues executing the previous action chunk while the policy computes the next one. The new action becomes available in a later control period.

Neither mode is automatically superior. ReflexBench reports that asynchronous execution can suffer more at low inference frequency because the observation-action mismatch is larger. At higher frequencies, it can benefit from continuous execution and outperform the synchronous setup on the two tasks used for this ablation (paper, Figures 4 and 5).

This result should be read as a protocol interaction, not a general law. The crossover depends on task motion, policy, chunk size, executed horizon, controller behavior, and latency. A benchmark should therefore publish the full schedule rather than labeling a system simply real time.

Action chunk size is not the executed horizon

A policy may predict several future actions in one call but execute only a prefix before replanning. ReflexBench distinguishes:

chunk size, the number of actions predicted by the policy;
action horizon, the number of those actions actually executed before the next policy update.

The paper reports that larger predicted chunks paired with shorter executed horizons generally worked better under its high-frequency asynchronous setup, although the best chunk varied by task. Subsequent comparisons use chunk size 8 and action horizon 2 (paper, Section V-B).

This distinction matters outside the benchmark. A long prediction can provide temporal coherence. Executing too much of it open loop can make the robot slow to correct. Executing too little can increase compute demand and expose the system to scheduling jitter.

What the ReflexVLA results do and do not show

The authors introduce ReflexVLA alongside the benchmark. The 1-billion-parameter policy adds latent future prediction, two-frame temporal fusion, batched visual encoding, and CUDA Graph replay. Under the authors’ matched ReflexBench protocol, it reports 50.4% average success, compared with 30.3% for its VLA-Adapter backbone and 50.2% for the 4-billion-parameter PUMA baseline (paper, Table I).

The average conceals a wide task distribution. ReflexVLA reports 100% on Whack-a-Mole, 77.1% on Rolling Ball Interception, 73.8% on Conveyor Belt Pick-and-Place, 31.7% on Ball Throwing, 12.4% on Rotating Peg Insertion, and 7.3% on Ball Catching. The benchmark is therefore not saturated merely because one task reaches 100%.

A progressive ablation on Conveyor Belt Pick-and-Place reports:

Configuration	Reported success	Reported latency
VLA-Adapter baseline	36.8%	81.522 ms
Frozen-target future prediction	62.8%	82.508 ms
Middle-feature temporal fusion	71.7%	125.107 ms
Batched encoding plus CUDA Graph	73.8%	64.991 ms

The comparison supports a bounded conclusion: under this task and implementation, prediction and temporal context improved success, while systems optimization recovered latency and improved the final result (paper, Table III). It does not isolate a universal causal effect across robots or prove that the same CUDA path is available on another deployment computer.

The paper also reports 97.2% average success on LIBERO, close to the 97.3% VLA-Adapter result in the same table. That is evidence that the dynamic-task changes did not materially reduce performance under this static simulated suite. It is not evidence that LIBERO predicts reaction-critical performance.

Physical tests add evidence, but remain bounded

ReflexVLA is also tested on an AgileX Piper arm for three tasks. Each policy is evaluated 20 times per task. The authors report:

Conveyor Belt Pick-and-Place: 16 successes in 20 attempts for ReflexVLA, 13 in 20 for PUMA, and 2 in 20 for SmolVLA;
PressButtons: an average of 22.5 correct presses within 30 seconds for ReflexVLA;
CatchBalls: an average of 6.7 catches out of 10 balls for ReflexVLA.

These physical results are more relevant to real sensing and execution than simulation alone. They remain one arm, three tasks, author-run evaluation, and 20 trials per policy-task condition. The sources do not provide long-duration operation, independent reproduction, safety events, intervention counts, maintenance, energy use, or customer workflow evidence.

Why final success still is not enough

Latency-aware task success reveals whether a policy acted in time. It still compresses the path into one outcome. Consider four rollouts:

the robot fails before touching the object;
the robot almost completes the task, then drops the object;
the robot loses progress, recovers, and succeeds;
the robot succeeds after repeated stalls and corrections.

Binary evaluation maps the first two to failure and the last two to success. An engineer needs the differences to decide what data to collect, what subsystem to improve, and when human intervention may be necessary.

PRM-as-a-Judge 1.5 proposes an outcome, process, and diagnosis framework. It receives a task instruction and rollout video, optionally with other views. A process reward model estimates normalized progress over time. Metrics are then calculated from the resulting curve (paper, Section 2).

The progress metrics in plain language

Metric	What it asks	Better direction	Eligibility
Milestone Coverage (MC@q)	What share of rollouts reached a declared progress threshold?	Higher	All rollouts
Max Progress (MP)	How far did the rollout get at its best point?	Higher	All rollouts
Path-weighted Progress Length (PPL)	How directly did execution reach its best point?	Higher	All rollouts
Cumulative Regret Area (CRA)	How far and how long did execution stay below its best-so-far state?	Lower	All rollouts
Stagnation Ratio (STR)	How often was progress change below a noise threshold?	Lower	All rollouts
Failure Near-Success (FNS)	How close did failed rollouts come to meaningful later milestones?	Higher	Failed rollouts only
Drawdown Recovery Ratio (DRR)	How much of the largest setback was later recovered?	Higher	Rollouts with a drawdown only
Success Quality Score (SQS)	How efficient and stable were successful rollouts?	Higher	Successful rollouts only

The conditioning is essential. FNS should not be averaged over successes. DRR excludes rollouts without a drawdown. SQS applies only to successful rollouts. The paper reports continuous metrics as the median over eligible rollouts and milestone coverage and success as percentages (metric definitions).

Failure Near-Success

FNS combines maximum progress with whether a failed rollout crossed the 50% and 75% milestones. A late-stage failure receives more credit than an early failure.

That does not make late failure acceptable. It makes the failure more diagnostic. A near-success may point to final alignment, grasp retention, termination logic, or recovery data. An early failure may point to perception, grounding, reachability, or task initialization.

Drawdown Recovery Ratio

DRR finds the largest loss from the rollout’s best-so-far progress, then asks how much of that loss was regained. A value of 1 represents full recovery after the largest drawdown. A value near 0 means the rollout did not regain the lost progress.

This metric connects naturally to robot intervention and handoff evidence. Autonomous recovery, assisted recovery, manual reset, and task restart should remain separate labels. A progress curve should not make hidden human support look like policy recovery.

Success Quality Score

SQS combines path efficiency, low regression, and low stagnation for successful rollouts. It separates a direct completion from a success that contains repeated failed attempts or long pauses.

A composite score should never replace its components. Two rollouts can receive a similar SQS for different reasons. Publish PPL, CRA, and STR alongside SQS, plus the progress curve and representative videos.

The judge must be evaluated too

A learned progress curve is not ground truth. It is a model output. PRM-as-a-Judge 1.5 introduces RoboPulse++ to evaluate whether process judges recognize rising and falling progress in robot video.

The paper describes 700 trajectories, 275 task entries, 17,052 frames, and 2,244 human-annotated intervals. Of the trajectories, 439 are from real-world sources and 261 are from simulation. Annotators view the complete trajectory and label contiguous intervals as rising or falling based on task-relevant state change (paper, Appendix C).

The best reported specialized setting, Robo-Dopamine Forward, reaches 0.77 macro F1 and 0.84 accuracy. Its rising-progress F1 is 0.92, while falling-progress F1 is 0.61. The best falling F1 in the table is 0.63. The authors conclude that recognizing regression remains the principal weakness of the evaluated judges.

That limitation matters because recovery metrics depend on detecting a real drawdown. If the judge misses a dropped object or an undone subgoal, DRR and SQS can be misleading.

Temporal context also changes results. On a context-dependent subset, the sequence-style RoboMeter reports 0.901 falling accuracy versus 0.408 for the pair-style Robo-Dopamine Forward setting. This supports a narrow point: a state can look locally reasonable while reversing progress established earlier. It does not make every sequence judge more accurate on every task.

What the process assessment found

PRM-as-a-Judge 1.5 applies its metrics to released RoboDojo rollout videos. It does not retrain the evaluated policies. The paper analyzes 4,606 simulation and 1,470 physical rollouts when comparing progress-derived and benchmark-reported success rates (paper, Appendix E).

Three results are useful for benchmark readers:

Rankings change by metric. Success-rate ordering is not fully consistent with recovery, near-success, or successful-execution quality.
Simulation and physical rankings diverge. Across nine shared models, the paper reports Spearman correlations from 0.18 for maximum progress to 0.58 for success rate. Precise alignment and complex interaction show larger physical degradation in its shared-task analysis.
The progress judge approximately recovers reported aggregate success. Model-level progress-derived success differs from RoboDojo’s reported rate by a mean absolute 1.57 percentage points in simulation and 1.32 points in physical evaluation, with rank correlations of 0.88 and 0.96 respectively.

All three are author-run analyses using one released benchmark and the available rollout videos. They should motivate richer reporting, not establish a permanent ranking of the named models.

A complete dynamic-evaluation card

Before accepting a dynamic robot result, reconstruct these fields.

1. Task dynamics

Describe target speed, acceleration, path distribution, appearance window, contact dynamics, timing tolerances, distractors, and randomization. Average success without a motion distribution cannot be transferred to another line speed or object path.

2. Timing boundary

Define where latency starts and ends. Prefer sensor exposure to effective actuator command, with camera, preprocessing, policy, middleware, queueing, controller, and actuator components measured separately.

3. Latency distribution

Report median, tail percentiles, worst observed delay, jitter, warm-up, and missed deadlines. One mean can hide occasional delays that dominate reaction-critical failures.

4. Execution schedule

State synchronous or asynchronous inference, observation age, queue replacement rule, action chunk size, executed horizon, replanning frequency, dropped-action behavior, and what happens when computation misses a deadline.

5. Compute and system configuration

Name robot, cameras, control computer, accelerator, numeric precision, batch size, software versions, and whether optimizations such as graph capture were enabled. Do not convert model latency into system frequency without the complete path.

6. Outcome denominator

Publish trials per task, task weights, training seeds, evaluation seeds, excluded attempts, aborts, resets, timeouts, and confidence intervals. Keep simulation and physical denominators separate.

7. Process metrics

Publish milestone coverage, maximum progress, path efficiency, regression, stagnation, recovery, and successful-execution quality where the judge is validated for the task.

8. Judge validity

Report the process judge, version, sampling rate, smoothing, viewpoints, task instruction, calibration, human agreement, rising and falling accuracy, and known failure classes. Compare against instrumented state when possible.

9. Intervention and safety

Label autonomous recovery, remote assistance, operator control, manual resets, emergency stops, collisions, and damaged objects. Functional timing success does not establish safe operation.

The adaptive robot safety case guide explains how to keep task success, safe task completion, engagement, formal requirements, runtime-monitor coverage, and physical validation as separate evidence layers.

10. Artifact and replication state

Release task definitions, simulator version, demonstration data or generation method, evaluation code, checkpoints, raw trial outcomes, progress curves, and representative failures. Then state whether another team has reproduced the result.

A recommended scorecard

A deployment-minded benchmark should publish four connected panels:

Longer tasks also need explicit state and planning interfaces. Our guide to planning, memory, and whole-body execution extends this timing and process framework to high-level subtask search and humanoid control.

Panel	Minimum measures
Outcome	Success, failure, abort, timeout, denominator, uncertainty
Timing	Observation age, end-to-end latency percentiles, inference frequency, action schedule, missed deadlines
Process	Milestones, max progress, regression, stagnation, drawdown recovery, success quality
Operations	Interventions, resets, safety stops, recovery time, throughput, uptime, quality defects

No one panel substitutes for the others. ReflexBench strengthens the timing panel. PRM-as-a-Judge strengthens the process panel. A buyer still needs the operations panel from a representative physical workflow.

Evidence boundaries

Claim	Evidence class	Confidence	Boundary
Configurable latency changes dynamic-task success	Author-run controlled simulation	High for the reported setup	Not independently replicated across hardware or task distributions
ReflexVLA improves the VLA-Adapter backbone on ReflexBench	Author-run matched benchmark and ablation	Medium-high	Method and benchmark come from the same team
ReflexVLA works on three physical dynamic tasks	Author-run physical evaluation	Medium	Twenty trials per policy-task condition, no long-duration field evidence
Progress metrics distinguish failures, recoveries, and successful paths	Defined toolkit applied to released rollouts	Medium-high as a diagnostic framework	Depends on the process judge and metric design
Falling progress is harder for current judges	RoboPulse++ author-run comparison	Medium-high for the evaluated models	Dataset and annotations need outside reproduction
Process metrics predict production reliability	Not established	Low	No customer uptime, throughput, safety, or economic validation

Verdict

ReflexBench makes one hidden systems variable explicit: the environment continues moving while the policy computes. PRM-as-a-Judge 1.5 makes another hidden variable visible: two identical final outcomes can contain very different progress, regression, and recovery paths.

Together they support a stronger evaluation rule:

Measure whether the robot acted in time. Measure what happened between start and finish. Validate the measurement system itself. Then keep benchmark evidence separate from recurring field performance.

Frequently asked questions

Why are static robot benchmarks insufficient for dynamic tasks?

Static benchmarks often pause the environment while a policy computes its next action. Moving objects do not pause on a physical robot, so observation age, inference delay, execution scheduling, and reaction timing can determine whether an otherwise capable policy succeeds.

What does ReflexBench measure?

ReflexBench tests six simulated reaction-critical manipulation tasks while decoupling simulator stepping from robot control. It supports synchronous and asynchronous inference, configurable latency, and evaluation of how inference frequency, action chunks, and execution horizons affect task success.

What is a robot progress curve?

A progress curve estimates how close a rollout is to completing its task at each sampled time. It can expose advancement, stagnation, regression, recovery, and inefficient successful execution that one final success label cannot show.

Can PRM-as-a-Judge replace ground-truth task success?

No. A learned process reward model is another measurement system with its own errors. Use it alongside explicit task success, instrumented state when available, human review, denominators, and uncertainty. RoboPulse++ shows that recognizing negative progress remains materially harder than recognizing positive progress.

What should a dynamic robot benchmark report?

Report the task dynamics, simulator or physical setting, observation and control rates, complete observation-to-action latency distribution, synchronous or asynchronous schedule, action chunk and executed horizon, trial counts, seeds, success rule, progress and recovery metrics, interventions, exclusions, failure classes, hardware, artifacts, and independent replication status.