Robot Demo Evidence Checklist: Autonomy, Resets, Speed, and Intervention

The short answer

A robot demonstration proves only the behavior that was actually shown under the disclosed conditions. It does not automatically prove autonomy, repeatability, useful speed, recovery, safety, or production readiness.

Use eight questions before repeating a demo claim:

1. What exact task was attempted?
2. Was the released run continuous and shown at normal speed?
3. How many total attempts produced how many successes?
4. Who held control during each part of the task?
5. What happened after an error?
6. How much did objects, layouts, and starting states vary?
7. Did the result meet a useful timing and quality requirement?
8. What operating evidence exists beyond the demonstration?

A polished clip may still document a real technical achievement. The purpose of this checklist is not to dismiss demonstrations. It is to keep the claim at the level the evidence supports.

A 60-second evidence check

When a new robot video appears, start with this compact card.

Field	Strong disclosure	Warning sign
Task	Inputs, required output, timeout, and success rule	A broad label such as “warehouse work”
Continuity	Full attempt is available, with edits identified	Rapid cuts hide setup, failure, or recovery
Attempts	All attempts, successes, failures, and exclusions	One selected success with no denominator
Control	Autonomous, teleoperated, supervised, and stopped time are separated	“Autonomous” with no authority record
Resets	Object resets, robot resets, fault clears, and human touches are counted	Support between clips is omitted
Speed	Playback rate and task-time distribution are reported	Accelerated video or one best-case cycle
Variation	Objects, poses, layouts, lighting, and disturbances are enumerated	One fixed setup presented as generality
Outcome	Accepted output, defects, damage, and recovery are reported	Motion completion substitutes for task quality
Operations	Hours, shifts, throughput, downtime, maintenance, and customer acceptance	A demo is described as deployment

If most fields are missing, the defensible label is demonstrated behavior with undisclosed repeatability and assistance.

Even a perfect disclosed attempt count leaves sampling uncertainty. The worked confidence-interval examples in our robot benchmark guide show why twenty successes do not establish a failure-free system.

1. Define the task before judging the motion

A robot can look capable while the task boundary remains vague. “Folding laundry,” “sequencing parts,” or “material handling” can describe many different experiments.

A useful task definition includes:

• starting state and permitted setup;
• object types, dimensions, weight, and presentation;
• required final state and quality tolerance;
• time limit and stop conditions;
• allowed fixtures, markers, maps, and prior knowledge;
• whether navigation, grasping, placement, and recovery all belong to the evaluated system;
• conditions that invalidate or exclude an attempt.

This matters because a demonstration can solve only one layer. A robot may execute a preselected motion while a person chooses the object. A planner may choose the right action while a separate controller executes it. A mobile system may navigate to a station but rely on a fixed fixture for manipulation.

Name the complete system under test. Do not transfer a successful component result to the whole robot.

2. Check continuity, editing, and playback speed

An uncut video can establish that the shown sequence occurred continuously. It cannot establish how often the sequence succeeds.

For video evidence, ask:

• Is the entire attempt available?
• Are cuts, time lapses, overlays, and repeated angles identified?
• Is playback at normal speed?
• Is task time measured from a declared start event to a declared completion event?
• Is waiting, planning, charging, operator response, and recovery included or removed?
• Does the clip begin after perception, grasp selection, mapping, or setup has already occurred?

Editing is not inherently deceptive. A short communication clip may need cuts. The problem is using an edited highlight as if it were a complete trial record.

The strongest release includes both a concise video and a protocol page with uncut representative successes, failures, timing, and attempt counts.

3. Demand the full attempt denominator

One success proves possibility under at least one observed condition. Repeatability needs a denominator.

At minimum, report:

• total attempts;
• successful autonomous attempts;
• successful assisted attempts;
• failed and aborted attempts;
• excluded attempts and exact exclusion rules;
• resets and repeated starts;
• number of days, operators, robots, and software versions;
• uncertainty or task-level variation when the sample supports it.

Do not combine assisted and autonomous completions without labels. Do not remove safety stops, operator rescues, or invalid starts from the denominator without explaining why.

Our robot benchmark success-rate guide expands this into task distributions, trial counts, checkpoint selection, repeated training seeds, failure categories, and artifact availability.

4. Reconstruct who controlled the robot

Autonomy is an operating record, not a visual style.

Boston Dynamics explicitly says the product Atlas supports autonomous, teleoperated, and tablet-steered control (Boston Dynamics). That is useful disclosure because the same robot can use different authority modes during setup, normal execution, exceptions, and recovery.

A complete authority record separates:

1. autonomous policy control;
2. scripted or preplanned execution;
3. continuous teleoperation;
4. brief operator intervention;
5. tablet or waypoint steering;
6. remote support without direct motion control;
7. protective stop and manual reset;
8. human work between attempts.

Report both event counts and time. Ten one-second corrections differ from one ten-minute teleoperated segment, even if total operator-control time is equal. Human attention also matters. An operator may monitor continuously while controlling the robot only briefly.

The robot intervention evidence guide provides a fuller record for cut-in, cut-out, switching latency, operator time, recovery, and post-correction evaluation.

5. Count resets, recovery, and hidden labor

A task can finish successfully while depending on substantial human support.

Count at least four kinds of recovery:

• Policy recovery: the robot corrects the error without authority changing.
• Assisted recovery: a person briefly takes control and returns the task to autonomy.
• Reset: the robot, object, fixture, or environment returns to an earlier state.
• Abort: the attempt ends without a valid completion.

Also record fault clearing, robot repositioning, object replacement, charging, calibration, cleaning, network repair, and software restart. These actions may be reasonable during development. Omitting them creates a false picture of operating burden.

A strong demo report shows representative failures. Failure video reveals whether errors are harmless, recoverable, damaging, or likely to require a technician.

6. Measure speed without hiding quality

Fast motion is not the same as useful throughput. Slow motion is not automatically a failure.

Report:

• median, lower-percentile, and worst observed task time;
• planning and perception latency;
• motion time and waiting time;
• retries and recovery time;
• accepted output per active robot-hour;
• defects, dropped objects, damaged items, and rework;
• the workflow’s actual cycle-time requirement.

Figure’s BMW report illustrates why target and result must remain separate. Figure published an 84-second total cycle requirement, above 99 percent successful placement per shift, and a goal of zero human pauses or resets per shift for Figure 02. The public source does not publish achieved distributions for those targets (Figure).

The production totals are still meaningful. BMW says Figure 02 supported more than 30,000 BMW X3 vehicles over ten months. Figure reports more than 90,000 parts and more than 1,250 runtime hours. Those figures support recurring work on one bounded task, but they do not turn target values into measured outcomes.

7. Test variation and disturbance deliberately

A fixed scene can be a valid engineering milestone. Generality requires controlled variation.

Describe what changed across attempts:

• object identity, geometry, weight, texture, and pose;
• container fill, clutter, occlusion, and presentation order;
• workstation position and layout;
• lighting and background;
• floor, slope, friction, and obstacles;
• payload and reach region;
• instructions and paraphrases;
• network, compute, and sensor conditions;
• disturbances introduced during execution.

Variation should not be implied by a montage. Publish the tested values and results by condition. A robot that succeeds on five objects is not automatically robust to an open object category.

Failures under variation are especially informative. They show where perception, planning, contact, balance, or recovery stops transferring.

8. Separate a demonstration from operating evidence

A site video can establish that a robot performed a behavior at that site. It does not establish customer acceptance or recurring production.

Agility Robotics describes a three-stage path for Digit: controlled solution setup, a cordoned customer-site proof of concept, then 90 days of production measurement covering uptime, throughput, reliability, operational impact, and return on investment (Agility Robotics). The stages are useful because they prevent a setup test from inheriting the status of an operating deployment.

BMW’s Figure 02 record provides a stronger operating case. The customer confirms the site, bounded task, ten-month period, and vehicle-production context (BMW Group). Figure adds runtime and part totals. Important fields remain unpublished, including fleet size, intervention totals, downtime, maintenance, safety denominators, and commercial terms.

Use this evidence ladder:

Level	Minimum evidence
Announced	A company describes a planned system or capability
Demonstrated	The named robot performs the bounded behavior under stated conditions
Piloting	The robot is evaluated at a partner or customer site with a defined protocol
Commercially accessible	A buyer has a real path to contract, order, or qualify for access
Production deployed	A customer accepts recurring work with operating time, output, support, and reliability evidence

A result can occupy more than one commercial category, but later stages must not be inferred from earlier ones.

A publication template for robot demos

Teams can attach this compact disclosure to a video or live event.

System

• Robot, hardware revision, end effector, sensors, policy, controller, and software version
• Onboard and offboard compute
• Network dependencies

Task

• Starting state, required output, tolerance, timeout, and failure rule
• Objects, fixtures, layout, and prior information
• Tested variations and exclusions

Trial record

• Dates, robots, operators, total attempts, successes, failures, aborts, and exclusions
• Uncut representative successes and failures
• Playback rate and timing distribution

Human support

• Time and events by autonomous, teleoperated, intervention, stop, and reset mode
• Monitoring time, remote support, on-site support, and between-attempt labor
• Recovery outcomes and repeat handoffs

Outcome

• Accepted output, quality defects, retries, damage, and rework
• Throughput per active hour and scheduled hour
• Safety stops, near misses, incidents, and corrective actions

Evidence boundary

• What the demonstration establishes
• What it does not establish
• Whether a customer or independent team confirmed any part of the result
• Links to protocol, logs, artifacts, and longer operating records

Common claims and defensible replacements

Weak claim	Better wording
“The robot is autonomous.”	“The robot completed the shown run without disclosed direct control; intervention and support across all attempts were not published.”
“The task is reliable.”	“The company showed one or more successful runs; the complete attempt denominator and failure distribution were not published.”
“The robot works at human speed.”	“The shown run took the reported time under the stated setup; a matched workflow timing distribution was not published.”
“The robot recovered from failure.”	“The shown failure was followed by recovery; recovery success across all relevant failures is unknown.”
“The robot is deployed.”	“The robot was demonstrated or piloted at the named site; recurring accepted operation is not yet established.”
“No human was involved.”	“No direct control was disclosed during the shown attempt; setup, monitoring, resets, remote support, and excluded attempts remain undisclosed.”

Precise wording protects both readers and credible teams. A bounded result can still be impressive.

Evidence boundaries in the four primary examples

Evidence	What it supports	What remains missing
Atlas has autonomous, teleoperated, and tablet-steered modes	Multiple declared authority modes for one product	Time, attempts, and outcomes in each mode
Agility defines setup, site proof of concept, and 90-day production measurement	A deployment and measurement process	A complete public customer dataset from that process
BMW confirms Figure 02 production context for ten months	Recurring customer operation on one bounded workflow	Fleet, intervention, downtime, maintenance, safety, and economics denominators
Figure reports 90,000+ parts and 1,250+ runtime hours	Substantial vendor-reported throughput and runtime	Attempts, achieved target distributions, support burden, and independent audit

These sources do not create a universal standard. Together they show why control mode, stage, customer confirmation, duration, throughput, and missing denominators should be reported separately.

Verdict

Robot demonstrations are valuable evidence when the claim stays bounded. A selected clip can establish that a named system performed a behavior. An uncut run can add continuity. Repeated trials can add reliability evidence. Declared authority modes can clarify autonomy. Customer-confirmed hours and output can establish recurring operation.

No one layer substitutes for the next.

The practical rule is simple: preserve the full denominator. Count attempts, control modes, interventions, resets, time, variation, failures, accepted output, and operating support. Then describe exactly what that record proves, and nothing more.

Frequently asked questions

How can you tell whether a robot demo is autonomous?

A video alone rarely proves autonomy. Look for a declared control mode, the share of task time under autonomous control, teleoperation or operator-control time, interventions, resets, protective stops, and support between attempts. If those fields are absent, classify autonomy as undisclosed rather than assuming it.

Does an uncut robot video prove repeatable performance?

No. An uncut run can establish continuity for the shown attempt, but it does not provide the number of attempts, failed runs, selection process, environment variation, reset labor, or repeatability across days and operators.

What is the most important denominator in a robot demonstration?

There is no single universal denominator. At minimum, ask for total attempts, autonomous attempts, successful completions, interventions, resets, elapsed task time, and the number of tested objects, layouts, and days. Field claims also need scheduled and active operating hours.

Should a slowed robot demo be considered a failure?

Not automatically. Slow operation can be appropriate during development or for safety. The evidence problem appears when playback speed, task time, timeout, or production cycle requirements are hidden. Report the measured speed and compare it with the intended workflow.

What separates a demonstration from a production deployment?

A demonstration shows a behavior under a bounded setup. A production deployment adds customer acceptance, recurring shifts, accepted output, intervention and support records, reliability, maintenance, safety controls, and a defined reporting window.

What should a company publish with a robot demo video?

Publish the robot and software version, task and success rule, complete attempt count, uncut representative runs, failures, control modes, interventions, resets, playback speed, timing distribution, tested variation, exclusions, safety controls, and links to logs or a fuller protocol where possible.