When Should a Robot Hand Control to a Human? An Intervention Evidence Guide

The short answer

A robot should hand control to a human when continuing autonomously is less defensible than requesting bounded help, and it should return control only after both the state and the proposed action are supported again. Measuring that process requires more than a final success rate.

A useful intervention report answers four separate questions:

1. Detection: Did the system recognize that autonomous execution had become unsupported?
2. Handoff: Did control move to the operator quickly, without unnecessary transfers?
3. Recovery: Did the operator restore a state from which the policy could continue?
4. Learning: Did the retained correction improve later unassisted performance?

AutoIntervene, a research preprint posted on August 7, 2026, provides a concrete physical test of this sequence. The method monitors short action chunks proposed by a robot policy, compares them with successful visual and action references, transfers control to an operator after repeated rejection, and returns control after repeated acceptance. Successful correction segments become training data for the next policy round (paper, project page).

The authors report meaningful results on two fixed robot arms and nine bimanual tasks. In the seven-task main benchmark, average unassisted task success rose from 30.9% for the initial policy to 80.0% after two AutoIntervene adaptation rounds. Manual switching reached 68.6%. AutoIntervene also used less recorded operator-control data on average.

Those numbers support a bounded conclusion: automatically selected correction segments improved these policies under the authors’ lab protocol while using less operator-control time than manual switching. They do not establish safe unattended operation, general failure detection, production staffing economics, or independent replication.

A separate approach can keep the large policy frozen and learn only which proposed actions have higher value. Our Q-Planning evidence audit compares that failure-reuse loop with the human resets, terminal labels, physical denominators, and missing intervention path that still belong in the operating record.

Why intervention is its own system metric

A robot can complete a task after human help and still be commercially weak. It can also fail a task after correctly requesting help, because the operator could not recover it. One aggregate success number collapses distinct system behaviors. The robot demo evidence checklist provides the wider review frame for continuity, edits, speed, variation, resets, and deployment status around this intervention record.

Consider three rollouts that all end successfully:

• the policy completes autonomously;
• a person controls the robot for two seconds after one failed grasp;
• a person controls most of the task after several resets.

All three can be labeled successful, but their autonomy, support cost, and learning value differ. The same distinction matters when reading physical AI deployment evidence or planning a 90-day humanoid pilot.

Intervention should therefore be logged as a control mode with timestamps, triggers, outcomes, and attribution. It is not a footnote to task success.

The FolDeX recovery-data audit adds a complementary case. The authors report a 14.25 percentage-point average success gain after recovery trajectories were included, but do not disclose the number of interventions, recovery duration, or complete operator effort. A downstream gain does not by itself quantify the handoff system that produced the data.

How AutoIntervene structures the handoff

AutoIntervene targets action-chunking policies. Instead of predicting one command at a time, these policies propose a short sequence of future joint and gripper commands. Smooth chunks can improve temporal consistency, but smooth motion can still be wrong after a missed contact, perception error, or accumulated execution drift.

The method adds a deployment monitor with four steps.

1. Build a memory from successful behavior

Each training trajectory contributes visual embeddings and matching action chunks. The memory represents combinations of scene appearance and action that occurred in successful demonstrations.

Visual similarity alone is not enough. Two task phases can look alike while requiring different motions. AutoIntervene also compares the proposed chunk with nearby reference actions, normalized by the variation in recorded actions.

2. Use phase-local support during autonomous control

While the policy controls the robot, retrieval is constrained to forward-moving windows on matched training trajectories. This is intended to prevent a visually similar frame from an earlier or different task phase from validating the wrong action.

A proposal is accepted only when visual support is high enough and action risk is low enough. Repeated rejection transfers control to the operator.

3. Use global support during recovery

The operator may change the object state or complete part of the task. The policy continues making background predictions while the person controls the robot. AutoIntervene searches the full memory during this period because recovery may return the system to any supported task phase.

Repeated acceptance returns control to the policy. Separate policy-side and operator-side thresholds create a form of hysteresis, reducing rapid reversal between modes.

4. Retain correction segments from successful rollouts

The operator-controlled interval begins at policy-to-operator transfer and ends when autonomy resumes or the episode terminates. Segments from successful rollouts are added as separate corrective trajectories. The next policy is trained on both prior data and the new intervention data.

This is important evidence discipline. The method does not call every operator second equally valuable. It attempts to collect data specifically from states the learned policy reached and could not handle.

For the wider role of demonstrations, teleoperation, and corrections, see how robot training data is made. Its DAgger collection and evaluation framework distinguishes expert action labels from control takeovers and keeps assisted collection separate from unassisted policy tests.

What the physical evaluation tested

The experiments used two fixed AgileX PiPER-X six-degree-of-freedom arms, stock parallel grippers, one overhead camera, and one wrist camera per arm. The policy observed joint and gripper values, measured torques, and RGB images. Policy commands ran at 30 Hz. Monitoring ran at 5 Hz during policy control and 30 Hz during operator control.

The nine tasks covered rigid, articulated, and deformable objects:

• peg disassembly;
• potato transfer between hands;
• towel folding;
• towel bagging;
• lidded box packing;
• plant sorting;
• towel box packing;
• two-towel box packing;
• towels-and-cable bagging.

For each task, the team collected 36 initial expert trajectories. Thirty trained the initial policy and built its memory. Six successful trajectories formed a held-out calibration set. Each intervention round retained corrections from five successful deployment rollouts per task. Unless otherwise stated, task success was then measured over 25 unassisted physical rollouts for each task and method.

This protocol separates data collection, calibration, and unassisted evaluation more clearly than a demonstration video. It remains a single-team evaluation on one fixed bimanual hardware setup.

The main result needs two denominators

The seven-task comparison reports both final task success and recorded control-data time.

Method after two rounds	Average task success	Average cumulative additional operator-control data	What the number means
Initial policy	30.9%	None	Policy trained on 30 initial demonstrations per task
Manual switching	68.6%	179.9 seconds	A person chose when to take and return control
AutoIntervene	80.0%	122.9 seconds	The monitor chose both handoff directions
Additional full demonstrations	56.0%	1,442.9 seconds total demonstration time	Ten more nominal full trajectories per task

The authors report that AutoIntervene improved mean success by 49.1 percentage points from the initial policy. They also state that it used approximately 74% less additional recorded control-data time than collecting full additional demonstrations.

The comparison supports targeted correction under this protocol. It does not show that 122.9 seconds is the total human labor required. An operator still monitored collection, handled the leader arms, reset trials, and participated in setup. The paper reports recorded control data, not attention time, response readiness, training cost, or a production support ratio.

Manual switching also improved the policy substantially. AutoIntervene’s advantage is not that people became unnecessary. Its reported advantage is more focused authority transfer and corrective data collection.

The controlled handoff test is promising and small

The paper separately compares handoff monitors on lidded box packing. Each method received 10 perturbed rollouts after a fixed five-centimeter box translation and 10 nominal rollouts, with each rollout capped at five complete handoff cycles.

AutoIntervene reported:

• 1.00 cut-in recall;
• 1.00 cut-in precision;
• 1.00 cut-out recall;
• 1.00 cut-out precision;
• 0.00 nominal false-trigger rate.

Prior monitors missed recoveries or triggered unnecessarily under the authors’ settings. Ablations also showed why both signals mattered. Removing visual support missed the workspace displacement. Removing action risk made return-to-policy decisions less reliable.

This is useful mechanism evidence, not a universal 100% handoff claim. Each rate comes from one task, one specified perturbation, 10 perturbed rollouts, and 10 nominal rollouts. The method still needs broader failure classes, operators, environments, lighting, occlusion, object variation, hardware faults, and longer operating periods.

Eight metrics every intervention report needs

1. Autonomous task success

Report attempts completed with no operator control. Keep this separate from assisted final success.

2. Intervention frequency

Use a denominator that fits the operation: interventions per attempt, task hour, motion hour, cycle, or item. Report all relevant denominators when task duration varies.

3. Operator-control time

Measure seconds under human authority and the fraction of task time. Also report human attention time, because an operator can remain on call even when not moving the robot.

4. Cut-in quality

Report missed intervention opportunities, valid cut-ins, extra cut-ins, precision, recall, and false triggers during nominal operation. Define who labels the moment when intervention became necessary.

5. Cut-out quality

Report whether autonomy resumed after a valid recovery, whether return was premature, and whether the system immediately handed control back. Bidirectional handoff is not complete if the monitor can detect failure but cannot recognize recovery.

6. Switching latency

Measure time from the first observable failure condition to autonomous motion stopping, then to effective operator control. Also measure recovery completion to resumed policy control. AutoIntervene provides example timeline timestamps, but the paper does not publish a latency distribution.

7. Recovery and final outcome

Separate successful recovery, failed recovery, abort, reset, collision, damage, timeout, and final task completion. One intervention can lead to another, so report repeat handoffs per rollout.

8. Corrective-data value

Evaluate the adapted policy in separate unassisted trials. Report how many correction segments, seconds, and task states were retained, then compare improvement across repeated training seeds. Otherwise, the claim may describe useful rescue without proving useful learning.

Our robot benchmark evidence audit provides the broader protocol fields needed around these intervention metrics.

A control handoff is not hazardous-energy isolation

Transferring authority to a person does not by itself make it safe to enter the robot’s work area. Remote correction, a protective stop, and servicing under an energy-control procedure are different operating states. An intervention report should not combine them into one category called “human help.”

For U.S. general-industry workplaces, 29 CFR 1910.147 addresses servicing and maintenance where unexpected energization, startup, or release of stored energy could injure employees. Its definition of servicing includes cleaning, unjamming, adjustments, and tool changes when that exposure exists. Normal production operations have a separate boundary: servicing during production is covered under the conditions in paragraph (a)(2)(ii), including guard removal or bypass and entry into a point of operation or associated danger zone. The section also has explicit exclusions and exceptions (GPO, 2025 annual CFR edition, paragraphs (a) and (b)).

This is an explanation of that published edition, not a determination of current legal compliance for a particular installation. Confirm current requirements, jurisdiction, task, and the complete system with a qualified workplace-safety specialist. Do not extend this workplace rule into a universal claim about home robots or every robot intervention.

Stopped motion and isolated energy are different claims

Paragraph (b) explicitly says that push buttons, selector switches, and other control-circuit devices are not energy-isolating devices. A pause command or control handoff is therefore not evidence that the machine has been physically isolated from its hazardous energy sources. This distinction does not make protective stops unimportant. It means the evidence must match the activity and the claimed protection (GPO, paragraph (b)).

The same text identifies electrical, mechanical, hydraulic, pneumatic, chemical, thermal, and other energy sources. Paragraph (d) separates shutdown, isolation, lockout or tagout application, control of stored energy, and verification. Potentially hazardous stored or residual energy must be rendered safe. Where energy can reaccumulate to a hazardous level, verification continues until servicing is complete or that possibility no longer exists. The authorized employee verifies isolation and deenergization before work begins (GPO, paragraphs (b) and (d)).

For a robot buyer, the practical question is not just “Did motion stop?” It is “What protects the person during this specific recovery activity, and what evidence supports that protection?” The robot safety standards guide places that question within application-level risk assessment and lifecycle access.

A quick reset is not automatically an exception

The minor-servicing exception in paragraph (a)(2)(ii) is conditional. The activity must take place during normal production, be routine, repetitive, and integral to equipment use for production, and use alternative measures that provide effective protection. Short duration alone is not the exception. Calling jam clearing a “reset” does not settle which requirements apply (GPO, paragraph (a)(2)(ii)).

When temporary energization is needed for testing or positioning during covered servicing, paragraph (f)(1) describes a specific sequence, including clearing tools and materials, removing employees from the equipment area, removing devices under the specified procedure, testing or positioning, then deenergizing and reapplying energy controls before servicing continues. It is not a general permission to resume autonomy while someone is still working in the danger zone (GPO, paragraph (f)(1)).

Add a recovery-access record beside the intervention metrics

As an editorial reporting recommendation, record:

• Activity and access: remote control from outside the hazard area, physical reset, unjamming, cleaning, inspection, or repair, with the person’s location.
• Protection and authority: the assessed recovery mode, applicable energy-control procedure, responsible authorized personnel, and any documented exception with its supporting protection.
• Time boundaries: fault detection, motion stop, isolation and verification when applicable, physical work, release checks, and authorized return to service.
• Outcome: recovery completed, task aborted, damage, near miss, repeated entry, and whether the event was excluded from reported autonomous performance.

Paragraph (e) separately addresses the equipment condition, employee positioning, notification, and device removal before return to service. That release process should not disappear inside a single “operator-control seconds” figure (GPO, paragraph (e)).

These fields are not a substitute for a site-specific safety procedure. They prevent a metric from hiding physical support work. AutoIntervene’s recorded correction time remains a learning-data measure, not proof of safe access, isolation, or complete recovery labor. Our runtime and uptime guide explains why recovery time also belongs beside the denominator for an operating-hours claim.

A compact intervention evidence card

Before accepting a claim about low human involvement, reconstruct this card.

Field	Minimum disclosure
System	Robot, policy, monitor, sensors, controller, compute, and versions
Task scope	Objects, sequence, environment, speed, timeout, and success rule
Authority modes	Autonomous, operator, protective stop, remote support, and reset states
Trigger	Signal, threshold, calibration set, persistence rule, and update policy
Trial denominator	Attempts, operating hours, task hours, and repeated policy seeds
Cut-in	Valid, missed, extra, precision, recall, and latency
Cut-out	Valid, missed, premature, repeated reversal, and latency
Human work	Control seconds, attention time, response time, resets, and staffing ratio
Outcomes	Autonomous success, assisted success, recovery, abort, damage, and exclusions
Learning	Retained correction data, mixture rule, unassisted reevaluation, and replication

A report that publishes only “95% autonomous” leaves most of this card blank.

What AutoIntervene does not establish

The public evidence does not establish:

• operation on mobile robots, humanoids, or customer production systems;
• detection of every physical, semantic, software, or safety failure;
• independent replication;
• operator performance across different skill levels;
• calibrated probabilities of failure;
• switching latency distributions or worst-case stop time;
• safety-rated monitoring or functional-safety certification;
• unattended operating hours, uptime, throughput, or return on investment;
• complete human attention, reset, maintenance, and supervision cost;
• robustness to camera failure, network delay, hardware faults, changing calibration, or adversarial scenes;
• whether results persist across repeated policy training seeds.

The monitor recognizes support relative to its successful memory. That is not the same as proving that an action is safe or that an unseen state is unrecoverable. A familiar visual-action pattern can still violate a force, collision, or task constraint that the memory does not represent.

A deployment must keep learned handoff logic separate from safety-rated stops and the system-level risk controls described in our robot safety standards guide.

The adaptive robot safety case guide shows where human handoff fits beside safe-success measurement, formal requirements, runtime monitors, protective controls, distribution-shift tests, and update governance.

Evidence verdict

Claim	Classification	Confidence	Why
AutoIntervene improved task success after adaptation	Author-run physical evaluation	Medium-high	Seven main tasks used disclosed physical trial counts, but the study has no independent replication or repeated training seeds
It used less operator-control data than manual switching	Matched author-run comparison	Medium-high	The same intervention-learning pipeline was compared with automatic and manual switching, but recorded control time is not total labor
It detected and exited failures perfectly	Supported only in one controlled test	Medium	Perfect reported rates cover 10 perturbed and 10 nominal lidded-box rollouts under one fixed perturbation
It is compatible with several action heads	Bounded physical ablation	Medium	ACT, Diffusion Policy, and Flow Matching improved over three rounds on one longer task
It proves safe autonomous deployment	Not supported	High	No safety certification, customer operation, long-duration reliability, or broad fault evaluation is reported
Intervention data can replace full demonstrations	Not established generally	High	Targeted segments were more efficient here, but all policies began with full demonstrations and results remain task-specific

What to watch next

1. A public implementation, configuration, evaluation harness, and versioned result logs.
2. Independent reproduction on different arms, mobile manipulators, and humanoids.
3. Multiple operators with skill, fatigue, and response-time distributions.
4. Cut-in and cut-out latency percentiles measured at the complete control boundary.
5. Longer nominal runs that can reveal rare false triggers.
6. More perturbation classes, including occlusion, camera loss, object damage, calibration drift, and hardware faults.
7. Repeated policy training seeds with uncertainty on post-adaptation gains.
8. Complete accounting for monitoring, resets, discarded rollouts, and operator attention.
9. A comparison with pause, safe-stop, remote reset, and full teleoperation strategies.
10. Customer-side operating evidence with autonomous hours, support ratio, throughput, safety events, and cost.

Bottom line

AutoIntervene makes a valuable point concrete: human help is part of the deployed control system and should be measured as such. Its strongest result is not a claim that intervention disappears. It is that automatically bounded corrections produced better unassisted policies than the tested alternatives while recording less operator-controlled data.

The next standard is stricter. Report when the robot asked for help, when it failed to ask, how quickly authority moved, how the person recovered the task, when autonomy resumed, what the support cost, and whether the correction improved later unassisted trials.

A robot is not autonomous because intervention is hidden. It becomes more assessable when every handoff, recovery, and learning outcome has a denominator.

Frequently asked questions

Does handing control to a person make robot servicing safe?

Not by itself. A control handoff, a protective stop, and hazardous-energy isolation are different states. The 2025 annual edition of 29 CFR 1910.147 distinguishes shutdown from isolation and verification for covered U.S. workplace servicing. Applicability, exceptions, current requirements, and the site-specific procedure need qualified review.

What is a robot intervention?

A robot intervention is a period when a person or another control system takes authority because autonomous execution needs correction, recovery, or a safety response. Reports should distinguish a brief handoff from a full teleoperated task and from support that occurs only between trials.

How does AutoIntervene decide when to hand control to a person?

AutoIntervene compares the policy's proposed action chunk with a memory built from successful demonstrations. It combines visual similarity with action consistency, requires repeated rejection before handing control to the operator, and uses separately calibrated criteria to decide when autonomy can resume.

Did AutoIntervene reduce operator time?

In the authors' seven-task physical benchmark, AutoIntervene used 122.9 seconds of cumulative additional operator-control data per task on average after two rounds, compared with 179.9 seconds for manual switching. The result measures recorded corrective control used for adaptation, not staffing cost or unattended production time.

Did the study test real robots?

Yes. The authors used two fixed AgileX PiPER-X arms on nine bimanual manipulation tasks. Main task success rates used 25 unassisted physical rollouts for each task and method unless otherwise stated. The work remains an author-run lab study, not independent replication or customer deployment.

What intervention metrics should a robot company publish?

Publish autonomous task success, interventions per attempt or operating hour, operator-control time, missed failures, unnecessary handoffs, switching latency, recovery success, repeat handoffs, resets, excluded runs, final task outcome, and the effect of corrective data on a separately evaluated policy.

Does a low intervention rate prove that a robot is safe or reliable?

No. A low rate can reflect an easy task, missed failures, permissive thresholds, slow operation, or unreported support between trials. Intervention evidence must be read with task scope, failure severity, success rules, operating time, safety controls, and complete support accounting.