How Robot Training Data Is Made: Teleoperation, Egocentric Video, EEG, and EMG

The short answer

Robot training data is manufactured through a physical production process. People set up tasks, calibrate cameras and robots, operate hardware or wear capture devices, reset scenes, label outcomes, inspect synchronization, and decide which failures to preserve. The resulting record can be far more useful than ordinary video because it links what was seen to what action was taken and what happened next.

The main collection methods solve different problems:

Robot teleoperation produces observations and actions in the robot’s own embodiment, but it occupies hardware and requires skilled operators.
Leader-follower rigs let a person command one mechanism while a second robot mirrors the motion, creating directly aligned robot trajectories.
Handheld interfaces such as UMI capture human manipulation with tracked gripper poses, offering more mobility and task coverage than a fixed robot.
Egocentric video can scale across workplaces and homes, but ordinary video does not automatically provide robot actions, joint states, force, or reliable outcomes.
Exocentric video captures human work from third-person cameras, which can reveal body, object, and scene context while retaining the human-to-robot action gap.
Simulation generates action-labeled interaction and privileged state at scale, but policies still face differences between simulated and real sensing, contact, latency, and object dynamics.
General vision-language data supplies broad semantic and reasoning coverage, but it does not natively describe executable robot actions or physical consequences.
Language labels and task metadata make episodes searchable and condition policies, but generated labels can be wrong.
Intervention data captures where a policy starts to fail and how a person recovers it.
EEG and EMG are experimental side channels intended to expose intent, surprise, error, muscle activation, or hidden hand motion. Public evidence has not yet shown that either improves a robot policy.
Embodiment editing replaces a human hand or arm in egocentric frames with a rendered robot embodiment. It may reduce the visual gap between human and robot data, but does not automatically recover executable actions or physical interaction.
Online adaptation begins with a pretrained policy and a small demonstration set, then lets the robot collect success and failure experience. It can reduce task-specific demonstrations while increasing robot time, reset, reward, and safety requirements.

Scale matters, but hours alone are not the result. XDOF publishes 3,553 hours across 134,806 bimanual episodes and 195 tasks in ABC-130K. Tencent reports more than 10,000 hours of UMI demonstrations for Hy-VLA, with more than 2,000 hours released. Xiaomi reports more than 100,000 hours of handheld UMI trajectories plus a separate robot-alignment mixture. Generalist AI says its internal corpus exceeds 500,000 hours across about 9,000 end-effector variations. These figures describe unlike mixtures, collection systems, access levels, and evaluation protocols. They should not be ranked as if one hour meant the same thing in every dataset.

The defensible conclusion is: the best robot dataset is not the one with the largest headline. It is the one whose provenance, synchronization, task distribution, embodiment, labels, interventions, failures, licenses, and measured downstream effect can be inspected.

Data modalities at a glance

Method	What it records	Main advantage	Main evidence limit
Direct robot teleoperation	Robot cameras, state, and commanded actions	Actions already match the robot	Slow, hardware-bound, operator-dependent
Leader-follower control	Human commands mapped through paired mechanisms	Precise robot trajectories and natural corrections	Collection hardware may differ from deployment hardware
UMI or tracked handheld tools	Human video plus tracked gripper or tool motion	Broader environments without moving a full robot	Requires retargeting and later robot alignment
Egocentric or exocentric video	First- or third-person views of human work	Large task, scene, and environment coverage	No native robot action, state, or force signal
Simulation	Synthetic observations, actions, states, contacts, and outcomes	Parallel generation and precise labels	Sim-to-real gap in sensing, physics, and system behavior
General vision-language data	Images, video, text, spatial labels, and question-answer pairs	Broad semantic, visual, and reasoning coverage	Indirect supervision for physical action
Language and event labels	Task, subtask, action, correction, and outcome descriptions	Search, conditioning, segmentation, and analysis	Human and model-generated labels can be inconsistent
Intervention traces	Policy rollout, takeover point, correction, and return of control	Concentrates data around failures	A corrected run is not an autonomous success
EEG tags	Electrical brain activity synchronized to a task	Possible labels for error, intent, surprise, or effort	Early trial, no published policy improvement
EMG tags	Muscle activation from forearm sensors	Possible hand-pose or effort signal when vision is incomplete	Early experiment, no published policy improvement
Embodiment-edited human video	Human scenes with robot hands or arms rendered into the frames	Robot-oriented visual observations across many embodiments	No native robot action, force, state, or policy result
Online policy adaptation	Demonstration seed plus autonomous successes and failures	Targets the current policy’s own errors on hardware	Requires reward, resets, safe exploration, and robot time

A five-layer data pyramid, and what it does not prove

The July 2026 survey Data Pyramid for Embodied Manipulation offers a useful way to organize the field. It places five complementary sources along a broad tradeoff between scalability and robot alignment:

Real-robot data records observations, states, actions, and outcomes on physical hardware. Its actions are the most directly tied to the robot that produced them, but every episode consumes hardware time, supervision, and resets.
UMI-style data removes the robot from collection while retaining tracked end-effector motion. It can reach more environments, but proprioception is missing and demonstrations still require retargeting and robot-specific alignment.
Egocentric and exocentric data preserves real human interaction and environmental physics at broader scale. It does not provide robot-native actions, so pose reconstruction, action inference, and human-to-robot transfer remain separate problems.
Simulation data can produce executable synthetic actions, exact state, contact labels, and outcomes in parallel. Its usefulness depends on assets, task construction, and how well simulated sensing and dynamics transfer to the physical system.
General data includes images, video, language, 3D data, planning examples, and other broad corpora. It can support perception and reasoning, but it is the least directly grounded in robot action and contact.

The ordering is not a leaderboard. The paper evaluates each layer through six dimensions: scalability, robot alignment, quality, diversity, reusability, and physical fidelity. A source can be strong on one dimension and weak on another. Simulation, for example, can return action labels and precise state at low marginal cost, yet still approximate the physics that matters during deployment.

The survey also reports that recent robot foundation models increasingly mix layers rather than relying on one source. That observation supports analyzing the data recipe, not declaring a universal winning layer. For any model, readers should ask which sources were used in pretraining, alignment, fine-tuning, and evaluation, how their action spaces were reconciled, and which ablations show that each source added value.

This paper is a survey and taxonomy. It does not introduce a matched benchmark proving that the five-layer framework, or any particular mixture, improves a policy. Its companion open repository makes the reviewed resources easier to inspect, but a curated list is not independent replication of the underlying results.

What a useful robot episode contains

A video file is not automatically a robot-learning episode. A complete record commonly needs:

Observations: synchronized RGB, depth, tactile, force, audio, or other sensor streams.
Robot state: joint positions, end-effector pose, gripper state, velocity, and relevant controller state.
Actions: the command sent to the robot, with an explicit representation and control rate.
Task context: instruction, objects, scene, initial condition, and embodiment identifiers.
Timing: clocks aligned well enough to connect each observation, action, and outcome.
Outcome: success, partial completion, failure type, safety stop, or abandonment.
Human involvement: demonstration, intervention, reset, correction, or post-selection.
Provenance and rights: collector, site, consent, license, retention rules, and permitted use.

Missing any one field can change what the data supports. Unsynchronized action and video can teach the wrong cause-and-effect relationship. A success-only dataset hides recovery behavior. A task label without an initial-state description can merge easy and hard episodes. An intervention that is not marked can make assisted operation look autonomous.

The LeRobot practical guide shows how open tooling connects teleoperation, episodic recording, rollout evaluation, and human correction. Its current real-robot documentation makes episode duration, reset time, task text, and episode count explicit. Those controls are basic, but they make the collection protocol inspectable.

1. Direct teleoperation: actions in the target embodiment

In direct teleoperation, a person controls the robot through a leader arm, joystick, wearable, motion-capture system, or another interface. The dataset can preserve camera observations, robot state, and the action sequence sent to the target hardware.

This is valuable because the policy learns from actions the robot can execute. It also captures embodiment-specific limits such as reach, joint coupling, camera placement, gripper geometry, latency, and collisions.

The cost is physical. A robot, workcell, operator, objects, resets, maintenance, and safety supervision are tied up during collection. Operator style can also become a hidden variable. Two people may solve the same task with different paths, speed, force, and tolerance for near-failures.

TechCrunch observed Encord pilots using leader-follower rigs for tasks including pouring coffee and stacking poker chips. The paired setup is concrete evidence that data collection is happening. It is not evidence that a trained policy has mastered those tasks, because the report does not publish a downstream evaluation.

Quality checks for teleoperation

Calibrate the leader and follower before collection and log the calibration version.
Measure end-to-end latency and dropped frames.
Separate command actions from measured robot motion.
Preserve collisions, aborted attempts, resets, and operator takeovers.
Balance operators, objects, initial states, and camera conditions.
Evaluate a held-out policy rather than reporting demonstration completion as model success.

2. UMI: portable human demonstrations with robot-oriented signals

A Universal Manipulation Interface is a tracked handheld device that resembles a robot gripper. A person performs a task while cameras and tracking capture visual observations and gripper motion. The method can move through homes, workshops, commercial spaces, and outdoor environments without bringing a full robot to every site.

That portability helps explain the recent scale claims. Tencent says Hy-VLA uses more than 10,000 hours of UMI demonstrations across more than 70 tasks, collected through a custom fingertip interface with optical motion capture. Its repository says more than 2,000 hours are publicly released in a LeRobot-compatible schema. Xiaomi reports more than 100,000 hours of UMI trajectories across more than 1,700 scenarios, then a separate cross-embodiment post-training mixture that includes more than 7,200 hours from mobile manipulators and dual-arm robots.

The distinction is essential. UMI trajectories are real human manipulation data, but they are not 100,000 hours of robot runtime. A policy still needs a mapping from handheld motion and camera geometry to a robot’s action space, kinematics, gripper, and observations. Our Xiaomi-Robotics-1 audit examines one reported scaling result and its bounded real-robot tests.

The latent actions guide examines another route, extracting transition supervision from video before physical commands are available. That mapping is only part of the embodiment gap in robot foundation models. A complete transfer record also identifies target-robot data, modified components, calibration, control-interface work, physical adaptation, interventions, and recovery.

Useful UMI reporting should separate:

collected hours from hours actually used in training;
public data from proprietary data;
handheld trajectories from robot trajectories;
scenes and task labels from genuinely held-out evaluation conditions;
pretraining from embodiment alignment and task fine-tuning;
model improvement from raw data volume.

3. Egocentric video: coverage without native robot actions

Egocentric video records work from the person’s point of view. A head or body camera can observe many environments and long-horizon activities without placing a robot in the scene. This makes it attractive for learning visual concepts, task structure, object state changes, and human strategies.

Encord told TechCrunch that it collects egocentric data from factories and adds camera views and other signals. The article also reports that Ropedia’s HOMIE device uses a head-mounted panoramic camera to capture first-person motion, object interaction, and spatial context. These are data-production approaches, not measured policy outcomes.

Ordinary egocentric video has a fundamental gap. It may show the world changing, but it usually does not provide the robot command that caused the change. Hands can be occluded, camera motion can dominate the image, contact force is hidden, and a human body has different kinematics from the target robot.

Video can still support representation learning, future prediction, language grounding, and task segmentation. The FLUX-mimic evidence audit covers a related strategy that uses broad video pretraining before connecting latent video features to robot actions. The necessary claim boundary is the same: useful pretraining evidence is not the same as reliable action execution in deployment.

Ego2Robot tests a more direct conversion path. It retargets human hand motion, removes the human arms, renders robot morphologies into the scene, and attaches camera-frame end-effector actions. Our Ego2Robot evidence audit separates its 1,940 source-video hours from 18,561 hours of rendered training streams, then examines the fixed-frame comparisons and physical tests.

HandEdit: an editing benchmark, not yet a policy benchmark

HandEdit treats the human-to-robot appearance gap as a dedicated image-editing problem. The authors assemble more than 200 million image-level editing instances from over 300,000 clips in five public hand-object datasets. The target set covers 26 robot descriptions, split evenly between hand-only and hand-arm configurations. The accompanying benchmark evaluates 11 image editors using visual similarity, vision-language-model judgments, human evaluation, and embodiment-aware checks.

This is a meaningful infrastructure contribution. It provides a common test for whether an editor removes the human hand, preserves the object and scene, follows the specified robot morphology, and retains a plausible interaction. The project page links to code and a dataset repository, which makes the artifact path inspectable.

The reported result is narrower than reliable robot learning. HandEdit benchmarks edited images, not a robot policy trained with those images. A visually plausible robot hand can still have incorrect contact, impossible joint motion, missing force, no synchronized robot state, and no executable action label. The source data also carry their own licenses, including non-commercial research restrictions for some components. Before treating embodiment editing as a cheaper replacement for robot data, ask for a controlled policy ablation with fixed downstream tasks, robot trials, intervention accounting, and held-out embodiments.

4. Language labels: structure, not automatic truth

A physical episode becomes easier to search and condition when it includes language such as a task instruction, subtask boundary, object state, correction, or failure reason. TechCrunch reports that Encord annotates clips with physical descriptions such as a hand tightening a bolt. LeRobot v0.6.0 can store timestamped subtasks, plans, corrections, speech, and visual question-answer pairs, and it includes VLM-assisted annotation workflows.

Labels create at least four failure modes:

Ambiguity: “place the cup” does not define the target pose or acceptable error.
Temporal drift: a label may begin or end after the actual physical event.
Outcome confusion: a plausible action description may be attached to a failed result.
Model-label error: generated annotations can sound precise while misreading contact, object identity, or intent.

Dense annotation may be worth the cost for a narrow task, but a vendor’s value estimate is not a measured model effect. The appropriate test holds the data and evaluation protocol constant, changes the label treatment, and reports the resulting task performance with uncertainty.

5. Intervention data: failures converted into demonstrations

A policy rollout can be more informative than a clean human demonstration because it reveals where the current model leaves its training distribution. In an intervention workflow, a person monitors the robot, takes control when failure begins, performs a correction, and returns control.

LeRobot v0.6.0 explicitly supports saving intervention windows for another training cycle. Tencent’s Hy-VLA repository describes FlowPRO as using real-robot failure interventions for preference optimization, although the method is marked under review in that source.

The reporting boundary must remain visible. A run that succeeds after a human takeover is an assisted completion, not an autonomous success. Teams should retain:

attempts and fully autonomous completions;
intervention count, duration, and reason;
time or actions between interventions;
reset and safety-stop counts;
performance before and after adding the corrections;
whether evaluation scenes were kept out of the correction set.

Intervention accounting is both a training-data requirement and an operating-evidence requirement.

MiDAS: one demonstration plus hours of robot interaction

The phrase “learns from one demonstration” can hide most of the learning budget. MiDAS makes that boundary unusually clear. It first behavior-clones a pretrained vision-language-action policy on one or a few successful demonstrations. It then freezes that base policy and trains a residual policy through value-based online reinforcement learning. Successes and failures gathered during interaction provide the corrective signal that the single demonstration cannot supply.

The authors report simulation results on LIBERO-Long and RoboCasa, plus two bimanual YAM pick-and-place tasks. In 15 evaluation rollouts per physical task, the one-demonstration behavior-cloning policy scored 40% and 27%. After online adaptation, MiDAS scored 67% and 80%. The harder task used about five to six hours of autonomous interaction. These are author-run results on two bounded tasks, not independent replication or unattended deployment.

The practical interpretation is not that one example teaches a finished skill. One successful demonstration gets a pretrained policy close enough to attempt the task. The remaining training input includes robot-generated failures and successes, a sparse task reward, instrumented resets, compute for online updates, and a physical setup safe enough for exploration. The paper also reports sharp failures under larger state shifts. Swapping object positions drove both policies to zero on the tested LIBERO tasks, and changing object affordances substantially reduced performance.

MiDAS therefore moves cost rather than eliminating it. Compared with collecting many demonstrations, a team may need fewer expert trajectories. It still needs robot occupancy, reliable reward detection, resets, supervision around unsafe failure modes, and a declared stopping rule. This is complementary to human intervention and handoff accounting: one method learns from autonomous interaction, while the other decides when a person should take control. Both require assisted and autonomous outcomes to remain separate.

6. EEG: an experiment in latent task-state labels

TechCrunch observed an Encord pilot wearing a Zander Labs headset while performing a Jenga task. The report says the system measures brain activity associated with states such as error, intent, surprise, or effort. Encord described the work as a trial: build an initial EEG-tagged dataset, test it with customer robotics models, and decide whether to scale only after measuring performance.

That sequence is scientifically appropriate. It also means the current status is limited:

data collection was observed;
the intended label classes were described;
no customer model or task protocol was named;
no trial count, effect size, or baseline was reported;
no downstream improvement was published.

EEG also brings practical questions about sensor noise, motion artifacts, synchronization, individual calibration, label interpretation, privacy, consent, and whether the signal adds information beyond video, actions, outcomes, and operator button presses.

The useful next result is a before-and-after study. Train the same policy on the same episodes with and without EEG-derived features or labels, evaluate on declared held-out tasks, and report trial counts, confidence intervals, failure classes, and effect size.

7. EMG: estimating hidden hand motion and effort

Electromyography measures electrical activity associated with muscle activation. TechCrunch reports that Encord is testing forearm sensors because ordinary video can miss parts of the hand. The stated goal is to help reconstruct a three-dimensional hand pose for richer demonstrations.

This may add information when fingers are occluded or when visible motion does not reveal muscle effort. It does not directly measure contact force, and the mapping from forearm signals to hand pose or intent varies across people, placement, fatigue, and task.

The public report establishes an experiment, not downstream value. A useful evaluation would compare vision-only reconstruction with vision plus EMG, then test whether any pose improvement survives retargeting and changes robot task success.

Hours are not a common unit of value

Published dataset headlines use hours because duration is easy to communicate. Duration hides substantial variation.

Published figure	What the source says	Why it is not directly comparable
3,553 hours	ABC-130K bimanual teleoperation, 134,806 episodes, 195 tasks	Open dataset with a declared task taxonomy and evaluation artifacts
10,000+ hours	Tencent Hy-VLA UMI demonstrations, with 2,000+ hours released	Handheld, optically tracked data plus a separate model and benchmark stack
100,000+ hours	Xiaomi handheld UMI trajectories across 1,700+ scenarios	Proprietary pretraining corpus, distinct from later robot-alignment data
500,000+ hours	Generalist AI internal interaction data across about 9,000 end-effector variations	First-party aggregate without a public dataset or matched external audit

An hour can contain one long repeated task, many short episodes, idle time, operator mistakes, identical initial states, or unusual high-value failures. Better reporting adds:

usable frames after quality control;
unique tasks, objects, scenes, operators, and embodiments;
episode and duration distributions;
failure and intervention prevalence;
train, validation, and evaluation separation;
duplication and contamination checks;
downstream scaling curves under one fixed protocol.

XDOF’s ABC release is useful because it goes beyond a duration claim. It reports 134,806 episodes, 195 tasks, task categories, episode-duration range, more than 100 hours of real evaluations, and released data, code, rubrics, and logs. Its project page also reports lower held-out validation loss and action error as training data increased. Those are first-party results, but the open artifacts make them more inspectable than a private-hour total.

Generalist’s large corpus supports a different thesis: variation across hands and tools may expose a model to diverse contact physics. The company explicitly says not every end effector will add useful signal and that it is still evaluating contributions. Without a public task distribution, baseline, and matched real-robot results, the half-million-hour figure establishes reported scale, not universal transfer.

The economics of manufactured data

Physical data has costs that internet-scale text collection can avoid:

robot and capture hardware;
operators, supervision, and training;
workcell space and task materials;
calibration and synchronization;
resets, maintenance, and breakage;
storage, video encoding, and transport;
annotation and quality review;
privacy, consent, licensing, and site access;
evaluation runs that do not become training data.

TechCrunch observed roughly a dozen pilots at Encord’s San Leandro facility and reported the company’s estimate that dense task annotation costs 20 times more than low-quality egocentric data while being worth 100 times more for specific tasks. Both ratios are company estimates, not a published cost study or model ablation.

The right purchasing question is not “What is the price per hour?” It is “What measurable performance change did this declared data slice produce on our task after quality control, adaptation, and evaluation?”

A practical dataset audit

Before training or buying robot data, ask:

Provenance

Who performed the task, where, and under what consent and license?
Is the data human video, tracked handheld data, teleoperation, autonomous rollout, or a mixture?
Which sites, customers, or populations impose use restrictions?

Synchronization and embodiment

Which observations, states, and actions are synchronized?
What are the control rate, camera rate, latency, and dropped-frame policy?
How is human or source-hardware motion mapped to the target robot?

Coverage

How many tasks, scenes, objects, operators, robots, and initial states are represented?
Are long-duration figures dominated by a small number of repetitive workflows?
Which failures, interventions, and safety stops are included?

Labels and outcomes

Who or what created each label?
Are task success and partial credit defined before evaluation?
Can labels, interventions, and resets be audited at episode level?

Evaluation

What remains held out by task, object, site, operator, and embodiment?
Is there a matched baseline without the new data slice?
Are trial counts, uncertainty, failure classes, and all-run logs available?

Operations

What did collection cost after rejected episodes and quality control?
How quickly can new failures become corrected training examples?
Can a dataset version be reproduced after the robot, camera, policy, or labeler changes?

What the evidence does not establish

The current source record does not establish:

that EEG or EMG improves robot policy performance;
that more hours always beat better targeted data;
that unlike teleoperation, UMI, video, and interaction-hour totals are comparable;
that a dataset collected on one embodiment transfers safely to another;
that model-generated language labels are accurate enough without audit;
that a company-reported dataset size has been independently verified;
that strong offline loss or simulation results imply field reliability;
that an assisted rollout is an autonomous success;
that public access to data automatically provides consent, safety, or commercial-use clearance.

What to watch next

EEG and EMG ablations: fixed episodes, matched policies, declared held-out tasks, trial counts, and effect sizes.
Data-value curves: performance by data slice, not only total hours.
Cross-embodiment transfer: source and target hardware, alignment data, failures, and intervention rates.
Operator effects: results across collectors, experience levels, fatigue, and interface designs.
Failure-rich datasets: explicit recovery, intervention, and safety-stop labels.
Provenance standards: machine-readable consent, license, site, and collection metadata.
Open evaluation: task rubrics, all-run logs, baseline configurations, and confidence intervals.
Cost disclosure: accepted-hour cost after collection, rejection, annotation, and quality control.
Embodiment-editing ablations: fixed policies trained with and without edited human frames, evaluated on held-out robots and physical tasks.
Online-adaptation budgets: demonstrations, robot hours, attempts, resets, reward design, safety stops, and final evaluation trials reported separately.

Verdict

Robot-learning progress increasingly depends on data operations, not only model architecture. Teleoperation and leader-follower systems create direct robot actions. UMI increases human-demonstration coverage. Egocentric video supplies broad visual experience. Embodiment editing can make those frames look more robot-specific. Online adaptation can turn a small demonstration seed into targeted robot experience. Language labels expose task structure. Interventions concentrate examples around failures. EEG and EMG may add hidden state, but their downstream value remains unproved.

Large datasets from XDOF, Tencent, Xiaomi, and Generalist demonstrate that the field is building industrial-scale collection systems. They do not create a common leaderboard. Each figure covers a different mix of people, interfaces, robots, tasks, access, and evidence.

Treat every robot dataset as a production system with inputs, quality controls, rights, and measurable outputs. Count hours, but make decisions from held-out task performance, intervention rates, provenance, and cost per verified improvement.

Frequently asked questions

What is robot training data?

Robot training data is a synchronized record of observations, actions, task context, and outcomes used to train or evaluate a robot policy. A useful episode may include camera streams, robot state, human commands, action trajectories, timestamps, labels, interventions, resets, and success or failure.

What is the difference between teleoperation and egocentric video?

Teleoperation records a human controlling a robot, so observations and robot actions are aligned in the target embodiment. Egocentric video records work from a person's viewpoint and can cover more tasks and environments, but it usually lacks the robot action and state signals needed for direct policy training unless another system reconstructs them.

What is UMI robot data?

Universal Manipulation Interface data is captured with a handheld gripper-like device and tracking system. It records human manipulation in a form designed to transfer more readily to robots than ordinary video. It is still demonstration data, not robot operating time.

Has EEG been proved useful for robot training?

No. TechCrunch observed an Encord and Zander Labs trial that tags demonstrations with brain signals associated with states such as error, intent, or surprise. Encord said it plans to test the data with customer models before deciding whether to scale. No downstream performance result has been published.

What does EMG add to robot data?

EMG measures electrical activity associated with muscle activation. Encord is experimenting with forearm sensors to help reconstruct hand pose when video does not show the full hand. The public report does not establish that these signals improve a robot policy.

Is more robot data always better?

No. More hours can include duplicates, poor calibration, narrow task coverage, weak labels, or inconsistent operators. The useful test is whether a controlled increase in a declared data slice improves performance on held-out tasks or conditions under the same evaluation protocol.

What are the five layers of the embodied data pyramid?

A July 2026 survey groups embodied manipulation data into real-robot data, UMI-style data, egocentric and exocentric human recordings, simulation data, and general vision-language data. The framework highlights a tradeoff between collection scale and direct alignment with robot execution. It is a taxonomy, not evidence that one recipe is best.

Can human hand video be converted into robot-hand training data?

HandEdit provides paired images that replace human hands and arms with 26 robot embodiments, but its published benchmark measures editing quality rather than downstream robot-policy improvement. Converting pixels does not by itself recover robot actions, force, state, or reliable contact dynamics.

Can a robot learn a task from one demonstration?

MiDAS shows that one successful demonstration can initialize online adaptation for selected simulation and bimanual pick-and-place tasks. The robot still used hours of online interaction, sparse reward feedback, instrumented resets, and a pretrained policy. One demonstration was a starting point, not the complete training input or evidence of general deployment readiness.