Field guide
Robot Dataset Splits: What Does a Held-Out Test Actually Prove?
A practical audit of robot training, validation, and test splits, covering episodes, sites, objects, preprocessing, checkpoint selection, and generalization claims.
The short answer
A held-out test proves only what was actually held out. New frames, new episodes, new objects, new homes, and a new robot body are different evaluation boundaries. The word “unseen” is not enough to identify one of them.
A robot policy can be evaluated on an episode that never appeared in training while still seeing a familiar room, object instance, operator, camera geometry, and task. That can be a legitimate test of repeat performance within a declared distribution. It is not automatically evidence of transfer to a new workplace or embodiment.
This guide uses robomimic’s public dataset format and versioned split script as a concrete implementation example. It also uses scikit-learn’s first-party documentation on grouped splits and leakage. The robot-specific audit framework below is our editorial recommendation, not a claim that either project certifies generalization or that a named dataset contains leakage.
For collection methods and provenance, start with how robot training data is made. For trial denominators, uncertainty, and success definitions, use our robot benchmark evidence audit. Here the question comes earlier: what information was the evaluation allowed to share with training?
Choose the claim before choosing the split
A split is an experimental design decision. Choose the intended claim first, then identify the unit that must stay outside development data.
| Intended claim | Separation to inspect | What can remain familiar |
|---|---|---|
| New windows from a known recording | Window boundaries and shared source frames | The entire episode and its physical context |
| New episodes under known conditions | Parent episode identifiers | Site, objects, operator, task, and robot |
| New collection sessions | Session or day identifiers | Site and object inventory |
| New object instances | Physical instance identifiers, not only category names | Task and collection site |
| New sites or homes | Site identifiers and all associated recordings | Task family and robot platform |
| New task combinations | Declared task and composition rules | Individual skills or objects |
| New robot embodiments | Robot and control-interface provenance, including adaptation data | Some task semantics or visual context |
This table describes questions to ask, not a universal hierarchy. A within-site test can be exactly right for a fixed industrial cell. A site holdout can be inappropriate if the actual question is repeatability on that cell. Do not relabel the result as broader transfer after seeing a favorable score.
Cross-embodiment evaluation needs particular care. A model adapted with target-robot demonstrations is not being tested under the same conditions as a model receiving no target-robot data. Our embodiment-gap guide separates reusable model structure from calibration, action interfaces, and target adaptation.
What robomimic’s split actually does
The robomimic dataset documentation describes one HDF5 dataset with a group for each trajectory. Each demonstration can contain observations, actions, rewards, and other recorded fields. Filter keys under mask identify subsets by demonstration name.
The versioned split script reviewed for this guide reads demonstration identifiers, randomly assigns a validation subset, and writes train and valid filter keys. It can operate on all demonstrations or an existing filtered subset. The command-line entry point seeds NumPy for repeatability.
The important boundary is visible in the implementation:
- the partitioning unit is a demonstration identifier;
- the output roles are training and validation;
- the script does not create a separate final test partition;
- it does not group demonstrations by home, object, operator, or session.
That is not a defect in a convenience script. It means users must supply the grouping and evaluation design needed for their own claim. Distinct demonstration names do not establish distinct physical conditions.
The documentation also explains how training and validation use their corresponding filter keys. Having those masks in a file is not enough. Audit the actual loader configuration so it consumes the intended subsets rather than the whole dataset.
The original robomimic study explicitly identifies variability associated with stopping criteria and the different objectives used in training and evaluation. That is a reason to report how checkpoints are selected, rather than treating one favorable checkpoint as a property of the method.
Split parent recordings before making windows
Sequence policies often consume windows rather than entire episodes. A window may contain observation history and a future action chunk. Two windows from the same recording can overlap in those inputs or targets.
As a design recommendation, assign the parent recording to a partition before generating windows, crops, relabeled versions, or other descendants. Keep every descendant with that parent. Otherwise, a random window split can put shared source material on both sides of the boundary.
Consider a hypothetical demonstration that is exported twice, once with original images and once with resized images. Different filenames do not make the recordings independent. If the two exports enter different partitions, an episode-ID intersection check may pass while source-recording overlap remains.
Preserve at least:
- a canonical parent recording ID;
- the collection episode ID;
- session, site, operator, object-instance, and robot IDs when available;
- transformation lineage for derived clips or synthetic variants;
- a declared rule for duplicate and near-duplicate review.
Exact-content checks can identify identical artifacts. They do not establish that differently encoded videos, shifted windows, or recreated scenes are independent. Where provenance is missing, report that limitation instead of calling the split clean.
This recommendation prevents a particular overlap mechanism. It does not prove that every overlapping split inflates every score, and it does not measure the size of any inflation.
Grouping is more than changing the random seed
GroupShuffleSplit produces randomized splits using supplied group identifiers. Its documentation makes two useful distinctions: split sizes refer to groups rather than samples, and test sets from repeated splits need not be mutually exclusive.
Applied to robot data, the grouping field should match the claimed boundary. To assess a new site, group all its episodes together. To assess new physical object instances, use instance identifiers rather than an object category such as “mug.”
A composite label needs care. Grouping by site + session can separate sessions while allowing the same site into both partitions. It does not implement a site holdout. If several relationships must remain separated, define the partition rule explicitly and audit every required field.
Unequal groups also change the mixture. Holding out a fraction of sites does not guarantee the same fraction of episodes, hours, tasks, or failures. Report all relevant units and the task composition of each partition. Check whether an apparent site effect is entangled with a task that appears only at that site.
Repeated grouped splits can support sensitivity analysis. They are not automatically independent replications, and a reused episode should not become several independent pieces of evidence merely because it appears in several reports.
Keep preprocessing and model selection inside development
The scikit-learn leakage guide says test data should not influence model choices and recommends splitting before preprocessing, then learning preprocessing transformations from training data. It illustrates how even preprocessing can expose information outside the intended training boundary.
For robot-policy work, our corresponding audit questions are:
- Which partition fitted observation or action normalization statistics?
- Which data informed feature selection, filtering thresholds, or learned encoders?
- Was the final test used to select a checkpoint, prompt, action horizon, or reward threshold?
- Were poor final attempts excluded after inspecting outcomes?
- Was an evaluation run added to later adaptation data and then reused as a final test?
Validation data can inform declared development choices. A final test reserved to assess the selected system has a different role. Once a test guides revisions, disclose that use and arrange a fresh final assessment if the claim requires one.
Offline prediction loss and executed robot behavior are also different measurements. A policy rollout encounters the states produced by its own actions. Do not substitute low validation loss for task completion, recovery, or safety. Our training-data guide’s DAgger explanation covers why learner-visited states matter.
Preserve the split, not only the seed
A seed is a recipe input, not a saved partition. To reproduce a dataset boundary, preserve the actual episode-to-partition assignments, the ordered input manifest, the grouping fields, the split configuration, and the implementation version. This is our audit recommendation. It makes the boundary inspectable without requiring a reviewer to reconstruct every random draw.
The scikit-learn randomness documentation distinguishes integer random_state values from mutable RandomState instances. With an integer, repeated calls to a splitter’s split reproduce its results for the same inputs. With a RandomState instance or None, successive calls can yield different results because the random state is consumed. That distinction concerns how randomness is supplied, not whether the grouping rule matches a new-home claim.
For example, suppose a hypothetical collection keeps the same seed but inserts a newly recorded episode into the list being split. The seed alone does not establish that the old episode assignments remain unchanged. Compare the saved partition manifests rather than assuming that matching configuration labels imply matching membership. If the dataset changes, identify the new release and explain whether the evaluation boundary changed with it.
Keep three kinds of repeatability separate
| Evidence layer | What to preserve | What it does not establish |
|---|---|---|
| Dataset partition | Parent episode assignments, ordered inputs, grouping logic, manifest identity, and split configuration | A repeated training trajectory or genuinely novel sites |
| Numerical training and evaluation | Code revision, checkpoint identity, preprocessing artifacts, software and device versions, random-number sources, and deterministic-operation settings | Identical results across every platform or library release |
| Physical assessment | Task protocol, object and site conditions, reset procedures, attempt records, interventions, and system configuration | Bitwise repetition of a real-world trial or field reliability |
PyTorch’s first-party reproducibility documentation states that completely reproducible results are not guaranteed across releases, commits, or platforms. It also warns that CPU and GPU executions can differ even with identical seeds. Seeding PyTorch therefore does not replace an environment record or a check of other random-number sources. The same documentation notes that applications using NumPy Generator objects need to seed those objects consistently, not only NumPy’s global generator.
PyTorch also distinguishes selecting an algorithm from making that algorithm deterministic. Disabling cuDNN convolution benchmarking can make algorithm selection repeatable, but the selected algorithm may still be nondeterministic. torch.use_deterministic_algorithms can select deterministic alternatives where available and raise an error for known nondeterministic operations without an alternative. These controls have a cost: the documentation warns that deterministic operations are often slower. Record the settings actually used rather than silently changing them between a quality comparison and a speed measurement.
None of those numerical controls makes a physical robot trial identical to an earlier trial. Our recommendation is to preserve the physical protocol and attempt ledger separately, then report outcomes across declared repetitions. For the denominator and uncertainty questions, use the robot benchmark success-rate guide. Do not select the best training seed after inspecting final results and present it as a preselected system. If several seeds were explored, disclose their selection role and distinguish development runs from final assessments.
A practical release check
Before sharing a result, ask another reviewer to inspect the saved partition assignments, confirm that the loader consumes them, and compare the final checkpoint and preprocessing artifacts with the declared run. Record whether they checked artifacts, reran offline evaluation, retrained the model, or repeated physical trials. Those are different review scopes. A successful artifact check is useful evidence, but it is not an independent robot reproduction.
Pretraining provenance can remain unknown
A clean fine-tuning split does not establish that a foundation model never encountered related material during pretraining. Request corpus versions, benchmark inclusion policies, deduplication rules, and any known overlap with evaluation sources.
If those records are unavailable, use a bounded statement: the disclosed fine-tuning partitions are separated at the episode level; overlap with upstream pretraining is not publicly established. Do not accuse a supplier of contamination merely because its corpus is private. Do not claim complete separation merely because the downstream masks are disjoint.
Synthetic data needs the same lineage review. A rendered test scene can have a new filename while reusing a training asset or parent trajectory. Whether that matters depends on the intended claim. Separate novel renderings from novel objects, behaviors, and physical conditions.
A compact split evidence card
Before accepting an “unseen” result, ask for this record:
| Field | What to publish |
|---|---|
| Intended claim | The task, population, and novelty boundary being tested |
| Dataset version | Immutable release or manifest identifier and collection scope |
| Split replay record | Saved assignments, ordered inputs, seed handling, and split implementation version |
| Numerical run record | Checkpoint, preprocessing artifacts, software and device versions, and randomness controls |
| Partition roles | Training, development validation, and final assessment |
| Grouping rule | Parent episode, site, session, object, or other declared unit |
| Overlap audit | Identifier intersections, source lineage, and unresolved provenance |
| Partition composition | Episodes, duration, tasks, groups, and relevant failure categories |
| Preprocessing | Fit scope for statistics and learned transformations |
| Selection policy | Checkpoint, hyperparameter, prompt, and exclusion rules |
| Prior and adaptation data | Upstream provenance and target-condition data used |
| Executed assessment | Frozen system, attempt counts, intervention rules, and outcomes |
| Evidence limit | Conditions not held out and claims not established |
Publish the manifest when permissions allow. Otherwise, publish the grouping logic and aggregate composition without exposing personal or sensitive site information. Distinguish “not shared” from “not recorded.”
What this audit does not prove
Disjoint partitions do not establish a safe robot, a commercially useful workflow, or reliable unattended operation. Nor does an unknown provenance field prove leakage. This is an evidence audit that identifies what a test can and cannot support.
The original source review is dated October 9, 2026. The reproducibility section was added after a first-party documentation review on October 12, 2026. The robomimic implementation citation is pinned to the reviewed file version. The documentation pages are living references. This guide reports no newly measured policy score, leakage prevalence, customer demand, or independent reproduction.
Name the held-out unit, preserve its lineage, keep development choices separate from final assessment, and match the claim to the actual partition. A percentage becomes much more useful when readers know which familiar conditions helped produce it.
Frequently asked questions
Does a held-out robot episode prove generalization to a new home?
No. An episode can be absent from training while its home, objects, operator, camera setup, and collection session remain represented. A new-home claim needs separation at the home level and an evaluation protocol matched to that claim.
Why should robot trajectories be split before creating training windows?
Overlapping windows can share observations and action targets from one parent recording. Assign the parent recording to a partition first, then generate windows within that partition. This prevents that specific overlap, but does not by itself hold out sites, objects, or embodiments.
Does robomimic's train-validation script create a held-out test set?
The reviewed version partitions demonstration identifiers into train and valid filter keys. It does not create a separate final test partition or group demonstrations by site, operator, object, or collection session.
Is validation loss the same as robot task success?
No. Validation loss evaluates a declared prediction objective on recorded examples. A rollout evaluates the policy's executed behavior under a task protocol. Keep checkpoint selection, offline losses, and final rollout outcomes separate.
Does using the same random seed reproduce a robot experiment?
Not by itself. Preserve the actual partition identifiers and input ordering, then record training randomness, software versions, device details, and deterministic-operation settings separately. PyTorch does not guarantee identical results across releases, platforms, or CPU and GPU execution even with identical seeds. Repeating a numerical run also does not reproduce a physical robot trial.
What should a robot dataset split report contain?
Report the dataset and manifest versions, parent episode identifiers, partition roles, grouping fields, overlap checks, preprocessing fit scope, checkpoint-selection rules, pretraining provenance, adaptation data, and final task attempts. State which generalization claims remain untested.