Field guide
Long-Horizon Robot Control: Planning, Memory, and Whole-Body Execution
An evidence guide to long-horizon robot control, using Tau-0-VLA and HAF to separate subtask planning, execution memory, low-level action, whole-body coordination, and online adaptation.
The short answer
Long-horizon robot control is not one model making one decision. It is a chain of interfaces that must stay consistent as the world changes. A planner must choose the next subtask. Memory must represent what actually happened. A low-level policy must turn the subtask into action. For a humanoid, locomotion, posture, and manipulation must remain coordinated. The system must then verify the outcome and decide whether to continue, retry, recover, or request help.
Two research releases from August 2026 expose different parts of that chain:
- Tau-0-VLA searches over high-level language subtasks using predicted visual outcomes, a progress value model, execution memory, and a reflective decision step. Its low-level VLA then executes the selected subtask across several robot embodiments (paper, project).
- HAF adapts a generalist flow-matching VLA to humanoid whole-body control. It generates locomotion, head, waist, and arm actions in ordered stages, then optionally refines a frozen VLA through low-dimensional offline-to-online reinforcement learning (paper, project).
They are not direct competitors. Tau-0-VLA focuses on which subtask to execute next and then hands that command to a low-level policy. HAF focuses on how a whole humanoid body should execute an action chunk and how that controller may be adapted after offline training.
| Control layer | Decision | Tau-0-VLA evidence | HAF evidence | Main unresolved risk |
|---|---|---|---|---|
| Task state | What has already happened? | Correctable textual execution memory | Current observations and robot state | Memory can diverge from physical reality |
| High-level planning | What should happen next? | Candidate subtasks, future-image prediction, value scoring, beam search, reflection | Language conditions the whole-body policy | A wrong high-level decision can be executed precisely |
| Low-level action | How should the robot move? | Unified 40-dimensional action interface across three embodiments | Flow-matching VLA with staged whole-body generation | Latency and control mismatch can destabilize execution |
| Adaptation | How should behavior improve? | Task-specific low-level fine-tuning | Spectral behavior cloning plus online SAC | Physical exploration can be unsafe or costly |
| Verification | Did the action work? | Milestones, progress, and next-step observation | Task scores and sparse terminal rewards | Partial progress can hide retries or human support |
Both papers report physical experiments. Both remain preprints with author-run evaluations. Neither supplies production deployment evidence.
Why long tasks fail at interfaces
A short manipulation can sometimes be described as one instruction and one action sequence. A longer task introduces state that must survive across many decisions.
Consider making a drink. The robot may need to locate a cup, place it, add ingredients in the correct order, attach a lid, and insert a straw. Failure can occur in several distinct ways:
- the planner selects an already completed step;
- memory says an ingredient was added when it was not;
- the selected subtask is correct, but the gripper misses;
- a retry changes the scene, but the planner reasons from the old state;
- the robot completes most of the task and fails at final contact;
- extra planning improves the decision but delays execution;
- a humanoid reaches correctly while its base or waist motion destabilizes the grasp.
A final success rate cannot identify which interface failed. Our dynamic robot benchmark guide explains why timing, regression, and recovery need separate measures. The long-horizon problem adds task memory and subtask selection to that measurement stack.
Tau-0-VLA: search before committing to a subtask
Tau-0-VLA separates control into a high-level policy and a low-level VLA.
At a high-level decision point, the system:
- uses the current observation and execution memory to propose a language subtask;
- accepts the proposal directly when token confidence is sufficient;
- otherwise generates alternative subtasks;
- predicts a terminal visual outcome for each candidate with a world model;
- scores predicted task progress with a value model;
- recursively expands promising branches through beam search;
- asks a reflective model to issue the final subtask;
- executes that subtask with the low-level policy;
- incorporates the resulting observation into memory for the next decision.
This is a propose, predict, evaluate, reflect, execute loop. The predicted future is used to compare high-level commitments, not to replace the low-level controller.
The distinction matters. Our world action model evidence guide covers whether predicted futures stay tied to supplied actions and valid physical interactions. Tau-0-VLA uses predicted terminal images to rank open-ended language subtasks. An incorrect world-model prediction can therefore misrank a plan even when the low-level policy is competent.
What correctable execution memory adds
Long tasks require more than a recent camera frame. Some state is visually ambiguous. An ingredient may already have been added. A room may have been cleared before the robot moved elsewhere. Repeating or skipping these steps can invalidate the task.
Event-selected visual and textual history offers another memory design. Our VLA control interfaces guide examines UniMem alongside prompt authority, typed spatial grounding, and behavior intent, while keeping its 15-trial hardware denominators separate from the Tau-0-VLA protocol here.
Tau-0-VLA maintains a textual execution record and trains the high-level policy to repair corrupted memories. Training perturbations make records lag behind, run ahead, or otherwise misrepresent demonstrated progress. The policy learns to correct those records without additional manual labels.
That is useful architectural evidence. It does not prove that every real execution error will be detected. Textual memory can be internally consistent and physically wrong. A field system still needs observable completion criteria, contradiction handling, timestamps, confidence, and a safe response when memory and sensor evidence disagree.
What the Tau-0-VLA experiments show
The low-level policy uses a unified 40-dimensional state and action representation spanning end effectors, joints, grippers, waist, and mobile base. The authors report training it on 40,115 hours of heterogeneous real-world data, including teleoperation, autonomous, and UMI data, with multimodal co-training.
That number is a training-data claim, not robot uptime or deployment time. The public paper does not provide enough detail to independently audit collection quality, duplication, embodiment balance, failure coverage, or how many hours directly support each evaluated task.
Four long-horizon tasks
The main long-horizon comparison covers Clean Room, Prepare Ingredients, Tomato and Egg Stir Fry, and Make Milk Tea. Each method and task uses 10 independently collected physical trials. Tasks contain 13 to 25 annotated milestones, with typical successful executions from about 3 to 10 minutes. Trial limits are longer.
| Method | Average success across four tasks | Average milestone progress | High-level planning condition |
|---|---|---|---|
| GR00T N1.7 | 2.5% | 45.29% | Direct whole-task execution |
| LingBot-VLA | 0% | 44.43% | Direct whole-task execution |
| Pi 0.5 | 22.5% | 73.05% | Direct whole-task execution |
| Tau-0-VLA | 27.5% | 80.10% | Direct whole-task execution |
| Tau-0-VLA hierarchical system | 45.0% | 87.85% | Plan Once, execution memory, no beam search |
The controlled Tau-0-VLA comparison holds the observation interface, action interface, and low-level policy fixed. The hierarchical version improves the average from 27.5% to 45.0%. The strongest bounded conclusion is that explicit subtask generation and execution memory improved these four author-run tasks under this protocol.
The result does not isolate every component, prove transfer to an unrelated task, or establish recurring reliability. There are only 10 trials per method-task cell. Final contact remains a major failure point. On Make Milk Tea, for example, the remaining failures concentrate on lid attachment and straw insertion after most preceding stages were completed.
More test-time compute improved three closed-loop tasks
The authors compare Plan Once with test-time computation while keeping the low-level policy fixed. Each condition again uses 10 physical trials.
| Task | Plan Once | Test-time computation | Progress change |
|---|---|---|---|
| Make Milk Tea | 5/10 | 7/10 | 91.92% to 95.38% |
| Book Organization | 6/10 | 9/10 | 66.67% to 93.33% |
| Clean Room | 5/10 | 7/10 | 94.80% to 97.60% |
Open-loop next-subtask accuracy also rises as compute increases and then approaches a plateau. The paper reports compute in PFLOPs per sample for the search study. It does not provide a complete deployment latency budget that joins planning time, network transfer, low-level inference, controller timing, and task interruption risk.
More search is therefore not free. A useful deployment should report when search is triggered, wall-clock decision latency, tail latency, energy or accelerator cost, timeout behavior, and whether the robot waits safely while the high-level policy reasons.
HAF: coordinate the body before executing the chunk
HAF addresses a different interface. A humanoid action space is not merely a larger arm controller. Base motion changes reachability and balance. Waist posture changes arm workspace. Head orientation changes perception. Arm actions can provoke compensating motion elsewhere.
HAF-VLA replaces one-shot full-body generation with three cumulative stages:
- generate locomotion and head actions;
- retain those clean actions in a cross-stage cache, then add waist actions;
- retain the expanded action state, then add arm actions;
- execute only the final complete action chunk.
Later stages can refine earlier action dimensions. The ordering establishes gross body motion and posture before fine manipulation.
This is hierarchical control inside the action generator. It differs from Tau-0-VLA’s hierarchy, where language subtasks sit above a low-level policy. The word hierarchical is not enough to compare two systems. Readers should ask what the levels represent, how often each level runs, and what information crosses the boundary.
What the HAF-VLA experiments show
The authors evaluate HAF on TienKung 2.0 and TienKung 3.0 humanoids. Demonstrations use an onboard head camera, kinematically matched master arms for the robot arms and hands, a joystick for locomotion and waist motion, and an IMU for head commands. They report 120 teleoperated trajectories per task.
Seven household tasks cover laundry loading, clothes retrieval, table tidying, basket transfer, toy storage, ball tossing, and box transfer. Each method-task cell uses 10 physical rollouts. The primary metric is normalized milestone score rather than full-task success.
| Method | Average normalized score across seven tasks |
|---|---|
| HAF-VLA | 70.5% |
| Pi 0.5 | 53.3% |
| GR00T N1.7 | 38.1% |
| Cosmos Policy | 27.6% |
| ACT | 25.2% |
HAF-VLA is best or tied best on each task in the reported table. The gap is largest on several tasks that combine locomotion with later manipulation.
The metric boundary is important. A 70.5% normalized task score is not a 70.5% full-task success rate. Partial milestones contribute credit. The paper provides a useful process measure, but a buyer would still need complete-task success, retry accounting, intervention, safety events, and the time distribution for each task.
The ablation supports ordering, not just more denoising
On Laundry Loading, HAF-VLA reports 66.7% normalized score. All-joint denoising with the same 30 total steps reports 53.3%. An arm-first hierarchy reports 50.0%, while Pi 0.5 with 30 steps reports 20.0%.
The authors also report that increasing Pi 0.5 from 10 to 30 denoising steps raises inference latency from 0.075 seconds to 0.115 seconds and reduces performance in this setup. Their interpretation is that delayed locomotion predictions amplify mismatch between velocity commands and execution.
This is a useful reminder: extra computation can improve planning in one layer and harm control in another. The relevant quantity is not model intelligence in isolation. It is whether each decision arrives inside the timing window of its control layer.
HAF-Steer: online adaptation with a frozen VLA
HAF-Steer adapts the controller without updating the full VLA backbone. It reverses demonstrated action chunks into the flow model’s initial noise, compresses temporal noise with a discrete cosine transform, and retains the first eight coefficients. A smaller actor is initialized by behavior cloning, then updated with mixed offline and online Soft Actor-Critic training. The frozen VLA decodes the resulting spectral action back into a full action chunk.
The design aims to reduce the exploration dimension while preserving smooth temporal variation. The paper evaluates Toy Storage and Basket Transfer in in-distribution and shifted-destination settings, using both Pi 0.5 and HAF-VLA backbones. Each evaluation condition uses 10 physical rollouts.
HAF-Steer improves both backbones in all four task and distribution settings in the reported figure. However, the paper does not disclose a complete accounting of online robot interactions, elapsed training time, resets, operator oversight, or safety-stop frequency for HAF-Steer.
A comparison method, DSRL, failed to complete training in several conditions because unsafe exploratory motions triggered early termination. That disclosure is important. It shows why online improvement cannot be evaluated only by the final policy. The exploration process itself needs a safety and cost denominator.
The two hierarchies fit together conceptually
A complete long-horizon humanoid system may need several clocks:
| Clock | Typical responsibility | Evidence to record |
|---|---|---|
| Task clock | Select a subtask, update memory, verify milestones | Planner calls, search budget, memory corrections, completion errors |
| Policy clock | Convert a subtask and observations into an action chunk | Inference latency, observation age, chunk size, replanning rule |
| Whole-body clock | Coordinate base, waist, head, arms, and hands | Control rate, command delay, stability, tracking error, missed deadlines |
| Recovery clock | Detect failure and choose retry, replan, or handoff | Detection delay, autonomous retries, operator time, restored task state |
| Learning clock | Turn demonstrations and corrections into improved behavior | New robot interactions, training cost, regressions, safety stops |
Tau-0-VLA supplies evidence at the task and policy boundary. HAF supplies evidence at the policy, whole-body, and learning boundaries. Neither paper evaluates the combined stack.
A planner that produces better subtasks can still destabilize a humanoid if decisions arrive too slowly. A coordinated whole-body controller can still execute the wrong task because memory is stale. A successful retry can still hide costly operator support. These interfaces need joint evaluation.
A long-horizon evaluation card
Before accepting a long-horizon robot result, reconstruct these fields.
1. Task graph
List every required milestone, prerequisite, optional branch, prohibited action, and terminal condition. State whether the task can follow one fixed sequence or requires decisions from observed state.
2. Memory contract
Define what enters memory, when it updates, how outcomes are verified, what confidence is retained, and how contradictions are repaired. Separate remembered action from observed result.
3. Planning contract
Report whether the planner issues full tasks, language subtasks, latent goals, or action chunks. State trigger thresholds, branching factor, beam width, depth, compute, wall-clock latency, and timeout behavior.
4. Action interface
Name the state and action dimensions, robot embodiment, control mode, chunk length, executed horizon, replanning frequency, controller rate, and how unavailable joints or controls are masked.
5. Timing
Measure observation age and end-to-end latency at each layer. Report median, tail, worst case, jitter, and missed deadlines. Extra planning and denoising should be evaluated against their own timing windows.
6. Outcome and process
Publish full-task success, milestone progress, trials, uncertainty, time limits, retries, failure stage, regression, recovery, and execution time. Do not present normalized partial progress as full-task reliability.
7. Human support
Label teleoperation in demonstrations, intervention during evaluation, manual resets, safety stops, remote assistance, and task restarts. Our robot handoff evidence guide provides the complete intervention denominator.
8. Online learning safety
For physical reinforcement learning, report robot interactions, wall-clock training, reward design, exploration constraints, collisions, emergency stops, operator supervision, hardware damage, and regressions on previously learned behavior.
9. Artifacts and replication
Release evaluation definitions, model revision, training recipe, representative failures, raw trial outcomes, planner settings, memory traces, control logs, and timing measurements. Then state whether another team reproduced the result.
10. Operations
Keep lab task evidence separate from recurring work. A deployment also needs uptime, throughput, quality, intervention rate, recovery time, maintenance, incidents, and economics.
Evidence boundaries
| Claim | Evidence class | Confidence | Boundary |
|---|---|---|---|
| Explicit subtasks and execution memory improve the four Tau-0-VLA long tasks | Author-run matched physical comparison | Medium-high for the reported protocol | Ten trials per method-task cell, one research system |
| More test-time search improves three closed-loop tasks | Author-run physical comparison with fixed low-level policy | Medium | Extra compute and full latency budget are not operationally characterized |
| Tau-0-VLA supports multiple embodiments | Author-run direct execution on three robot configurations | Medium | Shorter cross-embodiment tasks do not test the high-level planner |
| Ordered whole-body generation improves HAF’s seven task scores | Author-run physical comparison and ablation | Medium-high for the reported setup | Normalized milestone score is not complete-task success |
| HAF-Steer improves two tasks under ID and OOD settings | Author-run offline-to-online physical adaptation | Medium | Online interaction, supervision, and safety denominators are incomplete |
| Either system establishes production autonomy | Not established | Low | No independent field deployment, uptime, intervention, safety, or economic evidence |
Verdict
Tau-0-VLA and HAF show why long-horizon robot control should be evaluated as a stack of bounded decisions.
Tau-0-VLA provides evidence that explicit subtask planning, correctable memory, and consequence-aware search can improve multi-stage physical tasks. HAF provides evidence that ordering locomotion, posture, and manipulation inside a whole-body action generator can outperform one-shot generation, and that low-dimensional adaptation can improve a frozen policy.
A different route is to keep the action policy fixed and improve value-guided selection around it. Our Q-Planning evidence audit explains how a separate critic learns from successful and failed rollouts, what the two physical task curves establish, and why human labels and safety accounting remain part of the operating loop.
The practical rule is simple:
Evaluate the planner, memory, low-level policy, body controller, recovery path, and learning process separately. Then test whether their timing and state interfaces remain correct when they operate together.
Frequently asked questions
What makes a robot task long horizon?
A long-horizon task requires several dependent decisions or skills over an extended execution. The robot must track what is complete, select the next useful subtask, execute it, verify the result, and recover when the physical state differs from the plan.
What is the difference between a high-level robot planner and a low-level policy?
A high-level planner chooses a subtask such as pick up the spoon or move to the laundry basket. A low-level policy converts that subtask and the current observations into robot actions. Good motor control cannot rescue a wrong subtask, while a correct subtask can still fail through poor control.
What does test-time computation mean for robot control?
It means allocating extra inference work before committing to a decision. Tau-0-VLA proposes candidate subtasks, predicts their visual outcomes, scores progress, searches retained branches, and reflects before issuing the next subtask. The reported benefit is bounded to author-run tasks and comes with additional compute.
Why does whole-body humanoid control need coordination?
Walking, waist posture, head direction, and arm motion affect one another. A humanoid that moves its base poorly can create compensating upper-body motion and lose manipulation accuracy. HAF generates locomotion and head actions first, then adds waist and arm actions before executing the final full-body chunk.
Do Tau-0-VLA and HAF prove production autonomy?
No. Both provide author-run physical-robot evidence with disclosed task protocols and small trial counts. Neither establishes independent replication, certified safety, long-duration uptime, customer throughput, intervention rate, maintenance cost, or production economics.
What should a long-horizon robot evaluation report?
Report the task graph, milestone and success rules, trials, time limits, planning frequency, memory state, compute and latency, action interface, controller rate, retries, interventions, resets, safety stops, recovery behavior, artifacts, and independent replication status.