VLA Control Interfaces: Prompt Authority, Spatial Grounding, Intent, and Memory

The short answer

A vision-language-action model does not control a robot through actions alone. It also depends on interfaces that decide which text may change the instruction, which geometry reaches the executor, which objective organizes an action sequence, and which past events remain available to the policy.

Four August 2026 preprints make those boundaries unusually visible:

• TOWN-VLA regulates whether retrieved text may alter a frozen policy prompt.
• Pointing-VLA exposes points, functional-region heatmaps, and visual trajectories through typed prediction heads.
• INDI supervises the local objective of demonstrated behavior inside the action decoder.
• UniMem stores event-selected textual and visual memory under the same VLA backbone used for control.

These methods address different failure modes. Their percentages should not be ranked as one leaderboard. The robots, tasks, training changes, execution wrappers, and trial protocols differ.

Interface	Question it answers	When it acts	Main reported physical result	Important boundary
Prompt authority	May external text change the policy input?	Before a rollout	TOWN-VLA: 79/150 to 118/150 successes	One PiPER task, human-scored outcomes, oracle-free routing remains unresolved
Typed grounding	What spatial target should execution consume?	Before grasp and place stages	Pointing-VLA: 79/150 to 121/150 successes	Fixed pick-place scaffold and external motion wrapper
Behavior intent	What local objective should the next behavior serve?	Learned during training, recovered during execution	INDI: 62.0% to 68.7% average across reported physical conditions	Teacher-generated training targets, 50 trials per task-condition
Event-selected memory	Which past events and images should remain in context?	During a rollout	UniMem: 80.0% average across four hardware tasks	15 trials per method-task, tasks lasting minutes rather than hours

The useful engineering lesson is not that one extra module fixes VLA control. It is that each boundary needs an explicit contract, a failure mode, a timing budget, and its own evaluation denominator.

Why interfaces matter as much as model size

A VLA policy receives observations and an instruction, then predicts robot actions. Real systems usually add other components around that core:

• retrieval or planning that proposes extra instructions;
• visual grounding that identifies where to grasp or place;
• training signals that tell the decoder what a behavior is trying to accomplish;
• memory that preserves progress or an earlier scene state;
• motion planning, safety monitoring, and recovery around execution.

Every added component creates a conversion boundary. A retrieved plan becomes prompt text. Multimodal features become coordinates. A demonstration becomes an internal objective. A camera frame becomes a retained memory. Errors at those boundaries can dominate the strength of the base model.

This complements our guide to the embodiment gap, which covers calibration, sensors, control, contact, and target-robot adaptation. The interfaces here sit inside that larger stack. They determine what information reaches action generation and in what form.

Interface 1: prompt authority

Retrieval can supply a frozen VLA with a plan, memory, correction, or contextual hint without retraining its weights. That sounds low risk because the base policy remains unchanged. TOWN-VLA shows why this assumption is incomplete.

When retrieved text enters the executed instruction, it becomes a control input. The relevant question is not only whether the text is useful. It is whether the policy accepts that prompt form without losing its learned instruction boundary.

The prompt-form collapse result

The TOWN-VLA authors hold the policy, initial states, and execution protocol fixed. In one matched audit, the base prompt achieves 92.47% mean success, while raw appended text reduces success to 3.00%.

A separate 500-state control is more diagnostic:

• meaningful task-aligned appended text succeeds on 0/500 states;
• length-matched meaningless appended text also succeeds on 0/500 states;
• restoring the exact base prompt succeeds on 499/500 states.

This does not show that every VLA fails when text is appended. It shows that this frozen policy was highly sensitive to departure from its trained prompt template. Semantic quality did not rescue the changed form.

Proposal is not authority

TOWN-VLA separates generating a candidate from authorizing it. The interface:

1. retrieves and reranks candidate strategy capsules;
2. checks at most two candidates against fixed eligibility rules;
3. converts an accepted candidate into a canonical compact instruction;
4. otherwise restores the exact base prompt;
5. leaves the action-generating policy frozen.

In a 900-route audit, 525 routes returned to hash-identical base prompts. All 375 authorized prompts preserved the task signature. This makes rejection reversible and auditable.

Across a matched LIBERO-Plus evaluation with 10,030 episodes per method, reported success rises from 69.46% to 73.07%, a difference of 362 successes. On a PiPER arm with a frozen pi0.5 checkpoint, the result rises from 79/150 to 118/150. Trials are randomly interleaved across three scene conditions, with 50 trials per scene and method.

The physical comparison is meaningful because the backbone, cameras, feedback loop, criterion, and controller settings remain fixed. It is still a controlled single-task study with one operator and human-scored outcomes.

The unresolved admission problem

The authors’ oracle routing analysis can halve slow-path calls while preserving successes, but the tested oracle-free selectors do not reliably identify which calls should be removed. The learned selector authorizes only 2 of 36 held-out cells and merely matches the base policy.

A deployable prompt-authority system therefore needs more than a safe fallback. It needs a calibrated, oracle-free rule for when external text is worth admitting. It also needs a safety monitor for actions, because restoring a familiar prompt does not prove that the resulting robot motion is safe.

Interface 2: typed spatial grounding

A multimodal model may understand where an object or functional part is located, yet still expose that understanding through a brittle output. Text coordinates create two separate failure points:

1. generation may not produce parseable geometry;
2. a parseable coordinate may still be spatially wrong.

Pointing-VLA treats points, functional regions, and trajectories as different output types rather than forcing all geometry through text.

Three geometries, three jobs

Pointing-VLA adds geometry-specific readouts to a frozen Embodied-R1 backbone:

• Pointing emits a normalized image coordinate.
• Object-functional grounding (OFG) emits a heatmap over the part or region useful for interaction.
• Visual trajectory generation (VTG) emits a sequence of visual waypoints.

The deployed pick-place contract assigns OFG to the pick stage and Pointing to the place stage. An external wrapper maps image-space targets into the robot frame, and CuRobo handles collision-aware motion planning.

This is an important separation. A point, an actionable region, and a path are not interchangeable merely because all can be represented with numbers.

What the evaluations establish

On four Bridge/WidowX tasks, the fixed OFG-pick and Pointing-place combination averages 72.9% over 24 episodes per task. The paper reports no Bridge-specific fine-tuning for this evaluation.

A transferred OFG readout on NORA-1.5 preserves or improves the reported task results and reduces recorded controller time by more than 20 times under the shared wrapper. Separate typed heads are reported as 6.68 to 6.90 times faster than Embodied-R1 text decoding on an external suite. These are system-specific timing comparisons, not universal speed ratios.

The physical study uses an AgileX PiPER and a pi0.5 action policy. Each method receives 50 trials in three visual contexts:

Scene	pi0.5	pi0.5 with Pointing-VLA
No distractor	20/50	36/50
Yellow cylinder	26/50	42/50
Red cylinder	33/50	43/50
Total	79/150	121/150

Pre-lift grasp failures fall from 47 to 16, while tray-arrival failures fall from 13 to 4. Post-grasp transfer failures remain nearly unchanged, from 8 to 9. The failure distribution supports the claimed interface mechanism: the largest gains occur where pick and place geometry is injected.

The result does not establish open-ended spatial reasoning. The task uses one fixed pick-place scaffold, known target types, deterministic interface assignments, and an external execution stack. The paper identifies closed-loop geometric correction, broader task decomposition, and cross-robot transfer as future work.

Interface 3: behavior intent

Behavior cloning tells a robot policy which motor command followed an observation and instruction. It does not explicitly tell the decoder what local objective that command served.

INDI defines behavior intent as the local objective of the forthcoming segment, such as approaching the green block so it can be grasped and stacked on the yellow block. This differs from predicting one future frame or trajectory. Several executions may serve the same objective, and visually similar motions may serve different objectives.

How intent supervision enters the decoder

During training, a frozen teacher vision-language model receives:

• the current observation;
• the task instruction;
• a coarse action summary;
• video of the demonstrated execution segment.

The teacher identifies the behavior objective and produces a multimodal representation. The VLA action decoder learns to recover that representation from its normal deployment inputs at an intermediate layer. It then uses the recovered state to organize action prediction and related grounding predictions.

The teacher and target-generation modules are removed at deployment. Runtime overhead remains in the added decoder representations. The paper reports GR00T-N1.7 growing from 3.46 billion to 3.50 billion parameters and policy-query time rising from 56.1 to 61.5 milliseconds. For pi0.5, the reported change is 3.62 billion to 3.64 billion parameters and 24.4 to 29.7 milliseconds.

Simulation and physical evidence

On SimplerEnv-Bridge, INDI improves the matched GR00T-N1.7 baseline from 64.3% to 84.7% across four tasks. A future-supervision control reaches 68.0%, which helps distinguish behavior-objective supervision from simply adding future information.

On 24 RoboCasa Kitchen tasks with 100 demonstrations per task, the matched GR00T-N1.7 result rises from 64.1% to 70.3%. The same method improves pi0.5 from 34.9% to 41.4%. The reported values are means over three evaluation runs.

The physical study covers four tabletop tasks under clean, held-out-object, and distractor conditions, with 50 trials for each task and condition. Macro-averaged across the three conditions, success rises from 62.0% to 68.7%. The condition-level results are:

Condition	Baseline	INDI
Training objects, no distractors	71.0%	76.0%
Held-out substitute objects	62.0%	68.0%
Unrelated distractors	53.0%	62.0%

The largest gains occur on the longer Cross-Bin Stacking and Drawer Storage tasks. Controlled representation interventions also reduce success when task or phase content is replaced, providing evidence that the learned state affects execution rather than acting only as an unused auxiliary target.

The evidence remains bounded. Teacher-generated interpretations can encode systematic mistakes or shortcuts. The comparison does not isolate every target-generation choice, and the physical tasks remain tabletop experiments. No customer operation, intervention rate, safety-event log, or independent reproduction is reported.

Interface 4: event-selected memory

Some robot tasks are non-Markovian. The current camera image is not enough to choose the correct action because the required action depends on what happened earlier.

A robot may see the spoon over the beans after its first, second, or third scoop. Those frames can look similar, but the next action differs. UniMem calls this perceptual aliasing.

Adding old frames at fixed intervals is not a complete solution. Arbitrary history can miss the decisive moment, consume context, add latency, and encourage spurious correlations. UniMem instead learns when an event is worth storing.

One backbone for memory and action

UniMem builds on pi0.5 and maintains two linked forms of history:

• textual memory records discrete events and progress;
• visual keyframes preserve spatial information from selected moments.

A lightweight event classifier reads the policy’s latent state and decides when to update memory. Selected visual features are cached so the backbone does not re-encode every historical image at every control step. Both memory types condition the same backbone and action expert used for control.

The evaluated memory holds the initial image plus up to three past milestones. The authors report about 90 milliseconds for the policy configuration used in their discussion and a six-times speedup over the hierarchical memory baseline. In a simulated four-camera, 16-keyframe benchmark on an RTX 4090, caching adds about 25 milliseconds beyond a single-frame, two-camera base policy. Those timings depend on the hardware, camera streams, context size, and implementation.

What the memory tasks establish

Five simulation tasks use 25 evaluations per method-task. UniMem averages 93.4%, compared with 68.2% for a fixed-interval video-encoder baseline and 72.6% for the strongest single-memory ablation.

Four xArm6 hardware tasks use 15 trials per method-task:

Hardware task	Hierarchical MemER	UniMem
HammerMeasure	87%	87%
BeanScoop	67%	93%
TableClean	13%	80%
TapScoopPour	7%	60%
Average	43.5%	80.0%

The task differences reveal why both memory types matter. Text can count discrete scoop events, while a visual keyframe can preserve which cup a person tapped or where an object sat before removal. Neither text-only nor keyframe-only memory matches the full method across the evaluated hardware tasks.

The study does not establish durable memory over shifts, days, or open-ended operation. The evaluated context spans several minutes. Event labels are produced through automated offline scripts, with the human tap explicitly marked during data collection. Memory is capped, and the policy does not yet edit, consolidate, or retain experience in its weights.

Our long-horizon robot control guide covers the broader problem of task planning, execution memory, whole-body timing, recovery, and adaptation. UniMem supplies a more specific answer to one layer: how selected visual and textual history can condition low-level action without a separate high-level memory model.

The four interfaces are not substitutes

These methods can be arranged along a control pipeline, but they were not evaluated together.

1. Prompt authority decides whether external textual guidance may alter the instruction.
2. Typed grounding turns multimodal spatial reasoning into geometry an executor can consume.
3. Behavior intent organizes the action decoder around the local objective learned from demonstrations.
4. Event-selected memory supplies earlier task state when the current observation is ambiguous.

A system can succeed at one interface and fail at another. Canonical prompt admission cannot correct a bad grasp heatmap. Correct spatial grounding cannot rescue stale memory. A useful intent representation cannot guarantee that retrieved instructions are safe. Accurate memory cannot compensate for a low-level controller that misses its timing deadline.

This is why the Q-Planning failure-learning result belongs in the same systems conversation without being the same interface. Q-Planning scores action proposals with a learned value model. It changes selection after the policy has proposed actions, while the methods here regulate information entering or organizing the policy before final action execution.

A practical interface audit

When evaluating a VLA system, document each information boundary separately.

1. Authority

• What components may change the executed instruction or control target?
• Is a proposal automatically obeyed, or separately authorized?
• What exact fallback is used after rejection?
• Can the accepted input and fallback be logged and reproduced?

2. Representation

• Does downstream execution need text, a point, a region, a trajectory, an action chunk, or a value?
• Is parsing deterministic?
• What coordinate frame and calibration transform are used?
• Can a syntactically valid output still be geometrically wrong?

3. Training supervision

• Which labels come from people, robot sensors, scripts, simulators, or teacher models?
• Does the target describe an action, future state, motion, objective, success, or safety condition?
• Is the teacher present at deployment?
• What biases can enter through target generation?

4. Memory

• Which events trigger storage?
• What is retained as text, pixels, features, or latent state?
• How long can memory remain useful?
• How are contradictions, stale entries, and capacity limits handled?

5. Timing

• What is the end-to-end observation-to-action latency?
• Which components run once per task, once per stage, or every control step?
• What are median, tail, and worst-case delays?
• What happens when retrieval, decoding, memory update, or motion planning misses its budget?

6. Evaluation

• Are the base policy, initial states, wrappers, and success rules matched?
• How many trials support each percentage?
• Are failures classified by interface stage?
• Are human scoring, resets, interventions, and exclusions disclosed?
• Are results independently reproduced?

Our robot benchmark evidence checklist explains how to keep success rates attached to attempts, task distributions, evaluation checkpoints, and uncertainty. For systems that can change their behavior or invoke fallbacks, also retain the safety and change-control record described in our adaptive robot safety case guide.

Evidence boundaries

Claim	Evidence class	Confidence	Boundary
Appended text can collapse a frozen VLA despite useful semantics	Author-run matched control	High for the tested checkpoint	One prompt template and policy setup, not every VLA
Fail-closed prompt restoration can make rejected guidance reversible	Route audits plus simulation and one physical task	Medium-high	Oracle-free admission remains poorly calibrated
Typed spatial outputs can improve a fixed pick-place stack	Matched benchmark and 150-trial physical comparison	Medium-high	External wrapper, fixed stage assignments, one robot task
Behavior-intent supervision improves matched VLA backbones	Two simulation benchmarks and physical tabletop trials	Medium-high	Teacher-generated targets and no independent replication
Event-selected visual and textual memory improves non-Markovian tasks	Five simulation and four hardware tasks	Medium-high	Fifteen hardware trials per method-task and minute-scale context
These interfaces compose cleanly in one complete robot system	Not tested	Low	No joint evaluation across all four methods
Any of the four methods is production deployed	Not supported	High	No recurring customer operation or operational denominator

Verdict

The most important contribution of these studies is not a combined success rate. It is a clearer map of where robot behavior can change.

TOWN-VLA shows that external text needs authority, not merely relevance. Pointing-VLA shows that geometry should reach execution in a typed form. INDI shows that action imitation can be strengthened by supervising the objective a behavior serves. UniMem shows that history should be selected by events and exposed directly to control when the present view is ambiguous.

A credible VLA stack should make each interface explicit, preserve its inputs and decisions, measure its latency, classify its failures, and retain a safe fallback. The next evidence milestone is a joint system evaluated across unfamiliar tasks, robots, and longer operating periods, with independent replication and complete intervention and safety records.

Frequently asked questions

What is a VLA control interface?

A VLA control interface is a boundary where information is converted into something that can change robot behavior. Examples include admitting retrieved text into a policy prompt, converting visual reasoning into a grasp point, representing the local objective of an action sequence, and selecting past events for memory.

Why can adding useful text make a robot policy worse?

A frozen policy may depend on the instruction format it saw during training. TOWN-VLA reports that appending either meaningful or length-matched meaningless text caused complete failure in a 500-state control, even when the base prompt worked. The authors call this prompt-form collapse.

What is typed spatial grounding?

Typed spatial grounding exposes geometry in the form an executor needs, such as a point, functional-region heatmap, or visual trajectory. Pointing-VLA uses separate prediction heads instead of generating coordinate text that must later be parsed.

What does behavior intent mean in a robot policy?

Behavior intent is the local objective served by the next action segment, such as approaching a particular block to grasp and stack it. INDI uses a teacher model during training to supervise this objective, then removes the teacher at deployment.

Why does a VLA need event-selected memory?

Long tasks can contain visually similar moments that require different actions because earlier events differ. UniMem stores textual events and selected visual keyframes when a learned event classifier detects a task milestone, rather than sampling old frames at fixed intervals.

Do these studies prove production-ready robot autonomy?

No. They are author-run studies on selected benchmarks and laboratory tasks. They do not establish customer throughput, long-duration uptime, broad safety evidence, independent replication, or performance across unrelated robots and environments.