When Robot Rewards Change With Wording: Auditing Vision-Language Reward Models

The short answer

A robot reward model is not reliable merely because its average score is accurate. It must also give consistent judgments when the same goal is expressed in different words. If one phrasing turns a fixed trajectory into a failure and another turns it into a success, the learning system receives contradictory supervision even though the robot behavior did not change.

The September 2026 preprint Same Trajectory, Contradictory Rewards introduces ROBORMBENCH to isolate this problem. The authors hold each recorded real-robot trajectory fixed and vary only the natural-language goal. The benchmark contains:

• 2,390 real-robot trajectories;
• 21,673 filtered paraphrased instructions;
• 14 robot embodiments inherited from RoboRewardBench;
• exocentric and egocentric camera views;
• human-verified progress labels on a five-point scale;
• lexical, syntactic, and action-goal rewrites.

Many evaluated general-purpose vision-language models changed a fixed trajectory from failure-level to success-level scoring under equivalent instructions. Under the most divergent action-goal rewrites, the reported Score Crossing Rate ranged from 0.164 to 0.607 across the tested proprietary models and from 0.193 to 0.557 across the tested general-purpose open models. Two dedicated RoboReward models were more stable, at 0.086 and 0.114, but neither reached zero.

This is strong author-run evidence of an evaluation failure mode. It is not yet an online robot-learning result, an independent reproduction, or a deployment safety result. The downstream experiments select among recorded trajectories. They do not show how often wording instability changes a physical robot’s actions in closed loop.

Why reward wording can change robot learning

A vision-language reward model maps two inputs to a scalar score:

1. a visual record of robot behavior;
2. a text description of the intended goal.

The score may be used to label progress, rank demonstrations, select the best attempt, relabel collected data, or guide reinforcement learning. This creates a basic invariance requirement. If two instructions mean the same thing and the trajectory is identical, the model’s judgment should not cross from failure to success.

A wording-sensitive policy input is already a control risk. A wording-sensitive reward is different because it can corrupt the learning signal itself. The model may reward one failed behavior under one phrase, reject the same behavior under another, and gradually optimize for linguistic artifacts rather than task completion.

Our Q-Planning evidence audit shows why the reward layer matters even when the large action policy remains frozen. A separate value model can change which policy proposals are executed. If that evaluator is language-fragile, stable policy weights do not guarantee stable action selection.

What ROBORMBENCH actually varies

The benchmark begins with RoboRewardBench trajectories drawn from Open X-Embodiment and RoboArena. The original collection includes real-robot manipulation, multiple embodiments, two camera perspectives, and human-verified progress scores. The authors downsample the majority score-1 class before generating rewrites so the evaluation is less dominated by clear failures.

Each instruction is rewritten through three strategies:

Rewrite strategy	What changes	Example pattern	Main risk tested
Lexical substitution (LS)	content words	“put” becomes “place”	sensitivity to synonyms
Syntactic restructuring (SR)	sentence structure	imperative becomes a differently ordered sentence	sensitivity to grammar and form
Action-goal perspective shift (AGPS)	action description versus desired end state	“put blocks in the bowl” becomes “the blocks should end up in the bowl”	sensitivity to how task intent is framed

The three sets are evaluated separately. After filtering and minimum-count rules, the paper reports 613 trajectories for lexical substitution, 943 for syntactic restructuring, and 834 for action-goal shifts. These sum to 2,390 benchmark trajectory records with 21,673 accepted paraphrases.

The separation matters. A model that survives synonyms may still fail when the instruction changes from an action command to an equivalent goal state.

How semantic equivalence was checked

A paraphrase benchmark becomes invalid if its “equivalent” instructions quietly change the task. ROBORMBENCH addresses this with a conservative filter.

Three models independently evaluate every generated rewrite. A candidate is discarded if any validator flags a semantic difference, including modifiers such as “slowly.” Trajectories with fewer than five valid paraphrases after filtering are also excluded.

The authors then compare the ensemble with two human annotators on 210 sampled candidates. At least one human flag defines the human-flagged group. The resulting table contains:

• 24 candidates flagged by both the model ensemble and people;
• 1 human-flagged candidate missed by the ensemble;
• 17 candidates flagged only by the ensemble;
• 168 candidates flagged by neither.

That is 96% recall on the 25 human-flagged meaning changes in the sample. It supports the authors’ claim that the filter is conservative. It does not prove that all 21,673 retained rewrites are perfectly equivalent. The human audit is a 210-item sample, and the complete benchmark is English-only.

A strong reproduction should publish agreement by rewrite type, task category, and annotator, not only one pooled sample.

Three metrics that should stay separate

ROBORMBENCH reports robustness and accuracy as different properties.

Score Crossing Rate

The model predicts a progress score from 1 to 5. Scores 1 and 2 are treated as failure-level. Scores 4 and 5 are success-level. Score 3 is intermediate and excluded from the binary comparison.

A trajectory counts as a score crossing when at least one equivalent instruction produces a failure-level score and at least one produces a success-level score. Score Crossing Rate (SCR) is the fraction of trajectories with such a contradiction.

SCR is reference-free. It does not assume that the original wording is privileged. It asks whether any phrasing pair inside the verified set crosses the decision boundary.

Flip Rate

Flip Rate (FR) uses the original RoboRewardBench instruction as a reference. It measures the fraction of paraphrases whose failure-versus-success judgment differs from that original wording.

A model can have a modest FR while still producing a serious contradiction on a subset of trajectories. SCR finds how many trajectories are vulnerable. FR shows how frequently individual rewrites disagree with the reference.

Mean Error

Mean Error (ME) first averages a model’s predicted score across a trajectory’s paraphrases, then takes the absolute difference from the human progress score. The benchmark averages that error across trajectories.

ME measures average calibration against labels. It does not measure invariance. A model can be stable but consistently biased, or accurate on average while crossing the failure-success boundary under different wording.

Our robot benchmark success-rate guide makes the same general point for policy evaluation: one aggregate cannot replace the protocol and the failure distribution behind it.

What the model comparison reports

The paper evaluates seven proprietary VLM configurations, seven general-purpose open VLM configurations, and two dedicated RoboReward models. Lower values are better for SCR, FR, and ME.

The action-goal perspective shift is the most revealing condition because it changes how the same intent is framed.

Model	AGPS SCR	AGPS FR	AGPS ME	Defensible reading
Gemini 2.5 Flash Lite	0.607	0.264	0.970	more than half of tested trajectories crossed the boundary
Llama 4 Scout	0.557	0.216	1.109	high instability under goal-state reframing
GPT-4o	0.379	0.141	1.285	instability and comparatively high average error
GPT-5.1	0.300	0.101	1.015	lower than GPT-4o, but still 30% crossing
Claude Sonnet 4.6	0.164	0.047	0.970	lowest reported AGPS SCR among the tested general VLMs
RoboReward 4B	0.086	0.045	0.456	more stable and more accurate in this benchmark
RoboReward 8B	0.114	0.042	0.453	low average error, but nonzero crossing remains

These rows are selected to show the range, not to create a universal model leaderboard. Results belong to the paper’s prompts, model versions, generation settings, trajectory sample, label scale, and September 2026 evaluation.

The dedicated reward models have an important training advantage for this task. They were trained directly on trajectory and reward data. Their result supports the hypothesis that trajectory-grounded supervision matters more for this failure mode than general multimodal scale alone. It does not isolate every difference in architecture, data, prompting, or optimization.

Scale and reasoning were not reliable fixes

Within the tested Qwen3-VL and Gemma 3 families, SCR did not consistently improve with parameter count. The paper reports that larger variants often had higher crossing rates.

Explicit reasoning also failed as a universal remedy. Reasoning-enabled variants increased SCR in the paper’s main Qwen3-VL-235B-A22B and Gemini 3 Flash comparison while leaving average error broadly similar. Additional comparisons were mixed, improving GPT-5.1 robustness but reducing Claude Sonnet 4.6 robustness.

The correct conclusion is narrow:

For these evaluated model families and settings, more parameters and explicit reasoning did not reliably produce paraphrase-invariant robot rewards.

This is not evidence that scaling can never help. It shows that robustness must be measured directly rather than inferred from model size or a reasoning label.

How many paraphrases reveal the problem

SCR can only stay constant or increase as more equivalent instructions are tested because each added wording creates another opportunity to expose a crossing. The practical question is when the estimate becomes informative enough.

The authors subsample paraphrase sets at 3, 5, 7, 10, and 15 instructions. For Qwen3-VL-235B-A22B, SCR rises from 0.138 at three paraphrases to 0.212 at seven, then remains 0.212 through fifteen. For Gemini 2.5 Flash Lite, it rises from 0.125 at three to 0.263 at seven and reaches 0.290 at fifteen.

This suggests that seven paraphrases captured most observed crossings for these two tested models and this condition. Seven is not a universal standard. A production test should estimate saturation for its own task language, model, decision threshold, and rewrite distribution.

Why average accuracy was not enough

The strongest downstream evidence in the paper is an offline best-of-20 selection study. For each task, a reward model scores 20 recorded candidate trajectories and selects the highest-rated one. The authors compare the selected trajectory’s human progress score with the best available candidate.

Three proprietary models had similar AGPS Mean Error values:

Selector	Mean Error	SCR	Selected human score	Regret to best candidate
Gemini 2.5 Flash Lite	0.970	0.607	3.222	1.333
GPT-5.1	1.015	0.300	3.737	0.818
Claude Sonnet 4.6	0.970	0.164	3.980	0.576
Oracle	not applicable	not applicable	4.556	0.000

The similar error values do not produce similar decisions. The more paraphrase-stable model selects higher-progress trajectories and leaves less regret in this comparison.

Across 11 evaluated models, the paper reports a strong positive association between SCR and best-of-20 regret (Pearson r = 0.882, Spearman rho = 0.856). A separate pairwise preference test reports a negative association between SCR and pair accuracy (Pearson r = -0.893, Spearman rho = -0.855) and a positive association with regret.

These are correlations across author-evaluated models, not proof that lowering SCR alone causes better policies. The study also remains offline. It shows that reward instability can change which recorded trajectory is preferred, not how a physical robot learns after repeated closed-loop updates.

Two mitigations, two different costs

Average across equivalent instructions

The first mitigation generates five additional equivalent variants from each seed instruction and averages the reward predictions. The paper reports lower SCR and prediction error across five VLMs and all three rewrite strategies.

This method reduces dependence on one surface form, but it has three limits:

• it requires multiple reward-model queries;
• it still leaves nonzero score crossings;
• its quality depends on generating genuinely equivalent variants.

For a robot learning loop, the extra queries also add cost and latency. Those system effects were not evaluated in closed-loop hardware control.

Penalize reward variance during training

The second mitigation adds a training penalty when the same trajectory receives different scores across verified paraphrases. On Qwen3-VL-4B, the authors report:

Condition	Base SCR	Variance-reduced SCR	Base ME	Variance-reduced ME
Lexical substitution	0.076	0.042	1.046	0.239
Syntactic restructuring	0.168	0.048	1.003	0.319
Action-goal shift	0.200	0.057	1.020	0.372

The result is promising because it improves both reported stability and accuracy without extra inference queries. It is one model, one training recipe, and an author-run benchmark evaluation. A complete claim still needs held-out task families, unseen paraphrase generators, multiple training seeds, ablations, online policy outcomes, and independent reproduction.

What the benchmark does not establish

ROBORMBENCH has important boundaries:

1. Recorded trajectories, not live control. The visual input is fixed. The model does not change a physical robot’s next action during the benchmark.
2. End-of-episode rewards. The paper does not test step-level shaping, where wording errors could change many updates inside one trajectory.
3. English instructions only. Multilingual, code-switched, abbreviated, speech-derived, and operator-specific language remain untested.
4. Generated paraphrases. A conservative model filter and human sample improve quality, but they do not replace full human verification of every rewrite.
5. Five-point progress labels. Results may change with binary success, pairwise preference, continuous value targets, or safety penalties.
6. Author-run model access. Hosted models and prompts can change. Reproduction needs frozen outputs, version identifiers, and executable evaluation artifacts.
7. No safety outcome. Reward consistency is necessary for dependable learning, but a stable reward can still value unsafe behavior or omit a hazard.
8. No field deployment. The paper reports no customer operation, intervention rate, shift duration, damage, or production throughput.

Our VLA control interfaces guide covers a related but distinct problem. Prompt authority asks whether changed text should enter an action policy. ROBORMBENCH asks whether equivalent text changes the evaluator that scores completed behavior.

A practical reward-language audit

A team can adapt the paper’s method without treating one benchmark as certification.

1. Freeze the behavior evidence

Use the exact same trajectory frames, ordering, sampling rate, crop, and preprocessing for every instruction. Hash the media and model configuration. If visual input changes, the test no longer isolates language sensitivity.

2. Define the decision boundary

State how scalar scores become failure, intermediate, and success. Test at least one strict extreme-only boundary in addition to the operational threshold. Report how many predictions are excluded as intermediate.

3. Build task-native paraphrases

Include synonyms, syntax changes, action-to-goal reframing, references to object attributes, pronouns, word-order variation, concise operator language, and speech-recognition variants when voice is part of the system. Do not assume a generic paraphrase generator matches field language.

4. Verify meaning independently

Use domain experts or trained annotators who can see the task definition and relevant scene context. Record disagreements. Report equivalence precision and recall by rewrite type, then retain a frozen accepted set.

5. Report robustness beside accuracy

At minimum, publish:

• Score Crossing Rate;
• Flip Rate;
• Mean Error or task-appropriate calibration error;
• prediction range and standard deviation;
• results by task, progress label, rewrite type, and environment;
• uncertainty across trajectories and repeated model calls.

6. Test the downstream decision

Measure whether paraphrases change trajectory ranking, data retention, relabeling, policy checkpoints, or selected actions. Report selection regret against human or environment ground truth where available.

7. Test mitigations out of sample

Keep tasks, objects, operators, and paraphrase styles out of mitigation training. Compare aggregation, variance penalties, task-specific reward supervision, and non-language success detectors under matched query and compute budgets.

8. Preserve the safety boundary

A paraphrase-stable reward is not necessarily a safe reward. Evaluate unsafe contacts, forbidden regions, speed and force limits, human proximity, emergency stops, and intervention separately. Our adaptive robot safety case guide explains why task success and safe success need distinct evidence.

9. Re-run after every material change

Model version, system prompt, image sampling, threshold, task vocabulary, and paraphrase generator can all change SCR. Treat the audit as a versioned regression test, not a one-time paper result.

Evidence boundaries

Claim	Evidence class	Confidence	Boundary
Equivalent wording can change a fixed trajectory from failure-level to success-level scoring	author-run benchmark on recorded real-robot trajectories	high for the reported evaluation	no independent reproduction located
ROBORMBENCH contains 2,390 trajectories and 21,673 accepted paraphrases	paper dataset description	high	trajectories are divided across three rewrite sets, not 2,390 unique physical episodes established by this article
The semantic filter removes most meaning changes	210-item sampled human comparison	medium-high	one missed human flag, 17 conservative ensemble-only flags, and no full human audit
Dedicated RoboReward models are more robust than tested general VLMs	matched author-run model comparison	medium-high	training, architecture, prompting, and data differences are not fully isolated
Larger or reasoning-enabled models are inherently less robust	not established	low	tested families show no reliable benefit, not a universal scaling law
Lower SCR improves reward-guided decisions	offline selection and pairwise correlation	medium	association across models, not a controlled online policy-learning intervention
Aggregation and variance training solve wording sensitivity	not established	high confidence in the negative boundary	both reduce reported instability, but crossings remain or evidence is limited to one training setup
ROBORMBENCH establishes physical deployment safety	not supported	high	no closed-loop robot action, safety-event, or field-operation evaluation

Questions to ask about a robot reward model

1. What exact behavior, frames, and instruction does the model score?
2. Is the output progress, binary success, preference, value, or a safety judgment?
3. Which score boundary changes learning or action selection?
4. Does the same trajectory keep the same category across equivalent wording?
5. How were paraphrases generated and independently verified?
6. Are field operators, speech recognition, abbreviations, and multilingual inputs represented?
7. Are SCR and FR reported beside calibration or average error?
8. Which tasks, labels, or rewrite styles produce the most crossings?
9. Does instability change which data or action the system selects?
10. How many repeated model calls and paraphrases are needed before the estimate stabilizes?
11. What mitigation was trained, and on which held-out language and tasks was it tested?
12. What does mitigation cost in latency, queries, compute, and annotation?
13. Can a non-language sensor or environment predicate verify critical success conditions?
14. Are unsafe behavior and task failure scored separately?
15. Are model versions, prompts, outputs, and evaluation artifacts preserved for reproduction?

Verdict

ROBORMBENCH identifies a narrow but consequential failure mode. A reward model can look acceptable on average while issuing contradictory categorical judgments about identical robot behavior. The paper’s fixed-trajectory design isolates language better than evaluations that change both the instruction and the rollout.

Its most useful contribution is the metric split. Mean Error asks whether scores are accurate on average. Score Crossing Rate asks whether equivalent language can reverse the category. Flip Rate asks how frequently individual rewrites disagree with a reference. Offline selection regret then tests whether the instability changes a downstream decision.

The dedicated reward models and mitigation studies suggest practical routes forward, especially trajectory-grounded supervision and explicit consistency training. They do not establish a finished solution. The benchmark remains English-only, end-of-episode, author-run, and offline.

Before using natural language to supervise robot learning, test whether meaning stays fixed when wording changes. Then verify that the stable reward also represents the right task, the safety constraints, and the physical outcome.

Frequently asked questions

What is a vision-language reward model for robotics?

A vision-language reward model reads a robot trajectory together with a natural-language goal and predicts a score for task progress. That score can rank collected trajectories, label demonstrations, or guide policy improvement. The reward model evaluates behavior, it does not directly generate the robot's motor commands.

What is Score Crossing Rate?

Score Crossing Rate is the fraction of fixed trajectories that receive at least one failure-level score and at least one success-level score across semantically equivalent instructions. In ROBORMBENCH, scores 1 and 2 are failure-level, scores 4 and 5 are success-level, and score 3 is excluded from this binary comparison.

How is Flip Rate different from Score Crossing Rate?

Score Crossing Rate asks whether any contradiction exists within a trajectory's paraphrase set. Flip Rate uses the original instruction as a reference and measures how often an individual paraphrase changes the model's failure-versus-success judgment. Both are needed because one detects vulnerable trajectories while the other measures disagreement frequency.

Did larger or reasoning-enabled models solve paraphrase fragility?

No. In the author-run evaluation, larger models within the tested Qwen3-VL and Gemma3 families did not become consistently more robust. Explicit reasoning also did not reliably lower Score Crossing Rate. These are results for the evaluated models and prompts, not a universal claim about every future model.

Does ROBORMBENCH prove that paraphrase fragility makes a physical robot unsafe?

No. The benchmark evaluates end-of-episode reward predictions on recorded real-robot trajectories. Its downstream study covers offline trajectory selection and pairwise preference, not online policy training, physical deployment, safety events, or closed-loop robot behavior.

How should a team test a language-conditioned robot reward model?

Hold the trajectory fixed, create human-verified equivalent instructions, report Score Crossing Rate and Flip Rate beside prediction error, repeat the test across task and rewrite categories, then measure whether instability changes trajectory selection or policy updates. Keep safety validation separate from reward accuracy.