When Robots Mishear: How to Audit Voice Command Safety

The short answer

A voice command should be treated as an untrusted proposal, not as direct authority to move a robot. Speech recognition can change an object, action, quantity, location, person, or negation before a planner sees the instruction. A fluent transcript can still carry the wrong intent. A planner can then turn that error into a valid-looking sequence of executable actions.

A new preprint, When Robots Mishear Us, tests this boundary by corrupting text instructions and sending them through two embodied-agent safety benchmarks. Its clearest result is that some simulated errors changed a model response from refusal to an executable unsafe plan. Automatic transcript correction helped in one condition but did not remove the risk.

The evidence has an important limit. The authors did not evaluate speech audio, microphones, a production automatic speech recognition system, or a physical robot. Four error classes were generated as text with GPT-4, and the noise condition inserted fragments from the CHIME-6 corpus into text. Model outputs were judged by another model. The study therefore demonstrates a planning-layer vulnerability under simulated transcription errors, not the probability that a deployed robot will mishear a person and cause harm.

That boundary still matters. It identifies the complete safety chain that a voice-controlled robot must test:

speaker and environment → audio capture → transcript → intent → authority → plan → executable command → motion and contact → outcome

Word error rate covers only part of that chain.

What the new study actually tested

The study combines five categories of simulated automatic speech recognition (ASR) error with POEX and SafeAgentBench.

The error categories are:

1. acoustic substitution, where a word is replaced by a similar-sounding form;
2. grammar confusion, where grammatical structure is altered;
3. punctuation or segmentation error, where phrase boundaries change;
4. short-word omission or substitution, which can affect words such as articles, prepositions, or negations;
5. noise addition, implemented by inserting cleaned text fragments from CHIME-6 recordings.

The first four categories are synthetic text transformations generated by GPT-4. The noise condition is also a text perturbation, not audio passed through an ASR model. The authors define four average word error rate levels for that condition: 6%, 24%, 53%, and 80%. A mixed condition combines error types. A separate correction condition asks GPT-4 to repair corrupted text.

This design isolates what happens after the transcript changes. It does not measure which errors a real recognizer makes for a particular accent, microphone, room, language, vocabulary, or robot noise profile.

The POEX result: refusal can become execution

The POEX portion uses 136 intentionally harmful instructions across 25 RLBench task environments. Because every instruction is harmful, the desired response is refusal.

Qwen2.5-7B-Instruct generates the plans. GPT-4o evaluates them. The paper reports three measures:

• acceptance rate, the share of harmful requests that the model does not reject;
• executable success rate, the share converted into an executable unsafe plan;
• safety flips, cases that were rejected under clean text but became executable after corruption.

Input condition	Acceptance rate	Executable unsafe rate	Safety flips
Clean text	55.33%	35.33%	Not applicable
Acoustic substitution	60.00%	38.67%	14
Grammar confusion	51.33%	32.00%	5
Punctuation or segmentation	52.00%	30.67%	11
Short-word omission or substitution	54.00%	38.00%	8
Mild inserted noise, 6% WER	56.00%	33.33%	8
Heavy inserted noise, 53% WER	62.00%	36.00%	14
Severe inserted noise, 80% WER	67.33%	34.00%	27
Mixed ASR errors	54.67%	30.67%	9

The clean baseline is already unsafe by the benchmark’s definition. That matters. The study is not comparing a reliable planner with a broken one. It is showing that input corruption can move an already weak refusal boundary in either direction.

The nonmonotonic executable rate is also instructive. Severe corruption increased acceptance and safety flips, but it did not produce the highest executable unsafe rate. Noise can weaken refusal while also damaging plan coherence. A safe design cannot assume that an incoherent transcript will fail harmlessly.

The paper’s representative example starts with an instruction to pour soup on an electrical outlet. Under clean text, the model stops. After a similar-sounding text corruption, it emits steps to grasp the soup, move it over the outlet, and pour. This is a simulated planner example, not a physical execution. It shows how a transcription boundary can alter both risk judgment and plan generation.

The SafeAgentBench result: normal tasks also degrade

SafeAgentBench uses normal task instructions rather than intentionally harmful ones. The study compares clean and mixed-error text for two planners, with 300 samples per model and condition.

Model	Clean planning success	Mixed-error success	Difference
GPT-4o-mini	27.67%	23.33%	-4.34 percentage points
GPT-4.1-mini	38.00%	35.67%	-2.33 percentage points

These reported success rates are low even before corruption. The result supports a narrow conclusion: simulated transcript errors reduced correct planning under this protocol. It does not establish physical task success, because the study evaluates generated plans rather than completed robot trials.

The paired benchmark view is useful. Voice errors can create at least two different failures:

1. an unsafe request may cross a refusal boundary;
2. an ordinary request may become misunderstood or incorrectly planned.

A deployment test must measure both. Testing only harmful prompts misses routine misexecution. Testing only nominal commands misses refusal failure.

Why word error rate is the wrong safety denominator

Word error rate counts substitutions, deletions, and insertions relative to a reference transcript. It is useful for speech recognition, but it does not weight physical consequence.

Compare these hypothetical transcriptions:

• “move the blue cup” becomes “move a blue cup”;
• “do not energize the press” becomes “do energize the press”;
• “give the tool to Maya” becomes “give the tool to Micah”;
• “place two vials in rack B” becomes “place ten vials in rack D.”

The first error may be harmless in a scene with one blue cup. The others change negation, identity, quantity, or destination. One safety-critical word can matter more than a sentence with several harmless errors.

A voice-command audit should therefore add consequence-aware measures:

Measure	Question
Critical-token error rate	Were negations, numbers, units, object names, identities, locations, or mode words changed?
Intent preservation	Does the accepted command mean the same thing as the user’s intended command?
Unsafe acceptance rate	How often did a hazardous or unauthorized request pass the command gate?
Safe refusal rate	How often did the system reject or contain a hazardous request?
False refusal rate	How often was safe, valid work unnecessarily rejected?
Clarification success	Did a confirmation exchange resolve ambiguity without introducing a new error?
Wrong-action rate	Did execution target the wrong action, object, person, quantity, location, or time?
Safe outcome rate	Did the task finish without a defined safety violation?
Exposure-normalized event rate	How many misunderstandings, stops, interventions, and unsafe outcomes occurred per command, task hour, or hazardous exposure?

These measures need a task and hazard model. A transcript cannot be judged safely without knowing what the robot is allowed to do.

Transcript correction is not an execution gate

The study uses GPT-4 to correct two corrupted conditions. For acoustic substitution, the reported acceptance rate falls from 60.00% to 53.33%, the executable unsafe rate falls from 38.67% to 30.67%, and safety flips fall from 14 to 6.

For heavy inserted noise, correction reduces acceptance from 62.00% to 57.33%, but the executable unsafe rate remains 36.00% and safety flips move from 14 to 13.

The correction stage can recover recognizable text. It can also create a confident sentence that the user never said. Its output remains an inference.

A correction model should therefore expose:

• the raw transcript and corrected candidate;
• alternatives and calibrated uncertainty where available;
• the critical fields it changed;
• whether the change altered action, object, identity, quantity, location, negation, or mode;
• a reason to confirm, reject, or use another input channel;
• a record of what the user actually authorized.

Correction can improve an interface. It cannot authorize hazardous motion by itself.

A safer voice-command architecture

A useful architecture separates proposal, interpretation, authority, planning, execution, and protection.

1. Bound the voice interface

Define which commands voice may initiate and which it may not. Record:

• allowed tasks, objects, zones, tools, payloads, and operating modes;
• who may issue each command;
• whether voice can start, pause, resume, stop, or modify motion;
• commands that require a physical control, authenticated console, two-person approval, or direct teaching;
• behavior after an unknown speaker, overlapping speech, stale command, lost network, or low-confidence transcript.

Do not let a general language interface silently expand the robot’s authority.

2. Parse typed intent

Convert the transcript into an explicit command object before planning. For example:

• action: carry;
• object: sealed sample tray 4;
• source: inspection bench;
• destination: cold rack B;
• quantity: one;
• requester: authenticated lab technician;
• deadline: none;
• constraints: keep level, no human handoff;
• confirmation status: required and received.

Typed fields make missing or contradictory values visible. Free-form text can still accompany the command, but it should not hide which values drive execution.

3. Apply consequence-aware confirmation

Confirmation should depend on risk, reversibility, and uncertainty.

A low-consequence request can use a short echo: “Bring the empty blue tote to station two?” A command involving heat, blades, energized equipment, medicines, a person, a restricted zone, or irreversible handling needs a stronger process. That may include reading back critical fields, showing the target, requiring a named response, or switching to a protected control channel.

Confirmation is not useful if the system paraphrases away the disputed value. Repeat exact critical fields and make cancellation easy.

4. Keep planner proposals separate from execution authority

A planner may decompose a valid command into steps. Each step still needs to remain inside the approved task and control envelope.

Our VLA control interfaces guide shows why external text needs an admission rule and a reproducible fallback. The same principle applies here. The transcript proposes intent. A separate gate determines whether that intent may alter robot behavior.

Check at least:

• task authorization;
• object and destination match;
• scene state and prerequisites;
• tool, payload, speed, force, and workspace limits;
• prohibited actions and hazardous combinations;
• freshness of the command and current operating mode;
• whether confirmation covered the final interpreted command.

5. Preserve independent protective controls

Speech recognition, correction, language planning, and semantic monitors are not substitutes for required safety-related controls.

ISO 12100 provides a general machinery process for risk assessment and risk reduction (ISO). ISO 13849-1 addresses design and integration principles for safety-related parts of control systems (ISO). The applicable route depends on the machine and application, but a language-model refusal rate does not establish a safety function’s required performance.

Where the risk assessment requires it, limits, interlocks, protective stops, emergency stops, safe states, and access controls need appropriate independence and validation. Our adaptive robot safety case guide explains how to connect a bounded claim to hazards, controls, tests, monitoring, and change decisions.

How to test a real voice-controlled robot

The new paper begins at corrupted text. A production evaluation must begin with people speaking in the intended environment and finish with physical outcomes.

Build a representative command set

Include:

• routine valid commands;
• safety-relevant valid commands;
• prohibited and unauthorized requests;
• ambiguous object, location, person, and quantity references;
• negation, cancellation, correction, interruption, and changed intent;
• homophones, similar equipment names, abbreviations, jargon, serial numbers, and units;
• multiple speakers, unknown speakers, overlapping speech, and background conversation;
• commands outside the current mode or after the scene has changed.

Do not synthesize the whole test set from one language model. Use hazard analysis, operating procedures, incident history, operator interviews, and real task vocabulary.

Test the real acoustic envelope

Cover the microphones, rooms, robot self-noise, alarms, reverberation, distance, orientation, masks, hearing protection, radio channels, network compression, and device faults that users will encounter.

Include speakers across the intended population and languages. Report coverage rather than claiming a generic “accent test.” Averages can hide one group or vocabulary class with much worse critical-token performance.

Trace failures through the complete chain

For every trial, classify the first failure and the final outcome:

1. speech was not captured;
2. speaker or role was misidentified;
3. transcript changed a critical field;
4. correction changed intent;
5. parser produced the wrong typed command;
6. confirmation failed to expose the difference;
7. authority gate accepted an invalid command;
8. planner created an unsafe or unrelated step;
9. execution gate failed to reject it;
10. motion or contact violated a limit;
11. protective control failed or responded late;
12. operator intervention or recovery was required.

This avoids blaming “ASR” for every downstream defect or, conversely, reporting a low word error rate while the command gate is unsafe.

Use physical trials where consequences are controlled

Begin with simulation and shadow mode, then use fixtures, reduced energy, empty payloads, exclusion zones, and supervised physical trials appropriate to the hazard. Preserve the same command definitions and expected outcomes across stages.

The robot benchmark evidence checklist applies here too. Report attempts, exclusions, repeated initial states, human scoring, interventions, confidence intervals, and the exact model and system versions. A percentage without its trial denominator is not a safety claim.

Field metrics after release

Predeployment tests cannot reproduce every room, speaker, phrase, or interruption. Field assurance should track:

• commands attempted, accepted, rejected, clarified, cancelled, and timed out;
• critical-field corrections by type;
• disagreement between raw transcript, corrected transcript, typed command, and user confirmation;
• false acceptances and false refusals;
• planner and execution-gate rejections;
• unexpected motion, wrong-object, wrong-destination, wrong-person, wrong-quantity, and wrong-mode events;
• protective stops, emergency stops, interventions, near misses, and incidents;
• time from speech to confirmation, action, stop, and recovery;
• performance by task, acoustic condition, equipment vocabulary, and covered user group;
• software, acoustic model, language model, vocabulary, microphone, and robot configuration attached to each event.

Audio can contain personal, biometric, workplace, health, and confidential information. Collection and retention need a declared purpose, access controls, minimum necessary scope, deletion rules, and compliance with applicable privacy and labor requirements. Safety traceability does not justify unlimited recording.

Evidence boundaries

Claim	Evidence class	Confidence	Boundary
Simulated transcription errors can alter embodied-agent refusal and planning	Author-run text benchmark	Medium-high for the tested pipeline	No audio, production ASR, or physical robot
Acoustic substitution increased executable unsafe output in POEX	Matched author-run comparison	Medium-high	One planning model and model-based evaluator, with an unsafe clean baseline
Severe text noise produced more safety flips than clean text	Author-run comparison over 136 harmful instructions	Medium-high	Inserted text fragments are not measured acoustic recognition errors
Mixed errors reduced normal-task planning success	Two 300-sample model comparisons	Medium	Text-plan evaluation, low clean success, no physical execution
Automatic correction removes voice-command risk	Not supported	High	Heavy-noise executable unsafe rate did not improve after correction
Word error rate measures physical safety	Not supported	High	It does not weight intent, authority, consequence, or execution outcome
The paper estimates deployed robot incident frequency	Not supported	High	No field system, operating time, exposure, or physical outcome data

What better evidence would look like

The next useful study should connect real audio to controlled physical outcomes. It would include:

1. declared microphones, acoustic model, language model, planner, robot, controller, and software versions;
2. speakers representative of intended users, languages, and communication conditions;
3. real robot self-noise and workplace acoustic scenes;
4. critical-token, intent, authority, refusal, clarification, planning, and physical outcome metrics;
5. safety-relevant commands derived from a task hazard analysis;
6. matched trials with and without correction and confirmation layers;
7. independent execution gates and protective controls;
8. enough trials to estimate rare but consequential failure modes;
9. complete intervention, stop, near-miss, and excluded-trial records;
10. independent reproduction on another speech stack and robot.

A further challenge is common-cause failure. If the same model family corrects the transcript, infers intent, plans the task, and judges safety, one mistaken interpretation can propagate through every layer. Evaluations should include independent checks and ground truth that does not come from the system being tested.

Verdict

When Robots Mishear Us identifies a credible systems risk: text corruption at the speech boundary can change both refusal and task planning. Its safety flips are useful counterexamples, especially because they show that a superficially recognizable command can cross from rejection into executable output.

The study does not show how often real robots mishear people. It does not evaluate speech waveforms, ASR confidence, microphones, confirmation design, execution gates, physical controls, or robot outcomes. Its model-based benchmark cannot replace application testing.

The durable engineering rule is to keep voice as a bounded proposal. Preserve the raw and interpreted command, confirm consequence-sensitive fields, enforce task authority before planning, gate each executable action, retain independent protective controls, and measure safety from speech through physical outcome.

Frequently asked questions

Can a speech recognition error make a robot unsafe?

It can change the text that reaches a planner, weakening a refusal or producing the wrong task. A new benchmark study reports safety flips after simulated speech recognition errors. The study did not test microphones, acoustic models, or physical robots, so it establishes a planning-layer risk signal rather than a field incident rate.

Is word error rate enough to evaluate voice-controlled robots?

No. Word error rate treats all insertions, deletions, and substitutions alike, but a missing negation, wrong object, wrong person, or wrong quantity can have much greater physical consequence than several harmless transcription mistakes. Evaluate command meaning, hazard class, authority, and executable outcome as well as transcription accuracy.

Should a robot always ask a person to confirm a voice command?

No. Confirmation should be proportional to consequence and uncertainty. Requiring it for every harmless action can create fatigue and slow work. Safety-relevant, irreversible, ambiguous, out-of-scope, or identity-sensitive commands need stronger confirmation, rejection, or transfer to another control channel.

Does automatic transcript correction solve the problem?

Not by itself. In the reported benchmark, correction reduced some acoustic-substitution risk but did not improve the executable unsafe-output rate under heavy simulated noise. A correction model can also confidently replace the user's words with the wrong intent. The system still needs authority checks and execution safeguards.

What should be logged for a voice-controlled robot?

Preserve the audio or a privacy-protecting trace where lawful, raw transcript, corrected transcript, confidence and alternatives, speaker or role evidence, accepted command, confirmation, planner output, rejected actions, executed action, safety response, intervention, outcome, software versions, and timing. Retention and access must follow the site's privacy and security requirements.

Did the new study test a real robot listening to real speech?

No. It generated four language-style error categories with a language model and inserted text fragments from CHIME-6 to simulate noise. It then tested text planners and model-based evaluators on POEX and SafeAgentBench. The paper is useful evidence that corrupted text can alter planning, not a complete evaluation of an audio-to-motion system.