acceptodds
Under review as a conference paper at ICLR 2027

Robust to Noise, Broken by Turns: Benchmarking Speech-Driven Omni Agents in the Wild

Abstract

Speech is a natural interface for agents, yet most agent benchmarks assume typed instructions. We introduce OmniAgentBench, a benchmark for evaluating speech-driven multimodal planning across Schedule, GUI, and Embodied tasks, with matched text and audio inputs and controlled wild conditions. Native omni models perform comparably to ASR+LLM cascades (automatic speech recognition followed by an LLM) on clean speech, but they collapse when the same constraints are distributed across dialogue turns, with Qwen-Omni-3 dropping from 62.0 to 18.8 action matching score (AMS) on GUI navigation. Controlled experiments trace this degradation to the plan-every-turn protocol, not to speech perception or ASR. Because the model must respond after each turn, it commits early to an incomplete plan. Presenting the same turns as text produces a similar collapse, while suppressing intermediate responses largely restores performance. Training-free prompts that make the model wait for the last turn, or restate every constraint before planning, recover most of the loss without extra model calls, and the collapse replicates across four native omni models from three labs. We also recorded 116 human speakers reading the same instructions. On clean speech, human recordings score about as well as TTS on average, and the multi-turn collapse reproduces on real voices. OmniAgentBench contains 11,700 audio cases from 900 base instances and provides a reproducible protocol for studying speech-grounded multimodal planning under realistic instruction variation.

Primary area:
datasets and benchmarks
Keywords:
Robust to Noise, Broken by Turns: Benchmarking Speech-Driven Omni Agents in the Wild

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Atf9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OmniAgentBench, a benchmark covering Schedule, GUI, and Embodied tasks. It adapts existing benchmarks by converting text instructions into speech using TTS and introducing variations such as background noise, informal phrasing, and multiple dialogue turns. It then evaluates whether models can follow the spoken instructions and complete the tasks under these conditions.

Critical strengths
  1. As spoken interaction becomes more common, evaluating agents that follow spoken instructions is an important and timely research direction.

  2. The authors include human validation to check constraint preservation and audio intelligibility during benchmark construction. The additional evaluation with real human recordings also strengthens the benchmark. Creating benchmarks with human in the loop is largely appreciated.

Critical weaknesses
  1. Limited benchmark novelty. The benchmark largely reuses existing tasks, environments, and target plans, with instructions converted into speech through TTS. Although the added speech conditions are useful, I am not convinced that this adaptation provides a sufficiently distinct benchmark contribution.

  2. [minor] Unclear presentation and task definitions. Figure 1 provides a general pipeline overview, but the paper lacks a complete example showing the actual instruction, visual context, expected output, and evaluation procedure. This makes it difficult to understand what the benchmark measures. Similarly, Section 3.1 introduces the constraint set C and goal G, but without further explanations. Examples from each benchmark would help substantially.

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Response to Reviewer Atf9 (part 1 of 2)

Thank you for the review. We appreciate that you see spoken instructions for agents as an important and timely direction, and that you value the human validation and the real human recordings. We address your two concerns below. Every item, with its audio, context, and gold plan, is in the anonymous release linked in the reproducibility statement. The revised PDF also adds Appendix Q, Examples from Each Benchmark.

W1, benchmark novelty. To our knowledge, OmniAgentBench is the first benchmark in which spoken instructions drive multimodal agent planning under controlled wild conditions. Beyond the data, it contributes five new findings and a training-free method, set out below. The tasks, environments, and target plans are reused on purpose, so that only the spoken instruction changes. Each wild condition is then compared with clean speech and with matched text on exactly the same task. This contrast reveals a gap that the source benchmarks cannot show. On a single clean utterance, native omni models match ASR+LLM cascades (Table 2). Yet on EmbodiedBench, Qwen-Omni-3 falls from 56.2 to 32.1 Step-LCS when the same request is split into two turns (Table 6).

To address this gap, we propose Recap-and-Plan (Section 3.3). It adds one instruction to the final turn, asking the model to restate every constraint before it plans. With no training and no extra model call, it recovers essentially all of the lost EmbodiedBench performance for Qwen-Omni-2.5 (94 to 102 percent) and two thirds or more for Qwen-Omni-3 (67 to 71 percent, Table 6). It also carries over beyond Qwen and fully recovers the audio collapse of MiniCPM-o 2.6 from OpenBMB (Appendix E). A second training-free intervention, withholding the intermediate replies until the request is complete, lifts Qwen-Omni-3 on GUIBench from 18.6 to 60.8 AMS (Table 10).

The adaptation also goes well beyond converting the instructions with TTS.

  • New test cases. Every instance gets an informal rewrite and new two-turn and three-turn sequences. They are written as new spoken language and checked by annotators. An EmbodiedBench request grows from 9 words to 113 to 127 words.
  • New evaluation scenarios. They include real background noise at 5 to 15 dB SNR with a second talker over the whole request, silence-padded long audio, and delivery split across turns. Matched controls hold the audio fixed (Concat, Silent, Recap, Wait, End-marker).
  • Human speech. We collected 913 recordings from 116 speakers with 26 first languages and tested them for equivalence with TTS (G1).
Audio cases added to each source benchmark (Table 1)Cases
Clean speech300
Real background noise1,800
Silence-padded long audio900
Informal rewrite300
Two-turn and three-turn sequences600
Per benchmark, and over all three3,900 and 11,700

These evaluations lead to five findings that are new for speech-driven omni agents (G1).

  1. The agents are robust to noise. Real background noise moves the native omni average by less than one point on each benchmark (Tables 2 and 4), and human recordings score like TTS on average (-0.7 [-2.6, +1.1]).
  2. The agents break when a request is split across turns, and the cause is answering after every turn. For the Qwen models, matched text collapses about as much as audio (Table 7), and withholding the replies restores the score.
  3. The ranking reverses. On the full clean GUIBench set, Qwen-Omni-3 beats the best cascade (62.0 against 43.8 AMS, Table 2), but on the multi-turn set the cascade leads (48.0 against 18.8, Table 4).
  4. A newer, larger model collapses more (24.1 against 8.8 at two turns, Table 6). The collapse appears in six native omni models from five labs (Table R1) and on human speech.
  5. The model commits to a plan before the request is complete, even when told it may wait (G3a), and a user who adapts turn by turn does not prevent the loss (G3b).

Prior work also builds benchmarks on existing data. OmnixR (ICLR 2025) builds on MMLU-Pro, VoiceBench (TACL 2026) on text benchmarks spoken with TTS, and Speech Robust Bench (ICLR 2025) on existing speech corpora. In OmniAgentBench the reuse is by design. Only the spoken input changes between the clean and the wild conditions, so the gap between them is measured on the same tasks. This datasets and benchmarks submission is built to measure that gap, and the paper's two training-free interventions address it.

W2, examples from each benchmark, and C and G. Each instance pairs a spoken request with context M, a goal G, and constraints C. G is the requested end state, and C is the set of requirements stated in the request. Below is one item per benchmark. The EmbodiedBench and GUIBench outputs come from our rerun of the paper's setting with Qwen-Omni-3.

Continued in part 2.

AuthorsOpenReview ↗

Response to Reviewer Atf9 (part 2 of 2)

W2, examples from each benchmark (continued).

EmbodiedBench. We picked the item by a fixed rule. Among the items whose constraints map one to one onto the gold steps and whose Concat plan is fully correct, it has the median Concat-to-Multi-Turn difference. M is one egocentric image of a living room with a blue armchair, a white sofa, and a floor lamp, and the white shelves are behind the agent. G is the end state in which the agent holds the box and the floor lamp is on. C has four requirements.

  1. Turn around and go to the white shelves on the left.
  2. Pick up the open box from the bottom shelf.
  3. Turn around, walk toward the blue chair, and turn right to face the white sofa.
  4. Turn on the light.

The gold plan is GotoLocation(shelf), PickupObject(box), GotoLocation(floorlamp), ToggleObject(floorlamp). Requirement 3 brings the agent to the floor lamp, which is why it appears as GotoLocation(floorlamp). Turn 1 says "turn around and head over to those, um, white shelves ... grab, like, the open box from, um, you know, the bottom shelf". Turn 2 says "turn around again and walk towards, uh, the blue chair ... then just, like, turn on the light". Concat plays both turns as one utterance. Step-LCS is the longest common subsequence of the predicted and gold steps, divided by the number of gold steps.

ConditionModel outputGold steps matchedStep-LCS
ConcatGotoLocation(shelf), PickupObject(box), GotoLocation(armchair), GotoLocation(floorlamp), ToggleObject(floorlamp), NoOp4 of 41.00
Multi-TurnGotoLocation(bluechair), GotoLocation(sofa), ToggleObject(floorlamp), NoOp1 of 40.25

The Multi-Turn plan keeps only the steps from turn 2, so it never goes to the shelves or picks up the box.

ScheduleBench. The spoken request asks for a flight from Rome to London on March 20, 2025, with a transfer in Paris. It asks to "minimize the total cost", with a total price of at most 600 US dollars and "at least a one-hour interval between each pair of flights". M consists of two timetable images, Rome to Paris and Paris to London. G is arriving in London that day at the lowest price, and C consists of the Paris transfer, the 600-dollar budget, and the one-hour buffer. The agent outputs one itinerary. It scores 0 if the itinerary is infeasible, 0.5 if it is feasible but not the cheapest, and 1 if it is the cheapest feasible itinerary. Here only one itinerary is feasible, Rome to Paris from 04:45 to 07:00 and Paris to London from 12:30 to 14:05, for 437 dollars.

GUIBench. Turn 1 says "check out that app, uh, 'Chatty - AI Assistant' ... ask it about ... the Fundamental theorem of algebra". Turn 2 says "double-check all that stuff by, uh, you know, like, searching in Firefox". We score the second step of the episode. There, M is the current screenshot, with Chatty open on an empty chat, plus the history of one earlier click. G is the question answered in Chatty and the answer checked in Firefox, and C consists of the requirements in the two turns, which GUIOdyssey aligns to interface steps. The agent outputs one next action. A click gets full credit within 5 percent of the screen diagonal, and its credit decays to zero at twice that distance. Typed text, scrolls, and completion follow fixed rules. The gold action is a click on the message box. With Concat, the agent clicks the message box and scores 1.0. With Multi-Turn, it outputs COMPLETE and scores 0.0, declaring the task finished before it has started. In the paper, Multi-Turn raises premature COMPLETE actions from 24 to 88 of 300 predictions.

We hope the examples and the summary of new data make the contribution clearer, and we would welcome your reassessment of it. We invite you to check the revised PDF, where new and reworded text is marked in blue. Section 3.1 and Appendix Q add the definitions and the examples. If any concern remains, please let us know.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer Atf9,

Thank you for your review. The revised PDF adds the worked examples and the definitions of C and G that you asked for (Appendix Q, Section 3.1). Could you let us know whether the revision addresses your concerns?

Reviewer_o925OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OmniAgentBench, a benchmark for evaluating speech-driven multimodal planning across GUI navigation, scheduling, and embodied planning. It provides matched text and audio instructions with controlled variations, including noise, informal phrasing, silence padding, and multi-turn delivery. The main finding is that requiring models to respond after each instruction turn can substantially degrade final planning performance. Controlled comparisons with text inputs and silent accumulation implicate intermediate responses as an important contributor to this degradation. The paper also evaluates training-free interventions and validates key findings using human recordings.

Critical strengths
  1. The controlled diagnosis goes beyond reporting benchmark scores. The comparisons between text and audio, concatenated and multi-turn instructions, and intermediate responses versus silent accumulation help distinguish speech-perception difficulties from failures associated with the interaction protocol. The substantial recovery under silent accumulation provides meaningful evidence that intermediate responses contribute to the observed degradation.

  2. The empirical validation is reasonably broad. The paper evaluates multiple task domains and model families, and supplements synthetic speech with recordings from 116 human participants. The matched controls and paired statistical analyses strengthen the central finding. The reported exceptions, including weaker recovery on some models and tasks, also provide useful information about the limits of the interventions.

Critical weaknesses
  1. The ecological validity of the multi-turn protocol is limited. User turns are scripted, and the baseline requires a response after each turn even when the full request has not yet arrived. This is a useful controlled setting, but it does not establish how agents behave when allowed to acknowledge, clarify, or defer naturally. Moreover, the intervention prompts are explicitly informed of the final turn. Human recordings are read aloud rather than spontaneous, and embodied planning is evaluated offline rather than through task execution. These limitations warrant a narrower interpretation of the “in the wild” framing.

  2. Some broad claims require qualification. The small speech-versus-text gap observed for the Qwen models does not hold equally across all evaluated models. Likewise, Recap-and-Plan recovers much of the loss on EmbodiedBench but transfers poorly to GUIBench, and ScheduleBench does not exhibit the same collapse. The paper should distinguish the relatively strong evidence for protocol-dependent degradation from the more limited generality of its explanations and proposed fixes. Its novelty should also be positioned primarily as an extension and diagnosis of previously observed multi-turn degradation, rather than an entirely new failure phenomenon.

Questions for authors

Does the degradation persist under a more natural interaction policy that allows acknowledgments, clarification questions, or deferral without explicitly identifying the final turn? Such a control would help establish how much of the effect extends beyond the plan-every-turn protocol.

3 comments
AuthorsOpenReview ↗

Response to Reviewer o925

Thank you for the careful review, and for noting that the controlled diagnosis goes beyond reporting benchmark scores. Your question led to the most informative of our new experiments.

Question, does the degradation persist under a natural interaction policy? Yes, most of it does. We ran the free policy you suggested. G3a quotes the prompt in full, and the revised PDF reports the result in Section 4.3 and Appendix P (Table 29). The policy allows acknowledgment, clarification, or deferral, and it never marks the last turn. On Qwen-Omni-3, the agent commits to a plan on 97 to 98 percent of incomplete turns and asks a question on 2 to 3 percent. It recovers only 15 and 19 percent of the gap (+3.8 [+1.2, +6.2] and +6.3 [+3.3, +9.2] Step-LCS points over Multi-Turn).

The paper's Wait prompt is a useful comparison, because it shares the task prompt and output format with the free policy. Wait tells the agent to defer and marks the last turn. Under Wait, the agent defers on all 300 items and recovers 75 to 81 percent (Table 8). So the model is able to wait, but under the free policy it almost never chooses to. The adaptive simulated user (G3b, Appendix P, Table 30) also gives no final-turn flag, and reacting turn by turn did not prevent the loss.

W1, ecological validity. In the title, "in the wild" refers to controlled variations of the spoken input, namely real background noise, silence padding, informal phrasing, and delivery split across turns (G2). The revised introduction now states this definition (Section 1). These conditions are hard even for human listeners. The noise comes from six environments built from real MUSAN recordings, such as a coffee shop, a call center, and a second talker over the whole request, at 5 to 15 dB SNR. In our annotator study, listeners made out the full noisy instruction in 77 to 89 percent of ratings, against 95 to 99 percent for clean audio (Section 3.2). The revised Limitations (Section 5) state that the user turns in the multi-turn results are scripted, next to the read-aloud and offline-scoring limits. The free policy and the adaptive user (G3a, G3b) run without a final-turn flag. Wait and Recap do rely on an end-of-request cue, and the revised Section 4.3 and Conclusion now say so.

W2, qualifying broad claims. The collapse itself holds in all six models of Table R1. The revised paper scopes the explanations and fixes around it. The speech versus text gap is small for the Qwen models and Nemotron (Table 7, Appendix E), and the revised Section 1 now states it for these models only. Recap closes most of the EmbodiedBench gap for both Qwen models and MiniCPM-o (Table 6, Appendix E). On GUIBench, Recap transfers weakly (+4.7 AMS at 30B, Table 10). There, the intervention that transfers is withholding the intermediate replies, which lifts the score from 18.6 to 60.8 AMS. ScheduleBench does not collapse, and MiMo does not collapse there either (Tables 2 and 4, Appendix E). The revised Conclusion keeps the evidence and the fixes apart. It states the collapse and its cause. It scopes the fixes to EmbodiedBench and to a marked final turn. On novelty, the paper now presents the collapse as an extension of the multi-turn degradation known from text models. What it presents as new is the diagnosis of the cause in speech-driven omni agents. The table lists the wording changes in the revised abstract and Section 1.

Submitted textRevised text
"speech adds no penalty of its own" (Section 1) and "not to speech perception or ASR" (abstract)For the Qwen models and Nemotron, speech adds little penalty of its own (Section 1). The abstract traces most of the degradation to the plan-every-turn protocol.
"in the Wild" (title)Defined in the introduction as controlled variations of the spoken input, with real background noise and scripted turns.
"recover most of the loss without extra model calls" (abstract) and "a training-free Recap-and-Plan prompt recovers most of the lost performance" (Section 1)Both now say "on EmbodiedBench". Section 1 adds that, given an end-of-request cue, withholding the intermediate replies also recovers the gap on GUIBench.
"four native omni models from three labs" (abstract and Section 1)Six native omni models from five labs, with MiMo and Nemotron added (Table R1).
"This goes beyond the text-only degradation reported by Laban et al." (Section 1)"This extends the text-only degradation reported by Laban et al. and isolates its cause for speech-driven omni agents."

We hope the new controls answer your question. We invite you to check the revised PDF, where new and reworded text is marked in blue. The abstract, Section 4.3, the Limitations, and Appendix P answer your points. If any point remains unclear, please let us know.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer o925,

Thank you for the question about a natural interaction policy. The policy you suggested is now in the revised PDF (Section 4.3 and Appendix P). Even without a final-turn flag, the model commits to a plan on almost every turn, and most of the collapse remains.

AuthorsOpenReview ↗

Short reminder

Dear Reviewer o925,

The discussion closes soon, so we would like to point you once more to the revised PDF. It adds the free interaction policy you suggested (Section 4.3 and Appendix P) and narrows the claims you flagged in the abstract, Section 1, and the Conclusion. We would value your view on whether this answers your question

Reviewer_7bS4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OmniAgentBench, a benchmark for speech-driven multimodal planning across GUI navigation, scheduling, and embodied tasks, comprising 11,700 audio cases derived from 900 base instances. Its main finding is that generating intermediate plans during multi-turn interaction can substantially degrade performance, with matched text and silent-accumulation controls indicating that the response protocol, rather than speech perception alone, is a major contributing factor. Training-free interventions recover substantial performance in several settings, while additional experiments using recordings from 116 speakers support the relevance of these findings beyond synthetic speech.

Critical strengths
  • This paper addresses an important problem and fills a gap in existing evaluations of omni-modal agents.
  • The paper provides a comprehensive evaluation of synthetic and human-recorded speech across diverse scenarios, with analyses that reveal several limitations of current omni-modal agents.
Critical weaknesses
  • There may be a mismatch between the multi-turn interaction evaluation and real-world usage. In the paper’s setup, a predefined instruction is provided at each subsequent turn regardless of the model’s response, whereas human users typically adapt their instructions based on the model’s replies. As a result, the evaluation may overstate the model’s limitations in multi-turn interactions.
  • The paper primarily draws on existing benchmarks without introducing new evaluation scenarios or test cases, which somewhat limits its contribution.
  • The evaluation covers a relatively limited set of models and would benefit from including more models beyond the Qwen family.
Questions for authors

Please refer to Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7bS4 (part 1 of 2)

Thank you for the review, and for noting that the paper addresses an important problem and fills a gap in existing evaluations of omni-modal agents. We answer the three weaknesses in order.

W1, scripted turns and real users. A simulated user that reacts to each reply turn by turn leaves the collapse unchanged (G3b, with GPT-4o as the user on 100 EmbodiedBench items). In text and audio at two and three turns, the adaptive user scores within 1.2 Step-LCS points of the scripted user, and every 95 percent interval of the difference includes zero. Most intermediate replies gave the user nothing to correct. The earlier requirements went missing later, in the final plan. The score rises only when the user also checks the final plan and repeats what was dropped, for example from 30.9 to 45.0 in audio at two turns. The user had to do this for 46 to 85 percent of the plans, and even the corrected plans stay below Concat, the single-utterance baseline (60.2 in that setting). The agent also rarely asks for help. Even when it is allowed to ask, it asks a question on at most 3 percent of the turns before the request is complete (G3a). So the loss comes from the agent's final plan. A careful user can fix part of it, but only by remembering what the agent left out and saying it again. Section 4.3 and Appendix P of the revised PDF report both tests.

The scripted condition is a controlled design. It models a user with a fixed goal who adds details as they remember them, in informal speech with fillers such as "um" (Section 3.2 and Appendix B). It keeps the complete request and the visual context fixed and changes only how the turns are delivered and answered.

W2, new evaluation scenarios and test cases. The tasks, environments, and gold plans come from the sources on purpose, so every speech condition has a matched text baseline. What is new is the evaluation of how the spoken delivery of the same request changes the outcome. On the same 300 GUIBench steps, Qwen-Omni-3 scores 61.0 AMS when the request is one utterance, 18.6 when it answers after every turn, and 60.8 when it waits silently until the end, and text falls from 62.4 to 24.5 (Table 10). No score on the source benchmarks can show this. On top of the source tasks, OmniAgentBench offers three things.

  • To our knowledge, it is the first benchmark in which spoken instructions drive multimodal agent planning under controlled wild conditions. These are real noise at 5 to 15 dB SNR, including a second talker over the whole request, silence-padded long audio, informal disfluent speech, and requests split across turns.
  • It adds new evaluation scenarios. They are the paired speech cost and turn collapse on fixed tasks, controls that hold the audio fixed (Concat, Silent, Recap, Wait, End-marker), and an equivalence test of human against TTS speech on 913 recordings from 116 speakers.
  • It adds new test cases, with an informal rewrite and two-turn and three-turn sequences for all 900 instances, written as new spoken language and checked by annotators (G1).

These evaluations lead to new findings. On the full clean GUIBench set, Qwen-Omni-3 beats the best cascade (62.0 against 43.8 AMS, Table 2), but on the 300-step multi-turn set the cascade leads (48.0 against 18.8, Table 4). The newer, larger Qwen-Omni-3 also collapses more than Qwen-Omni-2.5 (24.1 against 8.8 at two turns, Table 6).

The paper also contributes a simple, practical method, Recap-and-Plan (Section 3.3 and Appendix D). It adds one instruction to the final turn, asking the model to restate every constraint before it plans. With no training, no change to the model, and no extra model call, it recovers essentially all of the lost EmbodiedBench performance for Qwen-Omni-2.5 (94 to 102 percent) and two thirds or more for Qwen-Omni-3 (67 to 71 percent, Table 6). It also carries over beyond Qwen and fully recovers the audio collapse of MiniCPM-o 2.6 from OpenBMB (Appendix E).

Prior work also builds benchmarks on existing data, such as OmnixR (ICLR 2025) on MMLU-Pro, VoiceBench (TACL 2026) on text benchmarks spoken with TTS, and Speech Robust Bench (ICLR 2025) on existing speech corpora. In the same way, the source tasks are only the base of OmniAgentBench. On that base, the spoken and wild conditions identify and evaluate a gap that current benchmarks miss. A model that looks strong on one clean utterance can fail once the same request is split across turns, and this datasets and benchmarks submission is built to evaluate exactly that.

Continued in part 2.

AuthorsOpenReview ↗

Response to Reviewer 7bS4 (part 2 of 2)

W3, models beyond Qwen. The paper already goes beyond Qwen in two ways, which we may not have made visible enough. First, it evaluates closed-source systems on clean speech and under all four perturbations (Tables 2 and 4). Gemini runs as a native omni model, and GPT-4o and GPT-4o-mini run as cascades behind GPT-4o-transcribe. On GUIBench, Gemini scores 38.8 AMS on clean speech and 13.0 under multi-turn delivery, and the GPT-4o cascade scores 20.9 and 7.6. On clean speech, the open Qwen-Omni-3 has the highest average score of all systems (42.2, against 29.3 for Gemini and 27.2 for the GPT-4o cascade, Table 2). Second, Appendix E runs the controlled contrast on two open non-Qwen models, MiniCPM-o 2.6 (OpenBMB) and Ming-Lite-Omni (inclusionAI). For the rebuttal we added two more non-Qwen families, MiMo-V2.5 from Xiaomi (310B parameters, 15B active) and Nemotron-3-Nano-Omni from NVIDIA (30B parameters, 3B active), described in G3c. The table gives the EmbodiedBench audio collapse for all six models (Concat minus Multi-Turn, Step-LCS, 300 items). Every 95 percent interval excludes zero (Appendix E).

ModelLabTwo turnsThree turns
Qwen-Omni-2.5Alibaba+8.8+12.3
Qwen-Omni-3Alibaba+24.1+32.9
MiniCPM-o 2.6OpenBMB+5.0+5.3
Ming-Lite-OmniinclusionAI+25.8+27.2
MiMo-V2.5 (new)Xiaomi+8.6+6.2
Nemotron-3-Nano-Omni (new)NVIDIA+14.8+20.3

The revised PDF adds the six-model table to Section 4.3 as Table R1 and reports MiMo and Nemotron in Appendix E, next to MiniCPM-o and Ming.

We hope these results address your three concerns. We invite you to check the revised PDF, where new and reworded text is marked in blue, in particular Section 4.3, Table R1, and Appendices E and P. If any concern remains, please let us know and we will address it during the discussion.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 7bS4,

Thank you for your review. Our response includes the experiments you requested. A simulated user who adapts turn by turn leaves the collapse unchanged (G3b), and the collapse now appears in six omni models from five labs (Table R1). Fixed user turns are also standard practice in multi-turn evaluation, as in MT-Bench, MultiChallenge, VoxDialogue (ICLR 2025), and Audio MultiChallenge (ACL 2026). None of them has an agent act on a spoken request in a visual scene, or plays the same spoken request both as one utterance and split across turns. Our benchmark adds both, together with real background noise, 913 human recordings, and training-free fixes. We would value your view on whether this addresses your concerns before the discussion closes.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (part 3 of 3)

G3. New experiments (continued)

(b) Adaptive user (7bS4). We let GPT-4o play the user. It knows only the turns it has to say, which are the scripted turns of the paper, and it never sees the gold plan. After each agent reply, it gives its next planned turn and points out anything the reply left out or got wrong. Once it has given every turn, it checks the plan the robot would carry out, and if something is missing it says so once, for example "it looks like you missed closing the laptop lid and picking it up". The agent is Qwen-Omni-3 with the paper's prompt and scorer, so it is still asked for a plan after every turn. In the audio runs, turn 1 is the released TTS clip, and a second TTS voice speaks the user's later turns.

Adaptive user, EmbodiedBench Step-LCS, first 100 items (99 at N3)Text N2Text N3Audio N2Audio N3
Concat57.657.260.259.5
Scripted Multi-Turn34.729.130.923.7
Adaptive user, reacting turn by turn34.728.732.124.9
Intermediate replies the user corrected3%13%4%19%
User-assisted, after one check of the final plan46.052.245.051.9
Final plans the user corrected46%78%49%85%

A user who reacts turn by turn gives the same scores as the scripted user. All four paired differences fall inside [-3.1, +3.7] (text -0.1 [-2.1, +2.0] and -0.5 [-3.1, +2.4], audio +1.2 [-1.1, +3.4] and +1.2 [-0.9, +3.7]). The user found nothing to correct in 81 to 97 percent of the intermediate replies. The loss appears in the final plan, which leaves out constraints from earlier turns.

We treat the final check as a separate, user-assisted condition, because it needs an extra round trip and a user who knows the whole request. In this check the user corrected 46 to 85 percent of the plans, and the score recovered 48 to 82 percent of the gap. This is close to what Recap recovers without any user (67 to 71 percent at 30B). Even so, the corrected plans score below Concat in every column, by +4.9 [+1.0, +8.9] to +15.2 [+11.4, +19.0].

(c) MiMo-V2.5 and Nemotron (7bS4). We added MiMo-V2.5 from Xiaomi, a mixture-of-experts omni model with 310B parameters, of which 15B are active per token. On EmbodiedBench (300 items), its audio collapse is +8.6 [+6.8, +10.6] at two turns and +6.2 [+4.0, +8.6] at three, from Concat scores of 42.9 and 41.7 to Multi-Turn scores of 34.3 and 35.5. On GUIBench, scored per episode over 177 episodes, it drops from 32.5 to 25.6 at two turns and from 35.3 to 29.2 at three. Like the paper's models, it does not collapse on ScheduleBench (FPS 24.5 and 26.3 for Concat, 27.3 and 27.8 for Multi-Turn).

We also added Nemotron-3-Nano-Omni from NVIDIA, a mixture-of-experts omni model with 30B parameters, of which 3B are active per token. On EmbodiedBench (300 items), its audio collapse is +14.8 [+12.7, +17.0] at two turns and +20.3 [+17.9, +22.8] at three, from Concat scores of 37.2 and 36.8 to Multi-Turn scores of 22.4 and 16.5, so it grows with the number of turns. Matched text collapses by about the same amount (+14.6 and +22.0). Audio and text differ by less than 2 points in every cell, so Nemotron shows no added audio penalty.

Changes to the paper

The revised PDF shows new and reworded text in blue. Table numbers from the submission are unchanged, and the one new main-text table is numbered R1.

  1. Section 4.3 summarizes the free policy and the adaptive user, and Appendix P gives the details.
  2. Section 4.3 summarizes the six-model replication in Table R1. Appendix E reports MiMo and Nemotron next to MiniCPM-o and Ming, and Appendix A lists their checkpoints.
  3. Appendix Q, Examples from Each Benchmark, adds one worked example per benchmark, and Section 3.1 defines C and G at first use.
  4. The abstract, Section 1, Section 4.3, and Section 5 scope the speech and Recap claims. The abstract, Section 1, and Section 4.3 also give the six-model count. The Limitations now add the two less scripted settings of Appendix P.
  5. Section 1 defines "in the wild" as controlled variations of the spoken input, as in our answer to o925 (W1).
AuthorsOpenReview ↗

General response (part 2 of 3)

G1. Novelty and contributions (continued)

Two training-free interventions. Withholding the agent's intermediate replies lifts Qwen-Omni-3 on GUIBench from 18.6 to 60.8 AMS (Table 10). Recap-and-Plan (Section 3.3) adds one instruction to the final turn, asking the model to restate every constraint before it plans, and it needs no training and no extra model call. It recovers 67 to 71 percent of the EmbodiedBench gap for Qwen-Omni-3 and 94 to 102 percent for Qwen-Omni-2.5 (Table 6).

G2. Definition of "in the wild"

In the title, "wild" refers to controlled variations of the spoken input, namely real background noise, silence padding, informal and disfluent phrasing, and delivery split across turns, all over fixed tasks. The clean condition is a single fluent utterance. The revised introduction states this definition, next to the limits that the Limitations already state (scripted turns, read-aloud human speech, offline embodied scoring).

G3. New experiments

(a) Free interaction policy (o925). We appended the following note to the paper's task prompt. "Interaction note. The user may describe this task over one or more turns, and you are not told how many turns there will be. You do not have to output a plan on every turn. After a turn you may instead acknowledge what you heard, ask one brief clarifying question, or wait for more information. Only when you judge that you have every requirement should you output the complete plan, in the JSON format described above, that satisfies all of them. Use only what the user has said and the image."

No turn is marked as the last, and we score the reply to the final user turn. To see what the model does before the final turn, we labeled every earlier reply automatically. A reply counts as committing if it contains an action from the action set, and as asking if it contains no action but does contain a question.

Qwen-Omni-3, EmbodiedBench audio, 300 itemsN2N3
Replies before the final turn that commit to a plan98%97%
Replies before the final turn that ask a question2%3%
Concat (paper)56.255.0
Multi-Turn (paper)32.122.1
Multi-Turn (rerun on our inference server, same session as Free)30.821.8
Free34.628.1
Free minus Multi-Turn, paired within the session+3.8 [+1.2, +6.2]+6.3 [+3.3, +9.2]

Even with permission to wait, the agent commits to a plan on 97 to 98 percent of incomplete turns, and most of the gap remains (15 and 19 percent recovered). The paper's Wait prompt uses the same task prompt and output format, but it tells the agent to defer and marks the last turn with "That is everything". Under Wait, the agent answers only "ok" before the last turn on all 300 items and recovers 75 to 81 percent of the gap (Table 8). The agent therefore defers when instructed, but it rarely chooses to defer on its own.

Continued in part 3.

AuthorsOpenReview ↗

General response to all reviewers (part 1 of 3)

We thank the reviewers for their detailed reviews. Reviewers found the problem "important and timely" (Atf9), said the paper "fills a gap in existing evaluations of omni-modal agents" (7bS4), and noted that the controlled diagnosis "goes beyond reporting benchmark scores" (o925). During the rebuttal period we ran three new experiments. First, a simulated user who adapts to the agent turn by turn does not reduce the multi-turn collapse. Second, the paper's main model keeps planning on almost every turn even when told it may wait. Third, two more model families from outside Qwen also collapse, MiMo-V2.5 from Xiaomi and Nemotron-3-Nano-Omni from NVIDIA.

Sections G1 to G3 answer points raised by more than one reviewer, and each reviewer section answers the rest. Concat plays all turns of a request as one utterance, Multi-Turn delivers them one at a time with a reply after each, and N2 and N3 mean two and three turns. All intervals are paired bootstrap 95 percent intervals (B = 2000). Cells marked "paper" are copied from the submission.

G1. Novelty and contributions

This submission is in the datasets and benchmarks track. It contributes a benchmark, the failure that the benchmark exposes in current omni agents, a controlled diagnosis of that failure, and two training-free interventions (Section 1).

The gap in current benchmarks. Agent benchmarks assume typed instructions, and speech benchmarks test speech understanding without an agent task (Section 1). None of them delivers a spoken request across turns, and this hides a large failure. On a single clean utterance, native omni models match ASR+LLM cascades (Table 2), and real background noise moves their average by less than one point on each benchmark (Tables 2 and 4). When the same request is split into two or three turns instead of played as one utterance, Qwen-Omni-3 loses 24.1 and 32.9 Step-LCS points on EmbodiedBench (Table 6). Turn-split delivery also reverses the ranking. On the full clean GUIBench set, Qwen-Omni-3 beats the best cascade (62.0 against 43.8 AMS, Table 2), but on the 300-step multi-turn set that cascade, GPT-4o-transcribe with Qwen3-VL, leads (48.0 against 18.8, Table 4).

New test cases. GUIOdyssey, MPCC, and EmbodiedBench contain no speech. OmniAgentBench builds 11,700 audio cases from their 900 instances (Table 1). Each instance gets a new informal rewrite and new two-turn and three-turn sequences, so every benchmark has 600 multi-turn cases written as new spoken language. In these sequences, an EmbodiedBench request grows from 9 words on average to 113 to 127 words, and the vocabulary grows from 287 to over 1,100 words (Table 27). In our annotator study, the rewrites kept the constraints (soft recall 0.965 to 0.996), and listeners could make out the full clean multi-turn instruction in 95 to 99 percent of ratings. One audio set drives every delivery condition and control (Concat, Multi-Turn, Silent, Recap, Wait, End-marker), so each contrast holds the audio fixed.

Human speech. We collected 913 recordings from 116 speakers who live in 25 countries and have 26 first languages. The benchmark is released under CC BY 4.0 with the human-speech stimuli, the recording tool, and the analysis code. On clean speech, human and TTS scores are equivalent on average within a pre-specified 5-point margin (Human minus TTS is -0.7 [-2.6, +1.1] over 19 model and benchmark cells, two one-sided tests).

New findings (Section 4.3). Multi-turn degradation is known for text LLMs (Laban et al.). Each of the following results is new for speech-driven omni agents.

  1. The collapse comes from answering after every turn. Matched text collapses about as much as audio (Text minus Audio is +1.1 and +3.6 at 30B, Table 7), so for the Qwen models speech perception is not the main cause. When Qwen-Omni-3 only lists the constraints after each turn, its list covers 96.9 and 92.8 percent of the gold sub-goals (Table 6), so the model can still state almost all of them. When Whisper transcribes the two-turn audio and the same model then plans once from the transcript, it rises from 32.1 to 53.7 Step-LCS on EmbodiedBench and from 18.6 to 62.6 AMS on GUIBench (Table 11).
  2. A newer, larger model collapses more. Qwen-Omni-3 scores 26.0 GUI AMS points above Qwen-Omni-2.5 (Table 3), yet its EmbodiedBench collapse is larger, 24.1 against 8.8 at two turns and 32.9 against 12.3 at three (Table 6).
  3. The collapse holds across voices, speakers, and models. It is significant in all 28 model-by-voice cells of the accent study (Table 9), it reproduces on the human recordings, and it appears in six native omni models from five labs (G3c).
  4. Letting the model wait, or letting the user adapt turn by turn, leaves most of the collapse in place (G3a, G3b).

Continued in part 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.