- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces OmniAgentBench, a benchmark covering Schedule, GUI, and Embodied tasks. It adapts existing benchmarks by converting text instructions into speech using TTS and introducing variations such as background noise, informal phrasing, and multiple dialogue turns. It then evaluates whether models can follow the spoken instructions and complete the tasks under these conditions.
Critical strengths
-
As spoken interaction becomes more common, evaluating agents that follow spoken instructions is an important and timely research direction.
-
The authors include human validation to check constraint preservation and audio intelligibility during benchmark construction. The additional evaluation with real human recordings also strengthens the benchmark. Creating benchmarks with human in the loop is largely appreciated.
Critical weaknesses
-
Limited benchmark novelty. The benchmark largely reuses existing tasks, environments, and target plans, with instructions converted into speech through TTS. Although the added speech conditions are useful, I am not convinced that this adaptation provides a sufficiently distinct benchmark contribution.
-
[minor] Unclear presentation and task definitions. Figure 1 provides a general pipeline overview, but the paper lacks a complete example showing the actual instruction, visual context, expected output, and evaluation procedure. This makes it difficult to understand what the benchmark measures. Similarly, Section 3.1 introduces the constraint set C and goal G, but without further explanations. Examples from each benchmark would help substantially.
Questions for authors
None
5 comments
Response to Reviewer Atf9 (part 2 of 2)
W2, examples from each benchmark (continued).
EmbodiedBench. We picked the item by a fixed rule. Among the items whose constraints map one to one onto the gold steps and whose Concat plan is fully correct, it has the median Concat-to-Multi-Turn difference. M is one egocentric image of a living room with a blue armchair, a white sofa, and a floor lamp, and the white shelves are behind the agent. G is the end state in which the agent holds the box and the floor lamp is on. C has four requirements.
- Turn around and go to the white shelves on the left.
- Pick up the open box from the bottom shelf.
- Turn around, walk toward the blue chair, and turn right to face the white sofa.
- Turn on the light.
The gold plan is GotoLocation(shelf), PickupObject(box), GotoLocation(floorlamp), ToggleObject(floorlamp). Requirement 3 brings the agent to the floor lamp, which is why it appears as GotoLocation(floorlamp). Turn 1 says "turn around and head over to those, um, white shelves ... grab, like, the open box from, um, you know, the bottom shelf". Turn 2 says "turn around again and walk towards, uh, the blue chair ... then just, like, turn on the light". Concat plays both turns as one utterance. Step-LCS is the longest common subsequence of the predicted and gold steps, divided by the number of gold steps.
| Condition | Model output | Gold steps matched | Step-LCS |
|---|---|---|---|
| Concat | GotoLocation(shelf), PickupObject(box), GotoLocation(armchair), GotoLocation(floorlamp), ToggleObject(floorlamp), NoOp | 4 of 4 | 1.00 |
| Multi-Turn | GotoLocation(bluechair), GotoLocation(sofa), ToggleObject(floorlamp), NoOp | 1 of 4 | 0.25 |
The Multi-Turn plan keeps only the steps from turn 2, so it never goes to the shelves or picks up the box.
ScheduleBench. The spoken request asks for a flight from Rome to London on March 20, 2025, with a transfer in Paris. It asks to "minimize the total cost", with a total price of at most 600 US dollars and "at least a one-hour interval between each pair of flights". M consists of two timetable images, Rome to Paris and Paris to London. G is arriving in London that day at the lowest price, and C consists of the Paris transfer, the 600-dollar budget, and the one-hour buffer. The agent outputs one itinerary. It scores 0 if the itinerary is infeasible, 0.5 if it is feasible but not the cheapest, and 1 if it is the cheapest feasible itinerary. Here only one itinerary is feasible, Rome to Paris from 04:45 to 07:00 and Paris to London from 12:30 to 14:05, for 437 dollars.
GUIBench. Turn 1 says "check out that app, uh, 'Chatty - AI Assistant' ... ask it about ... the Fundamental theorem of algebra". Turn 2 says "double-check all that stuff by, uh, you know, like, searching in Firefox". We score the second step of the episode. There, M is the current screenshot, with Chatty open on an empty chat, plus the history of one earlier click. G is the question answered in Chatty and the answer checked in Firefox, and C consists of the requirements in the two turns, which GUIOdyssey aligns to interface steps. The agent outputs one next action. A click gets full credit within 5 percent of the screen diagonal, and its credit decays to zero at twice that distance. Typed text, scrolls, and completion follow fixed rules. The gold action is a click on the message box. With Concat, the agent clicks the message box and scores 1.0. With Multi-Turn, it outputs COMPLETE and scores 0.0, declaring the task finished before it has started. In the paper, Multi-Turn raises premature COMPLETE actions from 24 to 88 of 300 predictions.
We hope the examples and the summary of new data make the contribution clearer, and we would welcome your reassessment of it. We invite you to check the revised PDF, where new and reworded text is marked in blue. Section 3.1 and Appendix Q add the definitions and the examples. If any concern remains, please let us know.
Follow-up on our response
Dear Reviewer Atf9,
Thank you for your review. The revised PDF adds the worked examples and the definitions of C and G that you asked for (Appendix Q, Section 3.1). Could you let us know whether the revision addresses your concerns?
Response to Reviewer Atf9 (part 1 of 2)
Thank you for the review. We appreciate that you see spoken instructions for agents as an important and timely direction, and that you value the human validation and the real human recordings. We address your two concerns below. Every item, with its audio, context, and gold plan, is in the anonymous release linked in the reproducibility statement. The revised PDF also adds Appendix Q, Examples from Each Benchmark.
W1, benchmark novelty. To our knowledge, OmniAgentBench is the first benchmark in which spoken instructions drive multimodal agent planning under controlled wild conditions. Beyond the data, it contributes five new findings and a training-free method, set out below. The tasks, environments, and target plans are reused on purpose, so that only the spoken instruction changes. Each wild condition is then compared with clean speech and with matched text on exactly the same task. This contrast reveals a gap that the source benchmarks cannot show. On a single clean utterance, native omni models match ASR+LLM cascades (Table 2). Yet on EmbodiedBench, Qwen-Omni-3 falls from 56.2 to 32.1 Step-LCS when the same request is split into two turns (Table 6).
To address this gap, we propose Recap-and-Plan (Section 3.3). It adds one instruction to the final turn, asking the model to restate every constraint before it plans. With no training and no extra model call, it recovers essentially all of the lost EmbodiedBench performance for Qwen-Omni-2.5 (94 to 102 percent) and two thirds or more for Qwen-Omni-3 (67 to 71 percent, Table 6). It also carries over beyond Qwen and fully recovers the audio collapse of MiniCPM-o 2.6 from OpenBMB (Appendix E). A second training-free intervention, withholding the intermediate replies until the request is complete, lifts Qwen-Omni-3 on GUIBench from 18.6 to 60.8 AMS (Table 10).
The adaptation also goes well beyond converting the instructions with TTS.
These evaluations lead to five findings that are new for speech-driven omni agents (G1).
Prior work also builds benchmarks on existing data. OmnixR (ICLR 2025) builds on MMLU-Pro, VoiceBench (TACL 2026) on text benchmarks spoken with TTS, and Speech Robust Bench (ICLR 2025) on existing speech corpora. In OmniAgentBench the reuse is by design. Only the spoken input changes between the clean and the wild conditions, so the gap between them is measured on the same tasks. This datasets and benchmarks submission is built to measure that gap, and the paper's two training-free interventions address it.
W2, examples from each benchmark, and C and G. Each instance pairs a spoken request with context M, a goal G, and constraints C. G is the requested end state, and C is the set of requirements stated in the request. Below is one item per benchmark. The EmbodiedBench and GUIBench outputs come from our rerun of the paper's setting with Qwen-Omni-3.
Continued in part 2.