acceptodds
Withdrawn from ICLR 2027

Real-Time Audio Tool Bench: Evaluating Tool Use in Multi-Turn Voice Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Voice-native models now call tools directly from speech, yet how reliably they do so in live, multi-turn conversation is largely unmeasured. Real-time voice raises problems that text tool-calling benchmarks cannot expose: arguments must be heard correctly, details arrive over several turns, and users interrupt, sometimes while a call is still pending. We introduce Real-Time Audio Tool Bench, a benchmark of 1040 tasks across 7 domains and 44 tools, each run live over a provider’s streaming API. Three suites cover explicit multi-turn requests (Reactive), implicit intent (Proactive), and mid-conversation changes of mind (Interruption). Without an LLM judge, deterministic checks grade which tools are called, with what arguments, and when; each failure is attributed to perception (e.g., a misheard name or number) or to reasoning (e.g., a missing, premature, or unexpected call). Across six models from four providers, the best reaches an overall score of 38.7%, and each suite exposes a different bottleneck: text transcripts in place of audio raise Reactive pass rates by 30 to 65 percentage points; models act on nearly all Strong-intent Proactive tasks but complete at most 21% of them; and when a pending call’s result never returns, no model recovers on more than 10.3% of tasks. Static test-time harnesses raise the overall score by at most 5.7 points, and each lowers accuracy on at least one Proactive band or Interruption phase. We release the benchmark, the evaluation framework, and all evaluation traces.

Primary area:
datasets and benchmarks
Keywords:
benchmark, tool use, realtime voice agent, audio tool use, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8huwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Authors present a benchmark for usage of tools in real-time voice agents, over a variety of tools in 7 different domains, such as travel, entertainment and life services.

Critical strengths

I like the idea in general, and think the benchmark is useful for many models. Extensive tool catalog, including many custom and interesting tools for useful real-world usage. The chain templates are interesting and reflect actual settings that these tools might be used. Extensive experiments done, all kinds of subsequent analyses performed in the appendix.

Critical weaknesses
  • I’d say I’m missing focus on longer-term tasks. In general I see voice agents being used over longer horizons (multi-turn), like having longer conversations with an agent to book things, and have a conversation at the same time, resulting in many back-and-forths between the user and the agent. I think something like this is kinda capthred in a benchmark like this, but I miss this a bit in the main text. One thing could be showing/knowing that with longer conversations, the accuracy of the tool-calling model drops.
  • I would like also an human baseline or comparison on the benchmark, knowing whether humans can pick up intent and know what actions to take (i.e. what tools to use) would be a good measurement for knowing how good these models currently perform. Since you have 44 tools in the benchmark, a baseline could be to pick when it has to pick a tool according to the benchmark, one of those 44 tools.
  • The (anonymous) codebase mentioned in the appendix still refers on its webpage that it is for the NeurIPS submission, and not for ICLR. Similarly, it seems like models evaluated on the benchmark are not updated since the NeurIPS submission (Gemini, GPT and Qwen models all released at least 6 months ago, not using newer versions released in the meantime up to ICLR submission deadline).
Questions for authors

I would like to see also open source models on this benchmark here, or at least a mention if there arent any, evaluating only closed source frontier realtime tool calling methods is a nice comparison, but I would have liked to see how good open source is, if only with cascaded versions like an ASR to Language Model backbone to a TTS model, if that would perform worse on the benchmark?

Reviewer_dD66OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Real-Time Audio Tool Bench, a 1040-task benchmark for evaluating tool calling by voice-native models in live, multi-turn spoken interaction. Tasks are run over providers’ streaming APIs with TTS-rendered user turns and mock tools, and are grouped into three suites: Reactive (explicit requests with arguments spread across turns), Proactive (implied intent, including a Weak band where the correct action is to abstain), and Interruption (corrections and insertions, including while a tool call is still pending). Scoring is rule-based with no LLM judge, and failures are attributed to perception or reasoning. Across six closed-API models the best reaches 38.7%, with each suite exposing a different bottleneck: audio perception, execution after intent detection, and tracking of pending calls. A test-time harness study shows that static fixes trade off across suites.

Critical strengths
  • New evaluation setting. Interrupting while a tool call is pending (the “dangling call” design with server-side result withholding) is not covered by any existing benchmark and isolates a real capability—tracking pending calls—that models clearly lack.

  • Clear, actionable findings. The three suites map onto three distinct bottlenecks (perception, execution, state tracking), and the text-vs-audio ablation and harness study make the diagnosis concrete rather than speculative.

  • Deterministic, validated scoring. No LLM judge; tool set, arguments, and timing are checked by rules, and the failure attribution is validated against human labels (0.87 macro-F1).

Critical weaknesses
  • The reported results cannot currently be trusted as stated. The main text evaluates gemini-3.1-flash, qwen3.5-omni-plus, and grok-voice-think, but Table 17 lists gemini-2.5-flash-native-audio-latest, qwen3-omni-flash-realtime, and grok-3-fast. It is unclear which models were run. In addition, the pass rates in Table 1 do not correspond to the stated task counts (Reactive values imply 198 tasks, Weak implies 118), Tables 27 and 30 report conflicting numbers for the same quantity, Table 33’s baseline (32.5%/37.0%) does not match Table 1 (41.9%), an example trace is attributed to a model not in the paper (gpt-5.4), and Appendix D.4.3 contains a leftover “TODO(authors)”. Individually minor, together these make it impossible to verify the central claims from the submission.

  • The synthetic setup lacks validation against human speech, and the scripted user limits interaction. Synthetic dialogues, TTS audio, and mock tools are reasonable choices for controlled evaluation. However, the paper does not establish whether its findings hold with human-recorded speech. A matched comparison on a subset of tasks using TTS and human recordings would help assess whether model rankings and failure patterns—particularly the conclusion that perception is the bottleneck—are robust to the audio source. Separately, all reported results use fixed user turns that cannot adapt to the model’s questions. Although clarification is formally “not penalized,” reasonable requests for confirmation or repetition may go unanswered, potentially disadvantaging appropriate clarification strategies. This is especially relevant to Medium-intent tasks, where confirmation may be preferable to immediate execution. The available responsive mode is not evaluated, leaving uncertainty about whether the low Proactive scores reflect model limitations or the interaction protocol.

Questions for authors

See weakness.

Reviewer_9LuPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how reliably voice models can call tools in live multi-turn conversations. The authors introduce Real-Time Audio Tool Bench, a benchmark of 1040 tasks across 7 domains and 44 tools. The proposed benchmark particularly evaluates multi-turn requests (Reactive), implicit intent (Proactive), and mid-conversation changes of mind (Interruption). Across six models from four providers, the best model reaches an overall score of 38.7%.

Critical strengths
  1. Introduced a new benchmark for evaluating voice agents in multi-turn conversations.
  2. Conducted extensive experiments to demonstrate the capabilities of existing omni-modal LLMs.
Critical weaknesses
  1. The novelty seems very limited in comparison to recent agentic benchmarks for the audio modality: Duplex-Bench-v3, VoiceAgentBench, and Tau-Voice.
  2. The paper seems incomplete. For example, in Sec D.4.3, the following is written: TODO(authors):Eleven Labs model ID and voice settings; how speakers were selected and assigned to tasks. Does this indicate this paper is largely written by an LLM?
Questions for authors

Address the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.