acceptodds
Under review as a conference paper at ICLR 2027

VoiceWorld-Bench: Evaluating AI Voice Agents Across Listening, Acting, and Talking in Realistic Environments

Abstract

As large language models evolve into real-time speech-interactive systems, AI voice agents can listen, reason, use tools, and speak across consumer and enterprise workflows. Yet existing benchmarks do not evaluate them end to end: agent benchmarks focus on text- or GUI-driven tool use, while audio benchmarks primarily measure speech understanding or dialogue quality, without requiring grounded action or task completion. To close this gap, we introduce VOICEWORLD-BENCH, a benchmark for AI voice agents in two realistic task environments: a voice-commerce online storefront and a voice-driven coding IDE. Each environment exposes MCP-based tools over persistent backend state, enabling agents to act in the environment and allowing deterministic verifiable judges to measure task completion. VOICEWORLD-BENCH evaluates AI voice agents along four dimensions: task completion, speech robustness, real-time responsiveness, and safety & security. We evaluate three agent designs: audio-native agents, which use a single model for speech interaction, reasoning, and tool calling; transcription-cascaded agents, which combine a dedicated transcription module, a text LLM, and a text-to-speech (TTS) module; and audio-cue-aware agents, which use an audio-capable LLM for paralinguistics-aware speech understanding before text-based tool execution. Our evaluation highlights several emerging findings. First, current AI voice agents are far from saturating VOICEWORLD-BENCH and exhibit distinct limitations across evaluation perspectives. Second, cascaded designs often achieve stronger grounded execution than fully audio-native agents, suggesting that text-based reasoning and tool use remain advantageous when task success depends less on audio-specific cues. Third, audio-cue-aware understanding benefits semantic intent recovery, but can come at the cost of degrading high-precision symbolic recovery. Fourth, safety and capability remain decoupled: agents that complete benign tasks successfully do not necessarily behave safely under adversarial conditions. Together, VOICEWORLD-BENCH and these findings motivate holistic evaluation of AI voice agents, and point to future directions in both agent-harness design and backbone-model adaptation for speech-aware, tool-grounded execution.

Primary area:
datasets and benchmarks
Keywords:
AI Voice Agent, Speech, Audio Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RHiMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors describe a benchmark to evaluate voice agents on how they would execute tasks within two environments: an online store and an IDE. The evaluations show that current models are not there yet, while on average audio-native agents work better than cascaded models, cascaded models perform better on task completion than native models.

Critical strengths
  • I think the idea is nice, and I see the use of voice agents for realistic envirnoments, definitely that of coding in IDE is something that is done more and more with voice agents.
Critical weaknesses
  • For all the manual reviews the authors have done in the paper, its good to provide some more context, statistics, inter-agreement scores between annotators, how many samples were rejected overall after manual review, etc. How were these reviews done, and what was the outcome of these manual reviews. 
For example for:
> (Line 293 & 294) [...] then manually refine the task specification, initial state, and expected outcome with domain-expert checks. 
> (Line 779 & 780) All generated inputs were manually reviewed for semantic. appropriateness, intelligibility, and qualitative plausibility. This manual review provides quality control for semantic fidelity, intelligibility, and qualitative plausibility of the generated waveforms.
> (Line 1157 & 1158) The generated inputs were manually reviewed for semantic appropriateness, intelligibility, and qualitative plausibility.

  • I think the paper would benefit from a bit more clearer writing. Some paragraphs are a bit too dense and telegraphic. For example, too many findings. Please structure the paper around one core idea with a takeaway key message, seven separate findings in the main text could also be condensed a bit more, some findings don't need their own section but can just be read from Table 1 with one extra sentence outlining that result. Also, because of the writing, I miss a bit the "why", why it is useful or necessary to have benchmarks like this, what would it solve. For the related research, I would also like a schematic overview table on how this benchmark is different than earlier works, for example what is different than VoiceBench or Real World Voice EQ Bench or t-Voice. Similarly, I believe the related work that is pasted after the references should be part of the main content, as well as the conclusion that is there after the references. I think the paper would benefit from a more traditional approach in sections, i.e. introduction, related work, method, results, discussion/conclusion.

Questions for authors
  • Line 315: Each task is paired with an automatic judge that checks whether the agent achieves the intended outcome in the environment 
What judge? What model? What prompt do the authors use to judge? How do the authors make sure there is no model bias?
  • I have a hard time understanding the use of VoiceGenAgent. The way I understand it the authors use VoiceGenAgent to synthesize audio, that is then used in the benchmark to evaluate other models on that are capable of synthetisizing audio. I think this results in a circular dependency, because for example one subsequently evaluates Google models on this generated benchmark based on Google models?
  • Also curious how these come into play: 
XTTS-v2 (Casanova et al., 2024) and CosyVoice (Du et al., 2024) for multilingual TTS and voice-cloning support. 
Are these mainly tools? Or how is the benchmark then used for multilingual and voice-cloning? The models evaluated on this benchmark dont necessarily support voice-cloning do they?
  • Line 417: Finding 2 (Cascaded voice agents are stronger at grounded task execution in coding.) 
Why is this the case?
Reviewer_JJsEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VOICEWORLD-BENCH, a benchmark for evaluating AI voice agents in two task environments: a voice-commerce online storefront and a voice-driven coding IDE. The authors evaluated 3 agent designs and observed critical limitations. For instance, the best system could only achieve 69.5% task success and 56.9% speech robustness. In short, the overall findings would motivate a holistic evaluation of AI voice agents.

Critical strengths
  1. Conducted extensive experiments on several LLMs across 3 agent designs and 4 evaluation dimensions. These experiments revealed critical limitations of LLM agents.
Critical weaknesses
  1. The novelty seems very limited in comparison to recent agentic benchmarks for the audio modality: Duplex-Bench-v3, VoiceAgentBench, and Tau-Voice.
  2. Native audio vs cascade performance for tool calling is already investigated in recent research but the authors did not demonstrate their novelty in comparison to the prior work. For example: https://arxiv.org/abs/2605.15104
  3. Meanwhile, as an evaluation paper, the paper is quite narrow in scope. Only two environments are evaluated.
Questions for authors

Address the weaknesses.

Reviewer_tW72OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces VoiceWorld-Bench, a benchmark for evaluating voice agents in two executable environments: an online storefront and a coding workspace. Both expose tools over persistent state, allowing evaluation of whether spoken requests produce the intended actions and outcomes. The benchmark contains 1,152 tasks covering task completion, speech robustness, real-time interaction, and safety/security. Its VoiceGenAgent pipeline synthesizes controlled speech conditions, including hesitation, emphasis, self-correction, background speech, and adversarial injections.

The evaluation compares 17 configurations spanning audio-native agents, conventional ASR–LLM–TTS cascades, and cascades that preserve audio cues through annotated transcripts. Outcomes are assessed using database/workspace assertions, tool traces, output checks, and limited task-specific LLM judgments.

Critical strengths

None

Critical weaknesses

On your novelty claim, there are already quite a few real-time audio agent benchmarks (for example, fullduplexbench, τ-Voice, just to name a few), and your related work compares none of them. These papers have been out for months. I'm not trying to be mean, but in my opinion, the direction that you are going for is already done/covered in those papers, and I don't see anything new here.

Where are your dataset queries and codebase?

There is no related work in the main body, only in the appendix. The paper also has no conclusion.

The queries are synthetically generated via TTS.

The task setup doesn't make sense. The commercial platform is reasonable, but regarding the coding agent scenario, are you measuring the agent's coding capability or its tool invocation capability? Also, a real-time agent doesn't usually seem to be related to a coding IDE scenario. I mean, VS Code doesn't even have a way for you to send real-time audio files.

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.