acceptodds
Under review as a conference paper at ICLR 2027

Dy-VoiceBench: Stateful Worlds and Living Users for Evaluating Dynamic Spoken Dialogue Agents

Abstract

Conversational agents are increasingly expected to complete tasks through interaction rather than generate isolated responses. However, many existing evaluations still treat dialogue as static prediction, where future turns, tool contexts and user behaviors are fixed and the model is not fully accountable for the consequences of its own trajectory. This is insufficient for spoken task completion, where model utterances and actions may change the external world, the user's ability to cooperate, and the episode-local knowledge available in later turns. We introduce Dy-VoiceBench, a benchmark for evaluating dynamic spoken dialogue agents through executable closed-loop episodes. Each episode specifies hidden states, visible observations, model-side and user-side actions, environment feedback, transition rules, process rubrics, and terminal success criteria. Rather than matching a reference transcript, Dy-VoiceBench scores the state trajectory induced by the evaluated model. It covers three complementary dynamics: world-state dynamics for executable task completion and user-action coordination, user-state dynamics for task-consequential spoken empathy and cooperation maintenance, and knowledge-state dynamics for acquiring, preserving, revising, and applying episode-local knowledge. Dy-VoiceBench contains 614 scenarios and 1,632 executable world-state action instances, providing a diagnosable testbed for state-centric spoken dialogue evaluation. Demo is available at https://dyvoicebench.github.io/.

Primary area:
datasets and benchmarks
Keywords:
Spoken Dialogue, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_RctyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Dy-VoiceBench, a benchmark for evaluating spoken dialogue agents across three dimensions: world-state dynamics, user-state dynamics, and knowledge-state dynamics. It combines executable environments, simulated users, and state-based evaluation to assess interactive task completion.

Critical strengths

The benchmark covers diverse scenarios and provides fine-grained evaluation metrics beyond conventional response-level accuracy. The inclusion of human validation is also a positive aspect.

Critical weaknesses
  • Lack of motivation: I found this manuscript's arguments are exaggerated, not sufficiently justifying their unique position, falling short of ICLR bar.

    • [Intro] Paragraph 2 is not true. Recent interactive agent evaluation includes turn-wise simulation, thus user turns are not fixed (ref: https://arxiv.org/abs/2603.13686, https://arxiv.org/abs/2506.07982)
    • [Intro] User-state changing in paragraph 4 is already existing. Previous literatures have explored changing user intents or states (ref: https://ojs.aaai.org/index.php/AAAI/article/view/37189, https://arxiv.org/abs/2306.01579).
    • [Intro] Knowledge evolving assumption is already existing (ref: https://arxiv.org/abs/2410.10813).
    • I only referred a few works, but there are several others. Although the authors mentioned a few in related works section, but this manuscript's unique position is not clearly justified (e.g., it is hard to understand what line 140 means). It would be incremental if they are simply combined.
    • I suggest preparing a supportive evidence showing that existing works cannot solve, with a significant revision of writing.
  • For overall writing, I recommend not using "too fancy" terms. For example, in user simulator:

    • While the simulator is just prompt-based simulator, using the term "hidden/latent state" may confuse readers.
    • Based on figure 9, only a few traits in line 170-171 are addressed.
    • Critically, it is unclear whether simply adding a few state variables and scoring instructions to an existing prompt-based LLM user simulator constitutes a sufficiently novel contribution to user-state modeling.
Questions for authors

N/A

Reviewer_CZcqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Dy-VoiceBench evaluates spoken dialogue agents through world-state completion, simulated user-state changes, and episode-local knowledge use. It contains 614 scenarios, including 341 executable world-state tasks. The world-state track separates agent-side tools from user-side actions and checks terminal environment states. A Gemini-based user simulator generates behavior-dependent interactions and user-state scores. The paper evaluates eight systems and reports simulator audits, human scoring comparisons, and a small matched human-interaction study.

Critical strengths

The executable world-state track addresses a meaningful gap in spoken-agent evaluation. Separating backend actions from local user actions makes instruction quality consequential, and terminal state assertions allow different valid solution paths rather than enforcing transcript matching.

The paper makes a substantial effort to audit its evaluation. It reports 1,004 audited D1 turns, human comparisons for D2 and D3 scoring, and 20 matched D2 episodes with human self-reports. The different model profiles across task completion, interaction, and knowledge use are also informative.

Critical weaknesses
  1. Simulator errors are frequent enough that their effect on the benchmark conclusions needs to be established. Table 2 reports incorrect behavior on 21.00% of D1-COOP turns and 9.56% of D1-HARD turns. Appendix F.5 includes role confusion, unsupported state assumptions, and invalid tool use among these errors. Executable terminal assertions prevent a textual judge from declaring success incorrectly, but they do not prevent a faulty user trajectory from changing task difficulty or the final state. The paper does not show how correcting these errors affects model scores or rankings. I therefore cannot yet treat D1 as a sufficiently reliable measure of the evaluated agents.

  2. D2's validity remains difficult to separate from the preferences of its simulator. Gemini 2.5 Pro updates the user state, generates the next interaction, and supplies the scores, while Gemini-2.5-Pro+TTS is also an evaluated system. This does not establish bias, but makes a simulator-sensitivity check important. The reported scoring agreement and 20-episode human study are encouraging, yet do not establish ranking stability across independent simulators or human users. The human study also needs participant counts, episode assignment, agent conditions, and a clearer account of how matched trajectories were constructed. This limits the interpretation of D2 as measuring human user-state regulation.

Questions for authors
  1. On the audited D1 episodes, what fraction of terminal outcomes change after simulator errors are corrected? Do the main model differences survive this check?
  2. How stable are D1 and D2 rankings across repeated runs and an independently chosen user simulator or scorer? A targeted subset would help assess this concern.
  3. For the 20 matched human episodes, how many participants and agents were involved, what information did participants receive, and how were human and simulated trajectories aligned? Please also clarify whether D2 score annotators were blinded to the simulator's proposed post-turn state.
Reviewer_eWiMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Dy-VoiceBench, a benchmark of 614 scenarios for spoken dialogue agents. It evaluates eight systems on their ability to complete tasks with a simulated user, adapt as the user becomes confused or reluctant, and apply rules introduced during the conversation. The top performer on the knowledge tasks completed 46% of the tasks that required the agent and user to act together.

Critical strengths
  • Evaluating whether spoken assistants actually help users complete tasks, rather than just produce plausible replies, is an important problem.

  • The 185 user-only scenarios and final-state checks provide a useful test of successful spoken guidance. The distractor-tool experiment also reveals robustness differences that baseline scores miss.

Critical weaknesses
  • The benefit of explicit user-state tracking remains uncertain. Cognitive-load trajectories correlate only moderately with human ratings (ρ = 0.43), with direction agreement in 12 of 20 episodes. Without a turn-level baseline, it is unclear whether state tracking adds meaningful evaluation value.

  • The simulated user behaved incorrectly on 15.04% of 1,004 audited turns. These errors could cause capable assistants to fail, but the paper does not quantify their impact on task success or model rankings.

Questions for authors

Have you compared the stateful simulator with a version that scores each reply without carrying forward trust or cognitive load? On the 20 matched episodes, does state tracking improve agreement with human turn-by-turn ratings?

Among the 109 audited D1 episodes, how often did incorrect simulated-user behavior contribute to task failure? How would these errors affect the reported success rates or model rankings?

Reviewer_fyKrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes Dy-VoiceBench, a benchmark for spoken dialogue agents covering task completion, user-state adaptation, and knowledge reasoning. It evaluates eight spoken or speech-enabled models on this benchmark, revealing some gaps in these models. However, the proposed user simulation and the selected baselines/models raise clear concerns about evaluation validity, speech-specific contributions, and benchmark difficulty. This work thus needs revision and is not ready for publication in its current form.

Critical strengths
  • This paper is well motivated. Spoken dialogue systems do need better evaluation benchmarks and methods. This work's formulation of dynamic spoken dialogue evaluation around world, user, and knowledge states, with track-specific interaction protocols is interesting and could provide some insights to the community.
Critical weaknesses
  • Critical: User simulator failures make the validity of the evaluation questionable. Many tasks in this benchmark simulates the user with an LLM. The final outcome depends heavily on its behavior. However, this user simulator sometimes violates its defined role and acts as an assistant instead. This shows that the proposed user simulation method/prompt/implementation has critical issues and many agent traces will become meaningless. Examples:
    • Section 4.4.1 audits a subset of trajectories by human and finds that 21.00% of the sampled turns have incorrect user simulation (violating role, logic, or tool-use constraints). This is a pretty high amount.
    • I manually looked at the "Live Demo" provided in the website from the paper and found repeated user role violations in the task "Dynamic World I" (Home WiFi Outage Fix):
      • Qwen3-Omni: In Round 2, the user suddenly flipped its role and becomes an assistant. It diagnoses the problem, guides the user about next steps, and even asks a question "First, can you find your modem?" The subsequent agent response is affected by this bug and repeats this guidance. This unexpected role switching completely violates the goal of this benchmark, making the outcome of this trajectory meaningless.
      • Kimi-Audio: Rounds 2, 5, 7, 9, 10, and 11 all have user role violation, where the user simulator suddenly becomes an assistant. Such repeated role violation makes this task trace useless. At the end of the conversation, the Kimi model being evaluated throws errors like Error: cannot reshape tensor of 0 elements into shape [1, 0, 18, -1, 2] because the unspecified dimension size -1 can be any value and is ambiguous, but the user simulator continues without caring about such obvious errors. This means the benchmark evaluation code has obvious bugs and cannot even properly handle model inference runtime errors.
      • MiMo-Audio: Rounds 2, 3, 4, 5, 6, 7, and 11 have user role violations. Again, this trace becomes useless.
  • The speech-specific contribution is unclear in this benchmark. This benchmark consists of two types of tasks:
    • D1 and D3 do not require information beyond the spoken words. These tasks are essentially text tasks synthesized into speech. A strong ASR → LLM → TTS system is therefore an important missing baseline.
    • D2 is designed to require paralinguistic information of user speech like emotion, but this is not validated or quantified. A cascade baseline should be evaluated to establish whether and how much paralinguistic information contributes beyond textual emotional cues and response content. Without these controls, the benchmark’s additional demands on speech understanding remain insufficiently demonstrated and it degrades to a text benchmark with ASR/TTS integrated.
  • The evaluated models and the selected user simulator and judge LLMs are old and lack explanation.
    • The user simulator and the judge are Gemini 2.5 Pro. The Gemini model pages says that its release date is June 17, 2025 and its retirement date is October 20, 2026. This is too old for a benchmark proposed recently. The authors need to upgrade their user simulator implementation, fix bugs, and validates newer results before the benchmark is usable.
    • The evaluated models such as Gemini2.5+TTS, GPT-4o, and Qwen3-Omni are old. The authors should evaluate newer LLMs and voice agent systems like newer Qwen-Omni, GPT-Live, Gemini-Live, and other realtime models. Note that this is not trivial but critical for a good benchmark. If newer models already saturated this benchmark, this work will bring limited value to this community and be less attractive.
Questions for authors

Please see my comments in "critical weaknesses" and address concerns accordingly. I don't think there is any easy question that will affect the overall assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.