- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces Dy-VoiceBench, a benchmark for evaluating spoken dialogue agents across three dimensions: world-state dynamics, user-state dynamics, and knowledge-state dynamics. It combines executable environments, simulated users, and state-based evaluation to assess interactive task completion.
Critical strengths
The benchmark covers diverse scenarios and provides fine-grained evaluation metrics beyond conventional response-level accuracy. The inclusion of human validation is also a positive aspect.
Critical weaknesses
-
Lack of motivation: I found this manuscript's arguments are exaggerated, not sufficiently justifying their unique position, falling short of ICLR bar.
- [Intro] Paragraph 2 is not true. Recent interactive agent evaluation includes turn-wise simulation, thus user turns are not fixed (ref: https://arxiv.org/abs/2603.13686, https://arxiv.org/abs/2506.07982)
- [Intro] User-state changing in paragraph 4 is already existing. Previous literatures have explored changing user intents or states (ref: https://ojs.aaai.org/index.php/AAAI/article/view/37189, https://arxiv.org/abs/2306.01579).
- [Intro] Knowledge evolving assumption is already existing (ref: https://arxiv.org/abs/2410.10813).
- I only referred a few works, but there are several others. Although the authors mentioned a few in related works section, but this manuscript's unique position is not clearly justified (e.g., it is hard to understand what line 140 means). It would be incremental if they are simply combined.
- I suggest preparing a supportive evidence showing that existing works cannot solve, with a significant revision of writing.
-
For overall writing, I recommend not using "too fancy" terms. For example, in user simulator:
- While the simulator is just prompt-based simulator, using the term "hidden/latent state" may confuse readers.
- Based on figure 9, only a few traits in line 170-171 are addressed.
- Critically, it is unclear whether simply adding a few state variables and scoring instructions to an existing prompt-based LLM user simulator constitutes a sufficiently novel contribution to user-state modeling.
Questions for authors
N/A