- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors describe a benchmark to evaluate voice agents on how they would execute tasks within two environments: an online store and an IDE. The evaluations show that current models are not there yet, while on average audio-native agents work better than cascaded models, cascaded models perform better on task completion than native models.
Critical strengths
- I think the idea is nice, and I see the use of voice agents for realistic envirnoments, definitely that of coding in IDE is something that is done more and more with voice agents.
Critical weaknesses
-
For all the manual reviews the authors have done in the paper, its good to provide some more context, statistics, inter-agreement scores between annotators, how many samples were rejected overall after manual review, etc. How were these reviews done, and what was the outcome of these manual reviews. For example for: > (Line 293 & 294) [...] then manually refine the task specification, initial state, and expected outcome with domain-expert checks. > (Line 779 & 780) All generated inputs were manually reviewed for semantic. appropriateness, intelligibility, and qualitative plausibility. This manual review provides quality control for semantic fidelity, intelligibility, and qualitative plausibility of the generated waveforms. > (Line 1157 & 1158) The generated inputs were manually reviewed for semantic appropriateness, intelligibility, and qualitative plausibility.
-
I think the paper would benefit from a bit more clearer writing. Some paragraphs are a bit too dense and telegraphic. For example, too many findings. Please structure the paper around one core idea with a takeaway key message, seven separate findings in the main text could also be condensed a bit more, some findings don't need their own section but can just be read from Table 1 with one extra sentence outlining that result. Also, because of the writing, I miss a bit the "why", why it is useful or necessary to have benchmarks like this, what would it solve. For the related research, I would also like a schematic overview table on how this benchmark is different than earlier works, for example what is different than VoiceBench or Real World Voice EQ Bench or t-Voice. Similarly, I believe the related work that is pasted after the references should be part of the main content, as well as the conclusion that is there after the references. I think the paper would benefit from a more traditional approach in sections, i.e. introduction, related work, method, results, discussion/conclusion.
Questions for authors
- Line 315: Each task is paired with an automatic judge that checks whether the agent achieves the intended outcome in the environment What judge? What model? What prompt do the authors use to judge? How do the authors make sure there is no model bias?
- I have a hard time understanding the use of VoiceGenAgent. The way I understand it the authors use VoiceGenAgent to synthesize audio, that is then used in the benchmark to evaluate other models on that are capable of synthetisizing audio. I think this results in a circular dependency, because for example one subsequently evaluates Google models on this generated benchmark based on Google models?
- Also curious how these come into play: XTTS-v2 (Casanova et al., 2024) and CosyVoice (Du et al., 2024) for multilingual TTS and voice-cloning support. Are these mainly tools? Or how is the benchmark then used for multilingual and voice-cloning? The models evaluated on this benchmark dont necessarily support voice-cloning do they?
- Line 417: Finding 2 (Cascaded voice agents are stronger at grounded task execution in coding.) Why is this the case?