- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Authors present a benchmark for usage of tools in real-time voice agents, over a variety of tools in 7 different domains, such as travel, entertainment and life services.
Critical strengths
I like the idea in general, and think the benchmark is useful for many models. Extensive tool catalog, including many custom and interesting tools for useful real-world usage. The chain templates are interesting and reflect actual settings that these tools might be used. Extensive experiments done, all kinds of subsequent analyses performed in the appendix.
Critical weaknesses
- I’d say I’m missing focus on longer-term tasks. In general I see voice agents being used over longer horizons (multi-turn), like having longer conversations with an agent to book things, and have a conversation at the same time, resulting in many back-and-forths between the user and the agent. I think something like this is kinda capthred in a benchmark like this, but I miss this a bit in the main text. One thing could be showing/knowing that with longer conversations, the accuracy of the tool-calling model drops.
- I would like also an human baseline or comparison on the benchmark, knowing whether humans can pick up intent and know what actions to take (i.e. what tools to use) would be a good measurement for knowing how good these models currently perform. Since you have 44 tools in the benchmark, a baseline could be to pick when it has to pick a tool according to the benchmark, one of those 44 tools.
- The (anonymous) codebase mentioned in the appendix still refers on its webpage that it is for the NeurIPS submission, and not for ICLR. Similarly, it seems like models evaluated on the benchmark are not updated since the NeurIPS submission (Gemini, GPT and Qwen models all released at least 6 months ago, not using newer versions released in the meantime up to ICLR submission deadline).
Questions for authors
I would like to see also open source models on this benchmark here, or at least a mention if there arent any, evaluating only closed source frontier realtime tool calling methods is a nice comparison, but I would have liked to see how good open source is, if only with cascaded versions like an ASR to Language Model backbone to a TTS model, if that would perform worse on the benchmark?