- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ConvoAudio, a benchmark for conversational audio understanding to fill the gap between existing broad audio-understanding benchmark which rely on aggregate scores from diverse real-world recordings,
The benchmark evaluates the model capability in 3 dimensions: dialogue content, interaction strucutre and temporal organization. It also generates 6 variants from the same dialogue source and contributes a dataset for SFT and evaluation,
The paper evaluated transcirpiton-based cascaded model and direct-audio model and find that both systems achieve similar performance on dialogue content. Direct-audio model has better capability in understanding interaction structure and acoustic-temporal organization,
The entire benchmark is synthetic generation and may not generalize to real-world scenarios.
Critical strengths
The key strength is on the design of the granular, controlled benchmark and the benchmark dataset provided. It has strong experimental coverage and provides detailed studies between ASR+LLM and audio-direct model on the 3 proposed evaluation dimensions.
The paper also demonstrates that fine-tuning on their dataset leads to significant performance gains and the benchmark is not just for evaluation but also a valuable resource for model improvement.
Critical weaknesses
The entire benchmark is on synthetically generated with limited evidence for real-world generation. Systems that perform well on this benchmark may not necessarily generalize to real-world scenarios. A key scenario in real-world conversation is overlapped speech, even just for 2 speakers. I would be interested to see how this affects the dialogue content understanding between the two configurations (ASR+LLM, direct-audio).
The strong gain in SFT may suggest highly regularised task or other highly learnable patterns. It would be good for the author to provide some analyses on the gain.
Questions for authors
(1) what is v in equation 1. (2) the most significant gaps between direct-audio and ASR+LLM is in the acoustic-temporal dimensions. Please provide a more granular breakdown of this result. Is the gain due to the localization of non-lexical vocal events and how do it affect the real-world application scenarios? Or the gain is due to better identification of turn boundaries?
3 comments
The Response to Reviewer WBeW (2/2)
Third, we conduct the question-only control to distinguish adaptation to the task interface from improved use of conversational evidence: (1) in the question-only condition, the model receives only the question, answer options, and response contract, without access to the audio; (2) in the complete-input condition, the same question, answer options, and response contract are paired with the corresponding audio. As shown in the following table, although SFT improves all three models under the question-only condition, the Base-to-SFT gains are consistently larger under the complete-input condition. This pattern shows that SFT does more than teach task-interface regularities: it indeed substantially improves the models’ ability to extract and use evidence from the input audio.
| Model | Question-only Overall: Base → SFT | Complete-input Overall: Base → SFT |
|---|---|---|
| Qwen2.5-Omni-3B | 23.75 → 41.47 (+17.72) | 23.79 → 72.13 (+48.34) |
| Qwen2.5-Omni-7B | 24.53 → 38.65 (+14.12) | 31.13 → 76.85 (+45.72) |
| Qwen3-Omni-30B-A3B-Instruct | 25.60 → 38.37 (+12.77) | 42.85 → 78.89 (+36.04) |
Taken together, the task-level and response-format analyses show that the gains are neither confined to a small set of tasks nor explained by better output compliance alone. More importantly, the question-only control shows that fine-tuning substantially increases the contribution of the audio across the benchmark. The strong SFT gains therefore reflect both adaptation to the task interface and the genuine gain in evidence-conditioned conversational audio understanding, rather than merely fitting the highly regularized task structure or superficial learnable patterns.
Q1.What is v in equation 1
A: Thank you for pointing out this missing definition. In Equation (1), denotes the role-to-voice assignment for dialogue , i.e., the mapping from conversational speakers or roles to their assigned voice profiles. We have explicitly defined after Equation (1) in the revised manuscript.
Q2.The most significant gaps between direct-audio and ASR+LLM is in the acoustic-temporal dimensions. Please provide a more granular breakdown of this result. Is the gain due to the localization of non-lexical vocal events and how do it affect the real-world application scenarios? Or the gain is due to better identification of turn boundaries?
A: As defined in Appendix A.4 and Table 6, the acoustic–temporal dimension can be further decomposed into three tasks: A-1: Voice attributes and nonlexical vocal events, A-2: Global audible structure of the exchange, and A-3: Condition-specific event timing and windows. Complete task-level results for these tasks across all evaluated configurations are reported in Appendix C.1 and Table 12.
For ease of presentation, we show one representative matched-answerer comparison in the following table. Here, we compare two configurations: (1) in the direct-audio configuration, Qwen3.5-Omni-Plus directly receives the audio input; (2) in the ASR+LLM configuration, the transcript produced by Qwen3-ASR-Flash is provided to the same Qwen3.5-Omni-Plus answerer. This matched comparison holds both the evaluation items and the downstream answerer fixed, isolating the difference between the two input pathways.
| Task | Direct-audio | ASR+LLM | Direct-audio gain | Contribution to the A-dimension gap |
|---|---|---|---|---|
| A-1 | 91.92 | 43.00 | +48.92 | +17.18 (56.66%) |
| A-2 | 41.25 | 35.67 | +5.58 | +1.96 (6.47%) |
| A-3 | 38.52 | 0.97 | +37.55 | +11.18 (36.87%) |
| A Overall | 58.23 | 27.91 | +30.32 | +30.32 (100%) |
This breakdown shows that the gap between direct-audio and ASR+LLM is driven primarily by A-1 and A-3. A-1 jointly evaluates voice attributes and nonlexical vocal-event states. Our field-level analysis shows that the A-1 gap mainly reflects voice-attribute recognition, for which direct-audio achieves 99.67%, compared with 46.33% for ASR+LLM. A-3 evaluates temporal localization at the event- or utterance-window level. On its numeric-window items, the rate at which both predicted window boundaries fall within ±2 seconds of the reference is 27.45% for direct-audio and 0.18% for ASR+LLM.
In summary, these finer-grained results show that the direct-audio advantage is mainly associated with stronger acoustic-cue recognition and temporal-window localization, rather than being primarily driven by turn-boundary identification. In real-world applications, these capabilities can support acoustic-cue review in meeting and customer-service recordings, as well as the retrieval of intervals containing side exchanges, handoffs, temporary takeovers, and returns to the main conversation.
[1] MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. ICLR 2025
[2] MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix. NeurIPS 2025
[3] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark. ICLR 2026
Follow-up on Our Rebuttal
Dear Reviewer WBeW,
We sincerely appreciate your thoughtful feedback and constructive comments, which have helped us improve our work.
Over the past few days, we have devoted considerable effort and resources to addressing each of your concerns, including generalization beyond synthetically generated conversations, the strong SFT gains, acoustic–temporal performance, and the symbol definition of Equation (1).
As the discussion period ends in less than two days, we would be grateful if you could review our rebuttal and let us know whether it adequately addresses your concerns.
Best regards,
The Authors
The Response to Reviewer WBeW (1/2)
Thanks for your appreciation of our work. Below, we resolve your concerns one by one.
A: First, we would like to emphasize that ConvoAudit is designed as a controlled diagnostic benchmark. Its synthesis-based construction allows us to preserve the underlying conversational content while systematically varying conversational organization, thereby isolating whether the model can identify and follow the relevant evidence pathway under controlled conditions.
Furthermore, to directly address the reviewer’s concern, we specifically select the speech subset of MMAU, the dialogue-reasoning subset of MMSU, and the speech subset of MMAR for the matched evaluation. Drawn from independent public audio benchmarks, these subsets directly correspond to the real-world scenarios and dialogue-content understanding settings highlighted by the reviewer.
Using exactly the same evaluation items, we compare the two configurations: (1) in the direct-audio configuration, Qwen3.5-Omni-Plus directly receives the audio input; (2) in the ASR+LLM configuration, the transcript produced by Qwen3-ASR-Flash is provided to the same Qwen3.5-Omni-Plus answerer. As shown in the following table, direct-audio outperforms ASR+LLM across all three datasets. The advantage is especially pronounced on MMAR Speech and MMSU Dialogue Reasoning, where direct-audio improves accuracy by 14.58 and 10.72 points, respectively.
Taken together, these results show that the advantage of direct-audio understanding extends beyond ConvoAudit’s controlled synthesis conditions to real-world scenarios and dialogue-content understanding settings. This finding also reinforces the paper’s central conclusion that preserving the original audio input enables more effective conversational understanding than relying on ASR transcripts, and demonstrates that this advantage is not confined to ConvoAudit’s controlled synthesis conditions.
A: Thanks for your valuable suggestion. Here, we analyze the gains from three complementary perspectives: task-level performance, response-format validity, and the question-only control.
First, as described in Appendix A.4 and Table 6, ConvoAudit decomposes conversational understanding into 14 tasks: seven dialogue-content tasks, four interaction-structure tasks, and three acoustic–temporal tasks. As reported in Appendix C.1 and Tables 11–12, all three models achieve gains on every task after SFT, but the magnitude of the gains varies substantially across tasks. The largest and most consistent gains occur in interaction-structure understanding. At the same time, the gains are clearly heterogeneous, and difficult tasks remain after SFT. The broad but non-uniform gains across all 14 tasks are therefore inconsistent with gains driven by a single highly learnable task pattern.
Second, we examine whether the gains mainly result from better compliance with the required response format. As shown in the following table, format validity increases substantially for the two smaller models, showing that adaptation to the response contract contributes to part of their gains. However, better format compliance cannot explain the full gain. Most notably, Qwen3-Omni-30B-A3B-Instruct gains 36.04 points for overall while its format validity increases by only 1.20 points.