acceptodds
Under review as a conference paper at ICLR 2027

ConvoAudit: A Controlled Diagnostic Benchmark for Conversational Audio Understanding

Abstract

Understanding conversational audio is essential for intelligent systems to support effective communication and assistance. However, evaluating this capability requires assessing not only the content they recover but also how they track participants, interpret interactions, and locate events in time. Although existing benchmarks offer increasingly detailed assessments, aggregate scores can obscure selective weaknesses, while multiple factors always vary together in real-world conversational audio, making their individual effects difficult to isolate. In this work, we introduce ConvoAudit, a controlled diagnostic benchmark for conversational audio understanding, to address these challenges. Specifically, ConvoAudit evaluates dialogue content, interaction structure, and acoustic-temporal organization as three complementary dimensions to profile system capabilities under different conversational conditions. These conditions are realized via synthesized variants of shared source dialogues, thus enabling us to examine how capability profiles and performance gaps between transcription-based and direct-audio systems vary with conversational organization. Our evaluation on over fifty system configurations reveals that transcription-based and direct-audio systems can achieve similar scores on dialogue content yet differ substantially on interaction structure and acoustic–temporal organization, with the relative advantage of direct-audio systems varying across conversational conditions. Matched transcript controls further demonstrate that the usefulness of structural evidence depends on both its availability and its correctness. Fine-tuning on ConvoAudit's training dataset also enables open-weight models to outperform strong proprietary baselines, highlighting its value for targeted model improvement. Together, these analyses connect where systems struggle with controlled tests of which evidence improves their judgments, thereby supporting more informative evaluation of conversational audio understanding.

Primary area:
datasets and benchmarks
Keywords:
Conversational Audio Understanding, Audio Language Models, Diagnostic Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WBeWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ConvoAudio, a benchmark for conversational audio understanding to fill the gap between existing broad audio-understanding benchmark which rely on aggregate scores from diverse real-world recordings,

The benchmark evaluates the model capability in 3 dimensions: dialogue content, interaction strucutre and temporal organization. It also generates 6 variants from the same dialogue source and contributes a dataset for SFT and evaluation,

The paper evaluated transcirpiton-based cascaded model and direct-audio model and find that both systems achieve similar performance on dialogue content. Direct-audio model has better capability in understanding interaction structure and acoustic-temporal organization,

The entire benchmark is synthetic generation and may not generalize to real-world scenarios.

Critical strengths

The key strength is on the design of the granular, controlled benchmark and the benchmark dataset provided. It has strong experimental coverage and provides detailed studies between ASR+LLM and audio-direct model on the 3 proposed evaluation dimensions.

The paper also demonstrates that fine-tuning on their dataset leads to significant performance gains and the benchmark is not just for evaluation but also a valuable resource for model improvement.

Critical weaknesses

The entire benchmark is on synthetically generated with limited evidence for real-world generation. Systems that perform well on this benchmark may not necessarily generalize to real-world scenarios. A key scenario in real-world conversation is overlapped speech, even just for 2 speakers. I would be interested to see how this affects the dialogue content understanding between the two configurations (ASR+LLM, direct-audio).

The strong gain in SFT may suggest highly regularised task or other highly learnable patterns. It would be good for the author to provide some analyses on the gain.

Questions for authors

(1) what is v in equation 1. (2) the most significant gaps between direct-audio and ASR+LLM is in the acoustic-temporal dimensions. Please provide a more granular breakdown of this result. Is the gain due to the localization of non-lexical vocal events and how do it affect the real-world application scenarios? Or the gain is due to better identification of turn boundaries?

3 comments
AuthorsOpenReview ↗

The Response to Reviewer WBeW (1/2)

Thanks for your appreciation of our work. Below, we resolve your concerns one by one.

W1.The entire benchmark is on synthetically generated with limited evidence for real-world generation. Systems that perform well on this benchmark may not necessarily generalize to real-world scenarios. A key scenario in real-world conversation is overlapped speech, even just for 2 speakers. I would be interested to see how this affects the dialogue content understanding between the two configurations (ASR+LLM, direct-audio).

A: First, we would like to emphasize that ConvoAudit is designed as a controlled diagnostic benchmark. Its synthesis-based construction allows us to preserve the underlying conversational content while systematically varying conversational organization, thereby isolating whether the model can identify and follow the relevant evidence pathway under controlled conditions.

Furthermore, to directly address the reviewer’s concern, we specifically select the speech subset of MMAU, the dialogue-reasoning subset of MMSU, and the speech subset of MMAR for the matched evaluation. Drawn from independent public audio benchmarks, these subsets directly correspond to the real-world scenarios and dialogue-content understanding settings highlighted by the reviewer.

Using exactly the same evaluation items, we compare the two configurations: (1) in the direct-audio configuration, Qwen3.5-Omni-Plus directly receives the audio input; (2) in the ASR+LLM configuration, the transcript produced by Qwen3-ASR-Flash is provided to the same Qwen3.5-Omni-Plus answerer. As shown in the following table, direct-audio outperforms ASR+LLM across all three datasets. The advantage is especially pronounced on MMAR Speech and MMSU Dialogue Reasoning, where direct-audio improves accuracy by 14.58 and 10.72 points, respectively.

DatasetDirect-audioASR+LLMDirect-audio gain
MMAU Speech70.8368.75+2.08
MMSU Dialogue Reasoning96.4385.71+10.72
MMAR Speech83.3368.75+14.58

Taken together, these results show that the advantage of direct-audio understanding extends beyond ConvoAudit’s controlled synthesis conditions to real-world scenarios and dialogue-content understanding settings. This finding also reinforces the paper’s central conclusion that preserving the original audio input enables more effective conversational understanding than relying on ASR transcripts, and demonstrates that this advantage is not confined to ConvoAudit’s controlled synthesis conditions.

W2.The strong gain in SFT may suggest highly regularised task or other highly learnable patterns. It would be good for the author to provide some analyses on the gain.

A: Thanks for your valuable suggestion. Here, we analyze the gains from three complementary perspectives: task-level performance, response-format validity, and the question-only control.

First, as described in Appendix A.4 and Table 6, ConvoAudit decomposes conversational understanding into 14 tasks: seven dialogue-content tasks, four interaction-structure tasks, and three acoustic–temporal tasks. As reported in Appendix C.1 and Tables 11–12, all three models achieve gains on every task after SFT, but the magnitude of the gains varies substantially across tasks. The largest and most consistent gains occur in interaction-structure understanding. At the same time, the gains are clearly heterogeneous, and difficult tasks remain after SFT. The broad but non-uniform gains across all 14 tasks are therefore inconsistent with gains driven by a single highly learnable task pattern.

Second, we examine whether the gains mainly result from better compliance with the required response format. As shown in the following table, format validity increases substantially for the two smaller models, showing that adaptation to the response contract contributes to part of their gains. However, better format compliance cannot explain the full gain. Most notably, Qwen3-Omni-30B-A3B-Instruct gains 36.04 points for overall while its format validity increases by only 1.20 points.

ModelFormat validity: Base → SFTOverall: Base → SFT
Qwen2.5-Omni-3B64.00 → 98.75 (+34.75)23.79 → 72.13 (+48.34)
Qwen2.5-Omni-7B78.37 → 99.28 (+20.91)31.13 → 76.85 (+45.72)
Qwen3-Omni-30B-A3B-Instruct97.70 → 98.90 (+1.20)42.85 → 78.89 (+36.04)
AuthorsOpenReview ↗

The Response to Reviewer WBeW (2/2)

Third, we conduct the question-only control to distinguish adaptation to the task interface from improved use of conversational evidence: (1) in the question-only condition, the model receives only the question, answer options, and response contract, without access to the audio; (2) in the complete-input condition, the same question, answer options, and response contract are paired with the corresponding audio. As shown in the following table, although SFT improves all three models under the question-only condition, the Base-to-SFT gains are consistently larger under the complete-input condition. This pattern shows that SFT does more than teach task-interface regularities: it indeed substantially improves the models’ ability to extract and use evidence from the input audio.

ModelQuestion-only Overall: Base → SFTComplete-input Overall: Base → SFT
Qwen2.5-Omni-3B23.75 → 41.47 (+17.72)23.79 → 72.13 (+48.34)
Qwen2.5-Omni-7B24.53 → 38.65 (+14.12)31.13 → 76.85 (+45.72)
Qwen3-Omni-30B-A3B-Instruct25.60 → 38.37 (+12.77)42.85 → 78.89 (+36.04)

Taken together, the task-level and response-format analyses show that the gains are neither confined to a small set of tasks nor explained by better output compliance alone. More importantly, the question-only control shows that fine-tuning substantially increases the contribution of the audio across the benchmark. The strong SFT gains therefore reflect both adaptation to the task interface and the genuine gain in evidence-conditioned conversational audio understanding, rather than merely fitting the highly regularized task structure or superficial learnable patterns.

Q1.What is v in equation 1

A: Thank you for pointing out this missing definition. In Equation (1), vsv_s denotes the role-to-voice assignment for dialogue ss, i.e., the mapping from conversational speakers or roles to their assigned voice profiles. We have explicitly defined vsv_s after Equation (1) in the revised manuscript.

Q2.The most significant gaps between direct-audio and ASR+LLM is in the acoustic-temporal dimensions. Please provide a more granular breakdown of this result. Is the gain due to the localization of non-lexical vocal events and how do it affect the real-world application scenarios? Or the gain is due to better identification of turn boundaries?

A: As defined in Appendix A.4 and Table 6, the acoustic–temporal dimension can be further decomposed into three tasks: A-1: Voice attributes and nonlexical vocal events, A-2: Global audible structure of the exchange, and A-3: Condition-specific event timing and windows. Complete task-level results for these tasks across all evaluated configurations are reported in Appendix C.1 and Table 12.

For ease of presentation, we show one representative matched-answerer comparison in the following table. Here, we compare two configurations: (1) in the direct-audio configuration, Qwen3.5-Omni-Plus directly receives the audio input; (2) in the ASR+LLM configuration, the transcript produced by Qwen3-ASR-Flash is provided to the same Qwen3.5-Omni-Plus answerer. This matched comparison holds both the evaluation items and the downstream answerer fixed, isolating the difference between the two input pathways.

TaskDirect-audioASR+LLMDirect-audio gainContribution to the A-dimension gap
A-191.9243.00+48.92+17.18 (56.66%)
A-241.2535.67+5.58+1.96 (6.47%)
A-338.520.97+37.55+11.18 (36.87%)
A Overall58.2327.91+30.32+30.32 (100%)

This breakdown shows that the gap between direct-audio and ASR+LLM is driven primarily by A-1 and A-3. A-1 jointly evaluates voice attributes and nonlexical vocal-event states. Our field-level analysis shows that the A-1 gap mainly reflects voice-attribute recognition, for which direct-audio achieves 99.67%, compared with 46.33% for ASR+LLM. A-3 evaluates temporal localization at the event- or utterance-window level. On its numeric-window items, the rate at which both predicted window boundaries fall within ±2 seconds of the reference is 27.45% for direct-audio and 0.18% for ASR+LLM.

In summary, these finer-grained results show that the direct-audio advantage is mainly associated with stronger acoustic-cue recognition and temporal-window localization, rather than being primarily driven by turn-boundary identification. In real-world applications, these capabilities can support acoustic-cue review in meeting and customer-service recordings, as well as the retrieval of intervals containing side exchanges, handoffs, temporary takeovers, and returns to the main conversation.

[1] MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. ICLR 2025

[2] MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix. NeurIPS 2025

[3] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark. ICLR 2026

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer WBeW,

We sincerely appreciate your thoughtful feedback and constructive comments, which have helped us improve our work.

Over the past few days, we have devoted considerable effort and resources to addressing each of your concerns, including generalization beyond synthetically generated conversations, the strong SFT gains, acoustic–temporal performance, and the symbol definition of Equation (1).

As the discussion period ends in less than two days, we would be grateful if you could review our rebuttal and let us know whether it adequately addresses your concerns.

Best regards,

The Authors

Reviewer_SBaTOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark called ConvoAudit which is a benchmark for assessing how audio-language models understand multi-speaker conversations, particularly changes in speaker participation, conversational flow, and event timing.

The main contributions of this paper are the construction of six controlled conversational variants using two audio-synthesis pipelines, producing 7,929 recordings and over 105K QA instances. There are three main capabilities evaluated, which are dialogue content, interaction structure, and acoustic–temporal understanding. Through experiments on over 50 system configurations, the authors compare direct-audio and transcript-based approaches, investigate the effects of timestamps and speaker labels, and demonstrate substantial gains from benchmark-specific fine-tuning.

The key finding of the authors is that accurate transcription or dialogue-content recovery does not necessarily imply an understanding of conversational structure. For example, Qwen3.5-Omni-Plus achieves nearly identical content scores with transcripts and direct audio (70.3 vs. 69.7), but direct audio substantially improves interaction-structure (30.6 to 46.3) and acoustic–temporal scores (27.9 to 58.2).

Critical strengths
  1. The authors address several potential confounds in synthetic benchmark construction which is great to see. They test robustness across two synthesis pipelines and two LLM judges, obtaining high rank correlations of 0.983 and 0.968, respectively. The human listening study further reports high 97.9% clear or mostly clear recordings and 91.5% agreement with reference answers (Section 5.6). These checks make the evaluation more credible than relying solely on automatically generated audio and annotations.

  2. Rather than simply concluding that direct audio performs better, the authors investigate which missing or inaccurate transcript information contributes to the gap. Table 3 shows that adding timestamps improves scores by 16.78-18.66 points, whereas adding speaker labels alone provides little benefit. Yet correcting speaker labels when all structural fields are available yields gains of 5.04-9.62 points. This distinction between the availability and correctness of structural information offers actionable insight into improving ASR–LLM pipelines.

Critical weaknesses

1.Table 4 reports substantial improvements after fine-tuning, including a 48.35-point overall gain for Qwen2.5-Omni-3B. Although the train ant test split is disjoint, both of the splits share the same synthetic conditions, synthesis pipelines, and question formats. Without evaluation on independent conversational audio, it might be unclear how much of the improvement reflects generalizable understanding rather than adaptation to ConvoAudit specific patterns.

  1. Several benchmark questions require speaker identities or timing information that plain ASR transcripts lack, such as identifying who consults whom or when a side conversation begins and ends (Figure 1). Thus, the direct-audio advantage in Table 1 may partly reflect missing information in the transcript baseline rather than differences in reasoning ability. Although Table 3 partially addresses this through structural-evidence experiments, a full comparison against transcripts with accurate speaker labels and timestamps would make the conclusion more convincing.
Questions for authors
  1. Would it be possible to evaluate whether models can answer some benchmark questions without listening to the audio? Given the structured QA generation process (Appendix A.4), a question-only baseline would help determine if any patterns or distributions provide exploitable shortcuts, particularly after fine-tuning.

  2. Would it be possible to break down the results between Chinese and English conversations? Do the relative strengths of direct-audio and transcript-based systems remain consistent across the two languages?

5 comments
AuthorsOpenReview ↗

The Response to Reviewer SBaT (1/2)

Thanks for your appreciation of our work. Below, we resolve your concerns one by one.

W1.Without evaluation on independent conversational audio, it might be unclear how much of the improvement reflects generalizable understanding rather than adaptation to ConvoAudit specific patterns.

A: To distinguish generalizable understanding from adaptation to ConvoAudit-specific patterns, we conduct two complementary evaluations: cross-route transfer between distinct synthesis pipelines and evaluation on independent public audio benchmarks.

First, we leverage ConvoAudit's dual-route design to examine whether the capability acquired through fine-tuning transfers across different synthesis pipelines. Using the same training configuration, we fine-tune Qwen3-Omni-30B-A3B-Instruct separately on Route-A and Route-B, then evaluate each resulting model under both routes. As shown in the following table, fine-tuning on either route produces clear gains not only on the corresponding route but also on the other synthesis route. This bidirectional transfer demonstrates that the learned capability generalizes across distinct synthesis implementations rather than depending on acoustic patterns specific to the single pipeline.

Route-A OverallRoute-B Overall
Before SFT42.5243.18
SFT on Route-A80.7376.88
SFT on Route-B77.0381.27

Second, we evaluate general audio-understanding performance before and after ConvoAudit fine-tuning on three independent public benchmarks. These benchmarks have independent audio sources, task definitions, and question formats, and cover broader speech, sound, music, and audio-reasoning capabilities rather than being tailored to ConvoAudit's conversational conditions. As shown in the following table, Qwen3-Omni-30B-A3B-Instruct improves consistently across all three external benchmarks after ConvoAudit fine-tuning. These results show that the improvement is not confined to ConvoAudit's conversational tasks: fine-tuning on ConvoAudit also preserves and slightly strengthens the model's general audio-understanding performance under independent data and task distributions.

DatasetBefore SFTAfter SFT
MMAU76.6077.90
MMSU70.8771.66
MMAR71.9072.10

Taken together, the gains persist across different synthesis pipelines and extend to public benchmarks with independent audio sources and question formats. The observed improvements therefore substantially reflect generalizable understanding rather than adaptation to ConvoAudit-specific synthesis.

W2.Although Table 3 partially addresses this through structural-evidence experiments, a full comparison against transcripts with accurate speaker labels and timestamps would make the conclusion more convincing.

A: As detailed in Section 5.4 and Appendix C.4, we have conducted extensive matched transcript controls to examine the role of speaker labels and timestamps, including:

  • Availability and correctness (Section 5.4, Table 3; Appendix C.4.2, Tables 21–22): We compare adding speaker labels or timestamps to plain transcripts and replacing ASR-provided values with reference values.
  • Field visibility (Appendix C.4.3, Tables 23–25): We evaluate all four visibility combinations of speaker labels and timestamps on ASR-provided and reference values, with transcript content fixed.
  • Field quality (Appendix C.4.4–C.4.5, Tables 26–29): We vary the quality of speaker labels and timestamps, with additional controls for speaker label substitution and timestamp corruption.

Here, we also provide the full comparison on the test set, using Qwen-Audio-3.0-ASR-Flash-Filetrans for ASR and Qwen3.5-Omni-Plus as the answerer throughout. As shown in the following table, adding ASR-provided speaker labels and timestamps improves Overall from 49.16 to 52.31, reference transcripts with accurate speaker labels and timestamps further improve it to 57.58, and direct audio still achieves the highest Overall (60.32).

InputDIAOverall
Plain ASR transcript69.6731.1726.6149.16
ASR transcript + ASR speaker labels and timestamps69.6133.2538.7652.31
Reference transcript + reference speaker labels and timestamps71.7741.9446.4657.58
Direct audio69.6646.3158.2360.32

These results show that the gap to direct audio narrows but persists even with accurate speaker labels and timestamps, supporting our finding that direct audio access offers advantages beyond content recovery, particularly in interaction structure (I) and acoustic–temporal organization (A).

AuthorsOpenReview ↗

The Response to Reviewer SBaT (2/2)

Q1.Would it be possible to evaluate whether models can answer some benchmark questions without listening to the audio? Given the structured QA generation process (Appendix A.4), a question-only baseline would help determine if any patterns or distributions provide exploitable shortcuts, particularly after fine-tuning.

A: Thanks for your valuable suggestion. Here, we conduct the question-only evaluation on the test set for all three SFT models reported in the paper together with Qwen3.5-Omni-Plus for reference. Under this setting, each model receives only the question, answer options, and response contract, without access to the audio.

As shown in the following table, the three SFT models achieve Overall scores of 41.47, 38.65, and 38.37 under the question-only evaluation, with their scores only slightly higher than that of Qwen3.5-Omni-Plus (37.75). When given the complete input (i.e., with the audio provided), their Overall scores increase to 72.13, 76.85, and 78.89, yielding large and consistent improvements compared to the question-only evaluation.

ModelQuestion-only OverallComplete-input OverallImprovement
Qwen2.5-Omni-3B-SFT41.4772.13+30.66
Qwen2.5-Omni-7B-SFT38.6576.85+38.20
Qwen3-Omni-30B-A3B-Instruct-SFT38.3778.89+40.52
Qwen3.5-Omni-Plus37.7560.32+22.57

These results show that the post-fine-tuning performance cannot be explained by exploitable patterns or distributional shortcuts in the questions, answer options, or response contract alone. The large and consistent improvements obtained when audio is provided demonstrate that these SFT models have indeed learned to extract and utilize evidence from the input audio, rather than merely fitting benchmark-specific patterns in the structured QA generation process.

Q2.Would it be possible to break down the results between Chinese and English conversations? Do the relative strengths of direct-audio and transcript-based systems remain consistent across the two languages?

A: Thanks for your valuable suggestion. We have added the complete language-stratified analysis in Appendix C.6 and Table 37, reporting D, I, A, and Overall separately for Chinese and English conversations.

For ease of presentation, we show one representative matched-answerer comparison in the following table. Here, we hold Qwen3.5-Omni-Plus fixed as the answerer in both pathways: the transcript-based pathway uses transcripts produced by Qwen3-ASR-Flash, whereas the direct-audio pathway provides the audio directly to the same model. This result along with the complete results in Appendix C.6 show that the principal pathway comparison is consistent across the two languages: transcripts preserve most dialogue-content information (D), whereas direct audio better captures participant interactions (I) and acoustic--temporal organization (A).

LanguageInput pathwayDIAOverall
ChineseQwen3-ASR-Flash transcript71.4229.5329.8850.07
ChineseDirect audio70.4144.9658.1860.20
EnglishQwen3-ASR-Flash transcript68.0333.0623.5648.59
EnglishDirect audio68.0749.3358.3560.59

[1] MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. ICLR 2025

[2] MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix. NeurIPS 2025

[3] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark. ICLR 2026

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer SBaT,

We sincerely appreciate your thoughtful feedback and constructive comments, which have helped us improve our work.

Over the past few days, we have devoted considerable effort and resources to addressing each of your concerns, including SFT generalization, speaker and timestamp information, potential question-only shortcuts, and performance across Chinese and English.

As the discussion period ends in less than two days, we would be grateful if you could review our rebuttal and let us know whether it adequately addresses your concerns.

Best regards,

The Authors

Reviewer_QhnLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ConvoAudit, a controlled diagnostic benchmark for conversational audio understanding. It constructs six conversational organization conditions while preserving the main content of the source conversations, and generates recordings through two audio synthesis pipelines. The benchmark separately evaluates conversational content, interaction structure, and acoustic–temporal organization. The paper compares transcript-based systems with direct-audio models, uses interventions on structural fields to analyze the roles of timestamps and speaker information, and examines improvements from within-benchmark fine-tuning.

Critical strengths

The controlled diagnostic design is valuable. Varying background activity, side conversations, and speaker transitions while retaining shared source conversations helps reduce confounding from content differences across heterogeneous real-world recordings. This design can reveal condition-specific weaknesses that aggregate scores may obscure. The two synthesis pipelines also allow the authors to examine whether the diagnostic findings depend on a particular synthesis implementation.

Critical weaknesses

w1. The conceptual advance of the main findings is limited, several of the paper’s main findings are largely expected:

a. Similar conversational content scores can coexist with substantial differences in interaction structure and acoustic–temporal performance (ll. 347–358).

b. More accurate timestamps and speaker labels improve performance (ll. 420–424).

These results provide useful quantitative evidence, but their conceptual contribution appears limited: plain lexical transcripts do not explicitly preserve all speaker and temporal structure, whereas the relevant tasks inherently depend on this information.

Meanwhile, MSU-Bench already separately evaluates speaker grounding and identification, dialogue structure, and contextual reasoning. The manuscript does not sufficiently explain what new diagnostic insights it yields beyond existing evaluations.

w2. Key information needed to assess benchmark reliability and reproducibility is missing (see the Question for details).

Questions for authors

q1. For the I dimension, particularly the failures in L2–L4, how much is attributable to errors in speaker segmentation/attribution, transcription, or temporal alignment, and how much persists when correct structural information is provided?

q2. What are the exact sample sizes and denominators for the reported 97.9% “clear or mostly clear” rate, 91.5% answer agreement among answerable questions, and 95.8% preference for the reference speaker in comparisons involving disagreement? How many annotators participated, how many evaluated each sample, and how were disagreements handled? What proportion of questions was unanswerable, and what were the results for each I/A task and for the L4/L5 conditions?

4 comments
AuthorsOpenReview ↗

The Response to Reviewer QhnL (1/2)

Thanks for your feedback. Below, we resolve your concerns one by one.

W1.The conceptual advance of the main findings is limited, several of the paper’s main findings are largely expected.

A: We respectfully disagree with the assessment that the conceptual contribution is limited. ConvoAudit’s contribution lies in the systematic diagnosis enabled by its controlled design, rather than in individual empirical observations viewed in isolation. It varies conversational organization while preserving the substantive mainline and controls the evidence available to the model, allowing us to identify which organizational changes expose capability weaknesses and which evidence helps address them. Specifically:

1. Regarding (a)

ConvoAudit creates six conditions from each source dialogue to examine how changes in conversational organization affect content, interaction, and acoustic–temporal understanding. For example, we evaluate Qwen3.5-Omni-Plus using either audio or transcripts as input (Section 5.1, Table 1). On interaction structure, transcripts lead by 4 points under condition L0 (i.e., the mainline baseline), but audio leads by 30 points under condition L3 (i.e., persistent handoff). Content scores differ by less than 0.5 points in both conditions (Appendix C.2, Table 14). This reversal demonstrates that conversational organization can change the relative strengths of audio and transcripts even when content understanding is closely matched.

The distinction between content and interaction understanding also appears when audio is used as input. For example, Qwen3.5-Omni-Plus scores 52.88 on interaction structure under condition L3 (i.e., persistent handoff) but 25.75 under condition L4 (i.e., temporary takeover followed by return). Its content scores remain around 70 in both conditions (Appendix C.2, Table 13). These results reveal the difficulty in tracking changes in participation despite access to audio, which cannot be attributed solely to information omitted from plain transcripts.

2. Regarding (b)

Our comparisons extend from plain transcripts to transcripts with ASR-derived or reference speaker labels and timestamps. For example, on the diagnostic subset covering A-2 and A-3, timestamps primarily support temporal localization, while adding ASR speaker labels alone yields little benefit. In the separate correctness controls, correcting speaker attribution consistently improves performance across four answerers (Section 5.4, Table 3; Appendix C.4.2, Tables 21–22). These results show how the usefulness of structural information depends on both its availability and its correctness.

The supporting analyses also examine all four combinations of speaker-label and timestamp availability (Appendix C.4.3, Tables 23–25), as well as eight combinations of ASR/reference text, time, and speaker values (Appendix C.4.4, Tables 26–27). Complementary timestamp-corruption and speaker-substitution experiments link annotation correctness to temporal localization (A-3) and scene recognition (A-2), respectively (Appendix C.4.5, Tables 28–29). Together, these comparisons examine the contribution of each type of evidence and how its value depends on the other information supplied.

Here, we further provide the full comparison on the test set, using Qwen-Audio-3.0-ASR-Flash-Filetrans for ASR and Qwen3.5-Omni-Plus as the answerer throughout. As shown in the following table, reference structured transcripts narrow the Overall gap to direct audio from 11.16 points for plain ASR transcripts to 2.74 points. The remaining gap thus supports our finding that direct audio access offers advantages beyond content recovery, particularly in interaction structure (I) and acoustic–temporal organization (A).

InputDIAOverall
Plain ASR transcript69.6731.1726.6149.16
ASR transcript + ASR speaker labels and timestamps69.6133.2538.7652.31
Reference transcript + reference speaker labels and timestamps71.7741.9446.4657.58
Direct audio69.6646.3158.2360.32

3. Regarding MSU-Bench

MSU-Bench evaluates speaker grounding, dialogue structure, and contextual reasoning, and includes analyses of speaker referencing and error types. However, ConvoAudit goes further by systematically varying conversational organization within the same source dialogue and controlling the evidence available to the model. This controlled design enables systematic diagnosis of which conversational changes expose capability weaknesses, how the direction and magnitude of performance differences vary across conditions, and how the availability and correctness of input evidence affect these differences. These comparisons connect conversational organization, available evidence, and model capabilities, thereby providing diagnostic insights beyond the general observations.

AuthorsOpenReview ↗

The Response to Reviewer QhnL (2/2)

W2&Q1.For the I dimension, particularly the failures in L2–L4, how much is attributable to errors in speaker segmentation/attribution, transcription, or temporal alignment, and how much persists when correct structural information is provided?

A: In the original manuscript, we have analyzed interaction-structure (I) performance across conversational conditions (Section 5.2, Figure 3; Appendix C.2, Table 13). Here, we further provide I-specific analyses to quantify the contributions of structural corrections and examine the errors that persist with reference information. The results are presented below:

1. Quantifying the benefit of reference structural information

We first compare ASR-derived and reference structure for I under L2–L4. As shown in the following table, reference structure improves I scores across all answerers. These consistent gains demonstrate the contribution of frontend corrections to interaction understanding.

AnswererASR-derived structureReference structureGain
GLM-5.226.6336.38+9.75
DeepSeek-V4-Pro23.7928.04+4.25

2. Disentangling transcription, timing, and speaker-attribution contributions

Following the field-quality design in Appendix C.4.4, we hold reference segmentation fixed and vary whether text, timestamps, and speaker assignments use projected ASR or reference values. As shown in the following table, the Shapley decomposition attributes most score recovery to speaker attribution, highlighting the importance of correctly associating speech with participants for I performance.

AnswererText contributionTimestamp contributionSpeaker-assignment contributionTotal recovery
Qwen3.5-Omni-Plus+2.080.00+13.89+15.97
Qwen3.7-Max−1.97+1.85+12.62+12.50

3. Examining interaction judgments with audio and reference structure

We further examine whether interaction-level errors persist when both audio and reference structure are available. With the same instruction, adding reference structure to audio raises Qwen3.5-Omni-Plus's I score from 41.13 to 57.71 under L2-L4. The field-level results clarify the nature of this remaining challenge: under condition L3 (i.e., persistent handoff), the model achieves 94% accuracy on the voice-attribute field, but only 21% on its channel and continuity fields. Thus, the model can identify the voice attribute with high accuracy while still struggling to determine how the additional participant changes the mainline. This shows that persistent-handoff and continuity judgments are not resolved simply by supplying accurate frontend structure, supporting the value of evaluating interaction relations separately from transcription and speaker attribution.

Taken together, these analyses provide the quantitative answer: speaker attribution is the dominant recoverable factor in the controlled field-quality comparison; reference structure yields substantial gains; and participation-relation judgments remain challenging even when audio and accurate structure are supplied. This supports the diagnostic role of interaction structure (I): it evaluates not only the recovery of words and speaker identities, but also the ability to track how participation changes the organization and continuity of a conversation.

W2&Q2.What are the exact sample sizes and denominators for the reported 97.9% “clear or mostly clear” rate, 91.5% answer agreement among answerable questions, and 95.8% preference for the reference speaker in comparisons involving disagreement? How many annotators participated, how many evaluated each sample, and how were disagreements handled? What proportion of questions was unanswerable, and what were the results for each I/A task and for the L4/L5 conditions?

A: In the human listening assessment described in Section 5.6, five annotators each evaluated the distinct set of 125 items, covering 625 items in total. Specifically, each item required annotators to select the appropriate options for audio clarity, speaker annotations, and the associated question. Of the evaluated items, 97.9% (612/625) were rated clear or mostly clear and thus considered answerable items, with human responses matching the reference answers on 91.5% of these answerable items (560/612). For speaker annotations, we further analyzed the 24 items where ASR and reference assignments differed, finding that reference assignments were preferred in 95.8% of these comparisons (23/24). Additionally, human–reference answer agreement among answerable items within each group was 88.9% (152/171) for interaction structure (I), 90.7% (117/129) for acoustic–temporal organization (A), 86.1% (87/101) under condition L4 (i.e., temporary takeover followed by return), and 89.0% (89/100) under condition L5 (i.e., combinations of conversational changes).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer QhnL,

We sincerely appreciate your thoughtful feedback and constructive comments, which have helped us improve our work.

Over the past few days, we have devoted considerable effort and resources to addressing each of your concerns, including the clarification of our contributions, the sources of I-dimension failures, and the details of the human listening assessment.

As the discussion period ends in less than two days, we would be grateful if you could review our rebuttal and let us know whether it adequately addresses your concerns.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-up on Your Review Update

Dear Reviewer QhnL,

Thank you again for your time and consideration.

We noticed the recent update to your review and wanted to check whether there were any accompanying comments that we might be unable to see. If you left additional feedback, would you mind checking whether it is visible to the authors?

We would greatly appreciate any clarification on whether our rebuttal has addressed your concerns.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.