acceptodds
Under review as a conference paper at ICLR 2027

AUDIOTRACE: EVIDENCE-GROUNDED QUESTION ANSWERING OVER LONG AUDIO

Abstract

Evaluations of large audio-language models (LALMs) have expanded beyond minute-scale clips to include recordings lasting several hours. Yet current long-audio QA evaluations face two key limitations. (1) Inflated scores can mask weaknesses in fine-grained audio understanding and reasoning. (2) Answer correctness is often assessed without verifying whether models identify the precise temporal spans and factual content supporting their predictions. To address these limitations, we introduce AudioTrace, a benchmark that disentangles answer correctness from evidence grounding in long-audio QA. The dataset comprises 120 Chinese and English recordings lasting 30–90 minutes each (99.8 hours total), with 1,033 pairs of multiple-choice and open-ended questions across five question categories. Our construction pipeline combines automated annotation with human review, yielding event-level temporal and content evidence for each answerable question. The evaluation protocol has three levels with independently generated responses: L1 assesses answer correctness alone; L2 additionally requires temporal localization of supporting evidence; and L3 further requires semantic grounding of that evidence. Experiments reveal a substantial gap between answer correctness and evidence-supported correctness. Across 17 model–paradigm configurations, mean L1 answer accuracy is 84.8% for multiple-choice QA and 43.8% for open-ended QA, whereas mean L3 joint accuracy is 22.1% and 10.9%, respectively. These results show that a correct answer alone does not establish accurate understanding of the supporting audio or reliable reasoning from it. Evaluating trustworthy long-audio QA therefore requires finer-grained checks of the temporal location and factual content of supporting evidence. The code and dataset are publicly available at https://anonymous.4open.science/r/open_audiotrace.

Primary area:
datasets and benchmarks
Keywords:
Large Audio Language Models, Long-Form Audio Understanding, Audio Question Answering, Evidence Grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_e1NROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AudioTrace, a benchmark for evidence-grounded question answering over long audio. It contains 120 recordings (99.8 hours) and 1,033 question pairs in Chinese and English. The main contribution is a three-level evaluation protocol that measures answer correctness (L1), temporal evidence localization (L2), and semantic evidence grounding (L3). The dataset is built using automated annotation and human verification. Experiments across 17 model configurations show a large gap between answer accuracy and evidence-grounded accuracy, suggesting that correctly answering a question does not necessarily mean the model can identify the supporting audio evidence.

Critical strengths

1. A useful benchmark for evaluating supporting evidence in long-audio QA.

The paper addresses an important limitation of answer-only evaluation. By providing timestamps and event descriptions for supporting evidence, AudioTrace allows a more detailed assessment of model performance. The inclusion of both multiple-choice and open-ended questions is also useful.

2. Careful dataset construction and extensive evaluation.

The annotation pipeline combines automated checks with independent human review. The authors also evaluate a broad range of models and include controls for question-only shortcuts and evaluation thresholds. These efforts strengthen the empirical value of the benchmark.

Critical weaknesses

1. Limited novelty and insufficient differentiation from prior work.

The central idea of evaluating answers together with supporting evidence is not new. EG-VQA (Huang et al., 2026) already evaluates answer correctness, temporal localization, and semantic evidence matching, and AudioTrace directly adopts its EG-F1 metric. Within the audio domain, AudioSpan (Huang et al., 2026), MuLA-Bench (Yang et al., 2026), TAG-Bench (Dai et al., 2026), Vox-Infinity (Cheng et al., 2026), and LongAudio-RAG (Vakada et al., 2026) have already explored long-audio understanding, temporal grounding, and evidence-based QA.

The paper also overlooks relevant audio hallucination studies, including Kuan et al. (Interspeech 2024), Kuan and Lee (ICASSP 2025; Interspeech 2025), and AHa-Bench (Cheng et al., NeurIPS 2025). These works have already investigated failures in factual audio understanding, including sound existence, attributes, and temporal relationships.

Overall, the paper mainly combines existing evaluation ideas into a new dataset. Its main finding that correct answers do not guarantee correct evidence is also well established. It is unclear what substantially new insights AudioTrace provides beyond these prior studies.

2. The proposed evaluation does not fully validate the paper's central claim.

The paper argues that correct answers may result from shortcuts rather than genuine audio understanding. However, L3 only checks whether the model can produce evidence matching the reference. A model could guess the correct answer first and then retrieve supporting evidence, without actually using that evidence to derive its answer. Therefore, passing L3 does not establish faithful reasoning.

Moreover, semantic evidence correctness is determined using embedding similarity with a threshold of 0.5, without direct human validation of this metric. Semantically similar but factually contradictory statements may still match. Since evidence correctness is the central contribution of the benchmark, this raises concerns about whether L3 reliably measures what the paper claims.

References

[1] Huang et al. (2026). EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence. arXiv:2606.24797.

[2] Huang et al. (2026). AudioSpan: Spanning the Duration and Depth of Audio Comprehension. arXiv:2608.26431.

[3] Yang et al. (2026). MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing. arXiv:2609.23416.

[4] Dai et al. (2026). TAG-Bench: Benchmarking Temporal Audio Grounding in Large Audio Language Models. arXiv:2609.01542.

[5] Cheng et al. (2026). Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models. arXiv:2609.22452.

[6] Vakada et al. (2026). LongAudio-RAG: Event-Grounded Question Answering over Multi-Hour Long Audio. arXiv:2602.14612.

[7] Kuan, C.-Y., Huang, W.-P., and Lee, H.-y. (2024). Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models. Interspeech 2024.

[8] Kuan, C.-Y., and Lee, H.-y. (2025). Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning. ICASSP 2025.

[9] Kuan, C.-Y., and Lee, H.-y. (2025). Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples. Interspeech 2025.

[10] Cheng et al. (2025). AHa-Bench: Benchmarking Audio Hallucinations in Large Audio-Language Models. NeurIPS 2025.

Questions for authors
  1. Novelty: Beyond applying existing evidence-grounding metrics to long audio, what new failure modes or findings does AudioTrace reveal that cannot be studied using prior benchmarks such as EG-VQA, AudioSpan, and MuLA-Bench?

  2. Evaluation validity: Have the authors validated EG-F1 against human judgments of factual evidence correctness? In particular, how does the metric handle evidence that is semantically similar but factually contradictory, or evidence that is only partially sufficient to support an answer?

  3. Faithful reasoning: How can the proposed evaluation distinguish a model that genuinely reasons from audio evidence from one that first predicts an answer and then finds plausible supporting evidence? This distinction seems important given the paper's central motivation.

6 comments
AuthorsOpenReview ↗

Answer to Q1

We thank the reviewer for the careful attention to AudioTrace’s novelty, the validity of its evidence evaluation, and reasoning faithfulness. These issues are directly related to the central motivation of the paper. In the revision, we add evidence-evaluation experiments and an independent rubric-based LLM judge to examine these concerns further.

1. Novelty

AudioTrace is not intended to reinvent a particular evidence-matching metric. Its novelty is to propose and systematically measure joint answer–evidence ability in long-audio question answering. For a question QQ, the model must provide both an answer AA and the supporting time interval and factual evidence EE. Our target is therefore:

P(A,E∣Q).P(A,E\mid Q).

This distinguishes AudioTrace from tasks that evaluate answer correctness alone, temporal localization alone, or retrieval of isolated audio segments. EG-VQA primarily studies video question answering. AudioSpan, MuLA-Bench, and related works involve long-audio question answering or temporal localization, but they do not require jointly verifying the answer and its supporting evidence for the same long-audio question, nor do they systematically evaluate the target quantity P(A,E∣Q)P(A,E\mid Q). Consequently, they cannot reveal the failure mode we observe: even with the same fixed set of L3 outputs, answer accuracy is 86.8% for MCQ and 43.9% for open-ended questions, whereas only 22.1% and 10.9%, respectively, provide acceptable supporting evidence together with the correct answer.

Question typeAnswer accuracyJoint answer–evidence accuracy
MCQ86.8%22.1%
Open-ended43.9%10.9%

This is not an existing audio-segment retrieval task applied to longer recordings. It requires the model to answer the same long-audio question while performing global temporal localization and evidence description. The gap above is a substantial capability gap measured directly in long-audio question answering; it cannot be inferred from video evaluation results or from independent temporal-localization performance.

In addition, we developed and fully released the first batch-generation pipeline of its kind for long-audio question answering. The pipeline generates questions from structured audio events, links each question to real timestamped supporting events, and produces auditable annotations through automatic screening and independent listening-based review. The construction code, prompts, and runtime configurations are publicly available, making the data-construction process reproducible.

AuthorsOpenReview ↗

Answer to Q2

2. Evaluation validity

The semantic similarity of event pairs and the set-level EG-F1 acceptance threshold are both set to strictly > 0.5. This is deliberately permissive for evidence matching: it tends to accept descriptions with different wording but similar meaning, and may also accept partially matching evidence. Even under these relatively easy-to-satisfy conditions, many answer-correct predictions still fail to satisfy the temporal and content requirements. Thus, the substantial drop from L1 to L3 is unlikely to be explained by an overly strict evidence criterion; it reflects the difficulty of explicitly locating and describing supporting events in long audio.

To further examine whether this finding depends on the scoring method, we used DeepSeek-V4-Flash to review all 35,122 L3 evidence predictions. The review follows a unified rubric that separately evaluates content correctness, evidence completeness, and evidence precision. A prediction is counted as accepted only when all three dimensions receive a score of 5. The overall acceptance rate is 12.87%. We also report model-level results under the original EG-F1 metric and the DeepSeek judge. Specifically, we compute evidence quality separately for all 17 configurations and compare their relative rankings. The Spearman correlation between the two model rankings is:

ρ=0.76\rho=0.76

Although the two scoring methods may disagree on individual examples, this model-level result shows that the rubric-based judge and the original EG-F1 metric produce broadly consistent relative rankings of model evidence quality.

Model-level acceptance rates for all 17 configurations

The table directly compares the acceptance rates of the two evidence metrics. An EG-F1 acceptance means that the evidence-set EG-F1 is strictly greater than 0.5. A DeepSeek acceptance means that content correctness, completeness, and precision all receive a score of 5.

ConfigurationEG-F1 > 0.5 (MCQ)DeepSeek 5/5/5 (MCQ)EG-F1 > 0.5 (Open-ended)DeepSeek 5/5/5 (Open-ended)
E2E_FireRed Audio40.30%19.94%23.74%12.78%
E2E_Gemini 3.1 Pro12.79%20.23%6.05%13.94%
E2E_Gemini 3.6 Flash29.79%32.14%17.01%19.75%
E2E_Qwen3.5 Omni Plus45.21%15.88%31.05%14.04%
E2E_Qwen3.8 Omni Flash67.81%37.37%54.91%29.62%
Cascade_Audio Flamingo5.82%6.49%5.02%5.32%
Cascade_FireRed Audio45.09%16.75%34.36%10.94%
Cascade_Gemini 3.1 Pro29.45%11.23%23.17%6.00%
Cascade_Gemini 3.6 Flash30.82%15.39%21.35%8.91%
Cascade_Kimi Audio3.77%12.00%2.51%9.87%
Cascade_MiMo Audio14.95%10.94%11.99%7.74%
Cascade_MiniCPM-o10.50%7.07%9.02%5.42%
Cascade_MOSS Audio13.70%8.52%11.87%5.52%
Cascade_Qwen3.5 Omni Plus17.58%12.20%14.27%8.81%
Cascade_Qwen3.8 Omni Flash27.63%14.04%20.78%9.29%
Cascade_Qwen3 Omni2.85%11.04%2.51%7.84%
Cascade_Step Audio26.51%5.81%5.25%4.74%

The full scoring rubric will be provided in the supplementary material.

AuthorsOpenReview ↗

Answer to Q3

3. Reasoning faithfulness

We believe the most appropriate faithfulness criterion requires a model to provide both a correct answer and reliable audio evidence for every question. If both outputs are correct, it is reasonable whether the model first forms an answer and then searches for supporting evidence, or first locates evidence and then answers from it. Our concern is that a model may guess the answer using only the question, the options, or other surface cues while failing to provide reliable evidence supporting that answer; such shortcut-based behavior is likely to fail on a broader set of questions. We therefore define reasoning faithfulness as joint answer–evidence correctness:

P(A,E∣Q),P(A,E\mid Q),

where QQ is the question, AA is the answer, and EE is the selected evidence. This joint probability can be decomposed according to different reasoning orders. For example, a model may first form an answer and then search for supporting evidence:

P(A,E∣Q)=P(A∣Q)P(E∣A,Q).P(A,E\mid Q)=P(A\mid Q)P(E\mid A,Q).

Alternatively, it may first locate evidence and then answer based on that evidence:

P(A,E∣Q)=P(E∣Q)P(A∣E,Q).P(A,E\mid Q)=P(E\mid Q)P(A\mid E,Q).

Our evaluation does not prohibit any particular generation order. It requires the model to satisfy both answer correctness and evidence correctness, thereby attaining a high value of P(A,E∣Q)P(A,E\mid Q).

Accordingly, we do not regard “forming an answer first and then searching for evidence” as an invalid paradigm. For MCQ, the options provide additional semantic constraints, and humans may also form a candidate answer from the question and options before returning to the audio to find supporting evidence. Thus, observing answer-conditioned behavior in MCQ does not by itself show that the model failed to reason from the audio; the key issue is whether the final answer is correct and whether the evidence genuinely supports it.

For open-ended questions, the data-construction process filters out questions that can be answered from commonsense or the question text alone. Without audio evidence, P(A∣Q)P(A\mid Q) should therefore be low, making it difficult for a model to determine the answer from the question alone. In other words, answer generation generally needs to rely on the audio content, which limits the feasibility of a “guess first, add supporting evidence later” shortcut.

Reviewer_TkBpOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed AudioTrace, a task of long auido question answering. This dataset contains 120 long recordings, total 99.8 hours, with 1033 question pairs. This dataset is a good contribution to the audio society.

Critical strengths
  • The long audio dataset is a good contribution to the audio socity.

  • Code and dataset are released to the public.

Critical weaknesses
  • Better to put "Benchmark" in title. So that readers know this is a benchmark paper.

  • There is a lack of definition of what MCQ is.

  • One limitation of this paper is that only a benchmark is proposed. The total contribution is about dataset and benchmark. There are no new methods and algorithms.

Questions for authors
  • See critical weaknesses.
1 comment
AuthorsOpenReview ↗

Answer to questions

We thank the reviewer for the positive assessment of the AudioTrace dataset, the release of the code and data, and for pointing out the issues concerning the title, the definition of MCQ, and the methodological contribution. We agree that these suggestions will help readers better understand the scope and positioning of the paper, and we will make the corresponding revisions.

Adding “Benchmark” to the title

We agree that explicitly adding “Benchmark” to the title would help readers immediately identify the nature of the paper. The primary goal of AudioTrace is to establish a systematic evaluation benchmark for long-audio question answering, rather than to introduce a new model or reasoning algorithm. We will therefore add “Benchmark” to the title in the revised manuscript and clarify this positioning in the abstract and introduction.

Definition of MCQ

MCQ stands for multiple-choice question. Each MCQ provides a question and several candidate options, and the model must select the correct answer. In AudioTrace, an MCQ also requires the model to provide audio evidence supporting its answer, including the corresponding time interval and event content. Thus, this task evaluates both answer selection and the ability to locate answer-relevant evidence in long audio. We will define the full term at the first occurrence of MCQ in the revised manuscript.

Regarding the benchmark-only contribution and the lack of a new method

We agree that the primary contributions of this work are the dataset, task definition, and evaluation benchmark, rather than a new model architecture or training algorithm. AudioTrace is designed to first expose a problem in long-audio question answering that has not previously been measured systematically: a model may produce a correct answer without being able to locate and describe the audio evidence that genuinely supports it. Our experiments show a substantial gap between answer accuracy and joint answer–evidence accuracy, indicating that reporting answer accuracy alone can hide important shortcomings in evidence localization and verifiable reasoning for long-audio models.

We believe that proposing a targeted model method before this problem has been clearly defined and quantified would lack a reliable diagnostic standard. AudioTrace therefore provides a reproducible data-construction pipeline, timestamped evidence annotations, a joint answer–evidence evaluation protocol, and benchmark results covering both end-to-end and cascade reasoning paradigms. The code and data are publicly released to facilitate fair comparison and error analysis in future work.

We will also make this scope and limitation explicit in the revised manuscript. Future work can build on the answer–evidence gap revealed by AudioTrace to study evidence-aware training, joint answer–evidence decoding, retrieval and verification based on audio events, and reasoning mechanisms that detect insufficient evidence during generation. In other words, this paper first establishes the problem, the data, and the measurement protocol, providing a foundation for future methods that address the problem directly. We will clarify this research direction in the discussion and limitations sections of the revised manuscript.

Reviewer_je2FOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AudioTrace is a benchmark for answering questions over long recordings while supplying supporting temporal intervals and event descriptions. It contains 120 Chinese and English recordings lasting 30-90 minutes, approximately 99.8 hours in total, and 1,033 paired multiple-choice/open-ended questions, including 157 unanswerable questions. Automated annotation and question construction are followed by evidence screening and unanimous approval from three human reviewers.

Critical strengths
  • The benchmark provides a genuinely useful resource for audio evaluation by pairing long recordings with dual response formats, explicit timestamp intervals, and unanswerable test items. Building an audio-specific implementation of evidence-grounded evaluation connects well to prior work like EG-VQA. The annotation pipeline also reflects careful quality control, combining automated checks with a three-person listening review.

  • In Appendix D.2, keeping the L3 outputs fixed still reveals a substantial drop between answer accuracy and joint accuracy: 86.8% versus 22.1% on multiple-choice questions, and 43.9% versus 10.9% on open-ended queries. This confirms that the observed drop is not simply an artifact of prompt differences across the L1 to L3 levels.

  • The evaluation metrics are thoroughly validated. The dual-judge decision rule achieves 96.8% binary agreement across a 500-response calibration set in Appendix C. In addition, the paper transparently highlights high text-only multiple-choice performance and clearly separates that from true source-audio reasoning.

Critical weaknesses
  • The benchmark's core evidence-acceptance metric has not been validated against human judgment. While Appendix C checks answer accuracy against human labels, Appendix D.3 admits there is no equivalent human agreement study for EG-F1. Relying on cosine similarity does not ensure factual entailment, and simply gluing transcripts to speaker attributes allows basic word overlap to hide flipped negations, incorrect emotions, or wrong speaker traits.

  • The event-matching gate is too forgiving of missing evidence. Predicting just one out of two required reference events yields an F1 score of 0.67 and can still pass if the duration threshold is met, even if the omitted event was vital to the answer. At the same time, models offering valid alternative timestamps or different event segmentations get penalized.

  • Sweeping acceptance thresholds does not fix the underlying matching errors. The sweeps in the paper only adjust the final cutoff gates, leaving the upstream rules for semantic similarity and temporal overlap untouched. Because the entire benchmark centers on evidence-backed correctness, measuring actual human false-acceptance and false-rejection rates is essential rather than an optional check.

  • Multiple-choice questions contain strong text shortcuts that bypass the audio. A text-only Qwen3.5-27B model gets 65.8% of answerable MCQs right without hearing any audio at all, climbing to over 86% on detail and aggregation questions (Table 24). This heavily weakens L1 MCQ as a metric for measuring whether a model actually listens to the recording.

  • The text-only controls fail to test post-hoc evidence hunting. The text-only baseline simply returns empty evidence, which guarantees an automatic zero on L2 and L3 items. This setup fails to test what happens when a model guesses an answer purely from the options and then uses that guess to retrieve plausible-looking timestamps.

  • The evaluations do not isolate how much of the performance gap stems from option bias. Because the text-only comparisons use a different model family rather than a direct audio ablation of the evaluated systems, it remains unclear how much of the audio models' MCQ accuracy or the large gap between MCQ and open-ended performance—is driven by option cues rather than true acoustic understanding.

  • AudioTrace’s temporal reasoning, contextual understanding, and open-ended QA substantially overlap MMAU-Pro. Beyond longer recordings, its main addition is explicit evidence annotation and joint answer–evidence scoring, following an approach already established in video QA. The paper should more clearly demonstrate what new audio-specific insights this evaluation enables.

Questions for authors
  • On a stratified sample of model evidence, how often do tIoU/EG-F1 acceptance decisions agree with human judgments that the evidence is factually correct and sufficient for the answer? Please include negation, paralinguistic errors, omitted essential events, and valid alternative evidence.

  • How do acceptance and model rankings change when pairwise semantic eligibility is varied or replaced with fact-aware matching, and when event segmentation or speaker-attribute inclusion changes? The current final-threshold sweep leaves these matching decisions fixed.

  • For models that can be evaluated both with and without audio, what happens on identical questions and options? Can you report results on an option-rewritten or independently screened subset and test a select-answer-then-retrieve baseline?

  • What proportions of answerable questions require multiple supporting events or widely separated spans? How do full-recording results compare with oracle-evidence inputs? This would separate long-context retrieval failure from local evidence-description and scoring failures.

6 comments
AuthorsOpenReview ↗

Answer to Q1

Thanks for the careful evaluation of AudioTrace and for the detailed feedback. We are pleased that the reviewer recognized the value of the benchmark for long-audio understanding, joint answer–evidence evaluation, unanswerable-question construction, and data-quality control. The comments focus on the human validity of the evidence-acceptance metric, the sensitivity of the matching rules, textual shortcuts in multiple-choice questions, post-answer evidence retrieval, and the effects of long time spans and multi-event evidence. These issues are directly related to the central evaluation goal of AudioTrace.

Question 1: Agreement between automatic evidence acceptance and human judgment

We use tIoU > 0.5 as the temporal-matching criterion. tIoU has a clear geometric interpretation: this threshold requires the intersection between the predicted and reference intervals to account for more than half of their union tIoU>0.5\mathrm{tIoU} > 0.5. It is therefore a relatively permissive but directly interpretable standard for temporal localization and generally does not require additional manual correction. We also set the semantic-matching and set-level EG-F1 acceptance thresholds to strictly > 0.5. This is deliberately permissive for content matching: it accepts descriptions with different wording but similar meaning, and may also accept partially matching evidence. Even under these relatively easy-to-satisfy conditions, many answer-correct predictions still fail to satisfy both the temporal and content requirements. Thus, the substantial drop from L1 to L3 is unlikely to be explained by an overly strict evidence criterion; instead, it reflects the difficulty of explicitly locating and describing supporting events in long audio.

To examine whether this conclusion depends on the scoring method, we additionally used DeepSeek-V4-Flash to review all 35,122 L3 evidence predictions. The review followed a unified rubric that separately evaluates content correctness, evidence completeness, and evidence precision. A prediction is counted as accepted only when all three dimensions receive a score of 5. The overall acceptance rate was 12.87%. We also report model-level results under both the original EG-F1 metric and the DeepSeek judge. Specifically, we computed evidence quality separately for all 17 configurations and compared their relative rankings. The Spearman correlation between the two model rankings is:

ρ=0.76\rho=0.76

This model-level result shows that, although the two scoring methods may disagree on individual examples, the rubric-based judge and the original EG-F1 metric produce broadly consistent relative rankings of evidence quality.

Model-level acceptance rates for all 17 configurations

The table directly compares the acceptance rates of the two evidence metrics. An EG-F1 acceptance means that the evidence-set EG-F1 is strictly greater than 0.5. A DeepSeek acceptance means that content correctness, completeness, and precision all receive a score of 5.

ConfigurationEG-F1 > 0.5 (MCQ)DeepSeek 5/5/5 (MCQ)EG-F1 > 0.5 (Open-ended)DeepSeek 5/5/5 (Open-ended)
E2E_FireRed Audio40.30%19.94%23.74%12.78%
E2E_Gemini 3.1 Pro12.79%20.23%6.05%13.94%
E2E_Gemini 3.6 Flash29.79%32.14%17.01%19.75%
E2E_Qwen3.5 Omni Plus45.21%15.88%31.05%14.04%
E2E_Qwen3.8 Omni Flash67.81%37.37%54.91%29.62%
Cascade_Audio Flamingo5.82%6.49%5.02%5.32%
Cascade_FireRed Audio45.09%16.75%34.36%10.94%
Cascade_Gemini 3.1 Pro29.45%11.23%23.17%6.00%
Cascade_Gemini 3.6 Flash30.82%15.39%21.35%8.91%
Cascade_Kimi Audio3.77%12.00%2.51%9.87%
Cascade_MiMo Audio14.95%10.94%11.99%7.74%
Cascade_MiniCPM-o10.50%7.07%9.02%5.42%
Cascade_MOSS Audio13.70%8.52%11.87%5.52%
Cascade_Qwen3.5 Omni Plus17.58%12.20%14.27%8.81%
Cascade_Qwen3.8 Omni Flash27.63%14.04%20.78%9.29%
Cascade_Qwen3 Omni2.85%11.04%2.51%7.84%
Cascade_Step Audio26.51%5.81%5.25%4.74%

The full scoring rubric will be provided in the supplementary material.

AuthorsOpenReview ↗

Answer to Q2 part 1

Question 2: Sensitivity to matching rules

To address the reviewer’s question about sensitivity to the matching rules, we vary the semantic-matching eligibility threshold and whether speaker_attr is included, while keeping the temporal IoU threshold fixed at >0.5, together with the event segmentation and all other EG-F1 rules. Acceptance is defined as a per-example EG-F1 strictly greater than 0.5; rank correlations are computed over the 17 configurations relative to the ranking at the corresponding official threshold 0.5.

Per-configuration threshold sweep with speaker_attr included (MCQ)

ConfigurationThreshold 0.1Threshold 0.3Threshold 0.5Threshold 0.7Threshold 0.9
E2E_FireRed Audio40.53%40.53%40.30%35.62%10.39%
E2E_Gemini 3.1 Pro13.47%13.47%12.79%11.19%1.37%
E2E_Gemini 3.6 Flash30.14%30.14%29.79%26.03%7.19%
E2E_Qwen3.5 Omni Plus46.00%45.78%45.09%38.58%10.39%
E2E_Qwen3.8 Omni Flash67.69%67.69%67.58%63.93%25.57%
Cascade_Audio Flamingo5.82%5.82%5.71%4.68%1.37%
Cascade_FireRed Audio45.55%45.55%45.09%39.95%9.59%
Cascade_Gemini 3.1 Pro29.68%29.68%29.34%25.46%5.48%
Cascade_Gemini 3.6 Flash31.28%31.28%30.82%26.14%7.65%
Cascade_Kimi Audio3.88%3.88%3.77%2.51%0.80%
Cascade_MiMo Audio15.07%15.07%14.84%12.79%3.88%
Cascade_MiniCPM-o10.16%10.16%10.16%8.68%2.74%
Cascade_MOSS Audio13.58%13.58%13.58%11.53%2.63%
Cascade_Qwen3.5 Omni Plus17.58%17.58%17.47%15.41%4.11%
Cascade_Qwen3.8 Omni Flash27.63%27.63%27.51%24.09%5.94%
Cascade_Qwen3 Omni2.74%2.74%2.74%2.28%0.34%
Cascade_Step Audio26.39%6.39%6.39%5.59%1.26%

Per-configuration threshold sweep with speaker_attr included (Open-ended)

ConfigurationThreshold 0.1Threshold 0.3Threshold 0.5Threshold 0.7Threshold 0.9
E2E_FireRed Audio23.86%23.86%23.74%20.09%5.82%
E2E_Gemini 3.1 Pro6.62%6.62%6.05%4.22%0.57%
E2E_Gemini 3.6 Flash17.24%17.24%17.01%14.84%5.82%
E2E_Qwen3.5 Omni Plus32.31%32.31%31.05%27.40%8.79%
E2E_Qwen3.8 Omni Flash54.91%54.91%54.79%51.71%20.89%
Cascade_Audio Flamingo5.14%5.14%4.91%4.11%1.26%
Cascade_FireRed Audio34.82%34.82%34.36%30.02%7.65%
Cascade_Gemini 3.1 Pro23.29%23.29%23.17%20.21%4.45%
Cascade_Gemini 3.6 Flash21.46%21.46%21.35%17.81%5.94%
Cascade_Kimi Audio2.51%2.51%2.51%1.83%0.80%
Cascade_MiMo Audio11.99%11.99%11.87%10.27%3.31%
Cascade_MiniCPM-o8.68%8.68%8.68%7.76%2.40%
Cascade_MOSS Audio11.76%11.76%11.76%9.70%1.94%
Cascade_Qwen3.5 Omni Plus14.27%14.27%14.16%12.44%3.65%
Cascade_Qwen3.8 Omni Flash20.78%20.78%20.66%18.15%4.68%
Cascade_Qwen3 Omni2.40%2.40%2.40%2.17%0.46%
Cascade_Step Audio25.25%5.25%5.14%4.11%0.80%

Overall acceptance and ranking stability with speaker_attr included

The table summarizes the mean acceptance rate across the 17 configurations and the Spearman correlation between the model ranking at each threshold and the ranking at the official threshold 0.5.

Semantic thresholdMean MCQ acceptanceMCQ rank ρMean open-ended acceptanceOpen-ended rank ρ
0.123.95%0.99917.49%1.000
0.323.94%0.99917.49%1.000
0.5 (official)23.70%1.00017.27%1.000
0.720.85%0.99915.11%0.992
0.95.92%0.9794.66%0.950

Increasing the threshold lowers the absolute acceptance rate, especially under the strict 0.9 condition. However, model rankings are nearly unchanged over 0.1–0.7, and remain highly correlated even at 0.9. Thus, semantic eligibility affects the absolute scale of acceptance but has limited impact on relative model performance.

AuthorsOpenReview ↗

Answer to Q2 part2

Comparison at the official threshold with and without speaker_attr

The table directly compares per-configuration acceptance rates at semantic threshold 0.5. When speaker_attr is excluded, semantic similarity is recomputed from the speech transcript alone; all other matching conditions remain unchanged.

ConfigurationMCQ: includedMCQ: excludedOpen-ended: includedOpen-ended: excluded
E2E_FireRed Audio40.30%36.11%23.74%21.01%
E2E_Gemini 3.1 Pro12.79%19.75%6.05%12.39%
E2E_Gemini 3.6 Flash29.79%33.49%17.01%20.43%
E2E_Qwen3.5 Omni Plus45.09%39.21%31.05%27.49%
E2E_Qwen3.8 Omni Flash67.58%57.02%54.79%47.05%
Cascade_Audio Flamingo5.71%8.62%4.91%7.26%
Cascade_FireRed Audio45.09%39.01%34.36%29.14%
Cascade_Gemini 3.1 Pro29.34%25.17%23.17%18.10%
Cascade_Gemini 3.6 Flash30.82%26.52%21.35%17.91%
Cascade_Kimi Audio3.77%12.68%2.51%10.07%
Cascade_MiMo Audio14.84%11.91%11.87%8.52%
Cascade_MiniCPM-o10.16%8.03%8.68%5.52%
Cascade_MOSS Audio13.58%9.29%11.76%6.49%
Cascade_Qwen3.5 Omni Plus17.47%12.39%14.16%9.87%
Cascade_Qwen3.8 Omni Flash27.51%20.23%20.66%13.94%
Cascade_Qwen3 Omni2.74%6.29%2.40%3.97%
Cascade_Step Audio26.39%4.65%5.14%3.10%

The speaker_attr-excluded condition uses the same temporal matching and EG-F1 decision procedure as the official evaluation and is compared directly with the original speaker_attr-included scores. The Spearman correlations between the two model rankings are MCQ $\rho=0.909$ and open-ended $\rho=0.873$. Thus, removing speaker_attr changes the absolute acceptance rate for some configurations, but the relative model performance remains substantially consistent.

We retain the speaker_attr-included setting as the official protocol because L3 explicitly requires the model to output both spoken content and audible speaker attributes. speaker_attr is itself a required speaker-evidence dimension. Including it in semantic matching therefore evaluates whether the model satisfies the full L3 output contract, rather than only matching the transcript and time interval.

Fact-aware matching rules and alternative event-segmentation strategies require additional event-relation annotations or a reconstructed matching protocol. We did not change these two protocol components in the current analysis; they remain directions for further ablation.

Comments on the paper

12 comments

Does this appear to be a response to another paper? Please check it.

Reviewer_je2FOpenReview ↗

Thanks

Apologies for the cross review. It got swapped amid multiple reviews. I am still going through the authors' rebuttal and will share my feedback soon.

Reviewer_je2FOpenReview ↗

Response to rebuttal

First of all, I thank the authors for such detailed rebuttal, it addresses few of my concerns - semantic-threshold sweep, speaker-attribute ablation, and multi-event statistics. Some of my previous concerns still remain unresolved:

  1. Human validation: The provided llm-as-a-judge evaluation using DeepSeek is still an automated cross-check and cannot validate against human judgements. Can the authors make human listeners assess a sample covering negotiations, speaker errors, omitted events and validated alternate evidence? Also can the authors report exactly what were the inputs to the DeepSeek judge?

  2. Shortcut controls: Can the authors provide within model audio/no-audio ablations? Passing the gold answer is informative but is different from fixing a model's predicted answer (text-only) and subsequently retrieving audio evidence for it. It is not inherently invalid to perform answer-first retrieval; the question is whether the resulting evidence actually supports the prediction.

  3. Oracle-evidence diagnostic: As I understand it, lower scores of the longer span questions do not isolate the local-description errors or the difficulty from event complexity. Can the authors evaluate a small single event subset using these clips with original time offsets, or score answers and local description correctness separately? It is intentional to remove retrieval in this ablation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.