- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AudioTrace, a benchmark for evidence-grounded question answering over long audio. It contains 120 recordings (99.8 hours) and 1,033 question pairs in Chinese and English. The main contribution is a three-level evaluation protocol that measures answer correctness (L1), temporal evidence localization (L2), and semantic evidence grounding (L3). The dataset is built using automated annotation and human verification. Experiments across 17 model configurations show a large gap between answer accuracy and evidence-grounded accuracy, suggesting that correctly answering a question does not necessarily mean the model can identify the supporting audio evidence.
Critical strengths
1. A useful benchmark for evaluating supporting evidence in long-audio QA.
The paper addresses an important limitation of answer-only evaluation. By providing timestamps and event descriptions for supporting evidence, AudioTrace allows a more detailed assessment of model performance. The inclusion of both multiple-choice and open-ended questions is also useful.
2. Careful dataset construction and extensive evaluation.
The annotation pipeline combines automated checks with independent human review. The authors also evaluate a broad range of models and include controls for question-only shortcuts and evaluation thresholds. These efforts strengthen the empirical value of the benchmark.
Critical weaknesses
1. Limited novelty and insufficient differentiation from prior work.
The central idea of evaluating answers together with supporting evidence is not new. EG-VQA (Huang et al., 2026) already evaluates answer correctness, temporal localization, and semantic evidence matching, and AudioTrace directly adopts its EG-F1 metric. Within the audio domain, AudioSpan (Huang et al., 2026), MuLA-Bench (Yang et al., 2026), TAG-Bench (Dai et al., 2026), Vox-Infinity (Cheng et al., 2026), and LongAudio-RAG (Vakada et al., 2026) have already explored long-audio understanding, temporal grounding, and evidence-based QA.
The paper also overlooks relevant audio hallucination studies, including Kuan et al. (Interspeech 2024), Kuan and Lee (ICASSP 2025; Interspeech 2025), and AHa-Bench (Cheng et al., NeurIPS 2025). These works have already investigated failures in factual audio understanding, including sound existence, attributes, and temporal relationships.
Overall, the paper mainly combines existing evaluation ideas into a new dataset. Its main finding that correct answers do not guarantee correct evidence is also well established. It is unclear what substantially new insights AudioTrace provides beyond these prior studies.
2. The proposed evaluation does not fully validate the paper's central claim.
The paper argues that correct answers may result from shortcuts rather than genuine audio understanding. However, L3 only checks whether the model can produce evidence matching the reference. A model could guess the correct answer first and then retrieve supporting evidence, without actually using that evidence to derive its answer. Therefore, passing L3 does not establish faithful reasoning.
Moreover, semantic evidence correctness is determined using embedding similarity with a threshold of 0.5, without direct human validation of this metric. Semantically similar but factually contradictory statements may still match. Since evidence correctness is the central contribution of the benchmark, this raises concerns about whether L3 reliably measures what the paper claims.
References
[1] Huang et al. (2026). EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence. arXiv:2606.24797.
[2] Huang et al. (2026). AudioSpan: Spanning the Duration and Depth of Audio Comprehension. arXiv:2608.26431.
[3] Yang et al. (2026). MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing. arXiv:2609.23416.
[4] Dai et al. (2026). TAG-Bench: Benchmarking Temporal Audio Grounding in Large Audio Language Models. arXiv:2609.01542.
[5] Cheng et al. (2026). Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models. arXiv:2609.22452.
[6] Vakada et al. (2026). LongAudio-RAG: Event-Grounded Question Answering over Multi-Hour Long Audio. arXiv:2602.14612.
[7] Kuan, C.-Y., Huang, W.-P., and Lee, H.-y. (2024). Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models. Interspeech 2024.
[8] Kuan, C.-Y., and Lee, H.-y. (2025). Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning. ICASSP 2025.
[9] Kuan, C.-Y., and Lee, H.-y. (2025). Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples. Interspeech 2025.
[10] Cheng et al. (2025). AHa-Bench: Benchmarking Audio Hallucinations in Large Audio-Language Models. NeurIPS 2025.
Questions for authors
-
Novelty: Beyond applying existing evidence-grounding metrics to long audio, what new failure modes or findings does AudioTrace reveal that cannot be studied using prior benchmarks such as EG-VQA, AudioSpan, and MuLA-Bench?
-
Evaluation validity: Have the authors validated EG-F1 against human judgments of factual evidence correctness? In particular, how does the metric handle evidence that is semantically similar but factually contradictory, or evidence that is only partially sufficient to support an answer?
-
Faithful reasoning: How can the proposed evaluation distinguish a model that genuinely reasons from audio evidence from one that first predicts an answer and then finds plausible supporting evidence? This distinction seems important given the paper's central motivation.
6 comments
Answer to Q2
2. Evaluation validity
The semantic similarity of event pairs and the set-level EG-F1 acceptance threshold are both set to strictly > 0.5. This is deliberately permissive for evidence matching: it tends to accept descriptions with different wording but similar meaning, and may also accept partially matching evidence. Even under these relatively easy-to-satisfy conditions, many answer-correct predictions still fail to satisfy the temporal and content requirements. Thus, the substantial drop from L1 to L3 is unlikely to be explained by an overly strict evidence criterion; it reflects the difficulty of explicitly locating and describing supporting events in long audio.
To further examine whether this finding depends on the scoring method, we used DeepSeek-V4-Flash to review all 35,122 L3 evidence predictions. The review follows a unified rubric that separately evaluates content correctness, evidence completeness, and evidence precision. A prediction is counted as accepted only when all three dimensions receive a score of 5. The overall acceptance rate is 12.87%. We also report model-level results under the original EG-F1 metric and the DeepSeek judge. Specifically, we compute evidence quality separately for all 17 configurations and compare their relative rankings. The Spearman correlation between the two model rankings is:
Although the two scoring methods may disagree on individual examples, this model-level result shows that the rubric-based judge and the original EG-F1 metric produce broadly consistent relative rankings of model evidence quality.
Model-level acceptance rates for all 17 configurations
The table directly compares the acceptance rates of the two evidence metrics. An EG-F1 acceptance means that the evidence-set EG-F1 is strictly greater than 0.5. A DeepSeek acceptance means that content correctness, completeness, and precision all receive a score of 5.
| Configuration | EG-F1 > 0.5 (MCQ) | DeepSeek 5/5/5 (MCQ) | EG-F1 > 0.5 (Open-ended) | DeepSeek 5/5/5 (Open-ended) |
|---|---|---|---|---|
| E2E_FireRed Audio | 40.30% | 19.94% | 23.74% | 12.78% |
| E2E_Gemini 3.1 Pro | 12.79% | 20.23% | 6.05% | 13.94% |
| E2E_Gemini 3.6 Flash | 29.79% | 32.14% | 17.01% | 19.75% |
| E2E_Qwen3.5 Omni Plus | 45.21% | 15.88% | 31.05% | 14.04% |
| E2E_Qwen3.8 Omni Flash | 67.81% | 37.37% | 54.91% | 29.62% |
| Cascade_Audio Flamingo | 5.82% | 6.49% | 5.02% | 5.32% |
| Cascade_FireRed Audio | 45.09% | 16.75% | 34.36% | 10.94% |
| Cascade_Gemini 3.1 Pro | 29.45% | 11.23% | 23.17% | 6.00% |
| Cascade_Gemini 3.6 Flash | 30.82% | 15.39% | 21.35% | 8.91% |
| Cascade_Kimi Audio | 3.77% | 12.00% | 2.51% | 9.87% |
| Cascade_MiMo Audio | 14.95% | 10.94% | 11.99% | 7.74% |
| Cascade_MiniCPM-o | 10.50% | 7.07% | 9.02% | 5.42% |
| Cascade_MOSS Audio | 13.70% | 8.52% | 11.87% | 5.52% |
| Cascade_Qwen3.5 Omni Plus | 17.58% | 12.20% | 14.27% | 8.81% |
| Cascade_Qwen3.8 Omni Flash | 27.63% | 14.04% | 20.78% | 9.29% |
| Cascade_Qwen3 Omni | 2.85% | 11.04% | 2.51% | 7.84% |
| Cascade_Step Audio2 | 6.51% | 5.81% | 5.25% | 4.74% |
The full scoring rubric will be provided in the supplementary material.
Answer to Q3
3. Reasoning faithfulness
We believe the most appropriate faithfulness criterion requires a model to provide both a correct answer and reliable audio evidence for every question. If both outputs are correct, it is reasonable whether the model first forms an answer and then searches for supporting evidence, or first locates evidence and then answers from it. Our concern is that a model may guess the answer using only the question, the options, or other surface cues while failing to provide reliable evidence supporting that answer; such shortcut-based behavior is likely to fail on a broader set of questions. We therefore define reasoning faithfulness as joint answer–evidence correctness:
where is the question, is the answer, and is the selected evidence. This joint probability can be decomposed according to different reasoning orders. For example, a model may first form an answer and then search for supporting evidence:
Alternatively, it may first locate evidence and then answer based on that evidence:
Our evaluation does not prohibit any particular generation order. It requires the model to satisfy both answer correctness and evidence correctness, thereby attaining a high value of .
Accordingly, we do not regard “forming an answer first and then searching for evidence” as an invalid paradigm. For MCQ, the options provide additional semantic constraints, and humans may also form a candidate answer from the question and options before returning to the audio to find supporting evidence. Thus, observing answer-conditioned behavior in MCQ does not by itself show that the model failed to reason from the audio; the key issue is whether the final answer is correct and whether the evidence genuinely supports it.
For open-ended questions, the data-construction process filters out questions that can be answered from commonsense or the question text alone. Without audio evidence, should therefore be low, making it difficult for a model to determine the answer from the question alone. In other words, answer generation generally needs to rely on the audio content, which limits the feasibility of a “guess first, add supporting evidence later” shortcut.
Answer to Q1
We thank the reviewer for the careful attention to AudioTrace’s novelty, the validity of its evidence evaluation, and reasoning faithfulness. These issues are directly related to the central motivation of the paper. In the revision, we add evidence-evaluation experiments and an independent rubric-based LLM judge to examine these concerns further.
1. Novelty
AudioTrace is not intended to reinvent a particular evidence-matching metric. Its novelty is to propose and systematically measure joint answer–evidence ability in long-audio question answering. For a question Q, the model must provide both an answer A and the supporting time interval and factual evidence E. Our target is therefore:
P(A,E∣Q).
This distinguishes AudioTrace from tasks that evaluate answer correctness alone, temporal localization alone, or retrieval of isolated audio segments. EG-VQA primarily studies video question answering. AudioSpan, MuLA-Bench, and related works involve long-audio question answering or temporal localization, but they do not require jointly verifying the answer and its supporting evidence for the same long-audio question, nor do they systematically evaluate the target quantity P(A,E∣Q). Consequently, they cannot reveal the failure mode we observe: even with the same fixed set of L3 outputs, answer accuracy is 86.8% for MCQ and 43.9% for open-ended questions, whereas only 22.1% and 10.9%, respectively, provide acceptable supporting evidence together with the correct answer.
This is not an existing audio-segment retrieval task applied to longer recordings. It requires the model to answer the same long-audio question while performing global temporal localization and evidence description. The gap above is a substantial capability gap measured directly in long-audio question answering; it cannot be inferred from video evaluation results or from independent temporal-localization performance.
In addition, we developed and fully released the first batch-generation pipeline of its kind for long-audio question answering. The pipeline generates questions from structured audio events, links each question to real timestamped supporting events, and produces auditable annotations through automatic screening and independent listening-based review. The construction code, prompts, and runtime configurations are publicly available, making the data-construction process reproducible.