- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MISHAP-Bench, a benchmark designed to measure hallucinations in large audio-language models (LALMs). Its evaluation focuses on whether generated claims are supported by the input audio or by externally verifiable evidence, rather than on answer correctness alone. The benchmark includes 3,000 audio clips from music, sound, and speech, as well as 12,000 open-ended questions covering two categories: context hallucination and knowledge hallucination. To assess model responses, the authors develop a groundedness judge that combines question-specific reference rubrics, human annotations, category-specific prompts, and web-based evidence retrieval. They evaluate ten LALMs and four inference-time hallucination mitigation methods. The results indicate that current models still hallucinate frequently: even Gemini 3.7 Flash exhibits a hallucination rate of 36.5%. Among the mitigation approaches, AAD achieves the most consistent improvements, although its effectiveness varies across models, audio domains, and hallucination types. Overall, MISHAP-Bench provides a relevant and potentially useful resource for studying the reliability of LALMs. However, the benchmark contains an uneven distribution of audio clip durations, which may affect the comparability of results across samples and the reliability of the evaluation. The role and processing procedure of the 1,000 human-annotated responses are not described in sufficient detail. The paper explains that these annotations are used to refine the reference rubrics and judge prompts, select in-context examples, and assess judge-human agreement, but it does not clearly specify how the annotated data are separated between prompt development, example selection, and independent evaluation. As a result, the reported agreement between the groundedness judge and human annotators may be affected by a potential calibration-evaluation overlap, and the extent to which the judge generalizes to the remaining samples is difficult to assess.
Critical strengths
-
The paper's main contribution is decoupling answer correctness from whether a response's claims are actually supported by the audio or external evidence. This is well-motivated for open-ended audio QA, where a partially-correct answer may carry unsupported details, and where refusing a false premise or expressing uncertainty is the reliable behavior. The shift closely matches what users actually care about in LALM deployment, and is especially apt given the heterogeneity of audio evidence.
-
Separating audio-grounded hallucination from audio-related external-knowledge hallucination explains where failures originate. The authors show the two metrics do not always move together, evidencing that a single hallucination rate masks distinct behavioral modes.
-
Per-clip generated questions with three context strategies and four knowledge strategies probe premise acceptance, assertion of unknowables, and appropriate abstention. Treating premise-correction and uncertainty-expression as grounded is a valuable correction to correctness-based scoring.
-
The paper not only includes a benchmark but also compares existing hallucination mitigation methods, thus expanding its academic value. The results expose trade-offs: higher context hallucination on music and sound, better context grounding but persistent knowledge hallucination on speech, and AAD generally more robust than VCD/MTI/DoLa yet still degrading in places.
Critical weaknesses
-
The role and processing procedure of the 1,000 human-annotated responses are not described in sufficient detail. The paper explains that these annotations are used to refine the reference rubrics and judge prompts, select in-context examples, and assess judge–human agreement, but it does not clearly specify how the annotated data are separated between prompt development, example selection, and independent evaluation. As a result, the reported agreement between the groundedness judge and human annotators may be inflated by a potential calibration–evaluation overlap, and the extent to which the judge generalizes to the remaining samples is difficult to assess.
-
The clips come from six existing datasets and inherit their domain, recording-condition, and language biases. Because four questions share the same audio and description, the effective independent sample size is closer to 3,000 than 12,000. The dataset's actual representativeness is therefore lower than claimed in the paper.
-
The duration distribution in the music domain is unbalanced and inconsistent with the other two domains. Unlike speech and sound, whose clip lengths are comparatively reasonable, the music clips concentrate on extremely short segments (under 1 s) and ultra-long segments (around 2 min), omitting the 10–30 s range. This skew makes cross-domain comparison unreliable, since a model's music hallucination rate may be driven by degenerate very-short or very-long inputs rather than by genuine difficulty.
-
The hallucination-mitigation experiments read more like an supplementary material than a native contribution of the benchmark.
Questions for authors
-
Why is the duration distribution in the music domain so unbalanced compared to speech and sound?
-
Could you explain in detail the role that the hallucination-mitigation experiments play in supporting the paper's main contribution?