- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates visually induced acoustic hallucinations in omni-modal LLMs. It introduces Audite-CapsBench, a diagnostic benchmark covering subtitle-induced substitution, truncation-induced completion, event misattribution, and speaker misattribution, together with corresponding pipelines for constructing targeted training data. The authors train Audite-Omni-4B through audio alignment, multimodal mid-training, supervised fine-tuning, and task-specialized GRPO followed by multi-teacher on-policy distillation. The resulting model achieves substantial improvements on the proposed benchmark and strong performance across public audio-visual benchmarks. Ablation experiments are conducted to examine their claims, especially the central assertion that introducing grounding supervision during mid-training is particularly effective because this stage shapes cross-modal dependence, whereas later SFT has limited ability to reverse established dependence.
Critical strengths
- The paper addresses an important failure mode: fluent and visually plausible descriptions can contain acoustic claims unsupported by the input. The four diagnostic tasks make distinct grounding failures concrete and measurable. The speech-related tasks are particularly useful for testing whether models preserve the spoken language and respect the boundaries of the observed utterance. Evaluating these failures within a comprehensive captioning task makes the benchmark relevant to free-form generation.
- The targeted data pipelines are well motivated and form a coherent response to the identified errors. Separating auditory and visual annotation is a useful way to avoid reinforcing cross-modal shortcuts in training labels. The instruction-nesting results in Appendix C.3 provide additional practical insight into transferring behavior learned from focused ASR supervision to comprehensive caption generation.
Critical weaknesses
-
The training-stage claims and attribution of gains need stronger support. SFT-only targeted supervision improves Audite-CapsBench from 46.5 to 77.8, while supervision at both stages reaches 81.7. These results support sustained supervision but do not establish that mid-training intervention is “profoundly more effective” or that SFT has limited corrective capacity. The authors should clarify targeted-data exposure across conditions and distinguish the proposed stage-dependent mechanism from ordinary forgetting when supervision is removed. The final model’s broad gains should also be separated from those attributable to the grounding intervention: the controlled ablation improves the eight-benchmark average from 56.54 to 57.69, whereas the final results reflect the full training pipeline. Uncertainty estimates would help assess smaller differences between conditions.
-
The benchmark needs clearer error decomposition and validation of its automated judgments. Aggregate scores combine hallucination, omission, and instruction-following failures. For example, correctly identifying subtitles but omitting speech transcription differs from falsely presenting subtitles as heard speech. Reporting source-attribution and coverage diagnostics, error distributions, and separate auditory results would clarify what improves. The interpretation of SM should also reflect its binary on-screen versus non-visual source distinction. Finally, a representative human audit of generated captions is needed, particularly for conflicting source claims. In Figure 7, the authors’ caption identifies an off-screen narrator but later describes the visible man with “as he speaks,” yet receives a correct verdict, while similar wording in another model’s caption is treated as erroneous. Clarifying how such inconsistencies are resolved would strengthen confidence in the scores.
Questions for authors
-
Please report the amount and exposure of targeted supervision in each stage-injection condition, preferably in both examples and tokens, including repeated exposure. Does the comparison match targeted-data exposure as well as the overall training budget? If exposure differs, please explain how this affects the stage-dependent interpretation and qualify the conclusions accordingly. Please also provide the quantitative results and training details underlying the claim that directly fine-tuning existing open-source models yields limited changes.
-
Please provide a manageable human audit of generated captions covering the four tasks and a small set of representative models, with agreement assessed for both claim extraction and final verdicts. Separately examining internally conflicting captions, such as Figure 7, would help clarify the judging rules. Please also report the SIS source and coverage diagnostics already defined in Appendix B.1, SIS/TIC error-category distributions, auditory-only EM results, and the on-screen/off-screen composition of SM.
-
To strengthen the explanation that mid-training shapes modality dependence, could the authors provide a limited set of paired input interventions across the relevant stage-ablation checkpoints, such as removing or replacing subtitles and swapping audio while holding the visual input fixed? The key question is whether earlier supervision changes sensitivity to the actual acoustic evidence beyond the improvements obtained through SFT-only supervision. The attention-share analysis is suggestive, but Mid+/SFT− has a higher audio-attention share than Mid+/SFT+ despite lower diagnostic accuracy. The counterfactual analyses of Selvakumar et al. provide a relevant methodological reference.
-
Please sharpen the distinction from directly relevant prior work, particularly AVHBench and Selvakumar et al., and separate gains established by the targeted-grounding ablations from the success of the complete training recipe. Paired confidence intervals estimated from existing per-example outputs would help assess the principal ablation differences, with audio and visual judgments kept paired within each EM example.
-
Could the authors provide a limited comparison with an available hallucination-mitigation method, such as ACPO, on Audite-CapsBench or a clearly specified subset, to assess the relative or complementary benefits of representative interventions? Training-based comparisons should report supervision exposure and training budget, while decoding-based comparisons should report additional inference cost.