acceptodds
Desk rejected at ICLR 2027

Eyes Over Ears: Mitigating Visually Induced Acoustic Hallucinations in Omni-Modal LLMs

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Omni-modal LLMs jointly process language, vision, and audio. In practice, audio is incorporated only after a vision-language backbone has been trained. We find that established visual priors in such backbones can override acoustic evidence on audio-related tasks, giving rise to *Visually Induced Acoustic Hallucinations*. To diagnose and mitigate these hallucinations, we develop a diagnostic benchmark and synthesize training data targeting these failure modes. Directly fine-tuning existing open-source omni-modal LLMs on these data yields limited changes to their learned modality-dependence patterns, motivating intervention earlier in training. We therefore introduce Audite-Omni-4B, a compact model trained through progressive audio alignment, Mid-Training, and SFT. Introducing targeted supervision from Mid-Training onward proves more effective, shifting multimodal attention toward audio tokens during audio-related reasoning and substantially improving performance on our diagnostic benchmark. The improvements also extend to open-source omni QA and captioning tasks, indicating generalization beyond the targeted failure mode. We further train task-specialized GRPO experts and consolidate their respective strengths into a single model through MOPD. Audite-Omni-4B scores 89.9 on our diagnostic suite, compared with 72.7 for the strongest closed-source baseline Qwen3.8-Omni-Flash. It also surpasses Qwen3-Omni-30B-A3B on 10 of 11 public omni QA and captioning benchmarks, establishing a new open-source SOTA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-modal Models, Omni-modal LLMs, Hallucination Mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dRcHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates visually induced acoustic hallucinations in omni-modal LLMs. It introduces Audite-CapsBench, a diagnostic benchmark covering subtitle-induced substitution, truncation-induced completion, event misattribution, and speaker misattribution, together with corresponding pipelines for constructing targeted training data. The authors train Audite-Omni-4B through audio alignment, multimodal mid-training, supervised fine-tuning, and task-specialized GRPO followed by multi-teacher on-policy distillation. The resulting model achieves substantial improvements on the proposed benchmark and strong performance across public audio-visual benchmarks. Ablation experiments are conducted to examine their claims, especially the central assertion that introducing grounding supervision during mid-training is particularly effective because this stage shapes cross-modal dependence, whereas later SFT has limited ability to reverse established dependence.

Critical strengths
  • The paper addresses an important failure mode: fluent and visually plausible descriptions can contain acoustic claims unsupported by the input. The four diagnostic tasks make distinct grounding failures concrete and measurable. The speech-related tasks are particularly useful for testing whether models preserve the spoken language and respect the boundaries of the observed utterance. Evaluating these failures within a comprehensive captioning task makes the benchmark relevant to free-form generation.
  • The targeted data pipelines are well motivated and form a coherent response to the identified errors. Separating auditory and visual annotation is a useful way to avoid reinforcing cross-modal shortcuts in training labels. The instruction-nesting results in Appendix C.3 provide additional practical insight into transferring behavior learned from focused ASR supervision to comprehensive caption generation.
Critical weaknesses
  1. The training-stage claims and attribution of gains need stronger support. SFT-only targeted supervision improves Audite-CapsBench from 46.5 to 77.8, while supervision at both stages reaches 81.7. These results support sustained supervision but do not establish that mid-training intervention is “profoundly more effective” or that SFT has limited corrective capacity. The authors should clarify targeted-data exposure across conditions and distinguish the proposed stage-dependent mechanism from ordinary forgetting when supervision is removed. The final model’s broad gains should also be separated from those attributable to the grounding intervention: the controlled ablation improves the eight-benchmark average from 56.54 to 57.69, whereas the final results reflect the full training pipeline. Uncertainty estimates would help assess smaller differences between conditions.

  2. The benchmark needs clearer error decomposition and validation of its automated judgments. Aggregate scores combine hallucination, omission, and instruction-following failures. For example, correctly identifying subtitles but omitting speech transcription differs from falsely presenting subtitles as heard speech. Reporting source-attribution and coverage diagnostics, error distributions, and separate auditory results would clarify what improves. The interpretation of SM should also reflect its binary on-screen versus non-visual source distinction. Finally, a representative human audit of generated captions is needed, particularly for conflicting source claims. In Figure 7, the authors’ caption identifies an off-screen narrator but later describes the visible man with “as he speaks,” yet receives a correct verdict, while similar wording in another model’s caption is treated as erroneous. Clarifying how such inconsistencies are resolved would strengthen confidence in the scores.

Questions for authors
  1. Please report the amount and exposure of targeted supervision in each stage-injection condition, preferably in both examples and tokens, including repeated exposure. Does the comparison match targeted-data exposure as well as the overall training budget? If exposure differs, please explain how this affects the stage-dependent interpretation and qualify the conclusions accordingly. Please also provide the quantitative results and training details underlying the claim that directly fine-tuning existing open-source models yields limited changes.

  2. Please provide a manageable human audit of generated captions covering the four tasks and a small set of representative models, with agreement assessed for both claim extraction and final verdicts. Separately examining internally conflicting captions, such as Figure 7, would help clarify the judging rules. Please also report the SIS source and coverage diagnostics already defined in Appendix B.1, SIS/TIC error-category distributions, auditory-only EM results, and the on-screen/off-screen composition of SM.

  3. To strengthen the explanation that mid-training shapes modality dependence, could the authors provide a limited set of paired input interventions across the relevant stage-ablation checkpoints, such as removing or replacing subtitles and swapping audio while holding the visual input fixed? The key question is whether earlier supervision changes sensitivity to the actual acoustic evidence beyond the improvements obtained through SFT-only supervision. The attention-share analysis is suggestive, but Mid+/SFT− has a higher audio-attention share than Mid+/SFT+ despite lower diagnostic accuracy. The counterfactual analyses of Selvakumar et al. provide a relevant methodological reference.

  4. Please sharpen the distinction from directly relevant prior work, particularly AVHBench and Selvakumar et al., and separate gains established by the targeted-grounding ablations from the success of the complete training recipe. Paired confidence intervals estimated from existing per-example outputs would help assess the principal ablation differences, with audio and visual judgments kept paired within each EM example.

  5. Could the authors provide a limited comparison with an available hallucination-mitigation method, such as ACPO, on Audite-CapsBench or a clearly specified subset, to assess the relative or complementary benefits of representative interventions? Training-based comparisons should report supervision exposure and training budget, while decoding-based comparisons should report additional inference cost.

Reviewer_BpPfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates visually induced acoustic hallucination in omni-modal LLMs and proposes a new benchmark to study this problem. Specifically, the benchmark includes four tasks that can be categorized into speech-content hallucinations and audio-visual attribution hallucinations. This paper further proposes a new model Audite-Omni-4B, which is trained in multiple stages and improves in multiple benchmarks.

Critical strengths
  1. This paper proposes injecting synthesized data at a mid-training stage, which is effective.
  2. The proposed Audite-Omni-4B model improves over baselines in multiple benchmarks.
  3. The paper is well-presented and easy to follow.
Critical weaknesses

1. Overlap with previous benchmark and unclear contribution.

  • This paper highlights that it uncovers systematic visually induced acoustic hallucinations in omni-modal LLMs (Line 103). However, this visually induced acoustic hallucination is not a new term and has been widely studied in previous studies, such as AVHBench[1] and CMM benchmark[2]. This makes the paper's contribution overclaimed and not clear.

  • Regarding the task definition, event misattribution task (e.g., asking models to identify the sound of flowing river water in Figure 8(d)) has been widely studied in AVHBench[1] and CMM benchmark[2]. These studies[1,2] also proposed other tasks, such as audio-driven hallucination tasks, making them more comprehensive than this paper.

[1] AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models. ICLR 2025.

[2] The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio. NeurIPS 2025.

2. Two of four tasks in the benchmark have been solved with almost 100% accuracy, leading to limited space or impact for future work.

Table 1 shows that Audite-Omni-4B achieves very high performance in the proposed benchmark, such as 98.0% for Subtitle-Induced Substitution and 97.2% for Truncation-Induced Completion tasks. This raises concern that Audite-Omni-4B in this paper has achieved almost 100% accuracy on these two tasks, thus there is no room or need for future work to follow or improve.

3.Regarding the proposed Audite-Omni-4B model, the current manuscript version lacks important results and details about the training data.

  • This paper compares the proposed Audite-Omni-4B model with baseline models, such as Qwen2.5-Omni. However, many studies[3-6] have improved the visually induced acoustic hallucinations, which are not compared.

[3] AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding. NeurIPS 2025.

[4] MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models. CVPR 2026.

[5] When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion? AAAI 2026.

[6] Don’t Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-visual Language Models. ECCV 2026.

  • As AVHBench[1] and CMM benchmark[2] are two widely applied benchmarks that investigate visually induced acoustic hallucinations, does the proposed Audite-Omni-4B model still improve over baseline models, including [3-6]?

  • What is the source and size of the training data for Audite-Omni-4B? Does Mid-training improve over SFT-only training on the AVHBench[1] and CMM[2] benchmarks?

Questions for authors

Please refer to the weakness section.

Reviewer_dPpYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how omni-modal large language models, when jointly processing audio and video, generate content without acoustic grounding because they rely on visual or subtitle cues. The authors build Audite-CapsBench to conduct controlled diagnostics on four error types: subtitle-induced substitution, truncation-induced completion, event misattribution, and speaker misattribution. They also construct corresponding training data and examine the effects of introducing such supervision at different training stages. On this basis, they propose Audite-Omni-4B, which undergoes audio alignment, mid-training, and supervised fine-tuning, with targeted data continuously added from mid-training onward, followed by GRPO to train task experts and MOPD to integrate their capabilities into a single model. Validation covers the diagnostic benchmark, public audio-video question answering and captioning benchmarks, training-stage ablations, and attention analysis. The authors report that the 4B model outperforms the compared open-source and closed-source baselines on the diagnostic suite and exceeds Qwen3-Omni-30B-A3B on 10 of 11 public audio-video benchmarks. Continuous use of targeted data in both mid-training and supervised fine-tuning performs better than using it in only one stage, and they observe an increase in the relative attention received by audio tokens.

Critical strengths
  1. The experiments are broad and well structured. In addition to the self-built diagnostic benchmark, the authors evaluate on public audio-video question answering and captioning tasks, and compare different training-stage settings using the same model. These experiments examine both whether the target errors are reduced and whether other tasks also benefit.
  2. The empirical results are strong, particularly given the model’s 4B scale. Audite-Omni-4B performs strongly on the diagnostic suite and exceeds Qwen3-Omni-30B-A3B on 10 of 11 public audio-video benchmarks. The paper also describes the data construction and training pipeline completely, which is useful as a reference for training omni-modal models.
Critical weaknesses

The problems and ideas of visual cues causing auditory hallucinations and targeted training to mitigate modality bias have already been studied in works such as AVHBench and OmniDPO. This paper mainly adds finer-grained diagnostics, corresponding data construction, and comparisons across different training stages, and the final model performs well. These results have practical value, but methodological novelty appears relatively limited. The discussion of what diagnostics and training experiments prior work has already done is relatively brief, which also makes it difficult to judge what new insights this paper adds.

Questions for authors
  1. In Figure 4, adding targeted data only in SFT raises the overall diagnostic score from 46.5 to 77.8, and adding it in both stages gives 81.7. When these data are removed from the SFT stage, performance decreases. So what exactly does the conclusion in §5 Conclusion that “Once disproportionate modality reliance is established, late-stage SFT shows limited capacity to reverse it” refer to? To what extent do these results show that earlier intervention is needed, and to what extent do they show that such supervision needs to be retained in later training?

  2. §4.4 states that the four experimental groups control the total data budget. Is the actual amount of targeted data also matched across groups? Can the authors list the cumulative number of targeted samples or tokens used in each group, the number of times samples are reused, and their proportion in the training data? These details affect the judgment of the two-stage benefit: earlier exposure to such data and seeing more such data in total may lead to different explanations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.