acceptodds
Withdrawn from ICLR 2027

Sight or Sound: Do Multimodal Large Language Models Have Blind Faith in Vision for Audio-Visual Understanding?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Audio-visual understanding requires Multimodal Large Language Models (MLLMs) to reason over what is seen and heard, yet aligned benchmarks rarely reveal which stream controls an answer when the evidence conflicts. We construct the Audio-Visual Incongruity Corpus (AVIC), a question-conditioned probe that preserves questions while pairing answer-disagreeing streams and constructing source-attribution options. Across four MLLMs, outputs favor visual-source answers even under audio-focused instructions; conflict-aware prompting increases abstention but leaves substantial visual preference. An option-free evaluation filtered by isolated-source alignment on an independent dataset reproduces the visual preference. On a competence-controlled two-model subset, probes find locally decodable audio information under cross-modal conflict and prompt-dependent decision transfer; paired patching restores conflict-induced answer margins through late decision states. Modality masking confirms that auditory evidence can guide answers when visual access is blocked. Finally, an exploratory attention-intervention study on AVHBench tests early audio boosting and late visual suppression, revealing task-dependent effects and larger aggregate improvements from suppressing high-attention rather than low-attention visual tokens. The results connect a gap between local auditory information and decision influence to attention targets for improving audio-visual reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-visual understanding, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_faKQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether multimodal large language models follow the requested modality when audio and visual evidence disagree. It constructs AVIC by pairing streams from AVQA samples with matching questions but different answers. Experiments on four models show a visual-source preference, with an option-free replication on an independent dataset.

Critical strengths
  • Selecting examples that both models answer correctly from isolated streams helps separate recognition failures from failures to use available evidence. Source-disjoint probe evaluation and matched prompt controls strengthen this analysis.
Critical weaknesses
  • Poor presentation and weak logical connections. The paper moves between conflict evaluation, representation probing, activation patching, and attention intervention without sufficiently explaining how the evidence supports each successive conclusion. Changes in prompts, answer options, and evaluation subsets make these results difficult to connect. The link between late decision-state sensitivity and the proposed attention intervention is particularly underdeveloped. Small figure labels further hinder readability.
  • Limited comparisons. The intervention is compared only against VCD and AVCD. Although high- versus low-attention masking is examined, random masking and simpler modality-reweighting controls are missing. These comparisons are needed to establish whether the proposed token selection and layer choices provide a meaningful advantage. Importing some baseline scores from prior work also leaves the comparability of evaluation configurations uncertain.
  • Insufficient experimental validation. The internal analysis is restricted to 300 animal-question examples and two models, while detailed intervention ablations cover only VideoLLaMA2. Intervention effectiveness is evaluated on a single benchmark, with settings selected on that same test set. Without independent evaluation, broader task coverage, and paired uncertainty estimates, the small improvements do not convincingly establish robustness or generalizability. The compared methods in main table is also limited.
Questions for authors

See weakness.

Reviewer_BhzwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AVIC, a question-matched audio-visual conflict dataset, and finds visual-source preference across four MLLMs. Competence-controlled probes and activation patching examine auditory information and decision states. Attention interventions are further explored on AVHBench.

Critical strengths
  1. Question-matched conflicts provide a useful way to test modality preference beyond accuracy on aligned inputs.
  2. The independent option-free replication and source-disjoint, competence-controlled probes strengthen the behavioral evidence.
  3. The paper carefully distinguishes local decodability, generated answers and margin recovery, and reports prompt-dependent effects.
Critical weaknesses
  1. Different source labels do not guarantee that each isolated stream supports its assigned answer, especially after truncation.
  2. The external filter checks relative similarity rather than correctness, and the ambiguity rule cannot reject every irrelevant response. How reliable are these assignments?
  3. Late decision-state patching can copy the reference answer preference, as Appendix D acknowledges. It therefore leaves the proposed link to modality routing unresolved.
  4. Both evaluations rely on embedding-based answer assignment. The external filter checks relative similarity rather than correctness, and the ambiguity rule cannot reject every irrelevant response. How reliable are these assignments?
Questions for authors
  1. How often are both source answers supported after re-matching and truncation?

  2. Do the mechanism findings survive a shared prompt and answer format?

  3. Do intervention gains remain on a held-out test set?

Reviewer_tWDAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies which modality MLLMs follow when audio and video conflict. It builds AVIC by pairing a video from one AVQA record with audio from another record that has the same question but a different annotated answer (15,926 items). Four MLLMs prefer visual-source answers even under audio-focused prompts; prompting shifts but does not flip this. Probes on 300 competence-controlled cases find audio information still decodable in the network but is lost at late decision states. An attention-based intervention on AVHBench gives small gains.

Critical strengths
  1. The question is well-motivated: aligned benchmarks cannot tell recognition from evidence use under conflict.
  2. The question-conditioned re-matching protocol is clean (same question, known answers per modality, fixed options across conditions), and the evaluation is careful in places: MCQ plus option-free replication on an independent dataset.
Critical weaknesses
  1. The main findings are already published. Beyond-Text-Dominance (arXiv:2604.16902) uses the same metric (its MSR = VD/AD) and the same last-token probe, and already reports for image+audio conflicts the same two conclusions: visual preference ("audio MSR remains below 21%") and mid-to-late-layer emergence. MMA-Bench (arXiv:2511.22826) already tests "misaligned audio-visual pairs" with white-box analysis. These findings are presented in this paper as newly established; the contribution is therefore one of protocol refinement, not discovery.
  2. Serious writing and presentation problems. Figures are effectively illegible at print size (legends in Figs. 3–5 are unreadable), Fig. 2(right) and Fig. 6 duplicate the same distribution, number formats are inconsistent within one table (Table 9 mixes "0.798" and ".998"), model naming drifts (Video-SALMONN2+ vs Video-SALMONN2-Plus), and the abstract contains unidiomatic phrasing ("The results connect a gap"). For a paper whose contribution is a measurement protocol, this presentation quality is far below the standard expected at the conference.
Questions for authors
  1. Novelty. Beyond-Text-Dominance already reports the same metric, the same last-token probe, and the same visual-preference and mid-to-late-layer findings. Which results in this paper are actually new, and can you provide a direct comparison?
  2. AVHBench attribution. The intervention study is evaluated on AVHBench (Sung-Bin et al., ICLR 2025), which is never cited. Why, and are the borrowed Table 5 numbers (VCD/AVCD from Jung et al.) obtained under the official AVHBench protocol?
  3. Protocol contradiction. Qwen2.5-Omni chooses the audio answer 266/300 times in the two-option audio-focused test but only AD=31.81% in the four-option setting. How do you reconcile these, and what does "visual dominance" mean given this dependence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.