acceptodds
Under review as a conference paper at ICLR 2027

When Vision Speaks for Sound

25 views

Abstract

Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Audio-Visual Grounding, Shortcut Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_afv5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies an "audio-visual Clever Hans effect" in video omni models: they appear to ground audio but mostly infer it from visual priors. The authors propose THUD, a diagnostic protocol with three counterfactual audio interventions (Shift, Mute, Swap) covering temporal sync, audio existence, and audio-visual consistency. Applied to both open and closed frontier models (Gemini, GPT-5.5, Qwen3-Omni, etc.), THUD exposes large accuracy drops. They then apply SFT + DPO on Qwen3-Omni-30B with a 10K preference mixture (THUD + FineVideo + long-form A/V QA), gaining ~28 points on the three dimensions while staying competitive on four general video benchmarks.

Critical strengths
  1. Identifies and quantifies the Clever Hans effect. The paper clearly shows Video-LLMs rely on visual priors instead of actually verifying audio, and uses controlled interventions to measure how strong this reliance is. A useful finding for understanding where current omni models actually stand.

  2. THUD is a clean, reusable diagnostic. The three interventions (Mute, Shift, Swap) cover three orthogonal aspects of audio-visual grounding: existence, timing, and content. It works uniformly across open and closed models, making it a general tool rather than a one-off test.

  3. The alignment recipe is empirically effective. With just 10K preference pairs, it gains 28 points on the three intervention dimensions while keeping general video and audio-visual benchmark performance intact.

Critical weaknesses

W1: Technical novelty is limited. The training recipe is standard SFT + DPO on counterfactually augmented data. The real original contribution lies in the Shift/Mute/Swap intervention design and the diagnostic analysis; there is no new algorithmic component.

W2: The three interventions are not equally valuable for an omni model. Mute (full silence) can be caught by a frontend VAD at near-zero cost, and global Shift is well-handled by specialized sync models like Synchformer. Only Swap is genuinely worth routing to an LLM, especially under background music or weakly synced footage. The paper should first clarify whether each failure mode warrants an LLM, instead of directly pushing LLM robustness on all three.

W3: All interventions are global perturbations; local perturbations are missing, yet they are more common and harder to detect in practice. For example, a drumming video where the first few beats align with the drum sound but the last beat fails to actually hit the drum (partial alignment + partial misalignment); or a clip where some segments correspond to the visuals while others are overlaid with background interference or locally swapped. These are precisely the cases frontend tools cannot handle and where an LLM should add value.

W4: A training-free baseline comparison would be useful. One straightforward idea is to compress the video into a caption and feed it together with the audio into the model [1]. An alternative is to caption video and audio separately and then combine the two captions for judgment [2]. These methods are cheap and are the only feasible option for closed models.

Minor issues:

  1. Fig 5 reveals a trade-off on the Synced bucket that deserves explicit discussion in the main text. Vanilla Qwen3-Omni scores 89% while Ours drops to 40%, meaning the trained model mislabels a substantial fraction of naturally aligned samples as desynced. There is a real cost on aligned samples in the sync task itself.
  2. Minor: value labels on the Gemini bars in Fig 6(a) overlap.

References: [1] Sound-VECaps: Improving Audio Generation with Visual Enhanced Captions.

[2] Omni2Sound: Towards Unified Video-Text-to-Audio Generation

Questions for authors

Please see weaknesses.

1 comment

Thank you for taking the time to review our paper and giving constructive feedback. We appreciate the opportunity to clarify our main findings and address the points you raised.

W1. Diagnostic contribution. We use standard SFT and DPO to study which failures respond to targeted supervision. The paired diagnosis reveals asymmetric errors. Across models, descriptions of sound on muted clips are much more frequent than reports of silence on audible originals (Fig. 3). Shift reveals a separate weakness by keeping the sound semantically appropriate while changing when it occurs. These results motivate evaluating presence, timing, and consistency separately.

W2. Role of specialized tools. A simple audio-level check can detect full silence, and dedicated synchronizers such as Synchformer are well suited to global offsets. THUD asks whether an omni model given both streams uses the relevant audio evidence in its answer. Full silence is a particularly direct test. Any claim of an audible impact on a silent clip is a grounding error, even if the impact is plausible from the visuals. We will discuss Synchformer as a specialized approach to global synchronization and clarify what THUD measures when an omni model receives both audio and video.

W3. Local perturbations. We agree that a missed drumbeat within an otherwise aligned clip tests a harder form of grounding. Our clip-level edits provide controlled cases for measuring sensitivity to each type of audio change. A local version would require identifying the affected event and scoring whether the model can localize the mismatch while correctly recognizing the surrounding aligned events. We will make this scope explicit.

W4. Training-free baselines. Both proposed captioning pipelines are relevant baselines for Swap. A direct comparison would use the same held-out pairs and give the judge either a visual-only caption with raw audio or independently generated captions for the two streams. For Shift, timestamped event descriptions would be important because a single clip-level caption can omit the timing information that the test measures. We will discuss Sound-VECaps and Omni2Sound in the revision and explain how the suggested captioning pipelines could be adapted into training-free baselines for THUD Swap.

Minor issues. Fig. 5 shows a trade-off on VGGSoundSync. Accuracy improves across the shifted bands, while accuracy on aligned clips drops from 89% to 40%. We will discuss both effects explicitly in §3.3 and fix the overlapping Gemini labels in Fig. 6(a).

Reviewer_HdSnOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that video-capable MLLMs often fake audio understanding: they infer what a video should sound like from vision instead of checking the real audio track. The authors call this an audio-visual Clever Hans effect. They propose THUD, a diagnostic protocol that perturbs the audio track in three ways (Shift: temporal offset, Mute: silence, Swap: audio from another video) and evaluates seven open and closed models. All of them collapse under these interventions relative to natural controls (Avg Gap 46 to 80pp). The paper also trains Qwen3-Omni-30B with SFT warm-up followed by DPO on intervention-derived preference pairs mixed with general video data, and reports a 28pp average gain on the three intervention dimensions without losing general benchmark performance.

Critical strengths
  1. The paired-control design is well thought out. Each intervened video keeps its natural original as a control, so shortcut reliance shows up as the accuracy drop under intervention. This is the main methodological contribution.
  2. The failure analysis is detailed, with eight failure modes including mirrored errors like False Silence and Audio Dodge, plus direction-of-offset and difficulty-band breakdowns. The effect holds across seven models and partly transfers to VGGSync (36.8 to 56.4).
Critical weaknesses
  1. The novelty is mostly in combining existing pieces: the phenomenon is documented in Selvakumar et al. 2026 (arXiv:2604.20605), AVHBench (ICLR 2025) and Baid et al. 2026 (arXiv:2604.14129), which already fixes the same failure mode with DPO on audio-swap preference pairs (swap axis only). The additions here (paired measurement, three-way separability over mute/swap/shift) are useful but moderate, and the mitigation is not compared with Baid et al. even on the shared swap setting.
  2. The evaluation rests on a parsing convention that favors the paper's claims: per Appendix G.3 the judge defaults ambiguous responses to "synced" on all three tasks, so hedged responses count as failures. The Mute prompt presupposes audio, and without sample sizes, CIs or seed variance, the small Table 2 differences are not interpretable. The qualitative collapse (about 100% vs 13%) is convincing, the precise numbers are not.
Questions for authors
  1. In Appendix G.3, ambiguous responses default to "synced". What fraction of predictions went through this fallback? Please re-report Mute Hallucination and Swap False-Match with ambiguous responses excluded or counted as uncertain.
  2. Does training convert hallucination into denial? Please report the eight failure modes (Fig. 3) before and after training, especially False Silence and Audio Dodge.
  3. Please re-run the Mute evaluation with a prompt that does not presuppose audio, for instance, "What do you hear, if anything?". If the conclusion holds under this variant, the diagnosis is considerably stronger.
  4. How does your recipe compare with Baid et al. (arXiv:2604.14129) on the same backbone? A comparison on the swap dimension would help position the mitigation contribution.

You can only answer the four questions above.

1 comment

Thank you for the detailed review and for recognizing the value of the paired controls.

Q1. Ambiguous responses. Appendix G.3 allows a synced fallback in task-level Mute and Shift parsing. The two error rates in Fig. 3 use the scoring rules in Appendix H. Mute Hallucination requires an assertion that concrete sound is present on a silent clip. Visual-only responses remain in the denominator and contribute to Audio Dodge. For Swap, unclear relationship judgments are already labeled uncertain and remain in the denominator. Swap False-Match requires an affirmative claim that the sound corresponds to the scene. Under these rules, the reported rates are 39–99% for Mute Hallucination and 63–95% for Swap False-Match across the six models. The aggregate synced count pools explicit and default assignments, so quantifying the fallback rate requires per-response parsing decisions.

Q2. Denial after training. Fig. 3 reports all eight failure rates for the base models. For vanilla Qwen3-Omni, False Silence and Audio Dodge are 0% and 3%, respectively, at the precision shown in the figure. Combined Mute accuracy rises from 47.5% to 73.9% in Fig. 7. An always-silent policy would score 50% on the equally weighted original and muted conditions, so the tuned model correctly recognizes audible content on some original clips. The combined score still leaves open whether false silence or avoidance increased. Resolving that question requires applying the eight Appendix H definitions to the per-clip outputs from both checkpoints.

Q3. Mute prompt. We use identical wording for the original and muted versions of each clip. The phrase “Describe the audio you hear” can still cue a sound description. Re-running both conditions with “What do you hear, if anything?” would directly measure how much the result depends on that wording.

Q4. Comparison with Baid et al. ACPO uses audio swaps in an input- and output-contrastive preference objective. Its published backbones include Qwen2.5-Omni. On THUD, our final Qwen3-Omni-30B checkpoint improves combined Swap accuracy from 56.3% to 65.1% (Fig. 7). A direct comparison requires applying ACPO to the same Qwen3-Omni-30B backbone and evaluating both checkpoints on the same held-out THUD pairs. We will expand Section 4 to explain how both approaches use swapped audio and how THUD’s Mute and Shift interventions separately probe sound presence and timing.

Reviewer_ooe4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether video-capable multimodal language models ground their sound-related responses in the audio stream or infer plausible sounds from visual context. It introduces THUD, a diagnostic protocol with three audio interventions: temporal shifting, muting, and soundtrack swapping. Evaluations reveal substantial performance drops when natural audio-visual correspondence is disrupted. The paper also investigates an SFT–DPO training recipe that combines intervention-derived preferences with general video data to improve audio verification while retaining general video understanding.

Critical strengths
  1. The paper exposes an important failure mode: models can produce plausible sound descriptions from visual priors without verifying the audio.
  2. The paired interventions cover several model families, and the alignment recipe improves synchronization, including transfer to VGGSoundSync.
  3. The paper is well written and easy to follow, with intuitive examples and informative failure-mode analyses.
Critical weaknesses
  1. Limited methodological novelty. Shift, Mute, and Swap are straightforward operations, and the mitigation follows a standard SFT–DPO pipeline. The contribution therefore rests mainly on the empirical diagnosis and data recipe. Given the related studies of audio-visual hallucination and visual dominance discussed in Section 4, the paper should articulate more clearly which new insight its evaluation establishes. This is my main concern about the significance of the contribution.
  2. Limited application scope. The intervention-driven data recipe targets three elementary audio–visual checks: sound presence, temporal offsets, and soundtrack mismatch. Improving these tasks through explicitly constructed supervision does not by itself establish a general solution to audio–visual understanding.
  3. Additional comparison with a recent omni model. An evaluation of Qwen3.8-Omni-Flash, which supports joint audio–video input, would help assess whether the observed failure modes persist in a more recent omni model.
Questions for authors

please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.