- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper identifies an "audio-visual Clever Hans effect" in video omni models: they appear to ground audio but mostly infer it from visual priors. The authors propose THUD, a diagnostic protocol with three counterfactual audio interventions (Shift, Mute, Swap) covering temporal sync, audio existence, and audio-visual consistency. Applied to both open and closed frontier models (Gemini, GPT-5.5, Qwen3-Omni, etc.), THUD exposes large accuracy drops. They then apply SFT + DPO on Qwen3-Omni-30B with a 10K preference mixture (THUD + FineVideo + long-form A/V QA), gaining ~28 points on the three dimensions while staying competitive on four general video benchmarks.
Critical strengths
-
Identifies and quantifies the Clever Hans effect. The paper clearly shows Video-LLMs rely on visual priors instead of actually verifying audio, and uses controlled interventions to measure how strong this reliance is. A useful finding for understanding where current omni models actually stand.
-
THUD is a clean, reusable diagnostic. The three interventions (Mute, Shift, Swap) cover three orthogonal aspects of audio-visual grounding: existence, timing, and content. It works uniformly across open and closed models, making it a general tool rather than a one-off test.
-
The alignment recipe is empirically effective. With just 10K preference pairs, it gains 28 points on the three intervention dimensions while keeping general video and audio-visual benchmark performance intact.
Critical weaknesses
W1: Technical novelty is limited. The training recipe is standard SFT + DPO on counterfactually augmented data. The real original contribution lies in the Shift/Mute/Swap intervention design and the diagnostic analysis; there is no new algorithmic component.
W2: The three interventions are not equally valuable for an omni model. Mute (full silence) can be caught by a frontend VAD at near-zero cost, and global Shift is well-handled by specialized sync models like Synchformer. Only Swap is genuinely worth routing to an LLM, especially under background music or weakly synced footage. The paper should first clarify whether each failure mode warrants an LLM, instead of directly pushing LLM robustness on all three.
W3: All interventions are global perturbations; local perturbations are missing, yet they are more common and harder to detect in practice. For example, a drumming video where the first few beats align with the drum sound but the last beat fails to actually hit the drum (partial alignment + partial misalignment); or a clip where some segments correspond to the visuals while others are overlaid with background interference or locally swapped. These are precisely the cases frontend tools cannot handle and where an LLM should add value.
W4: A training-free baseline comparison would be useful. One straightforward idea is to compress the video into a caption and feed it together with the audio into the model [1]. An alternative is to caption video and audio separately and then combine the two captions for judgment [2]. These methods are cheap and are the only feasible option for closed models.
Minor issues:
- Fig 5 reveals a trade-off on the Synced bucket that deserves explicit discussion in the main text. Vanilla Qwen3-Omni scores 89% while Ours drops to 40%, meaning the trained model mislabels a substantial fraction of naturally aligned samples as desynced. There is a real cost on aligned samples in the sync task itself.
- Minor: value labels on the Gemini bars in Fig 6(a) overlap.
References: [1] Sound-VECaps: Improving Audio Generation with Visual Enhanced Captions.
[2] Omni2Sound: Towards Unified Video-Text-to-Audio Generation
Questions for authors
Please see weaknesses.
Thank you for taking the time to review our paper and giving constructive feedback. We appreciate the opportunity to clarify our main findings and address the points you raised.
W1. Diagnostic contribution. We use standard SFT and DPO to study which failures respond to targeted supervision. The paired diagnosis reveals asymmetric errors. Across models, descriptions of sound on muted clips are much more frequent than reports of silence on audible originals (Fig. 3). Shift reveals a separate weakness by keeping the sound semantically appropriate while changing when it occurs. These results motivate evaluating presence, timing, and consistency separately.
W2. Role of specialized tools. A simple audio-level check can detect full silence, and dedicated synchronizers such as Synchformer are well suited to global offsets. THUD asks whether an omni model given both streams uses the relevant audio evidence in its answer. Full silence is a particularly direct test. Any claim of an audible impact on a silent clip is a grounding error, even if the impact is plausible from the visuals. We will discuss Synchformer as a specialized approach to global synchronization and clarify what THUD measures when an omni model receives both audio and video.
W3. Local perturbations. We agree that a missed drumbeat within an otherwise aligned clip tests a harder form of grounding. Our clip-level edits provide controlled cases for measuring sensitivity to each type of audio change. A local version would require identifying the affected event and scoring whether the model can localize the mismatch while correctly recognizing the surrounding aligned events. We will make this scope explicit.
W4. Training-free baselines. Both proposed captioning pipelines are relevant baselines for Swap. A direct comparison would use the same held-out pairs and give the judge either a visual-only caption with raw audio or independently generated captions for the two streams. For Shift, timestamped event descriptions would be important because a single clip-level caption can omit the timing information that the test measures. We will discuss Sound-VECaps and Omni2Sound in the revision and explain how the suggested captioning pipelines could be adapted into training-free baselines for THUD Swap.
Minor issues. Fig. 5 shows a trade-off on VGGSoundSync. Accuracy improves across the shifted bands, while accuracy on aligned clips drops from 89% to 40%. We will discuss both effects explicitly in §3.3 and fix the overlapping Gemini labels in Fig. 6(a).