acceptodds
Under review as a conference paper at ICLR 2027

Intervention, Not Shared Latents: Blocking Visual Shortcuts in Audio-Video Generation

Abstract

Joint audio–video (AV) generators are trained on data in which what an event looks like and what it sounds like are spuriously correlated. We present a controlled causal study of the resulting failure mode. In an AV structural causal model where the audio is, by construction, independent of the video's nuisance appearance, models that let audio read video directly—through cross-attention or a shared latent—learn a visual shortcut: they predict sound from appearance rather than the causal event and, when the appearance–event correlation is broken at test time, synthesize the wrong event's sound. Crucially, the popular remedy of routing both modalities through a shared common-cause latent does not fix this—a bottleneck, an unsupervised shared/private factorization, and a faithful shared-prior model all grab the appearance proxy and fail like the direct model. Blocking the shortcut instead requires an intervention on the nuisance: under the stated assumptions we prove that counterfactual invariance is necessary and sufficient to identify the causal predictor, and we verify the mechanism from feature-vector SCMs to procedural pixel video, real images with spectrogram audio, moving real digits, and a conditional generator. On a real, pretrained V2A generator (MMAudio), an input-intervention test shows the model is far from invariant to sound-irrelevant edits, though a generic-noise control reveals it is broadly input-brittle rather than specifically colour-shortcutting—clean isolation of the shortcut needs the controlled confounds our synthetic studies provide. We characterize when the shortcut occurs, compare the objective against supervised counterfactual augmentation, and isolate the unknown-nuisance regime—where the intervention cannot be applied—as the central open problem.

Primary area:
causal reasoning
Keywords:
audio-video generation, visual shortcut, counterfactual invariance, structural causal model, multimodal generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_amHfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work studies spurious correlations between audio and video in joint AV generation from the perspective of shared latent representations. Under strong assumptions (e.g., conditional independence of audio from the nuisance, recoverability of the event and nuisance from video, knowledge of the nuisance, and the ability to intervene on it), the paper theoretically explains that selecting a predictor invariant to the nuisance yields a true causal predictor. Several controlled experiments validate these theoretical insights.

The main issue, as the authors repeatedly acknowledge, is that the theoretical findings rest on multiple very strong assumptions, yet the paper does not discuss how each assumption can be satisfied in actual joint AV generation, or whether satisfying them is even possible. Although an experiment using MMAudio is presented as a connection to joint AV generation, it is unclear whether the experimental setup correctly simulates the spurious correlation discussed in the paper. Given that the Gaussian noise control produces comparable results, the outputs may simply be degraded due to out-of-distribution inputs.

While the reviewer acknowledges some contribution in clarifying the problem formulation, the content is merely standard from the perspective of causal inference, and the paper has not solved a new problem. I therefore vote for rejection.

Critical strengths

Clear formulation of the visual shortcut problem in joint audio-video generation.

Critical weaknesses
  • This paper mainly focuses on clarifying spurious correlations in joint AV generation. However, the paper does not discuss whether the assumptions used in the theoretical proofs can actually be satisfied in joint AV generation. Moreover, most empirical validation uses controlled settings unrelated to AV generation, and it is unclear whether these findings can transfer to actual joint AV generation. Consequently, the practical utility of this work must be assessed as extremely limited at this stage.
  • Although the problem is presented carefully, the insights the paper provides are not particularly novel. The paper claims novelty in formulating visual shortcuts in AV generation using a structural causal model (SCM) and in showing, both theoretically and through controlled experiments, that shared latent representations are insufficient to eliminate these shortcuts. It also claims that nuisance interventions are necessary, while clarifying that the unknown-nuisance setting remains an open problem. However, this amounts to saying that dependencies between modalities in AV generation can involve spurious correlations, that shared latent representations are not guaranteed to achieve complete disentanglement, and that intervening on a known nuisance to eliminate its influence yields robustness to that nuisance. This claim is extremely general and lacks specificity.
Questions for authors

Can the assumptions used in the theoretical analysis be satisfied in practical joint AV generation? In particular, how can the nuisance be identified and faithfully intervened on while preserving the event and audio semantics?

Reviewer_Yd8iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary:

This paper deals with a controlled study of the failure mode of common audio-video generators - models may learn spurious spurious correlations between visual appearance and sound-producing events rather than reasoning about the underlying event responsible for the audio. Models that let the audio read video directly, through techniques such as cross attention or shared latents, predict sound merely from the visual appearance. Counterfactual invariance, under the necessary and sufficient condition is both necessary and sufficient to identify the causal predictor of the audio. The authors formulate the problem through a Structural Causal Model (SCM). Experiments are conducted on a real V2A generator and a synthetic benchmark.

Critical strengths

The main strengths of the paper are as follows:

  1. The observation that forcing modalities to interact through a shared latent does not guarantee that the latent space would capture the underlying common cause is an important finding.
  2. The paper presents a formal proof that shows that when induced by perfect confounding, a shared bottleneck may admit a shortcut solution.
  3. The work presents a counter-factual intervention that is both necessary and sufficient to identify the causal predictor of the audio.
  4. The authors test the performance of competing methods on two different benchmarks, demonstrating the generalizability of the method.
Critical weaknesses

The key weaknesses of the paper are as follows:

  1. The proposed assessment assumes x_a and n are orthogonal to each other which is too rigid of an assumption. This may not always be the case. For instance, a metal cup sounds different from a ceramic cup and the visual appearance does provide strong cues towards this end.
  2. The aggressive image perturbations used for the real-world experiments, such as grayscale conversion and color channel rotation while easy to implement might produce images outside the distribution of real-world videos.
  3. The real-world evaluations of the proposed method are relatively limited.
  4. Works such as FlatSounds [a], explicitly target this failure mode.

References:

[a] Li, T., Gururani, S., Shih, K.J., Bhatt, G., Lee, S.G., Kong, Z., Goel, A., Anumanchipalli, G. and Liu, M.Y., 2026. Benchmarking Single-Factor Physical Video-to-Audio Generation. arXiv preprint arXiv:2605.30339.

Questions for authors

Comments/Questions:

  1. Does the proposed regularizer remain useful under imperfect counterfactual transformations?
  2. Typos: Abstract: "SCMs..." -> "Structural Causal Models (SCMs)..."
Reviewer_Mu3pOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper discusses an issue in joint video and audio generation models where the audio generation might learn shortcuts based on the appearance factors instead of actually generating the sounding events. They attribute this to the fact that most datasets contain nuisance regularities that teaches the model the visual shortcut to generate a specific sound when a specific object even appears in a frame, and when there is a distribution shift, the output sound becomes wrong.

The authors formulate a structured causal model to show that a shared latent representation is not enough to guarantee robustness to visual nuisances. The method they propose is called counterfactual consistency which is added to the training objective of joint generation models to regularize audio predictions toward invariance to changes in visual nuisance factors that preserve the underlying event and acoustic conditions. They setup controlled experiments using synthetic features, rendered images and videos and an experiment using colored MNIST paired with instrument sounds with varying pitch governed by the number class. The authors demonstrate the visual shortcut in these controlled experiments and then show the effectiveness of counterfactual consistency loss. They also run an experiment with a real video-to-audio generator model (MMAudio) and change factors that should not alter the sound to reveal sensitivity of the model, although generic perturbations produce comparable effects, and fine-tuning reduces sensitivity without establishing selective removal of an appearance shortcut.

Critical strengths
  1. The paper brings attention to a largely ignored issue in video-audio multimodal generation. Most existing systems pay a lot of attention to video generation and audio is usually treated as a sidecar. This leads to the problem setup where datasets are curated mainly for video quality and paired audio is used to train leading to the nuisance factors.
  2. The controlled settings are well setup and demonstrate the exact issues that the paper talks about from very simple, toy experiments, to more realistic scenarios involving image or video-conditioned audio models.
  3. The CC-CF loss allows a way to leverage unlabeled/silent videos in training as opposed to fully supervised training with counterfactual data augmentation.
Critical weaknesses
  1. The biggest issue I have with this paper is the lack of experiments shown on a real example of joint audiovisual generation. All experiments are only demonstrating the visual shortcut on video-conditioned audio generation tasks. The paper presents the nuisance as an issue for joint audio-visual generations and I would expect that to be the main focus of the work.
  2. The CC-CF loss seems to only be useful in synthetic scenarios and the main problem does remain that to do counterfactual synthesis with in-the-wild data, one needs to know beforehand what factors to alter which are irrelevant to the accompanying audio.
Questions for authors
  1. I would like the authors to clarify their choice of experiment setup. Why only choose toy settings and a real setting with MMAudio and not a joint-AV generator like LTX 2, Cosmos 3, etc.?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.