- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work studies spurious correlations between audio and video in joint AV generation from the perspective of shared latent representations. Under strong assumptions (e.g., conditional independence of audio from the nuisance, recoverability of the event and nuisance from video, knowledge of the nuisance, and the ability to intervene on it), the paper theoretically explains that selecting a predictor invariant to the nuisance yields a true causal predictor. Several controlled experiments validate these theoretical insights.
The main issue, as the authors repeatedly acknowledge, is that the theoretical findings rest on multiple very strong assumptions, yet the paper does not discuss how each assumption can be satisfied in actual joint AV generation, or whether satisfying them is even possible. Although an experiment using MMAudio is presented as a connection to joint AV generation, it is unclear whether the experimental setup correctly simulates the spurious correlation discussed in the paper. Given that the Gaussian noise control produces comparable results, the outputs may simply be degraded due to out-of-distribution inputs.
While the reviewer acknowledges some contribution in clarifying the problem formulation, the content is merely standard from the perspective of causal inference, and the paper has not solved a new problem. I therefore vote for rejection.
Critical strengths
Clear formulation of the visual shortcut problem in joint audio-video generation.
Critical weaknesses
- This paper mainly focuses on clarifying spurious correlations in joint AV generation. However, the paper does not discuss whether the assumptions used in the theoretical proofs can actually be satisfied in joint AV generation. Moreover, most empirical validation uses controlled settings unrelated to AV generation, and it is unclear whether these findings can transfer to actual joint AV generation. Consequently, the practical utility of this work must be assessed as extremely limited at this stage.
- Although the problem is presented carefully, the insights the paper provides are not particularly novel. The paper claims novelty in formulating visual shortcuts in AV generation using a structural causal model (SCM) and in showing, both theoretically and through controlled experiments, that shared latent representations are insufficient to eliminate these shortcuts. It also claims that nuisance interventions are necessary, while clarifying that the unknown-nuisance setting remains an open problem. However, this amounts to saying that dependencies between modalities in AV generation can involve spurious correlations, that shared latent representations are not guaranteed to achieve complete disentanglement, and that intervening on a known nuisance to eliminate its influence yields robustness to that nuisance. This claim is extremely general and lacks specificity.
Questions for authors
Can the assumptions used in the theoretical analysis be satisfied in practical joint AV generation? In particular, how can the nuisance be identified and faithfully intervened on while preserving the event and audio semantics?