- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses the issue of modality-specific information being suppressed due to global cross-modal contrastive alignment in audio-video representation learning. The authors introduce a concept dictionary based on BatchTopK-SAE into CAV-MAE, dividing audio-video representations into shared and modality-specific components, and applying contrastive learning constraints only to the shared components. Experiments show that this method outperforms the original CAV-MAE on both VGGSound and AudioSet-20K, and that the concept dictionary-based decomposition outperforms direct feature decomposition. However, the manuscript’s formatting appears to be incorrect, such as the font.
Critical strengths
- This paper examines the relationship between cross-modal alignment and the preservation of modality-specific information, which is a key issue in multimodal representation learning. Furthermore, the approach proposed in this paper is conceptually sound.
- The conceptual dictionary is only involved in optimization during the pretraining phase and is removed during inference; therefore, in principle, it does not increase downstream inference overhead. This training-only intervention has practical value.
Critical weaknesses
- The experiment only compared CAV-MAE; it lacks comparisons with other SOTA methods, making it difficult to demonstrate the algorithm’s superiority.
- The validity of some theories and methods requires further discussion. See Questions for details.
Questions for authors
- In Section 3.3, Equations 6–8, the authors use BatchTopK-SAE to obtain sparse concept representations , which they then partition into . Please explain the mechanism used to ensure that corresponds to true cross-modal shared concepts, while corresponds to modality-unique concepts.
- In Section 3.1, Equation 3, the authors argue that global InfoNCE suppresses modality-specific information. Subsequently, in Section 3.3, Equation 8, the authors apply InfoNCE only to the shared representation and conclude that unique concepts are thereby preserved. I would like the authors to further explain: Why does the fact that the unique component does not directly participate in the contrastive loss ensure that it remains unaffected during training?
- In Section 3.4, Equation (9), the authors not only modify the contrastive objective but also introduce a new $ \mathcal L_{\mathrm{sae}}$. Therefore, compared to the original CAV-MAE, both the contrastive alignment strategy and the sparse reconstruction regularization have actually been changed simultaneously. I would like the authors to further clarify: Does the final performance improvement stem from concept disentanglement, or from the additional SAE reconstruction regularization? If I have overlooked the experimental results, please point them out.
- The multiplication in Equation (6) does not appear to hold in the relevant dimension; please check whether this is a typo error.
- There are several inconsistencies in the data reported in the tables. Please explain the reasons for these inconsistencies. For example, VGGSound, Direct, in Tables 1 and 3; AS20K, Concept, and in Tables 2 and 4.