acceptodds
Under review as a conference paper at ICLR 2027

Preserving Unique Concepts in Different Modalities for Contrastive Learning: Improvements to Contrastive Audio-Visual Masked Autoencoder

Abstract

Contrastive learning is widely adopted in multimodal model training to narrow the distance between cross-modal feature representations, facilitating multimodal information fusion and unified representation learning. During training, contrastive loss guides the model to capture matched features of the same concept across different modalities. Nevertheless, it also causes the model to discard modality-specific conceptual information that exists solely within a single modality and cannot be shared cross-modally. To address this limitation, we decouple each modality’s features into shared conceptual components and modality-exclusive components. Cross-modal contrastive alignment is only imposed on shared components, which preserves modality-specific semantics while sustaining the model’s cross-modal reasoning capacity. Our experiments verify that globally aligning full features via contrastive loss degrades downstream task accuracy, whereas retaining modality-unique concepts yields steady performance improvements. Built upon the audio-visual multimodal model CAVMAE, we redesign its contrastive learning module to preserve exclusive conceptual information from both visual and audio modalities. Experiments across multiple datasets demonstrate that our method outperforms the vanilla baseline on downstream task. The results validate the rationality and effectiveness of our approach for audio-visual multimodal learning. Further analyses confirm that audio and visual modalities contain distinct exclusive concepts, which substantially boost the model’s general representation capability and downstream performance. Overall, our work proves the efficacy of preserving modality-specific conceptual information, especially for audio-visual multimodal tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Multimodal, Audio Representation Learning, Foundation Models, CAV-MAE

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_y9x1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the issue of modality-specific information being suppressed due to global cross-modal contrastive alignment in audio-video representation learning. The authors introduce a concept dictionary based on BatchTopK-SAE into CAV-MAE, dividing audio-video representations into shared and modality-specific components, and applying contrastive learning constraints only to the shared components. Experiments show that this method outperforms the original CAV-MAE on both VGGSound and AudioSet-20K, and that the concept dictionary-based decomposition outperforms direct feature decomposition. However, the manuscript’s formatting appears to be incorrect, such as the font.

Critical strengths
  1. This paper examines the relationship between cross-modal alignment and the preservation of modality-specific information, which is a key issue in multimodal representation learning. Furthermore, the approach proposed in this paper is conceptually sound.
  2. The conceptual dictionary is only involved in optimization during the pretraining phase and is removed during inference; therefore, in principle, it does not increase downstream inference overhead. This training-only intervention has practical value.
Critical weaknesses
  1. The experiment only compared CAV-MAE; it lacks comparisons with other SOTA methods, making it difficult to demonstrate the algorithm’s superiority.
  2. The validity of some theories and methods requires further discussion. See Questions for details.
Questions for authors
  1. In Section 3.3, Equations 6–8, the authors use BatchTopK-SAE to obtain sparse concept representations ZZ, which they then partition into Z=[Zs,Zu]Z=[Z^s,Z^u]. Please explain the mechanism used to ensure that ZsZ^s corresponds to true cross-modal shared concepts, while ZuZ^u corresponds to modality-unique concepts.
  2. In Section 3.1, Equation 3, the authors argue that global InfoNCE suppresses modality-specific information. Subsequently, in Section 3.3, Equation 8, the authors apply InfoNCE only to the shared representation and conclude that unique concepts are thereby preserved. I would like the authors to further explain: Why does the fact that the unique component does not directly participate in the contrastive loss ensure that it remains unaffected during training?
  3. In Section 3.4, Equation (9), the authors not only modify the contrastive objective but also introduce a new $ \mathcal L_{\mathrm{sae}}$. Therefore, compared to the original CAV-MAE, both the contrastive alignment strategy and the sparse reconstruction regularization have actually been changed simultaneously. I would like the authors to further clarify: Does the final performance improvement stem from concept disentanglement, or from the additional SAE reconstruction regularization? If I have overlooked the experimental results, please point them out.
  4. The multiplication in Equation (6) does not appear to hold in the relevant dimension; please check whether this is a typo error.
  5. There are several inconsistencies in the data reported in the tables. Please explain the reasons for these inconsistencies. For example, VGGSound, Direct, r=0.75r=0.75 in Tables 1 and 3; AS20K, Concept, r=0.50r=0.50 and r=0.75r=0.75 in Tables 2 and 4.
Reviewer_atboOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper adds SAE-based shared/private feature splitting to CAV-MAE and reports improved downstream audio-visual classification through selective contrastive alignment.

Critical strengths

The proposed training-time modification yields consistent classification gains without retaining the dictionary module during downstream inference.

Critical weaknesses
  1. Limited novelty and insufficient discussion of closely related work. The central motivation—preserving modality-specific information alongside cross-modal alignment—has already been addressed by CAV-MAE, FactorCL, and MISA, yet the paper inadequately discusses these precedents and omits the latter two. It does not establish what substantive contribution the proposed SAE-based splitting offers beyond existing shared/private representation learning approaches.

  2. Insufficient experimental details for reproducibility. The paper does not provide a sufficiently complete protocol to reproduce its results or verify the fairness of its comparisons. Missing or ambiguous details include the AS20K pretraining data, initialization checkpoints, train/validation/test splits, encoder-freezing scope, classification-head architecture, optimization hyperparameters, SAE and auxiliary losses, and the complete objective used in the experiment without MAE.

[1] Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive Audio-Visual Masked Autoencoder. International Conference on Learning Representations (ICLR), 2023.

[2] Paul Pu Liang, Zihao Deng, Martin Q. Ma, James Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. Factorized Contrastive Learning: Going Beyond Multi-view Redundancy. Advances in Neural Information Processing Systems, 36:32971–32998, 2023.

[3] Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis. Proceedings of the 28th ACM International Conference on Multimedia, pp. 1122–1131, 2020.

Questions for authors
  1. Novelty and related work. What is the substantive methodological contribution beyond CAV-MAE’s information-preservation motivation and existing shared/private approaches such as FactorCL and MISA? Please explain these distinctions and provide appropriate comparisons or controlled experiments demonstrating the specific benefit of SAE-based splitting.

  2. Reproducibility. Can the authors provide the complete data, initialization, training, and evaluation configurations for every method and ablation? In particular, please clarify the AS20K pretraining source and independent evaluation split, the downstream trainable parameters and classification head, and whether SAE reconstruction remains active in the experiment without MAE.

Reviewer_7q9dOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether global audio–visual contrastive alignment can suppress information unique to individual modalities. Building on CAV-MAE, the authors compare two strategies for restricting contrastive learning to a putatively shared representation. Direct Feature Splitting partitions the original features by a predefined ratio. Concept Splitting instead applies BatchTopK-SAE to learn sparse dictionary activations, partitions those activations into Shared and Unique groups, and aligns only the features reconstructed from the Shared group. The latter is trained with MAE reconstruction, selective InfoNCE, and SAE reconstruction objectives. The central hypothesis is that aligning complete audio and visual embeddings may overemphasize their common factors, even when the input also carries task-relevant information available in only one modality. Experiments on VGGSound and AudioSet-20K show consistent but relatively modest downstream gains for Concept Splitting over CAV-MAE and Direct Splitting. The findings support the usefulness of the training modification, but do not yet establish that the gains arise from genuine disentanglement or preservation of modality-unique concepts. At the shared ratio of 0.75, the reported five-seed means improve from 59.52% to 60.61% accuracy on VGGSound and from 0.3307 to 0.3364 mAP on AudioSet-20K.

Critical strengths

The tension between cross-modal alignment and preservation of modality-private cues is important, and selective alignment is a sensible direction. The five-seed improvements over CAV-MAE and Direct Splitting on two datasets provide useful evidence that the training modification benefits downstream recognition. These strengths should be distinguished from the still-unverified semantic explanation.

Critical weaknesses

(1) The semantic interpretation of Concept Splitting exceeds its current validation. Sparse reconstruction and selective InfoNCE create a meaningful inductive bias, but do not guarantee that the designated Unique group represents modality-private concepts. The paper needs direct evidence of this property and its preservation relative to Direct Splitting. No particular form of semantic supervision is required. The central issue is the gap between architectural roles and demonstrated semantics, rather than the absence of any separation mechanism. (2) The experiments do not separate SAE representation effects from selective alignment effects. The missing SAE-plus-global-alignment control leaves the causal account incomplete. Tables 1 and 2 already provide useful matched multi-seed comparisons, but do not isolate that factor. Broader ratio sweeps, clearer downstream feature-path documentation, and correction of the Table 2 SD would improve rigor. Additional contemporary baselines would help contextualize competitiveness, while direct branch-level measurements remain the most important test of the main claim.

Questions for authors
  1. Are audio and visual dictionaries shared or separate, and how are Shared/Unique coordinates selected? Since Equation (8) aligns reconstructed D-dimensional features, one-to-one matching of dictionary indices across modalities is not inherently required. Please explain the actual parameter sharing and show whether the resulting groups differ in cross-modal predictability and retention of modality-private cues.
  2. What empirical evidence distinguishes improved sparse representation quality from improved preservation of unique semantics? A quantitative comparison with Direct Splitting and a controlled intervention on the Unique group would be especially useful. Cross-modal predictability should be assessed with matched-capacity probes and held-out data.
  3. Can the authors compare SAE plus global contrastive alignment with SAE plus selective alignment under the same dictionary, sparsity, remaining losses, and optimization settings? Please report downstream metrics and multi-seed variability, ideally accompanied by SAE reconstruction error. An all-shared control is sufficient if it changes only the intended alignment restriction.
  4. Please reconcile the Table 2 SD, explain the seed selection in Tables 3–4, and state whether Table 5's no-MAE condition uses Direct or Concept Splitting and retains the SAE objectives. Correct the matrix order in Equation (6) if it is only a typographical error. Finally, identify the downstream classifier input and clarify how the freezing statements in Section 3.4 relate to excluding the SAE dictionary during inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.