- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper theoretically shows that implicit cross-modal alignment of representations between independently trained unimodal encoders is driven by augmentation-based self-supervised objectives. Authors connect the InfoNCE loss to CKA with a Gaussian kernel; the two theorems argue that low losses in both modalities are CKA lower-bound, and that a high SSL loss in either modality upper-bounds it.
Authors provide extensive empirical experiments to test the hypothesis using 53 frozen text, vision, and audio encoders. Results show that minimizing or maximizing InfoNCE in each modality raises or hurts cross-modal alignment.
Critical strengths
This paper provides a theoretical mechanism and analyze on the emergence of cross-modal aligment, together with solid empirical experiments covering common data modalities. I think both the theorems and the empirical results are very informative to the community.
Critical weaknesses
One concern I have is on the initialization of the text encoders used. Some text encoders like sentence-transformers were trained on some supervised datasets (e.g. NLI) so not solely SSL. The authors did not explain why the initialize the ViT from scracth but use pretrained text encoders (line 323).
Another concern is the missing of extrinsic evaluations. All the results use mutual kNN to evaluate nearest neighbor overlap. I think some extrinsic task results, e.g., cross-modal retrieval tasks would further support the claims.
Questions for authors
-
Why do you initialize the text encoders from SBERT/BERT? Have you traied to do SSL from scratch as the ViT?
-
Figure 3 (b), it is kind of surprising to me that the supervised contrasctive experiment performed worse than augmentation. Why is this the case and do you have any hypotheses?
2 comments
Response to Reviewer 6mQ7 (2/2)
W2: Extrinsic evaluation (cross-modal retrieval)
We thank the reviewer for this suggestion. Following the comment, we evaluated image-to-text and text-to-image retrieval between DINOv2 and each text model. Since the encoders are trained independently, we map each text encoder to DINOv2 with an orthogonal Procrustes map, fitted on only 100 paired COCO samples after projecting both spaces onto their top principal components. A rotation preserves distances and angles, so retrieval directly reflects the geometry of the frozen encoders, without learning a new representation. We evaluate on 400 held-out pairs, averaged over 10 random splits.
| Text model | I→T R@1 | I→T R@5 | I→T R@10 | T→I R@1 | T→I R@5 | T→I R@10 |
|---|---|---|---|---|---|---|
| MiniLM (Aug) | 12.0 | 39.6 | 56.9 | 13.0 | 40.9 | 57.5 |
| RoBERTa (Aug) | 12.4 | 40.7 | 58.5 | 13.9 | 41.9 | 58.4 |
| Sentence-T5 (Aug) | 12.1 | 39.2 | 56.3 | 12.4 | 39.4 | 56.2 |
| BLOOM (NTP) | 7.4 | 26.0 | 38.9 | 7.1 | 25.3 | 38.4 |
| OpenLLaMA (NTP) | 8.2 | 27.2 | 40.1 | 7.3 | 25.0 | 38.5 |
| LLaMA (NTP) | 6.8 | 23.6 | 34.7 | 6.3 | 21.8 | 33.4 |
| Pythia (NTP) | 5.9 | 20.6 | 31.2 | 5.2 | 18.9 | 30.1 |
| GPT-2 (NTP) | 6.9 | 24.0 | 36.3 | 6.7 | 22.1 | 34.9 |
| All Aug (9 models) | 12.1 | 39.7 | 57.0 | 12.9 | 40.4 | 57.1 |
| All NTP (15 models) | 7.0 | 24.2 | 36.2 | 6.5 | 22.6 | 35.1 |
| Chance | 0.25 | 1.25 | 2.5 | 0.25 | 1.25 | 2.5 |
Cross-modal retrieval (%) on MS-COCO with DINOv2 after orthogonal Procrustes alignment fitted on 100 pairs; 400 test pairs, mean over 10 splits. Results are averaged over the models in each family.
Every Aug family outperforms every NTP family on all six metrics, with R@10 of 57% vs. 36% on average, in agreement with the mutual NN results of Sec. 4.2, and even the largest LLMs (up to 13B parameters) do not close the gap. Retrieval is also predicted by the unimodal text InfoNCE loss (Spearman , Pearson against the negative loss on COCO), mirroring Fig. 5. We have added this evaluation to the revised manuscript (App. K), together with larger numbers of fitting pairs and the WIT dataset, where Aug models again outperform NTP models on average (R@10 of 20.5% vs. 15.5%).
Q2: Why supervised contrastive underperforms Aug-SSL
We thank the reviewer for this question. We attribute it to two factors:
Held-out evaluation. Fig. 3b evaluates CMA only on classes held out from supervised training (46 of 79 classes, App. P). Supervised objectives specialize the representation to the 33 seen classes, whereas Aug-SSL learns general-purpose structure that transfers to unseen semantics.
Labels carry less structure than augmentations. Labels only indicate class membership: they contain no information about which samples within a class are more similar, nor which classes are closer to each other. In SupCon, all same-label samples are positives, so the label overrides the augmentation structure, and supervised training is known to drive within-class variation toward collapse [1], discarding the instance-level semantics that the paired captions express. Aug-SSL retains this graded structure, which is shared with the text modality, and therefore yields higher CMA. We have added this discussion to the revised manuscript (App. E).
[1] Papyan, Han, and Donoho. "Prevalence of Neural Collapse during the Terminal Phase of Deep Learning Training." PNAS 2020.
We hope that our responses have addressed all your concerns and highlighted the contributions of our work. If any questions remain, we would be happy to provide additional details. We hope these clarifications will be taken into account in your final evaluation of the paper.
Response to Reviewer 6mQ7 (1/2)
We thank the reviewer for their insightful comments. We appreciate you finding that our work provides a theoretical mechanism for the emergence of cross-modal alignment, together with solid empirical experiments covering common data modalities, and that both the theorems and the empirical results are very informative to the community. Following your feedback, we have trained the text encoder from scratch in the controlled experiments, and added an extrinsic cross-modal retrieval evaluation. Below, we address each comment in detail.
W1: Initialization of the text encoder
We thank the reviewer for this comment.
Controlled experiments (Sec. 4.1). The text encoder is initialized from
bert-base-uncased(App. P, Tab. 13), which is pretrained with self-supervised masked language modeling and next-sentence prediction, without NLI or any other supervised data. The only contrastive sentence-level objective it is exposed to is our Aug-SSL training on Qwen paraphrases.Training from scratch. Following the reviewer's comment, we repeated the experiment of Fig. 3a with a randomly initialized text encoder, and obtained the same results (table below): minimizing both losses raises CMA from 0.016 to 0.059, 4.8× chance and on par with BERT initialization, while maximizing either loss drives CMA to chance. BERT pretraining is therefore not needed for the effect. The full training curves are in the revised manuscript (App. D, Fig. 8).
Final CMA (mutual kNN) of the controlled experiment in Fig. 3a with BERT-initialized vs. randomly initialized text encoders. Results are mean ± std over three seeds.
Q1: Training the text encoder from scratch
Please see our response to W1 above, including the new from-scratch results.