acceptodds
Under review as a conference paper at ICLR 2027

Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment

Abstract

Recent works have observed implicit cross-modal alignment between independently trained unimodal models. However, the mechanisms underlying the emergence of this phenomenon remain unresolved. In this paper, we demonstrate that unimodal augmentation-based self-supervised objectives implicitly induce cross-modal alignment. We show that low unimodal losses force high cross-modal alignment, whereas a high unimodal loss prevents such alignment. We theoretically connect the widely used InfoNCE loss to local CKA, the popular measure of cross-modal alignment. We further provide extensive empirical evidence supporting this claim, both in controlled settings and across 53 unconstrained frozen text, vision, and audio unimodal encoders. In both cases, unimodal augmentation-based objectives control cross-modal alignment. Overall, our results provide a theoretical and empirical foundation for understanding the emergence of implicit cross-modal alignment by establishing a direct connection between unimodal training objectives and cross-modal structure.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
The Platonic Representation Hypothesis, Self-supervised Learning, Cross-modal Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6mQ7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper theoretically shows that implicit cross-modal alignment of representations between independently trained unimodal encoders is driven by augmentation-based self-supervised objectives. Authors connect the InfoNCE loss to CKA with a Gaussian kernel; the two theorems argue that low losses in both modalities are CKA lower-bound, and that a high SSL loss in either modality upper-bounds it.

Authors provide extensive empirical experiments to test the hypothesis using 53 frozen text, vision, and audio encoders. Results show that minimizing or maximizing InfoNCE in each modality raises or hurts cross-modal alignment.

Critical strengths

This paper provides a theoretical mechanism and analyze on the emergence of cross-modal aligment, together with solid empirical experiments covering common data modalities. I think both the theorems and the empirical results are very informative to the community.

Critical weaknesses

One concern I have is on the initialization of the text encoders used. Some text encoders like sentence-transformers were trained on some supervised datasets (e.g. NLI) so not solely SSL. The authors did not explain why the initialize the ViT from scracth but use pretrained text encoders (line 323).

Another concern is the missing of extrinsic evaluations. All the results use mutual kNN to evaluate nearest neighbor overlap. I think some extrinsic task results, e.g., cross-modal retrieval tasks would further support the claims.

Questions for authors
  • Why do you initialize the text encoders from SBERT/BERT? Have you traied to do SSL from scratch as the ViT?

  • Figure 3 (b), it is kind of surprising to me that the supervised contrasctive experiment performed worse than augmentation. Why is this the case and do you have any hypotheses?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6mQ7 (1/2)

We thank the reviewer for their insightful comments. We appreciate you finding that our work provides a theoretical mechanism for the emergence of cross-modal alignment, together with solid empirical experiments covering common data modalities, and that both the theorems and the empirical results are very informative to the community. Following your feedback, we have trained the text encoder from scratch in the controlled experiments, and added an extrinsic cross-modal retrieval evaluation. Below, we address each comment in detail.


W1: Initialization of the text encoder

We thank the reviewer for this comment.

Controlled experiments (Sec. 4.1). The text encoder is initialized from bert-base-uncased (App. P, Tab. 13), which is pretrained with self-supervised masked language modeling and next-sentence prediction, without NLI or any other supervised data. The only contrastive sentence-level objective it is exposed to is our Aug-SSL training on Qwen paraphrases.

Training from scratch. Following the reviewer's comment, we repeated the experiment of Fig. 3a with a randomly initialized text encoder, and obtained the same results (table below): minimizing both losses raises CMA from 0.016 to 0.059, 4.8× chance and on par with BERT initialization, while maximizing either loss drives CMA to chance. BERT pretraining is therefore not needed for the effect. The full training curves are in the revised manuscript (App. D, Fig. 8).

PairingBERT initRandom init
min-ViT / min-SBERT0.0576 ± 0.00010.0592 ± 0.0005
min-ViT / max-SBERT0.0139 ± 0.00110.0126 ± 0.0005
max-ViT / min-SBERT0.0143 ± 0.00070.0144 ± 0.0008
max-ViT / max-SBERT0.0123 ± 0.00040.0128 ± 0.0004
Chance0.01220.0122

Final CMA (mutual kkNN) of the controlled experiment in Fig. 3a with BERT-initialized vs. randomly initialized text encoders. Results are mean ± std over three seeds.

Q1: Training the text encoder from scratch

Please see our response to W1 above, including the new from-scratch results.

AuthorsOpenReview ↗

Response to Reviewer 6mQ7 (2/2)

W2: Extrinsic evaluation (cross-modal retrieval)

We thank the reviewer for this suggestion. Following the comment, we evaluated image-to-text and text-to-image retrieval between DINOv2 and each text model. Since the encoders are trained independently, we map each text encoder to DINOv2 with an orthogonal Procrustes map, fitted on only 100 paired COCO samples after projecting both spaces onto their top principal components. A rotation preserves distances and angles, so retrieval directly reflects the geometry of the frozen encoders, without learning a new representation. We evaluate on 400 held-out pairs, averaged over 10 random splits.

Text modelI→T R@1I→T R@5I→T R@10T→I R@1T→I R@5T→I R@10
MiniLM (Aug)12.039.656.913.040.957.5
RoBERTa (Aug)12.440.758.513.941.958.4
Sentence-T5 (Aug)12.139.256.312.439.456.2
BLOOM (NTP)7.426.038.97.125.338.4
OpenLLaMA (NTP)8.227.240.17.325.038.5
LLaMA (NTP)6.823.634.76.321.833.4
Pythia (NTP)5.920.631.25.218.930.1
GPT-2 (NTP)6.924.036.36.722.134.9
All Aug (9 models)12.139.757.012.940.457.1
All NTP (15 models)7.024.236.26.522.635.1
Chance0.251.252.50.251.252.5

Cross-modal retrieval (%) on MS-COCO with DINOv2 after orthogonal Procrustes alignment fitted on 100 pairs; 400 test pairs, mean over 10 splits. Results are averaged over the models in each family.

Every Aug family outperforms every NTP family on all six metrics, with R@10 of 57% vs. 36% on average, in agreement with the mutual kkNN results of Sec. 4.2, and even the largest LLMs (up to 13B parameters) do not close the gap. Retrieval is also predicted by the unimodal text InfoNCE loss (Spearman ρs=0.74\rho_{s}=0.74, Pearson r=0.94r=0.94 against the negative loss on COCO), mirroring Fig. 5. We have added this evaluation to the revised manuscript (App. K), together with larger numbers of fitting pairs and the WIT dataset, where Aug models again outperform NTP models on average (R@10 of 20.5% vs. 15.5%).

Q2: Why supervised contrastive underperforms Aug-SSL

We thank the reviewer for this question. We attribute it to two factors:

Held-out evaluation. Fig. 3b evaluates CMA only on classes held out from supervised training (46 of 79 classes, App. P). Supervised objectives specialize the representation to the 33 seen classes, whereas Aug-SSL learns general-purpose structure that transfers to unseen semantics.

Labels carry less structure than augmentations. Labels only indicate class membership: they contain no information about which samples within a class are more similar, nor which classes are closer to each other. In SupCon, all same-label samples are positives, so the label overrides the augmentation structure, and supervised training is known to drive within-class variation toward collapse [1], discarding the instance-level semantics that the paired captions express. Aug-SSL retains this graded structure, which is shared with the text modality, and therefore yields higher CMA. We have added this discussion to the revised manuscript (App. E).

[1] Papyan, Han, and Donoho. "Prevalence of Neural Collapse during the Terminal Phase of Deep Learning Training." PNAS 2020.


We hope that our responses have addressed all your concerns and highlighted the contributions of our work. If any questions remain, we would be happy to provide additional details. We hope these clarifications will be taken into account in your final evaluation of the paper.

Reviewer_1nHBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper poses the question of when different modalities align and why. The main claim is that higher alignment is a result of augmented unimodal SSL - when models are trained to embed augmented views of the same sample (with correlated semantics) in the same vector. They provide theoretical justification for this claim by bounding the CKA between two embedding spaces using functions of the InfoNCE loss of both embedders. The paper continues by empirical verification of the theory, measuring mutual knn and CKA between various models, modalities and datasets.

Critical strengths

Great presentation, clear motivation and relevance.

The critical strengths are the significance and relevance of the paper. The question of which model representations align and why is a well researched one, to which the paper adds a novel and well grounded perspective, both theoretically and experimentally, focusing on SSL models which are commonly used and researched.

The paper also provides a link between the InfoNCE loss and CKA - both commonly used and researched metrics, which I find insightful.

Critical weaknesses

I have one concern about the paper, regarding why low unimodal loss should imply high CKA. From my understanding, the lower bound is mainly driven by how augmentation-invariant the encoders are, together with how well they separate different samples. If both encoders are perfectly augmentation-invariant, the CKA lower bound then depends on the semantic structure induced by the augmentations, and in particular on how much the augmentations cause semantic overlap between different samples (for example an augmentation that switches an image of a dog to a cat). A simple example: both encoders are augmentation-invariant, but one of them encodes the class-level structure as dog-cat-frog and the other as dog-frog-cat. While augmentation invariant, they disagree on the cross-class similarities, so their CKA is not high. To my knowledge augmentations usually don’t change the class-level semantics of images or texts. This means the augmentations carry little information about cross-class similarities, so in practice the lower bound gives little guarantee that the two encoders agree on them.

I think this corresponds to the mutual knn numbers in the experiments, which are pretty low and decrease for larger datasets (WIT vs MSCOCO). I assume that if the mutual knn was measured on a dataset that includes both the clean version and augmentations, the numbers would be much higher.

Generally I still find the paper very interesting and would be more than happy to discuss this point.

Questions for authors
  1. Following the critical weakness - from lines 227-230 it seems that high CKA is mainly driven by similarity between augmented variations of the same data, but not by similarity between different datapoints. Could you elaborate on this?

  2. Could you report the numerical value of the lower bound for the trained models? Could you compare the mutual knn metric between measuring on a clean dataset vs clean + augmented dataset?

  3. From figure 33 it seems that a it suffices for min⁡max⁡{LX,LY}\min \max \{L_X,L_Y\} to be small for the lower bound to be 1, meaning it is almost independent of LYL_Y for example given that LXL_X is small. Am I understanding this correctly? If so this substantially limits the power of the lower bound as high CKA is promised independent of what one of the encoders does, as long as the other converges.

  4. Following the above discussion - do the farthest nearest neighbors overlap in any way? CKA agreement is supposed to capture this as well no? If models do not agree on there, I think it might support my claim.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1nHB (1/3)

We thank the reviewer for their insightful comments. We appreciate you finding our work to have great presentation, clear motivation and relevance, and to add a novel and well-grounded perspective, both theoretically and experimentally, to the question of which representations align and why. We are also glad you found the link between the InfoNCE loss and CKA insightful. Following your feedback, we have added a rank-banded mutual kkNN analysis that measures agreement on near and far neighbors separately, and expanded our discussion of how augmentations shape the similarity structure between different samples. Below, we address each comment in detail.


W1: Why low unimodal loss implies high CKA

We thank the reviewer for this thoughtful analysis. The guarantee is indeed driven by how augmentations relate different samples. This is precisely what the augmentation concentration χ\chi quantifies, and we have made it explicit in the revision (Sec. 3.2). We address the concern in three points.

Augmentations relate different samples. In our model, an augmentation may slightly change the semantics of a sample (Eq. 2), so augmentations link semantically close samples to each other. This is well documented in practice: aggressive crops make different samples share nearly identical views, e.g., two different cars cropped to their wheels [1], and augmentation-overlap theory [1,2] shows that such overlaps between different samples, rather than instance invariance alone, are what allow contrastive learning to recover semantic structure. The augmentation graph of [3] makes this structure graded: augmented samples are connected according to how likely they are to originate from the same natural sample, and the learned representation is determined by these connections. Overlaps occur mostly within a class but not exclusively (with well-tuned crops on CIFAR-10, 3.6% of the augmentation-graph edges are inter-class [1]), and they are more likely between semantically closer classes, which share more visual and textual content. This addresses the reviewer's example: augmentations that never relate different samples correspond to χ=∞\chi = \infty, which lies outside our regime 0<χ<∞0 < \chi < \infty. Within this regime, augmentations link different samples, including across classes (e.g., dogs and cats share more augmented views than dogs and frogs). The cross-class structure is therefore encoded in JJ, and must be captured by any encoder that attains a low loss.

The alignment measure is local. Both our theory and our experiments concern local alignment: CKA-RBF with a bandwidth set from the kk-th nearest-neighbor distance (Sec. 3.1, App. P), and its discrete counterpart, mutual kkNN (Fig. 2b). Local CKA measures whether the two encoders agree on each sample's neighborhood, which is exactly where augmentations are most informative, rather than on the ordering of far-apart clusters. This is consistent with recent evidence that cross-modal alignment between independently trained models is predominantly local [4], and our new rank-banded analysis (Q4) confirms that the Aug advantage is concentrated there.

(continued in the next comment)


[1] Wang et al. "Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap." ICLR 2022.
[2] Zhang, Wang, and Wang. "An Augmentation Overlap Theory of Contrastive Learning." JMLR 2025.
[3] HaoChen et al. "Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss." NeurIPS 2021.
[4] Gröger, Wen, and Brbić. "Revisiting the Platonic Representation Hypothesis: An Aristotelian View." ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer 1nHB (2/3)

W1 (continued)

Mutual kkNN values and clean + augmented evaluation. All reported mutual kkNN values are computed on clean, distinct datapoints, so they measure exactly the agreement between different samples. Relative to chance, k/(n−1)≈0.049k/(n-1) \approx 0.049 for k=50k=50 and n=1024n=1024, Aug models on COCO reach roughly 7–8× chance. WIT values are lower for all models, since its short, encyclopedic captions describe less of the visual content. A clean + augmented evaluation is not well defined across modalities: augmentations are drawn independently in each modality, so there is no correspondence between a given crop of an image and a given paraphrase of its caption, and each point's nearest neighbors would trivially become its own augmented copies.

Q1: Clarifying lines 227–230

We thank the reviewer for pointing out that lines 227–230 can be misread; we have rewritten them (Sec. 3.2). The CKA is computed only between different datapoints, so augmented variations of the same datapoint never enter it. Augmentations only affect training: a low loss requires each sample to be more similar to its augmentation than to unrelated samples. Since augmentations link semantically close samples, this shapes the similarities between different datapoints according to their semantics. Both encoders are therefore pushed toward the same similarity structure between different datapoints, which yields a high local CKA.

Q2: Numerical value of the bound and clean + augmented evaluation

(1) Numerical bound. Thm. 1 is a worst-case guarantee: it holds for every pair of encoders attaining the given losses. Evaluating it numerically requires, besides the measurable losses and the known τ\tau and σ\sigma, the augmentation concentration χ\chi. This constant measures how much of a sample's latent semantics its augmentations retain (χ=0\chi=0 when augmentations discard them), and since the latent semantics are not observed, it cannot be computed directly on real data. Estimating χ\chi for natural augmentations such as crops or paraphrases requires further analysis, which is beyond the scope of this paper and is an interesting direction for future work. The prediction of the theorem, that low unimodal losses in both modalities lead to high alignment, is verified at every level of our experiments: in controlled training (Fig. 3a, and now also from random initialization, App. D) and across 53 pretrained encoders (Fig. 5, Tab. 1), with statistical significance (Tab. 3).

(2) Clean + augmented evaluation. As detailed in our response to W1, a clean + augmented evaluation is not well defined across modalities, since augmentations are drawn independently in each modality, and each point's nearest neighbors would trivially become its own augmented copies.

AuthorsOpenReview ↗

Response to Reviewer 1nHB (3/3)

Q3: Does the bound depend on both losses?

We thank the reviewer for this question. The x-axis of Fig. 37a is max⁡{LX,LY}\max\lbrace L_{X}, L_{Y} \rbrace, so a small value means that both losses are small. The bound requires both: if one encoder has a large loss, the bound gives no guarantee, regardless of the loss of the other. Fig. 37a plots the diagonal LX=LYL_{X} = L_{Y}, which is the worst case under max⁡{LX,LY}≤L\max\lbrace L_{X}, L_{Y} \rbrace \le L (App. P.3). Moreover, Thm. 2 shows the complementary direction: a high loss in either encoder caps the CKA, regardless of how well the other converges. This is also observed empirically in Fig. 3a, where pairing a loss-minimizing ViT with a loss-maximizing SBERT drives CMA to chance. We have clarified the axis in the caption of Fig. 37.

Q4: Agreement on far neighbors (rank-banded analysis)

We thank the reviewer for this question. The CKA we analyze is CKA-RBF with a local bandwidth (App. P), so very distant pairs contribute little, and Thm. 1 is a statement about neighborhoods, in line with evidence that cross-modal alignment is predominantly local [4]. App. I (Figs. 28, 30) already shows the Aug advantage for k∈{10,20,50,100}k \in \lbrace 10, 20, 50, 100 \rbrace. To separate near and far neighbors, following the reviewer's comment, we computed a rank-banded mutual kkNN: for each band of neighbor ranks [r1,r2][r_{1}, r_{2}], we measure the overlap between the two modalities' neighbors within that band only, normalized by its chance level.

Ranks1–1011–5051–100101–200201–500501–1023
Aug models34.6 ± 0.96.1 ± 0.22.9 ± 0.21.6 ± 0.11.1 ± 0.01.2 ± 0.0
NTP models18.6 ± 2.94.4 ± 0.52.3 ± 0.21.5 ± 0.11.1 ± 0.01.2 ± 0.0

Rank-banded mutual kkNN with DINOv2 on MS-COCO (n=1024n=1024), normalized by chance (1.0 = chance). Mean ± std over the 9 Aug and 16 NTP models.

In the first three bands, i.e., the nearest 10% of the dataset, every Aug model exceeds every NTP model, with nearly twice the agreement in the nearest band (34.6 vs. 18.6× chance). Beyond rank 100, all models are close to chance, as expected from a local measure. This is exactly the structure our theory predicts: augmentations link semantically close samples, so a low loss shapes each sample's neighborhood, which dominates the local CKA of Thm. 1. We have added this analysis to the revision (App. J, Fig. 32).

[4] Gröger, Wen, and Brbić. "Revisiting the Platonic Representation Hypothesis: An Aristotelian View." ICML 2026.


We hope that our responses have addressed all your concerns and highlighted the contributions of our work. If any questions remain, we would be happy to provide additional details. We hope these clarifications will be taken into account in your final evaluation of the paper.

Reviewer_dNxgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explains why cross-modal alignment (CMA) emerges implicitly from unimodal augmentation-based self-supervised learning (Aug-SSL). The authors argue that augmentations remove view-specific information, steering each modality's encoder toward shared semantic structure. They connect the InfoNCE loss to local CKA, proving that low unimodal losses in both modalities force high CMA, while a high loss in either modality restricts it, without assuming linearity. Empirically, controlled training experiments and 53 frozen pretrained encoders across vision, text, and audio confirm that unimodal InfoNCE loss predicts CMA, while model and corpus size do not.

Critical strengths
  1. The paper offers the first theoretical account of why cross-modal alignment emerges from unimodal augmentation-based SSL. Prior work documented that alignment emerges, but attributed it to dataset or task overlap empirically. This paper connects the InfoNCE loss to local CKA and proves both directions: low losses in both modalities guarantee high alignment (Thm. 1), while a high loss in either modality prevents it (Thm. 2). These results hold without assuming linearity or shared input statistics, which earlier theoretical work required. This is the consideration that most strongly supports acceptance, because it turns an empirical curiosity into a principled mechanism.

  2. The empirical validation is broad and well controlled. The paper validates the theory in both directions: controlled training experiments that isolate the effect of the objective (including a comparison against supervised baselines), and a large-scale analysis of 53 frozen encoders across vision, text, and audio. The key finding, that unimodal InfoNCE loss predicts cross-modal alignment while model and corpus size do not, is consistent across modalities and architectures, and is supported by statistical significance tests. This breadth matters because the claim is about a general property of Aug-SSL, not a single model family.

Critical weaknesses
  1. Figure 3b trains only the ViT, with augmentation-based InfoNCE, SupCon, and cross-entropy, and leaves the text encoder frozen. This paper should also freeze the vision encoder and train the text encoder under more than one objective, to see whether changing the text objective moves alignment.
  2. The lower bound of Theorem 1 is not computed for the trained models. It rises toward perfect alignment only when both unimodal losses are small enough, and it can fall to zero when the losses are not small enough. The paper plots the bound for selected temperatures and augmentation strengths, without placing any trained model on it. This paper should also evaluate the bound at the losses of the controlled training, to see whether it is still positive while alignment is rising.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dNxg

We thank the reviewer for their insightful comments. We appreciate you finding that our work offers the first theoretical account of why cross-modal alignment emerges from unimodal Aug-SSL, proving both directions without assuming linearity or shared input statistics, and thereby turning an empirical curiosity into a principled mechanism. We are also grateful that you found our empirical validation broad and well controlled, consistent across modalities and architectures, and supported by statistical significance tests. Following your feedback, we have added the mirror experiment of Fig. 3b, training the text encoder against a frozen vision encoder under several objectives, and clarified the role of the bound. Below, we address each comment in detail.


W1: Mirror experiment (frozen vision encoder, trained text encoder)

We thank the reviewer for this suggestion. Following the comment, we conducted the mirror experiment of Fig. 3b: we freeze the vision encoder (DINOv2 ViT-B/14) and train the text encoder on COCO under the same three objectives, Aug-SSL (InfoNCE over the Qwen paraphrases of Sec. 4.1), SupCon, and cross-entropy, from both random and BERT initialization, with the same evaluation as in Fig. 3b.

Text objectiveRandom initBERT init
Untrained (step 0)0.0560.268
Supervised Contrastive0.202 ± 0.0020.349 ± 0.002
Cross-entropy0.180 ± 0.0030.282 ± 0.002
Aug-SSL (InfoNCE)0.343 ± 0.0030.415 ± 0.003
Chance0.0320.032

Final CMA (mutual kkNN) between a frozen DINOv2 and a text encoder trained on COCO, on the held-out classes. Results are mean ± std over five evaluation sets.

Changing the text objective moves alignment exactly as changing the vision objective does: Aug-SSL yields the highest CMA under both initializations, with a large margin from random initialization (0.34 vs. 0.20 and 0.18). The full training curves are now in the revised manuscript (App. E, Fig. 9).

W2: Evaluating the bound of Thm. 1 for trained models

We thank the reviewer for this point. Thm. 1 is a worst-case guarantee: it holds for every pair of encoders attaining the given losses. Its role is to establish the mechanism and its direction, namely that low unimodal losses in both modalities force alignment. Evaluating it numerically requires, besides the measurable losses and the known τ\tau and σ\sigma, the augmentation concentration χ\chi. This constant measures how much of a sample's latent semantics its augmentations retain (χ=0\chi=0 when augmentations discard them), and since the latent semantics are not observed, it cannot be computed directly on real data. Estimating χ\chi for natural augmentations such as crops or paraphrases requires further analysis, which is beyond the scope of this paper and is an interesting direction for future work, toward further understanding the roles of different augmentations.

The prediction of the theorem, that low unimodal losses in both modalities lead to high alignment, is verified at every level of our experiments: in controlled training, where minimizing both losses raises CMA and maximizing either drives it to chance (Fig. 3a, and now also from random initialization, App. D), and across 53 pretrained encoders (Fig. 5, Tab. 1), with statistical significance (Tab. 3). We have clarified the role of Thm. 1 in the revised manuscript (App. O).


We hope that our responses have addressed all your concerns and highlighted the contributions of our work. If any questions remain, we would be happy to provide additional details. We hope these clarifications will be taken into account in your final evaluation of the paper.

Thanks for the responses. My concerns are well addressed. I think the theoretical account of why cross-modal alignment emerges from unimodal Aug-SSL in this paper is really interesting and insightful. Therefore, I maintain my score of 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thanking the reviewers

Dear Reviewers and ACs,

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that all reviewers recognized the theoretical contribution of our work, described as the first theoretical account of why cross-modal alignment emerges from unimodal Aug-SSL, which turns an empirical curiosity into a principled mechanism (dNxg), a novel and well-grounded perspective (1nHB), and a theoretical mechanism for this emergence (6mQ7). The reviewers likewise found our empirical validation broad and well controlled (dNxg), well-grounded (1nHB), and solid across common data modalities (6mQ7), the paper clear and easy to follow (dNxg, 1nHB, 6mQ7) with great presentation (1nHB), and the work significant and relevant (1nHB) and very informative to the community (6mQ7).

Following the reviews, we conducted new experiments, all of which further support our central claim: the effect holds in both directions across modalities, no pretraining is needed, the effect transfers to a downstream task, and the alignment is local, as the theory predicts. Together, these results show that the relationship between unimodal augmentation-based objectives and cross-modal alignment is not restricted to a particular modality, initialization, or evaluation metric. All new experiments and clarifications are included in the revised manuscript. We addressed each comment in detail below.

Thank you

Area_Chair_w5G7OpenReview ↗

Please read the rebuttal and engage in discussion

Dear Reviewers,

The authors have posted their rebuttal. Please read it and engage in discussion, in particular:

  • Whether each of your questions and concerns has been addressed.
  • Whether your rating and recommendation have changed, and why.
  • Any remaining concerns that must be settled before the meta-review.

Thanks for your time and contribution.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.