acceptodds
Under review as a conference paper at ICLR 2027

Imperfect Alignment, Reliable Similarity: Score–Encoder Separation in Multimodal Contrastive Learning

Abstract

Multimodal contrastive learning is often interpreted as aligning matched representations, yet each modality may contain noise irrelevant to cross-modal comparison. We ask whether reliable similarity requires the individual encoders to be denoised. We study population gradient flow for linear encoders with a learned temperature in a noisy two-modality model, where the score is factorized through the encoders. In a regular non-collapsing regime, every nuisance interaction in the score vanishes asymptotically while shared score signal remains. Generic initialization keeps encoder nuisance nonzero at every finite time, but permits asymptotic encoder denoising. Separately, when the output dimension exceeds the shared-signal dimension by at least two, an open basin of full-signal optima retains nuisance in both encoders asymptotically. Thus score- and encoder-level denoising are distinct. Controlled linear and nonlinear experiments support this separation; a real-data compression study further separates feature reconstruction, preservation of learned comparisons, and label retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal contrastive learning, representation learning, gradient flow, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jBHpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a method to understand and isolate if contrastive learning can produce reliable cross modal scores/similarities without removing nuisance information from individual representations from encoders.

Critical strengths

The topic investigated in the paper is relevant to the community.

Critical weaknesses

The introduction is unreadable. The motivation is unclear, the framing of the problem is unclear, the related works are unclear or they are not presented in a proper frame, the list of contributions begins with a result and not a contribution. I read the introduction and related work section and still I don't have any ideas of what the paper is about and what is the motivation.

Notation should be revised. bold for vectors and matrices, otherwise are scalars or varaibles.

The problem setup section cite theorem 1 and corollary 3(i)-(ii) but they are not presented there and the authors did not write where I could find them. similarly, there are a lot of mathematical terms that are not defined or explained.

Section 3: problem setup. Section 4: main results. The method is in the main results... the authors should severely improve the readability of the paper.

The paper does not have a logical flow, there are main results and the authors explain part of the method, than there is the experiment section still with equations.

The paper might be valuable as the topic is interesting and it would be interesting as well to explore it so the idea is good, but it is very hard to understand as it is written and presented.

Questions for authors

can the authors considerably improve the readness of the paper (including motivation, organization, related works, explanations, visualizations)? I understand that it would be very hard during rebuttal period, but this is also a suggestion.

3 comments

Thank you for reading our paper carefully, and we are sorry that the submitted version was so hard to follow. Following your suggestion, we have rewritten the main text from the beginning, revising the motivation, the organization, the related work, and the explanations, and adding Figure 1 to illustrate the main idea in the introduction. We are grateful that you saw value in the idea despite the presentation, and we hope the revised version makes it clear.

We rewrote the introduction so that it starts from the motivation, explains what earlier analyses leave open, and states our question before the results. We kept the list of contributions as the theorems we prove, because they are our contributions.

As you suggested, we set all vectors and matrices in bold, for example x\mathbf x, M\mathbf M, and WA\mathbf W_A, and kept scalars in plain type.

In Section 3, we removed the references to results that appear later and defined the terms that the theorems use, including the signal and nuisance coordinates, the signal block, the two notions of denoising, and score–encoder separation.

We also reorganized the paper so that Section 3 defines the model, the loss, and the gradient flow before Section 4 states the theorems with the mechanism behind each. In Section 6, we kept only one equation, the definition of the metric.

If any part of the revised paper is still hard to read, we would be grateful to know where, and we would be glad to revise it.

I thank the authors for the hard work of the rebuttal.

I think that the revision improved the paper. However, the revised version is almost a new paper with respect to the one originally submitted, in which most of the sections have been drastically changed and rewritten. Also, the original paper was 80 pages, now it is 38, and the authors have not specified what they removed and the reason behind the choice.

I think that submitting a non-ready paper and then take the one-month review time to completely revise it or to completely change it after rebuttal is a bit unfair with respect to authors who have carefully submitted a ready paper.

I am sorry but I have to confirm my score.

Thank you for reading our revision and for acknowledging that it improved the paper. We apologize that our previous response did not point you to item (5) of our global response. This item lists where each main result of the submission appears in the revised paper and why we removed the others.

Theorems 1 and 2 and Proposition 1 combine statements that we proved in the submission. To address the comments of Reviewers wP3q and AvFa, we added the non-Gaussian analysis in Section 5, the effect of cosine normalization, and the tests on pretrained image–text models in Section 6.2 (items (2)–(4)).

The reduction from 80 to 38 pages comes from the appendix, which we shortened from 67 to 26 pages. The proofs of the main results took 31 pages in Appendices A–C of the submission, and we rewrote them in 15 pages. For the reasons given in item (5), we removed the weight-decay analysis (Appendix D of the submission) and the compression and ranking results (Appendix F and part of Appendix E). Since we limited the appendix to what the main text uses, we also left out the additional related work (Appendix I) and other material such as the sweep over the output dimension and the comparison with an explicit encoder-alignment regularizer in Appendix G.

Reviewer_AvFaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the distinction between denoising individual representations and removing nuisance interactions from cross-modal similarity scores.

Critical strengths

The paper carefully distinguishes finite-time persistence from positive asymptotic nuisance.

Critical weaknesses
  1. The presentation and motivation need improvement. The motivation for analyzing population gradient flow, and its relevance to practical multimodal contrastive learning, should be explained more clearly. The theoretical section introduces numerous assumptions, definitions, and results with limited intuitive explanation of why they are needed and how they contribute to the central research question. The paper would benefit from a clearer explanation of the modeling choices and the practical insights enabled by this analysis.

  2. The experimental evidence is insufficient to establish the practical relevance of the theoretical findings. The controlled experiments primarily validate behavior within an idealized setting closely related to the theoretical model. The nonlinear experiments demonstrate residual encoder sensitivity at finite checkpoints, but do not establish substantial or persistent nuisance retention. Meanwhile, the real-data compression experiments illustrate differences between reconstruction and comparison-preservation objectives, rather than directly validating the proposed score–encoder separation mechanism. More direct experiments measuring both encoder and score sensitivity to controlled nuisance variations on real data would strengthen the paper.

  3. The connection to sample-normalized cosine similarity remains unclear. The theory uses expected-norm normalization, whereas sample-normalized cosine similarity can remain sensitive to nuisance through the embedding norms, even when all off-signal bilinear interactions vanish. As demonstrated in Appendix E.7, this can even change candidate rankings. Although the paper acknowledges this discrepancy, it does not establish under what conditions its theoretical conclusions transfer to cosine similarity. This limits the applicability of the central results to practical contrastive models.

  4. The significance of the dynamical contribution needs sharper positioning. Prior work already establishes that contrastive representations can retain modality-specific information, and the static possibility of score-encoder separation follows naturally from orthogonal private subspaces.

Questions for authors

See my weakness

2 comments

Thank you for reading our paper carefully, and we are sorry that it was still difficult to follow. We rewrote the main text around the distinction you saw as its strength, giving Theorem 2 one part for finite times and one for the limit.

W1: "the motivation for analyzing population gradient flow"

We analyze the population gradient flow because a finite checkpoint cannot tell whether the remaining nuisance is still decaying, and this flow is the large-batch, small-step limit of CLIP training (Section 3.1).

We introduced each assumption where it is used and stated why it is needed. For example, generic initialization (Definition 1) excludes the measure-zero set of initializations whose score collapses. Linear encoders turn the score into a fixed matrix whose blocks separate what the score depends on from what each encoder keeps (Eq. (8)), and we removed the Gaussian assumption in Section 5. We also added Figure 1 to illustrate the main claim.

For practice, strong retrieval does not certify an encoder for reuse. In pretrained image–text models, content-preserving transformations move the image embedding by 41% but change the ranks of the matching captions by only 5% (Section 6.2).

W2: "more direct experiments … on real data"

As you suggested, we measured both sensitivities on real data (Section 6.2). Across 13 pretrained image–text models and two datasets, all three content-preserving transformations affect the scores less than the image embeddings, while randomly initialized networks show no such gap. To test persistence, we extended nine nonlinear runs to 150,000 updates with a 100-fold learning-rate decay. At the end, each encoder's outputs for two nuisance draws still differ by 6–7 degrees (Section 6.1). We agree that the compression study does not test the separation, and we no longer use it as evidence.

W3: "under what conditions its theoretical conclusions transfer"

In Proposition D.1, we show that once the score is denoised, the cosine score changes only through ratios of output norms. A query's ranking is then preserved when these factors are equal across candidates, and the example after the proposition (Appendix E.7 in the submission) shows a reversal when they differ. In 13 pretrained models, this norm effect lowers image-to-text Recall@1 by about 19 points (Appendix E.4).

W4: "the static possibility of score-encoder separation"

We agree that retention and separation are both possible at minimizers, and Proposition 1 gives the exact condition, d≥r+2d\ge r+2. The open question was whether training reaches such a state while both encoders move. We prove that training removes nuisance from the score while both encoders keep it at every finite time (Theorems 1 and 2). For d≥r+2d\ge r+2, it also converges to separation from an open set of initial states. IndiSeek, for example, explicitly allows its CLIP-trained representation to retain modality-specific information, and Theorem 2 shows that training does leave it there.

We hope the revised version is easier to follow, and we would be glad to clarify anything that remains unclear.

Thank you for the detailed response and for running the direct test on pretrained models. I appreciate that the results are reported as they are, including where they fall short. However, my main concerns remain. In the pretrained models, the scores change about half as much as the embeddings, and, as the authors note, the movement is not orthogonal to the text embeddings, so the mechanism behind the theory is not observed in practice. The cosine-normalization analysis is helpful, but the 19-point drop in Recall@1 shows that the norm effect is large in real models, a setting that the authors agree the theorems do not cover. On the theory side, the non-Gaussian extension is a local result, the size of the open set in Theorem 2 remains unknown, and high initial temperatures leave little nuisance in the encoders. Overall, the evidence that encoders retain nuisance in a practically meaningful way is still limited. I therefore keep my rating at 2.

Reviewer_wP3qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the relationship between reliable similarity score and encoder denoising in contrastive learning. The key observation is that retrieval relies only on the similarity score formed by the two encoders, while modality-specific nuisnaces can be kept in the encoders but reside in the orthogonal component of the other encoder's output space. And this is called score-encoder speration in the paper.

The theory part of the paper analyzes the training dynamics of contrastive learning with linear encoders and a learned temperature, in a linear-Gaussian model with shared signal, modality-specific nuisance, and measurement error. The authors mainly show that the nuisance interactions, either with the other modality's shared signal, or the other modality's nuisance, vanish from the score during trianing, but is never fully removed from the encoder output at any finite training time.

The experiments support the separation in both synthetic and real data. A compression study on pretrained CLIP features further shows that reconstructing features, preserving the original model’s comparisons, and maximizing retrieval accuracy are distinct objectives.

Critical strengths

Clear seperation of different concepts with rigorous theoretical analysis. The paper clearly seperates denosing in scores and denosing in encoders, and points out that reliable similarity score does not require encoder denoising. This perspective is valuable for understanding CLIP-style retrieval, embedding compression, and representation auditing. The theory also analyzes the full training dynamics and handles nontrivial technical issues.

Transparent experiments and reporting. The synthetic experiments use controlled nuisance resampling, which directly measures how sensitive the score and the encoders are to nuisance. The compression study evaluates on held-out cohorts drawn after the method was frozen. The authors also report results that might appear unfavorable, such as encoder nuisance being almost fully removed at high initial temperature and CCA achieving higher retrieval accuracy than their method. These results are in fact consistent with the paper’s message.

Critical weaknesses

The theoretical setting is oversimplified, and its connection to practice is unclear. Under the paper’s assumptions, the objective reduces to a weighted least-squares factorization without the softmax nonlinearity, and score-level denoising follows almost directly from convergence. The stronger claim that both encoders retain nuisance is established only on an unquantified neighborhood of constructed optima, and the paper’s own experiments show encoders being fully denoised at high initial temperature. More importantly, several key ingredients do not carry over to real systems: with cosine normalization, nuisance in “invisible” directions still affects the score through the embedding norm, and nonlinear encoders do not admit a fixed score matrix to decompose.

The empirical evidence does not directly test the central claim on real models. The only real-data comparison of score and encoder variability conditions on class labels, so within-class variation reflects semantics rather than isolated nuisance. The nonlinear metric compares a scalar score with a high-dimensional embedding, which favors the score from dimensionality alone. The compression study does not test the separation itself. A direct experiment on pretrained CLIP would be more convincing, for example, measuring how much image embeddings move under semantics-preserving augmentations, and how much of that movement lies outside the span of text embeddings, compared with the resulting changes in retrieval rankings.

Questions for authors

Applicability beyond simplified setting. With the cosine normalization used in practice, nuisance placed in directions orthogonal to the other encoder’s outputs still changes the embedding norm and hence the score. Which of the theoretical conclusions do the authors expect to hold under cosine normalization? Can its effect on score-level invisibility be quantified, either theoretically or empirically?

Direct test on pretrained models. Could the authors test score–encoder separation directly on a pretrained CLIP model? For example, applying mild semantics-preserving augmentations (e.g., small crops, color jitter) to images, measuring how much the image embeddings move and whether this movement is largely orthogonal to the text embeddings, and comparing it with the resulting changes in retrieval rankings with a random-direction baseline. Substantial movement that is orthogonal to the text embeddings and leaves rankings stable would provide much stronger evidence for the paper’s central claim.

Practical implications. Beyond compression, what practical guidance do the authors draw from the score–encoder distinction for training or using contrastive models?

1 comment

Thank you for the careful review, and especially for designing a direct test on pretrained CLIP. We ran it on 13 models (Section 6.2) and report the outcome as it is, including where it falls short of full orthogonality.

W1: "score-level denoising follows almost directly from convergence"

Because training moves the encoders and the temperature rather than the score matrix, the score can collapse to zero along the flow. Theorem 1 shows that almost every initialization avoids this and converges to a denoised score with a nonzero signal block. The new Section 5 proves local convergence to score–encoder separation for non-Gaussian data, where the loss keeps the softmax nonlinearity (Theorem 3).

We agree that Theorem 2 leaves the size of its open set unknown, but its finite-time part covers almost every initialization. Starting from a separating minimizer whose encoders were rotated by up to 0.6 radians, all 24 trajectories returned to the minimum with at least 30% of each squared norm in nuisance (Section 6.1). High initial temperatures leave little nuisance, which Theorem 2 also allows.

W1 and Q1: cosine normalization and nonlinear encoders

We agree that our theorems cover neither setting, so we quantified the cosine effect both ways you asked. With the inner products fixed, the cosine score is multiplied by the ratio of old to new output norms (Proposition D.1). A query's ranking is then preserved when all candidate norms change by a common factor and can reverse otherwise. In 13 pretrained models, fixing every image–caption inner product and removing part of each caption norm lowers image-to-text Recall@1 by about 19 points (Appendix E.4).

Under cosine training, the score still depends less on nuisance than the encoders do (Figure 2(b)). Nonlinear networks have no score matrix, but training makes their rankings robust to redrawn nuisance while both encoders stay sensitive to it (Section 6.1).

W2 and Q2: "a direct experiment on pretrained CLIP"

Following your design, we applied three mild transformations, the strongest being color jitter of up to 40% with JPEG compression. We measured the change in each image embedding, in its scores against all captions, and in the rank of the matching ones, relative to replacing the image with a different one. For the strongest transformation these are 41%, 22%, and 5%, and in every model, setting, and dataset the scores change less than the embedding (Table 1, Appendix E.2).

The movement is not orthogonal to the text embeddings as a whole, but it leaves the rankings stable, and randomly initialized networks show no such gap. We did not use random directions as the reference, because in high dimensions they are nearly orthogonal to every caption embedding, so this baseline would mainly reflect each model's dimension, the same dimensionality issue you raised.

Q3: "beyond compression"

First, strong retrieval does not show that an encoder ignores nuisance. In Section 6.2 the embeddings move by 41% while the rankings change by 5%, so an encoder meant for reuse should be tested under controlled nuisance variation. Second, InfoNCE alone leaves nuisance in both encoders at every finite time (Theorem 2), so reuse for other tasks or modalities needs an explicit separation, as in DisentangledSSL and IndiSeek.

We hope these changes address your concerns, and we would be glad to clarify anything that remains.

Comments on the paper

1 comment

We thank the Area Chair and all reviewers for their time and careful reading. Because the reviews showed that the submitted version did not convey its question, we have rewritten the main text from the beginning. Almost every paragraph has changed, so we have not marked individual changes in color. We list the main changes below and answer each review in its own thread.

(1) Writing (Reviewers AvFa and jBHp)

We have rewritten the introduction so that it starts from the use of modality-specific information in downstream tasks, explains what existing theory leaves open, and states our question before the results. We have added Figure 1 to illustrate the main claim and reorganized the related work around the differences between earlier analyses and ours. We have also reorganized Sections 3 and 4 so that every term used in the theorems is defined before the first result and each theorem is stated together with the mechanism behind it.

(2) Non-Gaussian data (Reviewer wP3q)

We have added Section 5, which proves local convergence to score–encoder separation for non-Gaussian data (Theorem 3). For Gaussian data the loss is a quadratic function of the score matrix, whereas for non-Gaussian data it keeps the softmax nonlinearity.

(3) Pretrained image–text models (Reviewers wP3q and AvFa)

We have tested the separation directly in 13 pretrained image–text models by measuring how content-preserving image transformations change the encoder outputs, the scores, and the rankings (Section 6.2 and Table 1).

(4) Cosine normalization (Reviewers wP3q and AvFa)

We have derived the exact effect of sample-wise cosine normalization on the scores (Appendix D) and measured its size in the pretrained models (Section 6.2 and Appendix E.4).

(5) Results of the submission

In the submission, Theorem 1 showed that the off-signal blocks of the score vanish when the temperature stays bounded. Corollaries 2–4 then showed, for almost every initialization, that the loss becomes negative in finite time (Corollary 2), that the temperature stays bounded (Corollary 3(iii)), and that the signal weights and the score do not vanish in the limit (Corollaries 3(iv) and 4).

Theorem 1 of the revision states the combined conclusion of these four results for almost every initialization, and the intermediate statements are now steps of its proof (Appendix B). Corollary 3(ii), which states that the nuisance weights never vanish at finite times, is now the finite-time statement of Theorem 2. The table below lists where each main result of the submission appears in the revision.

SubmissionRevision
Theorem 1; Corollary 2; Corollary 3(iii)–(v); Corollary 4Theorem 1
Corollary 3(ii)Theorem 2, finite-time statement
Proposition 5; Proposition E.1Proposition 1
Proposition E.2Theorem 2, limit statement

We have removed results that the main theorems do not use. The bound on the probability that a matched pair scores higher than an unmatched pair (Section 4.5) and the output dimensions needed to preserve the scores or the encoder outputs under linear compression (Proposition 6) describe a fixed state with a denoised score rather than the training dynamics, and Section 6.2 now measures both effects in pretrained models. The weight-decay analysis (Appendix D of the submission) concerns a modified objective with a penalty on the encoder weights, and Corollary 3(i) concerns the signal weights rather than the nuisance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.