acceptodds
Under review as a conference paper at ICLR 2027

Uncovering the alignment and repelling tradeoff in CLIP: a path towards Gaussian geometry

Abstract

The contrastive learning framework introduced by CLIP learns image and text representations in a shared embedding space, aligning corresponding samples while repelling representations across the dataset. Building on the alignment–uniformity perspective of Wang and Isola, we identify the term responsible for uniformity in the CLIP objective as a repelling term. We show that its optimum is incompatible with perfect cross-modal alignment: improving the repelling term beyond the uniform distribution on the sphere necessarily requires a positive paired gap. To address this, we propose a prototype objective that shifts repulsion from individual embeddings to representations of image-text pairs. We prove that, under a mild condition on the inverse temperature, its unique population minimizer is the uniform distribution on the sphere with perfect cross-modal alignment. More generally, when perfect alignment is not possible, the optimal distribution is uniform on a sphere whose radius is determined by the HGR maximal correlation. We further show that fixed-dimensional projections of the optimal representations converge to an isotropic Gaussian as the embedding dimension increases. Experiments on MS-COCO and CC3M show that fine-tuning CLIP with the prototype objective substantially reduces the modality gap and improves spherical uniformity and coordinate Gaussianity, while maintaining comparable probe performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Contrastive learning, Multimodal learning, Representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ncqHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the geometric properties of population CLIP loss. It first characterizes the optima of the alignment and repulsion components of the loss, respectively. It then gives the asymptotic optimal bound for the uniform diagonal configuration. The authors further regard the midpoint of the image-text pairs as a prototype; under the temperature condition, the paper describes unrestricted and fixed-gap population optima for the prototype objective, deriving a Gaussian limit for fixed-dimensional projection. Experiments show improvements in terms of geometric metrics on MS-COCO and CC3M, but the downstream performance varies.

Critical strengths
  1. Theorem 5 provides a meaningful geometric characterization. The paper gives a sharp inverse-temperature threshold for the prototype objective to achieve both perfect alignment and spherical uniformity, and further characterizes the optimal prototype law for a fixed second moment. I find this is the paper's most distinct theoretical increment beyond previous alignment-uniformity explanations.

  2. The experiments support the claim that the proposed objective can substantially change the geometry from pretrained models. The comparison fixed the architecture, initialization, and batch size, and only replaced the loss. The centroid gap, uniformity, and coordinate KS all show improvements on MS-COCO and CC3M.

Critical weaknesses
  1. The theoretical increments are modest, and the positioning against previous work deserves more careful consideration.

Particularly, Cai et al. (2026) have analyzed cross-modal divergence from the population energy view. Dufumier et al. (2023) have proposed centroid repulsion and characterized spherical uniformity of the centroids in the asymptotic regime under perfect alignment. The derivation of Gaussian projections from spherical uniformity has already been proposed in Betser et al. (2026). So, the major contribution of this work should be more concentrated on the temperature thresholding and fixed-gap characterization in Theorem 5, rather than repulsion or Gaussianity themselves.

Also, Theorems 1, 2 only demonstrate that the optima of the two components are not compatible, but this does not mean the CLIP loss optimum necessarily entails a nonvanishing gap. Theorem 4 provides a necessary condition for outperforming uniform reference, which does not prove that such a configuration must exist. I realize this limitation is acknowledged, but these results appear insufficient to establish that this tradeoff explains the modality gap observed in pretrained CLIP-like models.

  1. The theoretical claim does not fully align with the experimental evaluation.

The fixed-gap results are derived from unrestricted triplets. Yet, a fixed gap cannot ensure that the optimal uniform prototype law on the sphere of radius RR can be realized by separate encoders from given data distributions. Lemma 4 does not address this realizability. Besides, HGR gives an upper bound of RR, which makes the claim that the "whose radius is determined by HGR" in the Abstract feel like an overclaim compared to what it supports.

Experiments did not evaluate this fixed-gap geometry directly either. For R<1R<1, Theorem 6 is for the prototype distribution, whereas Figure 4 evaluates the embedding coordinates pooled from the two modalities. Also, the paired gap in Table 1 is still about 1.17, which cannot be treated as a perfect-alignment situation. The current experiments support improvements in terms of coordinate statistics, but is insufficient to support the theoretical claims.

References:

Dufumier B, Barbano C A, Louiset R, et al. Integrating Prior Knowledge in Contrastive Learning with Kernel. International Conference on Machine Learning. ICML 2023.

Betser R, Gofer E, Levi M Y, et al. InfoNCE Induces Gaussian Distribution. ICLR 2026.

Cai Y, Zhang Z, Liu Y, et al. The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence. ICML 2026.

Questions for authors
  1. I suggest the authors to address the positioning of the work against previous analyses more carefully. For example, compare the analytical object, assumptions, and claims more clearly with Cat et al. (2026) and Dufumier et al. (2023). Besides temperature thresholding and fixed-gap characterization, what are the new findings that cannot be derived from prior work?

  2. For a fixed raw image-text joint distribution, what condition may characterize that the uniform prototype law on the sphere of radius RR can be realized from the separate encoders?

  3. When the paired gap is nonzero, why does Figure 4 report the pooled modality coordinate, not the prototype characterized in Theorem 6? Does the experiment align with theoretical predictions if you directly measure the radius distribution and the projection distribution of the prototype?

Reviewer_D4hiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the alignment and repulsion trade-off in CLIP through a population-level analysis of its contrastive learning objective. The authors propose a multimodal prototype objective that applies repulsion to image-text pair midpoints and theoretically characterize conditions under which its optimal representations exhibit perfect alignment, spherical uniformity, and asymptotically Gaussian coordinate distributions. Experiments on MS-COCO and CC3M demonstrate substantial improvements in embedding geometry, although these improvements do not consistently translate into better downstream classification or cross-modal retrieval performance.

Critical strengths
  1. Relevant theoretical investigation. The paper addresses an important question concerning the geometry induced by multimodal contrastive learning. Understanding how CLIP balances cross-modal alignment and representation repulsion is relevant to the design of more effective vision-language objectives.

  2. Mathematically structured analysis. Theorems 1 and 2 characterize the individual minimizers of the alignment and repelling terms, while Theorems 3 and 4 provide additional results concerning the uniform spherical reference and necessary conditions for improving upon it. These results give a precise mathematical account of competing incentives within the analyzed population objective.

  3. Interesting characterization of the prototype objective. Theorem 5 establishes conditions under which the multimodal prototype objective admits a uniform spherical optimum with perfect alignment. Its characterization of the optimal prototype distribution under a fixed alignment gap provides an additional theoretical contribution beyond the analysis of the original CLIP objective.

  4. Connection to maximal correlation and Gaussian geometry. The relationship between the optimal prototype radius and HGR maximal correlation is theoretically interesting. The asymptotic Gaussianity result also provides a connection between spherical representation geometry and recent theoretical investigations of Gaussian embeddings.

  5. Substantial improvements in geometric metrics. Table 1 shows that prototype fine-tuning reduces the centroid gap from 0.8219 for frozen CLIP to 0.0232, while spherical uniformity excess decreases from 2.0128 to 0.0421. The coordinate Gaussianity metric also improves considerably, supporting the effectiveness of the objective in reshaping embedding distributions.

  6. Controlled empirical comparisons. The experiments compare the standard CLIP and prototype objectives using the same architecture, initialization, and evaluation settings. This design helps isolate the effects of the training objective on the resulting embedding geometry.

  7. Transparent discussion of negative downstream findings. The authors acknowledge that improvements in Gaussianity and modality alignment do not consistently improve retrieval or zero-shot classification. This is a useful observation that highlights the distinction between geometric regularity and task-relevant discriminative structure.

Critical weaknesses
  1. Insufficient coverage of closely related theoretical literature. Although the paper cites foundational work on contrastive geometry and the modality gap, its discussion omits several directly relevant theoretical investigations. Most notably, Yin et al. (ICML 2026), "Towards Uniformity and Alignment for Multimodal Representation Learning," explicitly identify an alignment-uniformity conflict in multimodal InfoNCE and propose a decoupled objective to address it. Fahim et al. (2024), "It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap," similarly investigate the relationship between contrastive objectives, representation uniformity, and modality separation. These works are closely related to the central motivation of the submission and should be discussed explicitly.

  2. Insufficient differentiation of the theoretical novelty. The observation that multimodal contrastive objectives induce competing geometric incentives is not entirely new. Yin et al. (2026) already investigate an alignment-uniformity conflict, while Cai et al. (2026), which the paper cites, provide a population-level geometric account of persistent cross-modal divergence. Yaras et al. (2025) also analyze the emergence of the modality gap through gradient-flow dynamics. Although the mathematical formulations differ, the paper does not sufficiently clarify which conceptual insights are genuinely new and which are alternative characterizations of previously identified phenomena.

  3. Limited characterization of the complete CLIP objective. Theorems 1 and 2 establish that the alignment and repelling terms have incompatible individual minimizers, but this does not characterize the global minimizers of their sum. The paper explicitly acknowledges this limitation. Consequently, the analysis demonstrates competing optimization incentives but does not establish that the observed modality gap in practical CLIP models necessarily follows from the characterized conflict. The distinction between incompatible component-wise optima and the behavior of the complete objective deserves greater emphasis.

  4. Limited methodological novelty of the prototype objective. The proposed loss adapts the prototype construction of Dufumier et al. (2023), while the Gaussian convergence result follows from established high-dimensional spherical geometry. The principal novelty appears to lie in the multimodal population-level characterization and the connection to maximal correlation. The paper should position these results more explicitly as its central contributions rather than emphasizing the introduction of an entirely new solution to the alignment-repulsion conflict.

  5. Insufficient comparison with existing modality-gap mitigation methods. Previous work has proposed modifying contrastive objectives to improve alignment and uniformity, including Fahim et al. (2024), Yaras et al. (2025), and Yin et al. (2026). However, the empirical evaluation primarily compares the proposed objective against conventional CLIP fine-tuning. Comparisons with these existing approaches would help determine whether the prototype objective offers distinct geometric or practical advantages.

  6. Geometric improvements do not translate into consistently better downstream performance. Despite substantial improvements in alignment, uniformity, and Gaussianity, the proposed objective underperforms standard CLIP fine-tuning on cross-modal retrieval. Table 2 reports mean recall of 73.93% for the prototype objective compared with 77.93% for CLIP on CC3M validation, and 58.39% compared with 62.62% on MS-COCO. These consistent regressions weaken the practical motivation for replacing the standard CLIP objective.

  7. Insufficient justification for Gaussianity as a desirable multimodal representation property. The paper motivates Gaussian geometry through prior theoretical results concerning downstream prediction. However, those results do not necessarily establish that isotropic Gaussian representations are optimal for multimodal retrieval or zero-shot classification. The reported experiments reveal this limitation, suggesting that Gaussianity alone does not capture the semantic structure required for general-purpose vision-language learning.

  8. Incomplete connection between population-level theory and practical encoders. The theoretical results optimize over probability measures and couplings, whereas practical representations are constrained by the data distribution, encoder architecture, and finite training procedures. Although the analysis introduces maximal correlation to characterize limitations on alignment, it does not establish when the theoretically optimal distributions can be realized by actual neural encoders.

  9. Limited empirical validation of the maximal-correlation characterization. The relationship between optimal prototype radius and HGR maximal correlation is a potentially distinctive theoretical result. However, the paper does not estimate maximal correlation for its datasets or investigate whether the observed representation geometry agrees with the predicted relationship. This leaves an important theoretical contribution without direct empirical support.

  10. Limited evaluation of Gaussianity. The experiments primarily measure Gaussianity through one-dimensional coordinate-wise Kolmogorov-Smirnov distances. Although these measurements are relevant to the theoretical result, they do not establish joint multivariate Gaussianity or independence across representation dimensions. Additional tests involving random low-dimensional projections or cross-coordinate dependencies would strengthen the empirical analysis.

  11. Restricted experimental scope and statistical validation. The principal experiments use one pretrained CLIP ViT-B/32 architecture, MS-COCO, and a 100,000-pair subset of CC3M. The main geometric comparisons also lack repeated-run statistics. These limitations make it difficult to establish whether the observed behavior generalizes across model families, training scales, and random seeds.

  12. Insufficient explanation of the retrieval degradation. The proposed objective moves repulsion from individual image and text embeddings to pair midpoints, but the paper does not sufficiently analyze how this changes the preservation of instance-level semantic distinctions. Reduced marginal separation and greater spherical uniformity do not necessarily imply better discrimination between correct and incorrect image-text pairs. A more detailed analysis of retrieval margins and local semantic neighborhoods would help explain the observed performance losses.

  13. Incomplete discussion of related findings on modality gaps and downstream utility. Yi et al. (ICML 2026), "The Convergent Representation of Contrastive Vision-Language Models: Geometry, Modality Gap and Shared Space Alignment," investigates the geometric structure of convergent multimodal representations and the relationship between modality separation and downstream performance. This is particularly relevant to the submission's finding that reducing the gap does not necessarily improve retrieval. A more comprehensive comparison would strengthen the theoretical interpretation of these results.

Questions for authors
  1. Relationship to Yin et al. (ICML 2026). How does the proposed alignment-repulsion analysis differ from the alignment-uniformity conflict investigated in "Towards Uniformity and Alignment for Multimodal Representation Learning"? Since both studies analyze competing incentives in multimodal InfoNCE and propose alternative objectives to address them, a direct theoretical comparison is important for establishing novelty.

  2. Relationship to earlier contrastive-gap theories. How does the proposed explanation of modality separation differ from Fahim et al. (2024) and Yaras et al. (2025)? Can the authors identify specific mathematical conclusions that extend beyond these earlier investigations?

  3. Differentiation from Cai et al. (2026). The paper cites Cai et al., which also provides a population-level geometric analysis of cross-modal divergence. Could the authors explain more precisely which results are genuinely new relative to that framework, particularly concerning the alignment-repulsion conflict and the persistence of modality separation?

  4. Global geometry of the CLIP objective. Theorems 1 and 2 characterize the minimizers of the individual alignment and repelling terms rather than their sum. What conclusions can be drawn about the global minimizers of the complete CLIP objective? Can the authors clarify the extent to which the current analysis explains the modality gap observed in practical pretrained models?

  5. Novelty of the prototype-optimality result. Since the prototype objective is adapted from Dufumier et al. (2023), what additional mathematical difficulties arise specifically in the multimodal setting? Could the authors explain which parts of the optimality characterization cannot be obtained directly from the existing unimodal analysis?

  6. Comparison with existing geometric objectives. How does the proposed prototype loss compare with established modality-gap mitigation methods or alignment-uniformity regularizers under matched training conditions? Such comparisons would help establish whether its geometric improvements are distinctive.

  7. Geometry versus retrieval performance. Why does the prototype objective substantially reduce the centroid gap while degrading cross-modal retrieval? Could the authors examine positive-negative similarity margins, instance-level ranking consistency, or neighborhood preservation to identify which aspects of discriminative structure are lost?

  8. Justification of Gaussianity. What theoretical or empirical evidence supports Gaussianity as a desirable objective for multimodal representation learning specifically? Given that retrieval and zero-shot classification do not consistently improve, can the authors clarify the practical tasks for which the proposed geometry is expected to be beneficial?

  9. Relationship to Yi et al. (ICML 2026). How do the proposed geometric characterizations relate to prior findings connecting dimensional collapse, modality gaps, shared-space alignment, and downstream performance? Could these existing results help explain why reducing the modality gap does not necessarily improve retrieval?

  10. Realizability of theoretical optima. Under what assumptions can neural image and text encoders realize the theoretically optimal uniform and perfectly aligned distributions? How do finite model capacity, data ambiguity, and incomplete semantic correspondence affect this characterization?

  11. Empirical validation of maximal correlation. Can the authors estimate or approximate HGR maximal correlation on MS-COCO or CC3M and compare it with the observed paired gap or prototype radius? Such an experiment would help substantiate the practical relevance of the theoretical relationship.

  12. Joint Gaussianity. Could the authors examine random low-dimensional projections, cross-coordinate correlations, or multivariate normality to provide stronger empirical evidence for the Gaussian geometry predicted by the theory?

  13. Temperature-dependent behavior. The prototype-optimality theorem includes explicit temperature conditions. Can the authors provide experiments across different temperatures, including settings near the theoretical threshold, to assess whether the predicted changes in optimal geometry are reflected in practical training?

  14. Hybrid objective design. Since prototype training improves geometric regularity while CLIP preserves stronger retrieval performance, have the authors considered combining the two objectives? A controlled experiment could determine whether the geometric benefits can be retained without sacrificing cross-modal discrimination.

  15. Experimental generalization and reliability. Do the findings hold for other CLIP architectures, larger datasets, and multiple random seeds? Additional experiments would help determine whether the proposed geometric behavior is a general consequence of the objective rather than a property of the specific pretrained model and fine-tuning configuration.

Reviewer_NYxZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies CLIP geometry by separating its alignment and repelling terms. It analyzes a midpoint-based prototype objective with a uniform, aligned population optimum under temperature conditions, and derives a Gaussian projection limit. Fine-tuning improves geometric metrics, but downstream results are mixed.

Critical strengths
  • The characterization of the prototype optimum, including the sharp temperature threshold, is an interesting theoretical contribution.
  • The HGR-based bound shows how dependence between the input modalities limits achievable alignment for centered encoders, regardless of the training objective.
  • The distinction between paired and centroid gaps is useful. Experiments compare losses from the same initialization, and the paper openly reports downstream tradeoffs.
Critical weaknesses
  1. Incompatible minima of CLIP’s alignment and repelling terms do not establish a nonzero gap at the optimum of their sum. The conditional improvement result therefore leaves the explanation of the observed modality gap incomplete.
  2. The prototype analysis optimizes over arbitrary couplings without establishing realizability by separate encoders. HGR maximal correlation only bounds the radius, rather than determining it as suggested in the abstract.
  3. The Gaussianity diagnostics pool image and text embeddings, whereas the nonzero-gap theorem concerns prototypes. Temperature, dimension, and batch-size sweeps are also missing, leaving the theoretical predictions insufficiently tested.
  4. Evaluation is limited to one CLIP ViT-B/32 backbone, COCO, a small CC3M subset, and CIFAR-10/100. Comparisons with modality-gap reduction methods or simple alignment/uniformity regularizers are needed to establish the benefit of the prototype construction.
  5. Downstream benefits are mixed. NCM improves, but kNN gains are small and lack uncertainty estimates, while retrieval, zero-shot classification, and fitted probes favor CLIP fine-tuning. Better geometric metrics do not yet demonstrate a clear practical advantage.
Questions for authors
  1. Can you give a concrete setting where the full CLIP loss strictly improves on the uniform aligned reference?
  2. Under what data assumptions is the fixed-gap spherical prototype optimum realizable by separate encoders, and when is the HGR bound tight?
  3. Could you test prototype projections directly and report batch-size sensitivity? The appendix notes that the finite-batch training objective can have different minimizers from the population loss.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.