acceptodds
Under review as a conference paper at ICLR 2027

Alignment Governs the Modality Gap: Understanding Its Interplay with Uniformity and Downstream Performance

Abstract

Contrastively trained vision-language models exhibit a persistent modality gap, with image and text representations occupying distinct regions of a shared embedding space, yet what governs this gap remains unclear. While one line of prior work has attributed the modality gap to modality-wise localization, we instead investigate the role of cross-modal alignment. Through an analysis of InfoNCE gradients, we show that InfoNCE does not explicitly promote alignment between the two modality regions. Using a controlled intervention that explicitly strengthens this form of alignment, we substantially reduce the modality gap, as measured by both positive-pair similarity and centroid distance, while largely preserving zero-shot classification and cross-modal retrieval performance. Notably, within-modality uniformity deteriorates as the gap decreases, demonstrating that improved uniformity is not necessary for gap reduction. We further uncover an empirical trilemma among positive-pair similarity, within-modality uniformity, and downstream performance: across controlled and alternative interventions, we find no examined configuration that simultaneously improves positive-pair similarity and uniformity while preserving downstream performance. Finally, we find that improved uniformity provides an additional route to reducing centroid distance without improving positive-pair similarity, yet targeting higher uniformity during training does not consistently yield a more favorable modality-gap–performance relationship under distribution shift. These findings identify direct cross-modal alignment as a mechanism governing modality-gap reduction, while revealing the metric-dependent role and limitations of within-modality uniformity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Models, Contrastive Learning, Modality Gap, Cross-Modal Alignment, Uniformity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ra2VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper empirically investigates the relationship between the modality gap and downstream performance in vision-language models such as CLIP. First, the authors confirm that the InfoNCE loss used for CLIP training does not necessarily induce modality alignment. By constructing a setting in which the modality gap is reduced by InfoNCE, the authors propose AA-InfoNCE, a modification of InfoNCE that explicitly encourages modality alignment (absolute alignment). The authors then conduct controlled experiments using AA-InfoNCE. While prior work suggests that reducing the modality gap while improving uniformity is beneficial, this paper empirically demonstrates a trilemma among the modality gap, uniformity, and downstream performance; specifically, when uniformity is held fixed, there is a trade-off between downstream performance and positive-pair similarity.

Critical strengths
  • S1: The paper provides a detailed analysis of the relationship between the modality gap and uniformity, both of which have been considered important factors for improving downstream performance in prior work.
  • S2: The paper proposes AA-InfoNCE, which strengthens absolute alignment while preserving the form of InfoNCE.
Critical weaknesses
  • W1: The main experiments are conducted only with OpenCLIP ViT-B/16. It is unclear whether similar results hold for other models in the CLIP family (e.g., SigLIP and OpenAI CLIP). The scope within which the claims hold should be appropriately framed.
  • W2: Since positive-pair identification accuracy is not defined, the significance of the opposite trends observed in Figs. 2(a) and 2(b) is unclear.
  • W3: If the modality gap is intended to measure the discrepancy between the distributions of image and text embeddings, it seems more natural to use distributional distances such as MMD or the Wasserstein distance. While I understand that centroid distance and positive-pair similarity are commonly used to measure the modality gap, the fact that they exhibit opposite correlations depending on which metric is used, as shown in Fig. 4, also raises questions about whether these metrics are appropriate for measuring the modality gap.
Questions for authors
  • Q1: Are the reported downstream-task results averaged over all tasks? Does the relationship with metrics such as positive-pair similarity exhibit a similar trend for every individual task?
Reviewer_BfqtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relationship between cross-modal alignment, within-modality uniformity, and downstream performance in contrastively trained vision-language models. Through a gradient analysis of InfoNCE, the authors first distinguish between relative alignment, which emphasizes positive pairs over negative pairs, and absolute alignment, which directly minimized the distance between paired image and text. Based on this, the authors introduce Absolute Alignment InfoNCE (AA-InfoNCE) to explicitly strengthen positive-pair alignment. Experiments show that AA-InfoNCE can substantially reduce the modality gap largely preserving downstream performance, although within-modality uniformity deteriorates. The paper further introduces uniformity-targeting regularization and reports an empirical trilemma among positive-pair similarity, within-modality uniformity, and downstream performance. Finally, the authors show that improved uniformity can reduce centroid distance without increasing positive-pair similarity, but does not consistently improve the modality-gap–performance relationship under distribution shift.

Critical strengths

S1. The paper presents several interesting observations about the interplay between cross-modal alignment, within-modality uniformity, and downstream performance. In particular, the finding that the modality gap can be substantially reduced while within-modality uniformity deteriorates is interesting.

S2. The experimental design is thoughtful. By independently varying the strength of absolute alignment through the AA-InfoNCE margin and controlling within-modality uniformity through explicit target constraints, the authors are able to investigate the relationships among different representation properties in a relatively controlled setting.

Critical weaknesses

Weaknesses:

W1. The theoretical derivation in Sec. 3.1 is based on a simple sample-wise gradient analysis while the training dynamic is based on the population- or distribution-wise interaction among all samples [1] [2]. This makes the theoretical argument less persuasive. In particular, Equation (2) expresses the InfoNCE feature-update direction as a weighted combination of differences between positive and negative text representations. Based on this decomposition, the authors argue that InfoNCE does not explicitly induce a direction toward the paired text representation. However, the absence of an explicit positive-pair displacement vector in this decomposition does not imply that the gradient lacks an absolute-alignment component. In general, the gradient can have a nonzero projection onto the direction connecting the paired image and text representations. More importantly, the discussion of vanishing gap-reducing gradients relies on specific geometric assumptions, but the paper does not demonstrate whether these assumptions characterize the representations learned in the reported experiments. Section 3.2 further motivates the absolute-alignment direction through an approximation involving a hardest negative, assuming that positive pairs are already nearly aligned and that the corresponding negative image lies close to the image anchor. These assumptions are restrictive, particularly in the setting of a persistent modality gap. Moreover, identifying the direction of one negative contribution does not establish that this direction dominates the full InfoNCE gradient.

[1] The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence. Cai et al., ICML 2026.

[2] The Convergent Representation of Contrastive Vision-Language Models: Geometry, Modality Gap and Shared Space Alignment. Yi et al., ICML 2026.

W2. The AA-InfoNCE intervention explicitly introduces a gradient component that encourages positive-pair alignment. Its success in increasing positive-pair similarity therefore demonstrates the effectiveness of the intervention, but does not independently establish that insufficient absolute alignment is the underlying cause of the original modality gap.

W3. In Section 4.3, the authors control uniformity by adding a regularizer that constrains the image and text uniformity metrics toward prescribed targets. They then vary the AA-InfoNCE margin to obtain different levels of positive-pair similarity. Although this provides controlled comparisons at approximately fixed uniformity, it does not isolate the relationship among the three properties independently of the training objectives. Changing the uniformity target also changes the optimization dynamics and potentially affects other aspects of the learned representations, including the covariance structure, semantic neighborhoods, and relative alignment. Consequently, the observed degradation in downstream performance at higher uniformity targets could reflect limitations of optimization interventions rather than an intrinsic incompatibility between high uniformity, strong positive-pair alignment, and downstream performance.

W4. The empirical results support the alignment-uniformity-performance trilemma under the investigated interventions, but the broader trilemma interpretation remains insufficiently justified. Figure 2 shows that downstream performance can remain relatively stable across a substantial range of positive-pair similarity, with degradation occurring primarily under stronger alignment. The paper does not clearly specify what constitutes a meaningful improvement, comparable performance, or a violation of the proposed trilemma. A more convincing analysis would explicitly characterize the Pareto relationships among the three objectives.

Questions for authors

Please see weaknesses.

Reviewer_E6tZOpenReview ↗
Rating
2(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies what drives the modality gap in CLIP-style models. From the InfoNCE gradient, the authors argue that the loss only enforces relative alignment, and they propose AA-InfoNCE, which adds the stop-gradient midpoint of each image-text pair as an extra negative to pull pairs together. Training on CC12M and LAION-400M, they show that this closes the gap while keeping downstream performance up to a point, and that uniformity drops as the gap closes. They further claim an "empirical trilemma" among positive-pair similarity, uniformity and downstream performance.

Critical strengths
  1. The paper is clearly written and the setup is well controlled. The margin m is a clean knob, and all metrics go back to standard InfoNCE as the intervention weakens.

  2. Separating positive-pair similarity from centroid distance is a useful distinction, and the paper shows the two can behave differently.

Critical weaknesses
  1. The main findings are close to true by construction. AA-InfoNCE explicitly pulls each image toward its paired text, so it is no surprise that positive-pair similarity goes up and centroid distance goes down, and pulling pairs together will of course hurt uniformity. This does not show that alignment "governs" the modality gap in standard CLIP training, which is what the title claims. The gradient argument in Section 3.2 also assumes paired representations are already nearly aligned, which is exactly the case where there is no gap to explain. Moreover, prior work proposes uniformity as one way to close the gap, not as a necessary condition, so showing that "uniformity is not necessary" does not overturn anything.

  2. The evidence for the trilemma is narrow and hard to verify. It rests on one backbone, one main dataset and three uniformity targets, and "no examined configuration escapes it" only covers the authors' own handful of interventions. For a purely empirical paper, no code, checkpoints or other reproduction materials are provided, even though all datasets and models used are public. This limits the credibility and robustness of the reported results.

  3. Limited novelty and unclear significance. The intervention is close to geodesic multi-modal mixup (Oh et al., 2023), which also uses mixtures of image and text embeddings as hard negatives, yet the paper never compares with it. The other takeaways are largely known: the paper itself cites prior work showing that fully closing the gap can hurt performance and that alignment and uniformity conflict in CLIP. Finally, the paper never shows why a smaller gap is worth having. It lists tasks that benefit from it in the related work, but evaluates none of them, and on standard benchmarks the reduced gap brings no gain.

Minor issues:

  • Most representation metrics are measured on the CC12M training data, which says little about how the learned space behaves on new data.
  • The CC12M models are small and far from standard CLIP performance, so it is unclear whether the trade-offs hold at scale.
Questions for authors

See Weaknesses. Also,

  1. How does AA-InfoNCE differ from the mixed hard negatives in Oh et al. (2023), and how does it compare empirically?
  2. Beyond showing that an extra loss can close the gap, what does this tell us about why the gap appears in standard CLIP training?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewers for their constructive feedback and valuable suggestions. We have uploaded a revised manuscript addressing the concerns raised in the reviews. The main revisions are summarized below:

  1. Scope clarification: Revised the title, abstract, and Section 3.1 to clarify that our study focuses on modality-gap reduction rather than explaining the emergence or persistence of the modality gap.
  2. Gradient analysis: Clarified the interpretation of the InfoNCE gradient decomposition in Section 3.1 and the motivation for the idealized assumptions in Section 3.2.
  3. Empirical trilemma: Refined the wording in the abstract and Sections 1 and 4 to emphasize the systematic trade-offs observed under the examined interventions, without implying universal incompatibility.
  4. Generality and limitations: Expanded Section 5 to clarify the scope of our empirical findings and their implications for understanding the roles of cross-modal alignment and within-modality uniformity.
  5. Evaluation details: Clarified the definition of positive-pair identification accuracy in Section 4.1.

We have highlighted the revised text in the manuscript for ease of review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.