acceptodds
Under review as a conference paper at ICLR 2027

On the Origin of the Modality Gap: A Mechanistic Perspective via Correspondence Noise

Abstract

Contrastive vision-language models exhibit a persistent modality gap despite being trained to align images and text, yet its underlying mechanism remains unclear. We provide a mechanistic account demonstrating that a small number of high-offset dimensions disproportionately shift the ground truth retrieval rank of samples with weak, ambiguous, or erroneous image–text correspondence through their cross-modal interaction. Controlled perturbations of correspondence quality reproduce this behavior, providing causal evidence that correspondence noise contributes to the emergence and function of the gap. We introduce a correspondence noise score, show consistency across checkpoints and diverse pretrained models, and demonstrate that filtering correspondence noise before training jointly reduces the modality gap and improves downstream performance. Finally, we show that gap magnitude is positively but inconsistently associated with performance across converged models, making it an unreliable proxy for representation quality, while its functional importance is primarily geometric rather than semantic.

Primary area:
interpretability and explainable AI
Keywords:
modality gap, contrastive vision language models, explainable AI, CLIP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6MqZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper experimentally investigates the modality gap in vision-language models (VLMs), particularly CLIP. The experiments reveal that the modality gap is concentrated in a subset of embedding dimensions (modality gap dimensions) and that the influence of these dimensions varies across samples. The authors further formulate the influence of modality gap dimensions on each sample as a Correspondence Noise (CN) score and show that samples with higher CN scores are more likely to have captions that are inappropriate for their corresponding images. Based on this finding, the authors argue that correspondence noise in CLIP's training data is the cause of the modality gap. They further demonstrate that training CLIP using only data with low CN scores reduces modality gap and improves downstream task performance.

Critical strengths
  • S1: The paper analyzes the causes of the modality gap through extensive experiments and identifies correspondence noise, as quantified by the CN score, as its cause.
  • S2: The paper demonstrates that filtering out samples with high CN scores from the training data improves CLIP's downstream performance.
Critical weaknesses
  • W1: The paper evaluates the modality gap using MROS, which is based on the mean offset of embeddings from zero. However, MROS is computed based on the mean offset of each individual modality and does not directly evaluate the actual modality gap between images and text. The authors should demonstrate that MROS, which is computed using only a single modality, serves as a valid proxy for the actual modality gap.
  • W2: The paper should also discuss the distribution of offsets in the modality gap dimensions. Are the offsets caused by a small number of outlier samples, or do they occur consistently across all samples?
  • W3: In Fig. 8, the modality gap is evaluated using MROS, but how does the actual modality gap change? It would be useful to evaluate the gap using general modality gap metrics, such as positive pair similarity, centroid distance, and Wasserstein distance.
  • W4: For Fig. 7, it would be interesting to group models with similar CN score trends using methods such as spectral clustering.
  • W5 (minor): The method for calculating MROS should be explained precisely. The paper does not explain MAD⁡(⋅)\operatorname{MAD}(\cdot) (though it can be guessed as median absolute deviation) and the magic number 1.48261.4826 in Eq. (6) of the appendix.
Questions for authors

Please refer to the weaknesses.

Reviewer_TY6HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the relationship between correspondence noise and the modality gap in contrastive vision–language models. It analyzes how a few high-offset dimensions affect sample-level retrieval rankings and introduces a correspondence noise (CN) score. Through input perturbations and data-filtering experiments, the authors argue that noisy correspondence contributes to the gap and that gap magnitude alone is not a reliable indicator of representation quality.

Critical strengths

The sample-level analysis provides a useful perspective beyond aggregate measurements of the modality gap. Examining which pairs are sensitive to gap-dimension ablation helps characterize the functional role of these dimensions. The experiments keep retrieval candidates fixed when comparing clean and perturbed inputs, which is an appropriate control. The authors also correctly distinguish the benefits of cleaner training data from the claim that reducing the gap itself improves performance.

Critical weaknesses
  1. The novelty is incremental and needs clearer positioning. As acknowledged in the related work, previous studies have already connected correspondence quality to modality-gap magnitude. The main contribution here is the sample-level analysis and CN score, rather than the broader connection between noise and the gap. The claims about explaining the gap’s origin should be distinguished more carefully from these existing findings.
  2. The causal evidence does not fully establish how the gap originates. Section 4.2 changes inputs to pretrained models and measures CN, demonstrating sensitivity of an existing representation rather than explaining its formation. Section 4.4 provides more relevant training evidence, but uses only two runs with differently selected datasets. Differences in semantic composition and sample difficulty remain potential confounders, and variability across random seeds is not assessed.
  3. CN requires independent validation as a noise measure. Absolute rank changes can reflect retrieval ambiguity, similarity margins, and candidate composition, not only correspondence noise. Taking the absolute value also obscures whether gap dimensions help or harm retrieval. The main paper relies primarily on qualitative examples to associate CN with naturally occurring noise; evaluation against human annotations and simple similarity- or loss-based baselines would strengthen this interpretation.
  4. The interpretation depends on the coordinate system. Gap dimensions are selected through heuristic one-sided mean offsets. A shared orthogonal rotation preserves all cross-modal cosine similarities but can redistribute these offsets across coordinates. Rotation or subspace-level controls would clarify whether the proposed mechanism describes the learned coordinate system specifically or a more general property of the modality gap.
  5. The distinction between geometric and semantic contributions is not fully resolved. Mean replacement removes sample-specific variation only on the high-offset side; the corresponding coordinates in the other modality may still carry semantic information. The reported sensitivity to renormalization further complicates the interpretation. These experiments support a role for offsets and normalization, but do not conclusively establish that the dimensions’ importance is primarily geometric rather than semantic.
  6. Several presentation errors should be corrected. The statement that at most approximately 20% of samples are affected “across training” (lines 243–245) appears inconsistent with the substantially higher values at the earliest checkpoint in Figure 3(c,d). Minor corrections include “varies” → “vary” at lines 199–200, an unmatched parenthesis at line 239, and “images with image-not-available placeholder” → “images containing ‘image not available’ placeholders” at lines 290–291.
Questions for authors

Can the authors vary only the mismatch rate within a fixed training corpus, using multiple random seeds and matched training budgets, to isolate the effect of correspondence noise?

Does CN outperform similarity- and loss-based scores on human-annotated correspondence noise after controlling for original retrieval rank? Do signed rank changes support the proposed promotion of weak ground-truth matches?

Do the main conclusions hold under shared orthogonal rotations and mean replacement in both modalities, with normalization effects explicitly controlled?

Reviewer_RJVjOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper offers a mechanistic account of the modality gap in contrastive vision–language models. The authors observe that the gap is concentrated in a few coordinates with a large one-sided mean offset: one modality’s mean is far from zero and the other’s is near zero. Based on this observation, they propose a correspondence noise (CN) score, defined through the change in ground-truth retrieval rank after ablating individual gap dimensions. They further conduct controlled image and text perturbations, examine the consistency of CN rankings across checkpoints and pretrained models, and train CLIP models on datasets with different CN distributions. Finally, they investigate the relationship between the modality gap and downstream performance.

Critical strengths

S1. The paper combines checkpoint-level analyses, controlled perturbations, cross-model comparisons, and retraining experiments. In particular, the consistency of CN rankings across several independently trained models suggests that the proposed score may capture a reproducible signal related to the characteristics of image-text pairs.

S2. The empirical analysis covers many pretrained models, two retrieval environments, and a broad set of corruption families.

Critical weaknesses

W1. The paper states that 'the characteristic modality gap emerges only after sufficient contrastive training on sufficiently large datasets'. This is in contradictory with the observation, the modality gap emerges at the initialization, from previous papers. See Fig 1 in [1] and Fig 7c in [2]. This observation should be invariant to any metric of modality gap.

[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. Liang et al., NeurIPS 2022.

[2] Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models. Schrodi, et al., ICLR 2025.

W2. The paper claims that the modality gap is concentrated in a small number of high-offset dimensions. However, the identification of these dimensions and their individual ablation effects are inherently dependent on the chosen coordinate system. Applying an orthogonal transformation to both image and text embeddings preserves all cross modal dot products, while the mean offsets can be redistributed across coordinates. [3] shows that the cross modal information is captured alone with singular vectors rather than coordinates. This leads to a fundamental ambiguity. If the modality gap and all retrieval predictions remain unchanged, while the supposedly critical dimensions and their ablation effects can change, in what sense do these specific dimensions constitute an underlying mechanism of the modality gap?

[3] The Convergent Representation of Contrastive Vision-Language Models: Geometry, Modality Gap and Shared Space Alignment. Yi et al., ICML 2026.

W3. The proposed CN score is defined as the average absolute change in ground-truth retrieval rank after removing selected gap dimensions. While this provides a measure of retrieval sensitivity, its interpretation as correspondence noise is not sufficiently established. A large rank change does not necessarily indicate an incorrect or unreliable image-text pair. When there are multiple hard negatives in the dataset, GT rank can be sensitive to small changes in similarity scores. Therefore, every later claim depends on CN = noise, 'Filtering correspondence noise' (Sec 4.4) and 'consistency of correspondence noise rankings' (Sec 4.3), are, strictly, claims about gap sensitivity.

W4. Sec 4.2 is presented as an investigation of the causal effects of correspondence noise. However, the experiments manipulate images or captions after the representation model has already been trained and then measure changes in CN. They therefore establish, at most, that input corruption can causally affect sensitivity to existing gap dimensions. This is fundamentally different from demonstrating that correspondence noise causes those gap dimensions to emerge during contrastive training.

Furthermore, several of the strongest perturbations involve missing images, generic unavailable-image captions, or severe corruption. Such perturbations naturally disrupt retrieval behavior, and their effects on an ablation-based sensitivity metric are not particularly surprising. These experiments would be more convincing if they demonstrated an effect specific to gap dimensions rather than a generic consequence of making retrieval more ambiguous or difficult.

W5. In Sec 4.4, the experimental design does not adequately isolate correspondence noise. The authors compare models trained on low-CN and mixed-CN samples. Since CN is a model-dependent retrieval-sensitivity measure rather than independently established correspondence quality, the resulting datasets may differ along several dimensions unrelated to actual annotation errors. For example, low-CN filtering may preferentially select easy examples, specific captions, visually distinctive images, or samples with large retrieval margins. It may also change semantic coverage and the distribution of difficult negatives. Any of these changes could influence the learned representation geometry and downstream performance. The experiment therefore does not establish that correspondence noise itself causes the observed difference. A more direct test would train models on the same original dataset with controlled levels of artificially introduced image-text mismatch, while holding the training budget and the remaining data distribution fixed. Please see the synthetic noise scenario in [4]. Without such a controlled intervention, the conclusion that correspondence noise contributes causally to the emergence of the modality gap remains underdetermined.

[4] Noisy Correspondence Learning with Modality Gap Direction Correction. Wang et al., AAAI 2026.

Questions for authors

Please see weaknesses.

1 comment

Dear Reviewer RJVj,

We agree that there is a certain inconsistency within the manuscript that does not disentangle sufficiently well two observations wrt to the modality gap. (1) The original finding of Liang et al. that even at initilization the two mean centroids exhibit a systematic difference, which is measured by L2M. And (2) the finding by Schrodi et al., that trained CLIP models exhibit this sparse structure of a few offset dimensions that make up most of the originally described modality gap. Our framing was mainly building on the findings of Schrodi et al. and their way of presenting their results / describing the modality gap. We believe that some of the Reviewers concerns and critique are also applicable to the paper of Schrodi et al., which also partially confounds both observations (even though it got selected as an oral in 2025..). Even though we do not fully agree with the Reviewer that it invalidates all of our findings, we realized that the paper must better differentiate both phenomena, the current writing is not consistent with all the findings from the literature and has partially wrong claims.

Either way, we wanted to thank you for the helpful feedback which may enable a better and more consistent paper in the future. It was quite obvious you wrote this review fully manually and invested a good amount of time to review our paper.

There is one remaining point which we are quite curious about. How did you mean W2. We agree that in theory one could use an orthogonal transformation to "eliminate" the modality gap dimensions. We were not sure why this is a problem that one could shift the coordinate system, since we were rather describing a re-occuring phenomenon empirically for a large set of different CLIP models (in line with the findings of Schrodi et al.). We would be curious why this is an actual weakness, if in the given coordinate system the model operates, the offset dimensions occur after sufficiently long training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.