- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper experimentally investigates the modality gap in vision-language models (VLMs), particularly CLIP. The experiments reveal that the modality gap is concentrated in a subset of embedding dimensions (modality gap dimensions) and that the influence of these dimensions varies across samples. The authors further formulate the influence of modality gap dimensions on each sample as a Correspondence Noise (CN) score and show that samples with higher CN scores are more likely to have captions that are inappropriate for their corresponding images. Based on this finding, the authors argue that correspondence noise in CLIP's training data is the cause of the modality gap. They further demonstrate that training CLIP using only data with low CN scores reduces modality gap and improves downstream task performance.
Critical strengths
- S1: The paper analyzes the causes of the modality gap through extensive experiments and identifies correspondence noise, as quantified by the CN score, as its cause.
- S2: The paper demonstrates that filtering out samples with high CN scores from the training data improves CLIP's downstream performance.
Critical weaknesses
- W1: The paper evaluates the modality gap using MROS, which is based on the mean offset of embeddings from zero. However, MROS is computed based on the mean offset of each individual modality and does not directly evaluate the actual modality gap between images and text. The authors should demonstrate that MROS, which is computed using only a single modality, serves as a valid proxy for the actual modality gap.
- W2: The paper should also discuss the distribution of offsets in the modality gap dimensions. Are the offsets caused by a small number of outlier samples, or do they occur consistently across all samples?
- W3: In Fig. 8, the modality gap is evaluated using MROS, but how does the actual modality gap change? It would be useful to evaluate the gap using general modality gap metrics, such as positive pair similarity, centroid distance, and Wasserstein distance.
- W4: For Fig. 7, it would be interesting to group models with similar CN score trends using methods such as spectral clustering.
- W5 (minor): The method for calculating MROS should be explained precisely. The paper does not explain (though it can be guessed as median absolute deviation) and the magic number in Eq. (6) of the appendix.
Questions for authors
Please refer to the weaknesses.
Dear Reviewer RJVj,
We agree that there is a certain inconsistency within the manuscript that does not disentangle sufficiently well two observations wrt to the modality gap. (1) The original finding of Liang et al. that even at initilization the two mean centroids exhibit a systematic difference, which is measured by L2M. And (2) the finding by Schrodi et al., that trained CLIP models exhibit this sparse structure of a few offset dimensions that make up most of the originally described modality gap. Our framing was mainly building on the findings of Schrodi et al. and their way of presenting their results / describing the modality gap. We believe that some of the Reviewers concerns and critique are also applicable to the paper of Schrodi et al., which also partially confounds both observations (even though it got selected as an oral in 2025..). Even though we do not fully agree with the Reviewer that it invalidates all of our findings, we realized that the paper must better differentiate both phenomena, the current writing is not consistent with all the findings from the literature and has partially wrong claims.
Either way, we wanted to thank you for the helpful feedback which may enable a better and more consistent paper in the future. It was quite obvious you wrote this review fully manually and invested a good amount of time to review our paper.
There is one remaining point which we are quite curious about. How did you mean W2. We agree that in theory one could use an orthogonal transformation to "eliminate" the modality gap dimensions. We were not sure why this is a problem that one could shift the coordinate system, since we were rather describing a re-occuring phenomenon empirically for a large set of different CLIP models (in line with the findings of Schrodi et al.). We would be curious why this is an actual weakness, if in the given coordinate system the model operates, the offset dimensions occur after sufficiently long training?