- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper empirically investigates the relationship between the modality gap and downstream performance in vision-language models such as CLIP. First, the authors confirm that the InfoNCE loss used for CLIP training does not necessarily induce modality alignment. By constructing a setting in which the modality gap is reduced by InfoNCE, the authors propose AA-InfoNCE, a modification of InfoNCE that explicitly encourages modality alignment (absolute alignment). The authors then conduct controlled experiments using AA-InfoNCE. While prior work suggests that reducing the modality gap while improving uniformity is beneficial, this paper empirically demonstrates a trilemma among the modality gap, uniformity, and downstream performance; specifically, when uniformity is held fixed, there is a trade-off between downstream performance and positive-pair similarity.
Critical strengths
- S1: The paper provides a detailed analysis of the relationship between the modality gap and uniformity, both of which have been considered important factors for improving downstream performance in prior work.
- S2: The paper proposes AA-InfoNCE, which strengthens absolute alignment while preserving the form of InfoNCE.
Critical weaknesses
- W1: The main experiments are conducted only with OpenCLIP ViT-B/16. It is unclear whether similar results hold for other models in the CLIP family (e.g., SigLIP and OpenAI CLIP). The scope within which the claims hold should be appropriately framed.
- W2: Since positive-pair identification accuracy is not defined, the significance of the opposite trends observed in Figs. 2(a) and 2(b) is unclear.
- W3: If the modality gap is intended to measure the discrepancy between the distributions of image and text embeddings, it seems more natural to use distributional distances such as MMD or the Wasserstein distance. While I understand that centroid distance and positive-pair similarity are commonly used to measure the modality gap, the fact that they exhibit opposite correlations depending on which metric is used, as shown in Fig. 4, also raises questions about whether these metrics are appropriate for measuring the modality gap.
Questions for authors
- Q1: Are the reported downstream-task results averaged over all tasks? Does the relationship with metrics such as positive-pair similarity exhibit a similar trend for every individual task?
Official Comment by Authors
We thank the reviewers for their constructive feedback and valuable suggestions. We have uploaded a revised manuscript addressing the concerns raised in the reviews. The main revisions are summarized below:
We have highlighted the revised text in the manuscript for ease of review.