- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the relation between modal gaps and retrieval robustness in multimodal contrastive learning. The optimal robust embedding is defined by the average cosine gap of cross-modal nearest neighbor retrieval. It is proved that when the number of sample pairs satisfies , the configuration where the two modalities are perfectly aligned and the embeddings form a regular simplex has the optimal average gap, and it is consistent with the global optimal configuration of the CLIP loss. Based on this, the paper focuses on the existing explicit uniformity and alignment loss EMMUA, analyzing its global optimal solution and gradient optimization dynamics, and attempting to explain the mechanism by which it alleviates modal gaps and improves embedding robustness. The verification process includes directly optimizing the synthesized embedding vectors, training the model on MAD, CIFAR-100, and ImageNet, fine-tuning on MS-COCO, and comparing the performance under modal gaps, retrieval gaps, and different perturbations. The results show that EMMUA can significantly reduce modal gaps, increase retrieval gaps, and improve the resistance of embeddings to Gaussian noise and quantization errors, but the clean performance on some datasets has decreased, and the improvement for input image perturbations is relatively limited.
Critical strengths
- This paper focuses on the modal gap in multimodal models, paying attention to its impact on retrieval stability and embedding quantization. It links the geometric properties of the representation space with actual retrieval requirements.
- The cosine gap between correctly paired samples and the hardest negative samples is used to measure retrieval stability. Under the condition of , the optimal average gap is linked to the complete alignment rule simplex, providing a clear theoretical reference for evaluating embedding quality.
- This paper connects the optimal configuration of the loss function, the gradient training dynamics, and the final retrieval robustness, which helps to understand why the objective function has an ideal optimal solution, while the actual training may still retain the modal gap.
Critical weaknesses
- Theorem 7 uses the EMMUA loss without the denominator positive sample term. However, Appendix A.4 introduces another form that includes this term and writes both in Eq. 51 as being equal. The difference between the two losses is not constant, and the gradients may also be different. Therefore, the current dynamics proof cannot directly cover the actual training objective.
- The proof of modal gap contraction does not adequately handle the spherical constraint. Theorems 3.6 and the Appendix are mainly derived based on vector updates in the Euclidean space, while the theoretical issues and some experiments limit the embedding to the unit sphere. Normalization or projection onto the tangent space will change the update direction, and the relevant conclusions need to be re-proven. Additionally, the reduction of modal mean distance does not equal the complete alignment of paired samples, and it cannot directly lead to global convergence.
- Theorem A.9 gives an upper bound for the embedding norm, but its proof actually requires a positive lower bound, so the current input robustness guarantee does not hold. Theorem 3.5 claims that the conclusion applies to any , but it also omits the necessary condition: when , the two modalities do not have the constraint of having to be aligned with each other.
- The theoretical definition considers the average margin of all queries, but Fig. 5 and Table 2 mainly count the samples that are correctly predicted by each model. Since the sets of correct samples for the two models are different, this comparison is not sufficient to directly prove that the average margin and robustness on the complete test set have achieved the same improvement.
Questions for authors
See Critical Weaknesses.
2 comments
The authors answered my questions, and I maintain my original acceptance.
We thank the reviewer for their thoughtful comments.
Loss Consistency - Thank you for pointing this out. The objective we train and analyze is Eq. 7, which has no positive-pair terms in the denominators. The inconsistency was in the restatement of the loss in Appendix A.4. We replaced it with Eq. 7 itself, so that all of the dynamics analysis is derived for the trained objective. The gradients in Eq. 8 and Eq. 50 were already those of Eq. 7, so Theorem 3.6 is unchanged.
Spherical Constraint - we agree that our analysis is of a simplified setting of Euclidian space without a spherical constraint. However, as [2] showed in high dimensions the spherical and Euclidian analysis are rather similar. We also follow previous works [2,3,4] that perform similar analysis without the spherical constraint. Finally, we provide empirical evidence that the same phenomena seen in euclidian space (figure 2) occurs on the sphere (all other figures).
We agree that a vanishing mean gap implies neither pairwise alignment nor global convergence, and we do not claim global convergence and will state this explicitly. The implication runs the other way: every perfectly aligned solution has zero gap. We therefore also report pairwise alignment directly:
Alignment is higher with EMMUA on every dataset. The gain on CIFAR-100 is modest, although the gap there falls from 0.356 to 0.043. As discussed in Section 4.2, with one caption per class the optimum maps all images of a class to one point, and alignment is limited by how far the image encoder collapses each class. These numbers will be added to Table 1.
Theory:
Theorem A.9 - Thank you for the correction. We corrected this in the manuscript. As you note, this concerns only the theoretical extension, not the embedding noise results.
Theorem 3.5 - Thank you for pointing this out. This is supposed to be for any λ>0 as both you and reviewer v2WA pointed out. We fixed this in the manuscript.
Average margin vs Correctly classified margin - this is indeed a delicate point. To compare the two models on identical sets, we additionally report a margin defined for every query: with j=argmaxk⟨xi,yk⟩, the top-1 minus top-2 similarity ⟨xi,yj⟩−maxk=j⟨xi,yk⟩. It coincides with the margin definition on correctly retrieved queries and measures how stable the retrieved result is on the others.
EMMUA's value is ~2-5× larger on every evaluation set, including the three on which the MS-COCO-finetuned models are evaluated zero-shot. We present this as a measure of retrieval stability over the full set, not of correctness: it is non-negative by construction and is also larger for EMMUA on incorrectly retrieved queries. It is consistent with Table 5, where retrieval consistency under noise is higher for EMMUA on every dataset.
[1] "Decoupled contrastive learning", Yeh et al., 2022
[2] "Is the Modality Gap a Bug or a Feature? A Robustness Perspective", Chowers et al., 2026
[3] "Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data", Zhang et al., 2024
[4] "Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning", Yaras and Chen et al., 2026