- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the lack of compositionality in vision-language models such as CLIP and SigLIP. It shows that the information required to succeed in compositionality benchmarks already exists to some degree in the models, but this information has little effect on the cosine similarity score. In addition, the paper suggests a diagnostic tool (mean erasure), proves that it increases the group score to the level of the text score in 2x2 benchmarks, and empirically verifies these results on Winoground and ColorSwap. The authors later use a non-diagnostic tool (top-2 approximation) on Winoground that leads to a moderate group score increase.
Critical strengths
The 2x2 benchmark results (both theoretical and empirical) are interesting and explain part of the gap between Winoground and other 1x2 benchmarks such as SugarCrepe.
Critical weaknesses
-
Novelty. The paper is mainly diagnostic, and its main contributions (contributions 1 and 2 in the paper) expose an evaluation gap and not a direct improvement in compositionality. This gap was already introduced in Zhu et al. (2025), and the paper's contributions on top of it are minor (see question 1).
-
For the diagnostic tool, the increase in group and image scores on Winoground is substantial. However, the suggested approach based on this analysis (top-2 approximation) leads to only a small increase. In addition, applying it to the Winoground benchmark might use the negative caption, which will not be available in real scenarios.
-
The paper is hard to follow, with many results that are later scoped (e.g., L194-203, “What Corollary 2 does and does not buy us…”) and with references to previous versions of the same paper (e.g., L287-289 “An earlier version of this work instead estimated...”).
Questions for authors
-
Regarding contributions 1 and 2. From the related work section (L132-134): “For two images… Our contribution connects mean-erasure to that existing criterion and explains the Group–Text identity and its normalization boundary”. I understand that you use mean-erasure instead of modifying the Group Score, but since you use it as a diagnostic tool, I don’t understand the added benefit on top of Zhu et al (2025). (a) Will the new Image Score be substantially different compared to their GroupMatch due to normalization? (b) What does the information that Group Score equals Test Score tell us about compositionality, or is it just exposing an evaluation artifact? (c) Can the authors explain what insights one might get from this paper and not from Zhu et al. (2025) regarding the difference between 1x2 benchmarks and 2x2 benchmarks?
-
Regarding contribution 3 (the experiments in Section 6): I understand that in all SugarCrepe categories, the top-20% hold the majority of the influence on the cosine similarity (and are over-represented by 2.35-3.07x compared to the discriminative signal). In addition, you state that the top-5% most discriminative dimensions receive only ~0.5x of the share for swap-object. (a) Are those discriminative dimensions the same across all SugarCrepe categories and across other benchmarks? (b) Since swap-object includes only 245 samples, those dimensions are chosen and evaluated on the same samples. Will you get similar numbers if you choose the discriminative dimensions based on a subset of that category and evaluate on the rest of the samples?
-
Regarding contribution 4. The top-2 approximation for the Winoground dataset increases results by 5.5 points rather than 22 points for the diagnostic tool, and this happens when the other caption can be the distractor, which won’t necessarily exist in standard retrieval tasks. (a) Do you understand why the method didn’t pick only the distractor, which would in turn lead to 22 points increase? (b) What would be the results if the distractors can’t be used for mean erasure and only during evaluation?