acceptodds
Withdrawn from ICLR 2027

Where Compositionality Hides: Shared-Content Interference in Contrastive Vision-Language Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive vision–language models struggle with compositional retrieval, but a low retrieval score does not by itself establish that the relevant information is absent from their representations. We identify an exact, format-dependent bottleneck in the scoring geometry. Removing the mean of two normalized candidate captions from a frozen image embedding preserves every 1×2 caption decision, yet forces the 2×2 Group Score to equal the original Text Score. On CLIP ViT-B/32, this closes the full 22-point Winoground Group–Text gap, increasing Group Score from 9.0% to 31.0% and Image Score from 11.25% to 64.5%, with replication across five models and ColorSwap. An equivalent score decomposition shows that 213 of 355 raw Image-Score failures have a positive joint-matching margin but fail the cross-image offset threshold. A complementary probe-free analysis reports that the highest-magnitude 20% of coordinates carry 57.3% of absolute similarity mass but only 20.2% of the reported discriminative share on object swaps. We distinguish this basis-dependent observation from the exact projection result. Tested global corrections do not recover the paired diagnostic’s gain, while a top-K approximation recovers part of it in the evaluated setting. Together, these results establish retrieval geometry as a concrete bottleneck in paired compositional evaluation: useful discrimination is already present in frozen pooled features, although gap closure alone neither proves complete compositional representation nor establishes open-gallery retrieval improvement.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, compositionality, CLIP, contrastive learning, mechanistic interpretability, retrieval geometry, multimodal representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_e6U1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the lack of compositionality in vision-language models such as CLIP and SigLIP. It shows that the information required to succeed in compositionality benchmarks already exists to some degree in the models, but this information has little effect on the cosine similarity score. In addition, the paper suggests a diagnostic tool (mean erasure), proves that it increases the group score to the level of the text score in 2x2 benchmarks, and empirically verifies these results on Winoground and ColorSwap. The authors later use a non-diagnostic tool (top-2 approximation) on Winoground that leads to a moderate group score increase.

Critical strengths

The 2x2 benchmark results (both theoretical and empirical) are interesting and explain part of the gap between Winoground and other 1x2 benchmarks such as SugarCrepe.

Critical weaknesses
  1. Novelty. The paper is mainly diagnostic, and its main contributions (contributions 1 and 2 in the paper) expose an evaluation gap and not a direct improvement in compositionality. This gap was already introduced in Zhu et al. (2025), and the paper's contributions on top of it are minor (see question 1).

  2. For the diagnostic tool, the increase in group and image scores on Winoground is substantial. However, the suggested approach based on this analysis (top-2 approximation) leads to only a small increase. In addition, applying it to the Winoground benchmark might use the negative caption, which will not be available in real scenarios.

  3. The paper is hard to follow, with many results that are later scoped (e.g., L194-203, “What Corollary 2 does and does not buy us…”) and with references to previous versions of the same paper (e.g., L287-289 “An earlier version of this work instead estimated...”).

Questions for authors
  1. Regarding contributions 1 and 2. From the related work section (L132-134): “For two images… Our contribution connects mean-erasure to that existing criterion and explains the Group–Text identity and its normalization boundary”. I understand that you use mean-erasure instead of modifying the Group Score, but since you use it as a diagnostic tool, I don’t understand the added benefit on top of Zhu et al (2025). (a) Will the new Image Score be substantially different compared to their GroupMatch due to normalization? (b) What does the information that Group Score equals Test Score tell us about compositionality, or is it just exposing an evaluation artifact? (c) Can the authors explain what insights one might get from this paper and not from Zhu et al. (2025) regarding the difference between 1x2 benchmarks and 2x2 benchmarks?

  2. Regarding contribution 3 (the experiments in Section 6): I understand that in all SugarCrepe categories, the top-20% hold the majority of the influence on the cosine similarity (and are over-represented by 2.35-3.07x compared to the discriminative signal). In addition, you state that the top-5% most discriminative dimensions receive only ~0.5x of the share for swap-object. (a) Are those discriminative dimensions the same across all SugarCrepe categories and across other benchmarks? (b) Since swap-object includes only 245 samples, those dimensions are chosen and evaluated on the same samples. Will you get similar numbers if you choose the discriminative dimensions based on a subset of that category and evaluate on the rest of the samples?

  3. Regarding contribution 4. The top-2 approximation for the Winoground dataset increases results by 5.5 points rather than 22 points for the diagnostic tool, and this happens when the other caption can be the distractor, which won’t necessarily exist in standard retrieval tasks. (a) Do you understand why the method didn’t pick only the distractor, which would in turn lead to 22 points increase? (b) What would be the results if the distractors can’t be used for mean erasure and only during evaluation?

Reviewer_SzMKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

CLIP performs poorly on the Winoground combination matching task. The authors believe that some of the errors stem from the similarity scoring method, and that discriminative signals may already exist within the embeddings. SO they used Mean-erasure to eliminate the common direction in the candidate texts. Group Score improved from 9% to 31%. But the authors were unable to apply this method directly to cross-modal retrieval.

Critical strengths

This article presents an interesting finding: some of CLIP’s errors may stem from the way it compares similarity.

Critical weaknesses
  1. The paper does not sufficiently demonstrate that the reported improvements reflect meaningful recovery of compositional information.

  2. This paper presents several analyses, including mean-erasure, coordinate-level discriminability, and content-agnostic corrections, but their relationship is not convincingly established.

  3. The main improvement relies on access to paired candidate captions; it also did not demonstrate generalization to open-gallery retrieval.

Questions for authors
  1. The paper is generally difficult to follow. Several important terms, such as "1 × 2 caption decision" and "1 × 2 versus 2 × 2 evaluation", are introduced without sufficient explanation. The introduction also assumes familiarity with Winoground without clearly explaining the task and its evaluation metrics. Could the authors explain the problem setting and the main research objective more clearly? In particular, what is the central question this paper aims to answer?

  2. I am not convinced that the reported improvement in Group Score demonstrates meaningful recovery of compositional information. As shown in Corollary 2, the proposed transformation mathematically guarantees that Group Score equals the original Text Score. What additional evidence supports the claim that this result reveals previously inaccessible compositional information in CLIP representations? How does this finding differ from existing row-centering or joint matching approaches?

  3. Section 6.3 reports that the spectral properties do not predict the improvements from mean-erasure. Could the authors clarify how these findings collectively support the proposed shared-content interference explanation?

  4. The paper seems generated by AI the author has not removed the references to the AI generation in Appendix P.

"What can be checked without new encoder runs? The retained aggregate output supports Table 11, the normalization differences, the reported medians, and the failure-partition counts. The analysis script reconstructs integer counts at n = 400, generates the table, recomputes Wilson intervals and exact binomial probabilities, and verifies the projection identities on randomly generated unit embeddings. Those numerical checks support implementation consistency and do not substitute for the proofs or a fresh dataset run."

Reviewer_fMy3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper was extremely incomprehensible.

After careful inspection, it seems to be targeted towards addressing compositional retrieval, details of which difficult to understand based on how the paper is written and presented.

Critical strengths

The paper was extremely incomprehensible.

Critical weaknesses

The aim of the experiments are not clearly stated. Abstract fails to give a bigger picture of the paper. Introduction is extremely difficult to understand. The aim of the experiments are vague. Many times certain concepts are used without being defined.

Abstract: "We identify an exact, format-dependent bottleneck in the scoring geometry." --> what is format-dependent bottleneck ?

Abstract: “213 of 355 raw Image failures have a positive joint-matching margin but fail the cross-image offset threshold.” —> “joint-matching margin”and “cross-image offset threshold” are not properly defined anywhere.

Abstract: “only 20.2% of the reported discriminative share.” —> it is not clear where is this metric defined.

Section1: “Candidate dependence. The paired diagnostic is content-adaptive, and the global corrections tested here do not reproduce its gain.” —> This absolute does not make any sense. What is candidate dependence, content adaptive, which global corrections does it refer to ?

“The transformation uses both candidate captions but no correct-assignment label.” —> which assignment label are being referred to here.

Table 2: “These aggregate results are carried over from the submitted experiment; per-pair replication code accompanies this revision.” —> what submitted experiment and what is per-pair replication code and what revision ?

Inconsistent numbers across tables: CLIP ViT-L/14 baseline Text Score is 28.25 in Tables 5 and 11 but 28.5 in Table 7. LAION baseline Image Score is 11.25 in Table 11 but 11.5 in Table 7. SigLIP-base post-erasure Group Score is 33.00 in Table 5 but its Text Score is 32.8 in Table 7, even though Corollary 2 says the two must be equal.

Section 8: the top-K decay is described as “tracing the curve Section 7 predicts,” but Section 7 makes no prediction about a curve.

Questions for authors

See "Critical Weaknesses"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.