acceptodds
Under review as a conference paper at ICLR 2027

On Unidentified Representation Optimality of Vision-Language Compositional Reasoning

Abstract

Contrastive Language-Image Pre-training (CLIP) is a cornerstone of multimodal representation learning, while it is born with a persistent inability to reason about compositional relations. What's worse, a deficiency can be inherited by the Multimodal Large Language Models (MLLMs) that adopt its image encoder as their visual receiver. Why such failures arise has remained without a rigorous account. We develop a token-aware causal representation learning framework built on sequential, language-token Structural Causal Models (SCMs), recasting block identifiability for tokenized text and proving that CLIP's contrastive objective recovers the modality-invariant latent variables at the token level. It yields a principled identifiability-based diagnosis, i.e., the compositionality derived from a vanilla CLIP is nonidentifiable: pseudo-optimal text encoders attain perfect modality-invariant alignment while remaining provably invariant to the SWAP, REPLACE and ADD operators over atomic concepts, so no contrastive optimum guarantees separation between a caption and its hard negatives. Through the modality gap, the problem propagates to the image encoder, which leads to an algorithm-independent impossibility result: no projector, LLM capacity, or instruction tuning can lift a downstream MLLM above chance on the affected queries if they use the pseudo-optimal encoders to receive images. Guided by our theoretical recipe, we propose the remedy of hard negative strategies measurably repair both CLIP and MLLMs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
CLIP, theory of multimdal representation learning, MLLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8ixvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper delivers a causal-theoretic treatment of compositional reasoning vulnerabilities in CLIP and downstream Mllms. The authors construct a token-aware SCM of image-text generation, establishing that while contrastive alignment successfully identifies modality-invariant latents at the token level, it suffers from an intrinsic composition nonidentifiability: there exist "pseudo-optimal" text encoders that achieve near-zero alignment loss yet remain invariant to semantic perturbations such as SWAP. Leveraging the constant modality gap, the authors show this blindness propagates to visual representations, culminating in an algorithm-independent impossibility theorem showing that no downstream projector, LLM capacity, or instruction tuning can elevate an MLLM above random chance on affected queries. Guided by this diagnosis, the paper develops compound hard-negative data curation schemes (Multi-Calling / Needle-Tracking Multi-Calling) with paired synthetic negative images, yielding improvements across CLIP and MLLMs on SugarCREPE and MMVP while safeguarding zero-shot generality.

Critical strengths

Problem has a good merit, this is a long-standing problem associated with CLIP.

Rigorous theoretical foundation are provided.

Critical weaknesses

The theoretical proofs rely on multiple strong structural assumptions. I don't know how applicable they are, looks like its easy to doubt its meaningfulness. (I didnt check all Lemmas and I am not an expert in theory).

Proposed fix solution is quite expensive and involves multiple models (MLLM, T2I).

Baselines are very old. No good open-source MLLM verification (Qwen3.5/3.8 27B, Gemma...) More like 2024-2025 paper. Evaluation is also very limited. only 3 benchmarks are reported in MLLM part.

minor:

Please check lines 1907 and 1977 for wrong references.

Questions for authors

please see weaknesses

Reviewer_KWW4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why CLIP, and MLLMs that use CLIP’s image encoder, fail at compositional reasoning (binding attributes to objects, word order, negation). It models captions as token sequences in a causal model and proves that CLIP’s contrastive objective still recovers the shared latent variable (Theorem 1). It then argues that “pseudo-optimal” text encoders exist that reach the same loss but cannot tell a caption from its SWAP, REPLACE, or ADD hard negative (Theorems 3–5). Using a constant modality gap, it claims this collapse transfers to the image encoder (Prop. 6, Cor. 7), and then to any downstream MLLM, which cannot beat chance on the affected questions (Theorem 8). As a fix, it stacks the three operators to build compound hard negatives (MC) and a variant that picks the negative the current encoder is most blind to (NT-MC). Experiments on SugarCrepe, VTAB/ImageNet zero-shot, and MMVP/CV-Bench/GQA with swapped vision towers show gains.

Critical strengths
  • The question is important and the end-to-end story (text encoder -> image encoder -> MLLM) is a clear way to frame it.
  • The compound hard-negative idea (MC / NT-MC) is simple and cheap to add to existing pipelines.
  • The paper includes a falsification test (Table 4: gains should only appear in categories covered by the operators), which is good practice.
Critical weaknesses
  1. Most of the theory was already published. The token-level causal model, Theorem 1, Theorems 3–5, Table 1, and the proofs in Appendix B1–B6 appear almost word for word in “Understanding Hardness of Vision-Language Compositionality from a Token-level Causal Lens” (ICLR 2026). That paper is not cited, and the paper still lists this material as new contributions (I) and (II). What is actually new here is Prop. 6, Cor. 7, Theorem 8, NT-MC, and the MLLM experiments. The paper must cite the earlier work and say clearly what is new.

  2. The “pseudo-optimal” encoder is the same as the optimal one. The whole argument rests on a second encoder g** that gets the same loss as g* but cannot tell a caption from its hard negative. But the proof in Appendix B5 ends with g** = g* (lines 1561–1563). And Theorem 1 says any optimal encoder only depends on the shared meaning z_inv, so it can merge two captions only if they mean the same thing — in which case one is not a hard negative. So Theorems 3–5 do not show what they claim. Appendix D2 even admits the construction “might produce the empty set.”

  3. The modality-gap step contradicts itself. Prop. 6 uses two things at once: the loss is zero (so image and text embeddings are equal) and the embeddings differ by a fixed gap δ. Both cannot hold unless δ = 0. So the step from text encoder to image encoder is not proved.

  4. Theorem 8 is too strong to be meaningful. The impossibility proof never uses the encoder, the loss, or CLIP (the paper says so in B11(e)). The same argument, with the latent for “dog” swapped with “cat”, would prove that no model can answer “is there a dog?” above chance. The paper’s own Table 6 also shows LLaDA-V at 76.7 on MMVP with a contrastive tower, far above the 50% the theorem predicts.

  5. The experiments do not support the claims. Table 5, the only direct test that g** exists, has no numbers. The “witness” g** in Appendix D2 is trained with permuted captions as positives, which changes the objective, so it does not show CLIP’s own objective produces g**. Table 6 has mismatches (LLaDA-V GQA is +2.4 in the table but +5.4 in the text; the mean only works with +5.4), the Appendix says a ViT-B/32 was swapped in while Table 6 lists ViT-L and SigLIP towers, baselines are copied from other papers, and all results are single runs. The paper’s own sanity check also fails: the biggest MMVP gain is in a category the paper itself says its method should barely help (Table 4).

Missing related work. Kang et al. (ICCV 2025) already prove an impossibility result for CLIP-like spaces; Koishigarina et al. (ICLR 2026) show binding information is present in CLIP’s embeddings; Li et al. (ACL 2025) show the CLIP vision encoder keeps information LLaVA can use. These should be discussed, and the “first principled account” claim toned down.

Questions for authors
  1. Prior work. Assumption 12, Theorem 1, Theorems 3–5, Table 1 and Appendix B1–B6 match the ICLR 2026 paper Understanding Hardness of Vision-Language Compositionality from a Token-level Causal Lens, which is not cited. Please cite it and list exactly what is new in this submission.

  2. g** = g*. Appendix B5 (lines 1561–1563, 1621) concludes g** = g*. By Theorem 1, an optimal encoder maps X and π(X) to the same point only if they share z_inv. On which inputs does g** differ from g*? If none (D2 admits the set may be empty), what do Theorems 3–5 establish?

  3. Modality gap. Theorem 1's optimum has zero alignment loss, so f* = g* a.s. and δ = 0 in Assumption 13. B7 uses both Eq. 88 (loss → 0) and Eq. 89 (f* = g** + δ) for the same pair. How can δ ≠ 0?

  4. Theorem 8. The proof uses only that π is a bijection on Z that flips φ; it never uses the encoder or the loss (B11(e)). With π swapping the "dog"/"cat" latent slots, it would show no model can answer "is there a dog?" above chance. What rules this out? And Lemma 19 cites Theorem 3 for π, but Theorem 3 permutes token columns, not latent coordinates — where is a latent π constructed?

  5. Vision tower. Appendix C: a CC3M/CC12M ViT-B/32 replaces the LLaVA-1.5/InstructBLIP towers. Table 6: ViT-L/14-336, EVA ViT-g, SigLIP-SO400M, pre-trained from scratch. For each host, which encoder, resolution and data were used?

  6. Table 5. It has no numbers, only ℓ, m, Δ, "near chance", "partial recovery". Please report the values.

  7. Arithmetic. Table 6: LLaDA-V GQA is +2.4 in the table but +5.4 in the text (lines 2371, 2396); the mean +2.3 needs +5.4. Table 8(a): CC3M NegCLIP Overall is 57.18 but its cells average 56.45. Which are correct?

  8. Zero-shot. NegCLIP ImageNet top-1 goes 4.67 → 10.51 with compound negatives. Why would harder negatives double zero-shot accuracy? Please add a vanilla CLIP row (same data, same steps) to Table 8(b).

  9. Falsification test. Table 4 rates Structural & Physical "Weak/Low", yet it has the largest MMVP gain (+~14, line 469); "Strong/High" Presence of Features gains +~5. How does this fit the claim that gains appear only where the operators apply?

Reviewer_d2qVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates compositional reasoning failures in CLIP and downstream MLLMs through a token-aware causal representation framework. The authors argue that optimal contrastive alignment may fail to identify compositional structure and propose a downstream impossibility result. Motivated by this analysis, they introduce compound hard-negative generation and collapse-targeted selection, reporting improvements on compositional reasoning and visual question answering benchmarks.

Critical strengths
  • The paper offers an interesting theoretical perspective connecting contrastive alignment, compositional nonidentifiability, and downstream reasoning limitations.

  • The work is well-motivated by the gap between strong image–text alignment and unreliable compositional reasoning.

  • The reported improvements across CLIP and MLLM settings, together with budget-controlled ablations, support the practical value of the proposed negative-sampling strategy.

Critical weaknesses
  • The core proofs contain substantial gaps. Eq. (75) equates a union of intersections with an intersection of unions, which does not generally hold. Lemma 17 also appears to infer a constant shared latent variable across two families merely from an intersection condition. These issues undermine the proofs of Theorems 3 and 5.

  • The experimental evidence does not establish the claimed impossibility mechanism. Table 5 provides symbolic quantities and qualitative descriptions rather than concrete measurements. Furthermore, collapse of the final contrastive embedding does not necessarily imply information loss in the intermediate visual tokens consumed by MLLMs.

  • The downstream impossibility claim is overstated. Lemma 19 does not sufficiently justify why a nontrivial concept permutation must flip every query label. Moreover, a worst-case bound over observationally equivalent worlds does not establish that practical MLLMs cannot exceed chance after task-specific supervision.

Questions for authors
  • Can the authors correct Eq. (75) and clarify whether Theorem 3 still holds? What additional assumptions are needed for Lemma 17?

  • Under what explicit conditions does the permutation guarantee (\phi(\pi(z))=1-\phi(z))? Does the impossibility result cover labeled instruction tuning and updates to the visual encoder?

  • (Minor) Figure 5 in Appendix D3 is incomplete.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Confidential note regarding Submission 25031 and the novelty concern raised by Reviewer KWW4

Dear Area Chair,

We thank you and the three reviewers, d2qV, KWW4, and 8ixv, for their time and careful reading of our submission. We have studied every comment closely. Most of the technical points are well taken, in particular the issues around Eq. (75), the construction of g** in Theorems 3–5, the scope of Theorem 8, and the inconsistencies in Tables 5–6. We agree they need substantive revision, and we will address them fully in the next version of the manuscript.

However, we have decided not to post a rebuttal in this cycle. One main reason concerns Reviewer KWW4's central novelty claim, which we explain confidentially below.

1. The "previously published" claim of KWW4 is factually incorrect.

Reviewer KWW4 states that most of our theoretical contributions (Theorems 1 and 3–5) have "already been published" at ICLR 2026, citing arXiv:2510.26302. That work is in fact an earlier version of this submission by the same author(s). [It was submitted to ICLR 2026 and was not accepted (OpenReview ID: jA7y8mTUsN).] It has never appeared in any peer-reviewed archival venue. It therefore does not count as prior publication under the ICLR dual-submission policy.

We acknowledge that the arXiv PDF carried the header "Published as a conference paper at ICLR 2026". This was a template artifact from preparing the de-anonymized preprint, and we have now corrected it (arXiv vX). We regret any confusion it may have caused.

2. Our concern with how the claim was reached.

The two documents share most of their wording, notation, and formatting nearly verbatim. That alone strongly suggests they are versions of the same work, not a case of independent prior work. In such a situation, we believe a reviewer should first verify whether the cited work was actually accepted and published. That check takes a single click on OpenReview. Reviewer KWW4 instead presented the overlap as established prior publication. This claim shaped the "Poor" novelty assessment and the overall score of 1, and it turned a question about the relationship between two versions into a de facto originality concern. We are concerned that this level of verification falls short of what the ICLR Reviewer Guidelines expect, particularly for a claim that directly determines the score.

We also note that the review rates Soundness as "2: minor or fixable concerns" while concluding that the theorems do not hold. We respectfully ask the AC to consider this internal inconsistency when weighing the review.

Based on these, we are unsure whether the overlap assessment relied on automated AI tools. We can accept if you make AI-assisted review, but should have verified the output by yourself before you submitted your review.

3. Why we are not rebutting due to KWW4‘s concern.

We cannot answer this point publicly without revealing our identity, which would violate the double-blind policy. Combined with the substantive revisions the other comments require, we have concluded that the most responsible course is to revise the work thoroughly and resubmit it, rather than give a partial response now.

We remain grateful to all reviewers for their constructive technical feedback, which will substantially improve the paper.

Sincerely,

The Authors of Submission 25031

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.