acceptodds
Under review as a conference paper at ICLR 2027

Probing Fine-Grained Semantic Alignment Across Vision and Language with Minimal Pairs

Abstract

To what extent does alignment between vision and language preserve specific semantic distinctions? We introduce a new benchmark that examines this question across shallow mappings between independently trained vision and language models, CLIP-style contrastive models, and generative vision-language models. Each benchmark item leverages a minimal pair (of pairs) design comprising two minimally contrasting images and two corresponding captions, with contrasts spanning object properties and relations, physical events, and social interactions. By testing whether similarity scores or matching decisions consistently favor the correct image–caption correspondences over the mismatched alternatives, the benchmark probes whether alignment preserves the specific semantic distinction targeted by each item. Across these settings, we find that alignment is highly conditional. Shallow unimodal alignment recovers some shared object and scene structure, but performs poorly on minimal pairs that require preserving specific relational, causal, or social distinctions. Multimodal training improves performance, but gaps remain even for stronger models compared to human baselines. Further analyses show that implicit captions are consistently harder than explicit captions, suggesting that current models are better at matching named visual properties than inferring latent consequences from world knowledge. Together, our results suggest that broad semantic structure may support cross-modal matching, but fine-grained inferential distinctions needed to match closely related images and captions remain fragile even in multimodal systems.

Primary area:
datasets and benchmarks
Keywords:
Vision-language alignment, Cross-modal representation, Minimal-pair benchmarks, Multimodal models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J97rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces minimal image–caption pairs covering physical-causal, perspectival, and social knowledge, with explicit and implicit captions for the same images. It evaluates shallow cross-modal mappings, CLIP, and generative VLMs against human baselines. Results show weak shallow alignment and stronger multimodal performance, with remaining gaps in social interaction and implicit perspective judgments.

Critical strengths

The explicit/implicit comparison holds visual evidence fixed and provides a useful diagnostic. Evaluation spans multiple model families and includes both human consensus and single-rater baselines.

Critical weaknesses

Benchmark size and coverage are unclear, making its breadth difficult to assess. (Section 2 omits total and per-category item counts; Appendix A.3 reports only annotation totals.)

The claimed minimality of the generated pairs is insufficiently validated; high human accuracy does not rule out unintended matching cues.

there are several pairwise benchmarks need discussion.: MMVP, MLLM-CompBench, and CompareBench. The paper needs to establish what its semantic domains and explicit/implicit captions add beyond existing pairwise evaluations.

Questions for authors
  • How many unique image pairs and semantic contrasts does each subset contain?
  • Which capabilities are newly tested relative to EqBen, VisMin, and SPEC?
  • What fraction of generated pairs failed validation, and why?
1 comment
AuthorsOpenReview ↗

Response to Reviewer J97r

We thank the reviewer for their thoughtful comments and feedback, as well as for their encouraging assessment of our work. We have uploaded the revised manuscript PDF, with all changes highlighted in blue.


W1 / Q1: Benchmark size and coverage

The counts were missing from the paper and are now given in the new Appendix A.1.4 (Table 3). The suite has 369 items across six inferential distinctions:

domaindistinctionitemscaption levels
physical–causalpast event70implicit, explicit
physical–causalpast location13implicit, explicit
physical–causalfuture use20implicit, explicit
perspectivalapparent size30implicit, explicit
socialsocial inference70implicit, explicit
socialsocial interaction166literal (explicit only)
total369

W2 / Q3: Minimality validation (new Appendix A.7)

We added a quality-control appendix (new Appendix A.7) that measures every image pair and every caption pair against baselines of unrelated pairs, to verify that the two members of a pair are highly similar overall and differ only in the targeted way. Image pairs are compared at the pixel level, at the perceptual level (LPIPS, DreamSim) and in the DINOv2 and CLIP embedding spaces used for shallow alignment; caption pairs at the surface level (differing spans and word edits), at the level of sentence meaning (sentence embeddings, natural-language inference) and in the Llama and CLIP text embedding spaces. We found that:

Image pairs are minimal at every level (Figure 11). The matched partner is far closer than any unrelated pair, the edit is confined to a small, connected region of the frame, and in social interaction the partner is closer than other edits of the same scene.

Caption pairs are minimal for both explicit and implicit conditions (Figure 12). Explicit pairs differ at one point in the sentence, share the same frame, and are read as contradictions by an NLI model. Implicit pairs differ at one point fewer, since the explicit caption is the implicit caption plus exactly one span (the property word) in every contrast, and are closer still in embedding similarity.

Outlier flagging and handling is reported in Appendix A.6 Table 4.


W3 / Q2: Related work.

Thank you for the pointers; the revision adds a related-work section with a comparison table (Appendix B, to keep section numbers stable; it moves to Section 2 in the camera-ready). The benchmarks named sit on two sides of our design. MMVP, MLLM-CompBench and CompareBench sort fine-grained failures into visual categories, but compare images that differ in many respects and evaluate through a question, so a failure cannot be attributed to the category under test and a similarity space or alignment map cannot be scored at all. VisMin, EqBen and SPEC supply the control we build on: two images and two captions that differ in one respect, scored as a 2×2 group; we adopt that format directly and now credit EqBen and SPEC as its predecessors.

Our benchmark adds two things to this format. First, it extends the controlled minimal-pair design to new domains, physical-causal, perspectival and, above all, social: no existing benchmark tests social interaction or social inference as controlled image–caption pairs, and social content appears in prior work only as emotion labels in question-answering sets. Second, within each domain it adds a level of inference: the same image pair is tested with the property named and with it left implicit, so how far the alignment reaches beyond what is stated is measured with the visual evidence fixed.

Reviewer_fsmDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for fine-grained image–text semantic alignment. It uses minimal image–text pairs with explicit and implicit descriptions to assess understanding of physical causality, perspective, and social semantics. The authors evaluate shallow alignment methods, CLIP, and generative vision–language models against human baselines. Multimodal models perform better overall, but implicit descriptions are generally more challenging. Humans retain an advantage on some perspective and social interaction tasks.

Critical strengths
  1. Minimal image–caption pairs and explicit/implicit caption conditions enable targeted evaluation of fine-grained physical-causal, perspectival, and social distinctions while controlling visual content.

  2. Experiments cover shallow alignment, contrastive models, and generative VLMs, with scaling and alignment ablations. Human consensus and single-rater baselines provide useful references for interpreting domain-specific limitations.

Critical weaknesses
  1. The source of matching failures remains insufficiently isolated.

    a) Within-modality limitations. Poor matching may reflect failures in visual discrimination, language inference, or cross-modal alignment. Without separate within-modality evaluations, these explanations cannot be distinguished.

    b) Shared constraints across alignment methods. Although Appendix A.4 compares multiple mappings, all use the same 5,000 COCO pairs and feature extraction procedure. Limitations arising from training-data coverage or representation pooling therefore remain unresolved.

    c) Scope of the conclusions. The results establish limitations of the evaluated configurations, but provide less direct support for broader claims about representational convergence or the inability of scaling to recover additional distinctions.

  2. The interpretation and generality of the explicit–implicit gap require further validation.

    a) Lexical cues and ambiguity. Removing a property word eliminates a direct matching cue and may make the description less determinate. For example, an intention to discard a match does not necessarily imply prior use. The performance gap therefore cannot be attributed solely to inferential difficulty.

    b) Synthetic-image dependence. All images are generated, leaving uncertainty about whether the findings generalize beyond generation artifacts and stimulus-selection choices. Results on independently validated subsets would strengthen this aspect.

    c) Variation across models and domains. In §4.3, GPT-5.6-terra is not significantly below human consensus in any implicit condition, and its explicit–implicit decline is close to the human estimate. Claims of persistent deficits should therefore be qualified by model and domain, while distinguishing a nonsignificant difference from evidence of equivalence.

Questions for authors
  1. Can the authors distinguish alignment failure from within-modality limitations? The authors are recommended to separately evaluate visual discrimination and language-only inference, then report cross-modal matching on items where both succeed. This would clarify whether the bottleneck lies in alignment.

  2. Does the explicit-implicit gap reflect inferential difficulty or reduced textual specificity? Could controlled caption paraphrases or an infer-then-match protocol help separate reasoning limitations from lexical-cue dependence and caption ambiguity, particularly on perspectival items?

  3. Could depth selection inflate the explicit-implicit gap? Since the best depth is selected using explicit-caption performance, do the main conclusions hold when depth is fixed or selected on an independent validation set?

3 comments
AuthorsOpenReview ↗

Response to Reviewer fsmD (1/3)

We thank the reviewer for their thoughtful comments and feedback, as well as for their encouraging assessment of our work. We have uploaded the revised manuscript PDF, with all changes highlighted in blue.


W1a / Q1. Alignment failure vs within-modality limitations (new Appendix A.6).

We agree that a matching failure alone does not locate the failure. For the post-hoc alignment between unimodal models, the revision adds a within-modality control on each side (new Appendix A.6). We use the social-interaction subset, where each concept pair recurs across many items, so a linear probe can be trained on one modality at a time. The table below gives, for Llama-3.1-8B × DINOv2-giant at the validation depth cell, the probe accuracy within the vision model, the probe accuracy within the language model, and the cross-modal group score of their alignment map on the same items:

contrastimage pairswithin vision: probe accuracy (null 95th pct.)within language: probe accuracy (null 95th pct.)across modalities: group score [null]
comforting vs teasing68.99 (.72)1.00 (.68).015 [.008]
friendly vs hostile601.00 (.75)1.00 (.69).000 [.008]
cooperating vs competing38.73 (.63)1.00 (.70).000 [.008]
Bold: above the permutation null at p < .05.

Each distinction is linearly decodable from the vision model alone and from the language model alone, yet on the same items the map's cross-modal score sits at its permutation null. The information is present in both unimodal representations and is not brought into correspondence, which is what we mean by an alignment failure. Appendix A.6 gives the probe settings, the same analysis for the CLIP image and text towers and CLIP's cross-modal scores, and notes that the other subsets do not repeat concepts across items, so a supervised probe is not available there.


W1b. Training data of the mapping.

To test whether the maps fail because their 5,000 MSCOCO training pairs never required a fine distinction, we refit the same map on VisMin training data, which includes minimal pairs differ in object, attribute, count or spatial relation (5,000 pairs alone, added to MSCOCO, and 5,000 from each single category), and scored both held-out VisMin and our suite (new Appendix A.10 Figure 15 and Tables 8-9).

The gain is selective. On held-out VisMin it is confined to object and attribute: object rises from .28 to .37 and attribute from .09 to .13, while counting stays at its null and spatial relation moves by at most two points, even when the map is trained on counting-only or relation-only pairs.

Our benchmark shows a similar pattern. The only change is on explicit past-location physical-causal items (.08 to .23); explicit future-use items remain where they were (.25), and no perspectival, social or implicit condition rises above its null under any of the ten training sets (see table below):

contrastlevelMSCOCO 5k (paper)MSCOCO 5k + VisMin 5kVisMin 5k only
past eventexplicit / implicit.03 / .01.06 / .01.03 / .03
past locationexplicit / implicit.08 / .08.23* / .00.23* / .08
future useexplicit / implicit.25* / .00.20 / .00.25* / .00
perspectivalexplicit / implicit.07 / .00.00 / .03.00 / .00
social inferenceexplicit / implicit.01 / .01.01 / .00.03 / .03
social interactionliteral.01.01.01

Bold: above the permutation null at p < .05.

Therefore, the training data for the cross-modal alignment does not sufficiently explain the failures. Minimal-pair training amplifies the distinctions the two representations already share and creates none, so the failures lie in what the map can bring into correspondence, not in the data it was fit on.


W1c. Scope of the conclusions

We have reworded the conclusions so that they state what was measured: the revised Section 4.1 now says that a linear mapping between the unimodal representations does not carry most of the distinctions tested here, rather than that alignment does not extend to fine-grained distinctions, and the scaling sentence is limited to the ranges tested.

AuthorsOpenReview ↗

Response to Reviewer fsmD (2/3)


W2a / Q2. Explicit-implicit performance gap & inferential difficulty.

We agree that an implicit caption supports the target state by inference, not entailment: an intention to discard a match does not guarantee that it was used, so even a careful reader will sometimes pick the wrong image. Some decline from explicit to implicit captions is therefore expected for any reader, and the question is how much of a model's decline goes beyond that. The human raters answer it: they read the same implicit captions with the same images, so whatever the captions leave open appears in the human decline, and a model's decline beyond it is the model's own. The revision reports that excess per item, with bootstrap intervals, in each subset (new Appendix A.9).

The human decline is small and stable across subsets. The human panel declines .09 overall (95% CI .03–.15) and between .00 (future use, the reviewer's example) and .15 (past location) in every subset, with overlapping intervals. Whatever the implicit captions leave open costs people about the same amount everywhere, which makes the human decline a usable baseline for each subset. This indicates that the ambiguity cost by implicit inference is minimal.

The models' declines are not stable across subsets, and the instability is where the deficit lies. If the gap reflects only what the captions leave open, a model's decline should track the human one subset by subset. Decline in group score from explicit to implicit captions; bold marks a decline that exceeds the human one on the same items (paired bootstrap 95% CI excludes zero):

humanGPT-5.6-terraClaude Fable 5.1Gemini 3.6 FlashClaude Sonnet 4.5
past event.07.09.10.23.25
past location.15.04.12.12.31
future use.00.03.00.05.23
perspectival.13.25.50.45.27
social inference.11.11.05.20.23

The human row is flat; the model rows are not. For Fable and Gemini the excess over people appears only on perspectival items (+0.37, +0.32) and nowhere else: a systematic shortcoming in one implicit inference that humans do not show, which a caption equally open to every reader cannot produce. Sonnet, with an excess in every subset, is the one model whose pattern fits reliance on the cue.


W2b. Synthetic images & validation

The reviewer asks whether the findings hold on independently validated items. The human panel provides that validation: 314 of the 369 items are solved by the human consensus under every caption level, and these raters had no part in building the benchmark suite. Restricting every model to those items changes no group score by more than .09 and leaves the ordering of models and the perspectival decline in place.

Two further checks address generation artifacts and real images: the benchmark quality-control appendix (new Appendix A.7) shows that the two images of a pair differ locally (median 15–24% of the frame) and are far closer to each other than to other items (AUC ≥ .995), and VisMin, built from edited real images, shows the same reader ordering and the same spatial weakness (Figures 2–4), with the caveat that it has no implicit captions, so the explicit–implicit gap itself can be validated sufficiently only on our items.


W2c. Qualify by model and domain.

We agree with the reviewer’s suggestions. The revision changes the text as follows:

  • Claims are stated per model and per domain. The Section 4.3 heading "a persistent perspectival gap" becomes "implicit perspectival judgments remain hardest for every frontier model", and the paragraph gives each model's score and decline with its own confidence interval, naming GPT-5.6-terra as the model not distinguishable from consensus in any condition. The abstract, introduction and discussion are reworded in the same way: the two domains that remain hard for every frontier model (social interaction, implicit perspectival) are named as such, and claims about the explicit–implicit decline distinguish the two models that decline like people from the two that decline more.
AuthorsOpenReview ↗

Response to Reviewer fsmD (3/3)


Q3: Depth selection robustness

Thank you for raising this. The reviewer is right that choosing the depth cell on explicit performance can favour the explicit condition, and we have now tested this directly. Three changes appear in the revision.

1. The depth cell is now chosen independently of the benchmark. Instead of selecting the cell on explicit suite scores, we select the cell that gives the highest cross-validated alignment between the two models on 1,000 held-out MSCOCO pairs, which share nothing with the benchmark items. This picks language depth 80% and vision depth 100% for the three language models we fitted. We also now read three model pairs (Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B) rather than one, and average over them (Section 3.1, Figure 2, Figure 3). Appendix A.4.2 is also updated accordingly.

2. The full depth grid gives the same answer under every selection rule (new Appendix A.5, Figure 9). Across all 121 cells of the 11 × 11 grid, we compared (i) the original explicit-selected rule, (ii) the new independent rule (validation), (iii) a fixed last-block cell, and (iv) the per-condition best depth combination as an upper bound. Under every rule, explicit future-use items are the only suite condition above their permutation null; perspectival, social and all implicit conditions remain at their null. Even the per-condition best cell falls back to the null once it is cross-fitted, except for explicit future use.

Reviewer_KLc9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark, Minimal Pairs of Pairs, to evaluate the Platonic Representation Hypothesis (PRH), which states that independently trained models converge toward a shared representation of the world as they scale.

The dataset follows a format similar to VisMin: each sample contains two images and two corresponding captions, where the images differ in a single aspect. It covers three domains: Physical-causal, Perspectival, and Social, which complement VisMin's object, attribute, counting, and spatial contrasts. The dataset also has Implicit and Explicit caption versions using the same image pairs, depending on whether the distinguishing property is directly stated or must be inferred. For example, "the accountant lost hours of work" versus "the frustrated accountant lost hours of work."

The authors use a simple representation alignment algorithm to align independently trained Llama-3.1-8B and DINOv2 representations. They then check whether the cosine similarity of the correct image–caption pairs is higher than that of the incorrect pairs. They also test CLIP, which is already trained to align images and text, and prompted generative VLM baselines.

Overall, the authors find that shallow alignment between unimodal models fails on most fine-grained distinctions, performing around the random baseline for both explicit and implicit captions. CLIP performs better than unimodal alignment on explicit captions, but is still around random performance. On implicit captions, both unimodal alignment and CLIP perform substantially worse, and CLIP's advantage over unimodal disappears. Generative VLMs generally perform well on both implicit and explicit captions, with stronger frontier models approaching human performance on many tasks.

Overall, the results show that simple unimodal alignment doesn't preserve the fine-grained distinctions tested by Minimal Pairs of Pairs. This result challenges PRH but doesn't disprove it since the authors acknowledge the alignment algorithm might be too simple.

Critical strengths
  1. The explicit and implicit caption component of the proposed minimal-pair dataset is novel. The three knowledge domains complement those covered by VisMin and provide a useful benchmark for the research community.

  2. The results challenge PRH under the tested alignment algorithm. The authors are generally careful in their interpretation and acknowledge that the alignment algorithm itself might be the limiting factor. The topic is also exceptionally well suited to ICLR.

Critical weaknesses
  1. Limited coverage of related work. The paper lacks a dedicated related work section and does not adequately discuss closely related studies [1, 2, 3] on representational alignment between independently trained vision and language models.
  2. Limited novelty of the findings. Prior work [2] has already challenged PRH by showing that representational alignment captures coarse-grained distinctions (e.g., dogs vs. cars) but fails to preserve fine-grained distinctions (e.g., German Shepherd vs. Northern Inuit). The main findings of this paper are therefore not entirely new.
  3. Limited investigation of the alignment algorithm. Although the authors acknowledge that the alignment algorithm might be the limiting factor, they do not investigate whether more expressive alignment methods, such as those proposed in [1, 3], could recover the fine-grained distinctions. This makes it difficult to determine whether the failures arise from the underlying representations or the alignment method.

[1] Yoon, Yue, and Kim. Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models. UniReps Workshop, 2025.

[2] Koepke et al. Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale. 2026.

[3] Zhang, Yang, and Agrawal. Assessing and Learning Alignment of Unimodal Vision and Language Models. CVPR 2025.

Questions for authors

Line 84 states that "Each item consists of two images and two captions." Should it be four captions instead due to explicit and implicit variants?

1 comment
AuthorsOpenReview ↗

Response to Reviewer KLc9

We thank the reviewer for their thoughtful comments and feedback, as well as for their encouraging assessment of our work. We have uploaded the revised manuscript PDF, with all changes highlighted in blue.


W1: Related work (new Appendix B).

Thank you for the pointers; all three are now discussed in a new related-work section (Appendix B in the revision, to keep section numbers stable; it will move to Section 2 in camera-ready), together with the benchmark literature. These studies bracket our question from two sides: Koepke et al. [2] show, with aggregate geometry metrics, that the convergence we start from is coarse, holding between categories but not within them, and JAM [1] and SAIL [3] show how much of the gap a trained aligner between frozen encoders can close on retrieval and classification.


W2: Novelty.

We agree that the coarse-versus-fine conclusion is shared with [2], which we now cite as concurrent evidence. What our work adds is the question neither side asks: which semantic distinctions survive the correspondence and which are lost. We test this across physical-causal, perspectival and social distinctions, and at two levels of inference, since the same image pair is tested with the property named and with it left implicit, which probes how far the alignment reaches beyond what is stated.


W3: Additional alignment algorithms (new Appendix A.8).

We also agree that our orthogonal map is one shallow way to align, and the revision now varies the map, its training data and its depth (Appendix A.4-A.6), and adds SAIL [3] as a trained aligner (Appendix A.8). The answer is domain-graded rather than a uniform loss of resolution: physical-causal contrasts respond to all of these, while perspectival and social contrasts respond to none. That domain structure, and the explicit-implicit comparison, are not available from aggregate convergence measures and are what the paper contributes.


Q1: Minor correction.

Yes. Each image pair carries two captions per caption level, and every contrast except social interaction has both an explicit and an implicit level, so four captions in all. Section 2.1 now says so: "Each item consists of two images and two captions at a given caption level.”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.