acceptodds
Under review as a conference paper at ICLR 2027

Readout Blindness: VLM Scores Miss the Spatial Direction Their Frozen Encoders Retain

Abstract

CLIP-like vision-language models remain a cornerstone of multimodal systems, yet their scores stay near chance on directed spatial relations, such as whether one object is left of another. We call this failure readout blindness and analyze, theoretically and empirically, why deployed scores miss the direction: when scoring rules treat the subject and object symmetrically, direction cancels regardless of encoder training. Guided by this analysis, we introduce Antisymmetric Displacement Readout (ADR), which aligns caption words with image patches in the frozen features and scores each relation by the signed displacement between matched object centroids. Notably, ADR succeeds without additional training or learned parameters, thereby demonstrating that directional information remains in the frozen encoder. However, text and world priors can inflate accuracy, so we further introduce prior deflation, which measures the benefit of the image-text pairing as the grounded gain over a null that pairs each item with an unrelated image. Extensive experiments across encoder families show that ADR substantially improves over deployed scores, which remain near chance on most direction-balanced sets even for fine-tuned encoders. Compared with more complex readouts, ADR outperforms the evaluated MLLM likelihood readouts and is competitive with their chat inference at a small fraction of the computation. These results support our claim that directional information can be recovered from frozen features by an appropriate readout. Our implementation and evaluation kit will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, spatial relations, explicit alignment, optimal transport, training-free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FDKSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether the vision encoder already contain the directional information despite that vision-language matching scores fail on directed spatial relations. It introduces Antisymmetric Displacement Readout (ADR), which aligns words with image patches and scores signed displacements between object centroids. To solve the accuracy inflation problem brought by text and world priors, it further introduces prior deflation which measures the benefit of the image–caption pairing over a null. Experiments across nine encoder families show substantial recovery on several spatial subsets without weight updates, alongside comparisons with matching heads and MLLM likelihood/chat readouts. The results support the effectiveness of the readout design.

Critical strengths
  1. A constructive connection between analysis and method. The symmetry analysis identifies explicit conditions under which direction is lost, and the antisymmetric construction reduces to an interpretable centroid-displacement score.

  2. Substantial gains with frozen weights across encoder families. On VG-Relation left/right, CLIP improves from 49.64% accuracy with its deployed score to 90.45% with ADR's excess-coupling variant. The broader comparisons include multiple encoder families, while reporting the weaker results on What'sUp-A rather than implying uniform recovery. In addition, comparisons with half-crop, cross-encoder, and MLLM readouts help assess alternatives to the proposed approach.

Critical weaknesses
  1. Insufficient discussion of closely related work. The introduction frames whether frozen encoders retain information that their scores fail to expose as unresolved (lines 45–53), but [1, 2] already show that CLIP's failures can arise from downstream extraction or cross-modal alignment rather than missing information. MaskCLIP and ClearCLIP [3, 4] provide training-free dense-feature extraction relevant to Appendix D.1, while Test-Time Matching [5] studies capability obscured by evaluation procedures. The paper should discuss these connections explicitly and clarify the specific advances of its analysis, ADR, and prior-deflation protocol.
  2. The contributions of feature extraction and scoring are not isolated. ADR changes both how patch features are extracted (Appendix D.1) and how spatial relations are scored. The experiments evaluate these changes together. Holding the scoring rule fixed while comparing standard patch features with the modified extraction would clarify how much the extraction procedure contributes to the gains.
  3. Prompt robustness of the MLLM baselines is unclear. Appendix E.4 uses fixed prompt templates, whereas prior work [1] documents prompt sensitivity and reports substantially higher LLaVA-1.5 performance under related evaluation protocols. The authors should evaluate a small, predefined set of prompt variants, including relevant published templates, on the same items with fixed model and decoding settings. Reporting both real and null accuracy across these prompts would clarify whether the comparison with ADR is robust to prompting choices rather than specific to the selected templates.

[1] Siting Li, Pang Wei Koh, and Simon Shaolei Du. Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder. ACL, 2025.

[2] Darina Koishigarina, Arnas Uselis, and Seong Joon Oh. CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally. ICLR, 2026.

[3] Chong Zhou, Chen Change Loy, and Bo Dai. Extract Free Dense Labels from CLIP. ECCV, 2022.

[4] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference. ECCV, 2024.

[5] Yinglun Zhu, Jiancheng Zhang, and Fuzhi Tang. Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models. ICLR, 2026.

Questions for authors
  1. What are the specific advances over [1–5] in the diagnosis, readout design, and evaluation? How does Appendix D.1's feature extraction relate to MaskCLIP and ClearCLIP?
  2. How does ADR perform with standard patch features versus the modified extraction in Appendix D.1, keeping the word–patch alignment and directional scoring procedures fixed?
  3. Are the grounded-gain rankings and the identification of direction-balanced subsets stable across multiple derangements and relation/object-stratified null evaluations?
  4. How do LLaVA-1.5's real accuracy and grounded gain vary across a small, predefined set of prompts? Does its performance relative to ADR remain consistent?
5 comments
AuthorsOpenReview ↗

Official Response from Authors of Submission 4198 [1/2]

Dear Reviewer FDKS,

Thank you for your expert review. We appreciate your recognition of the constructive link between our symmetry analysis and ADR and the substantial gains with frozen weights across encoder families. We address your concerns below.


W1 & Q1: Advances over [1–5] and relation of D.1 to MaskCLIP/ClearCLIP.

We will cite and discuss all five works.

  • [1, 2]. These works show that deployed scores can miss encoded information. We further prove when context-free unary and symmetric-channel relational scores cannot distinguish direction. This leads to ADR, which reads direction from object locations without an MLLM decoder or learned alignment map.
  • Contextual cosine. It changes with word order, so our invariance conditions do not apply. In "Beyond the theorems," we write its score difference as the image embedding projected onto the text-embedding difference. Mismatched-image controls then test directional alignment. This separates exact invariance from empirical failures.
  • [3, 4]. ADR builds on their dense-feature extraction and adds directional scoring. D.1 follows MaskCLIP's unmixed value path. ClearCLIP also removes the final residual and FFN, but uses query-query attention.
  • [5]. TTM uses group matching and iterative test-time adaptation. Prior deflation instead tests, with fixed weights, whether accuracy depends on the correct image–caption pairing. We have not run a direct TTM comparison.

W2 & Q2: Separating feature extraction and scoring.

Table 1 fixes alignment and centroid scoring and varies only feature extraction. Both value-path and ClearCLIP features far outperform standard tokens. Recovery thus does not depend on the value path specifically.

Table 1. ADR-π̃ on CLIP ViT-B/16 with different features. Accuracy (%). WU = What'sUp.

FeaturesVG L/RCOCOWU-AWU-B
Standard final-layer23.3324.2621.8124.29
Value-path (D.1)90.4592.3331.6293.22
ClearCLIP q-q90.5888.8630.6495.48

Table 2 fixes the value-path features and varies the readout. Mean-pooling cosine and unary OT stay near the null, while ADR recovers direction. Both extraction and readout therefore contribute. This does not establish pooling as the cause of failure.

Table 2. Readouts on CLIP ViT-B/16. Real / null accuracy (%). All rows use value-path features except global cosine.

ReadoutVG L/RCOCOWU-AWU-B
Global cosine49.64 / 49.6945.30 / 47.2826.96 / 25.9853.67 / 52.82
Mean-pool cosine49.82 / 50.5148.02 / 50.9925.25 / 25.9851.13 / 49.44
Unary OT50.00 / 50.0052.97 / 48.7624.75 / 25.9848.87 / 48.59
ADR-dense90.22 / 49.4494.06 / 52.7241.18 / 25.2598.02 / 50.00
ADR-π̃90.45 / 49.0392.33 / 51.7331.62 / 25.7493.22 / 52.26

AuthorsOpenReview ↗

Official Response from Authors of Submission 4198 [2/2]

Q3: Stability across derangements and stratified nulls.

  • Derangements. Over 20 same-source-excluded derangements, ADR-π̃ keeps positive gain and a clear lead over global cosine on all subsets. Null SDs are small relative to the gaps.

Table 3. CLIP ViT-B/16, 20 derangements. Acc. (%) / gain (pp, mean ± SD).

ReadoutVG L/RCOCOWU-AWU-B
Global cosine49.64 / −0.02±0.5645.30 / −3.57±2.3726.96 / 2.08±1.1353.67 / 3.40±1.33
ADR-π̃90.45 / 40.42±0.9692.33 / 39.25±2.6431.62 / 7.06±1.1593.22 / 43.47±2.38

  • Stratified nulls. The null depends on replacement selection. On WU-B, the ADR-π̃ null is 55.82 ± 2.83% over five relation-stratified offsets, as same-relation images may keep the original direction. Under object-stratified replacement it is 6.78%. That control has one replacement per example (354 examples, 177 groups), so reseeding gives no independent stability test.
  • Direction-balanced subsets. Over 20 derangements, LLaVA-1.5-7B and Qwen3.6-27B caption-likelihood nulls stay near chance on VG L/R (49.96 ± 0.63, 49.93 ± 0.70) and What'sUp. LLaVA is higher on COCO (54.65 ± 2.02). This does not show balance across all prompts and stratified controls, so we will state the readouts and null protocol defining these subsets.

W3 & Q4: Prompt robustness of MLLM baselines.

With fixed LLaVA-1.5-7B weights and decoding, we tested the original prompt, a paraphrase, and relation questions adapted from [1] (Table 8). Adapted questions keep our candidate sets and average both answer orders. We also ran the official VQAScore of [1] with its original checkpoint, system prompt, preprocessing, and scoring.

Table 4. Real (%) / null (%) / gain (pp) on identical items. ADR uses SigLIP2. VQAScore is a separate scoring protocol, not a chat prompt.

ReadoutVG L/RCOCOWU-AWU-B
LLaVA caption lik.65.80 / 49.44 / 16.3679.21 / 54.21 / 25.0030.39 / 24.26 / 6.1363.56 / 52.82 / 10.73
Qwen caption lik.74.76 / 49.29 / 25.4788.12 / 49.01 / 39.1167.89 / 24.51 / 43.3879.10 / 50.56 / 28.53
LLaVA original chat65.74 / 50.10 / 15.6465.59 / 50.74 / 14.8544.49 / 21.57 / 22.9265.11 / 51.41 / 13.70
LLaVA paraphrase68.21 / 50.27 / 17.9466.09 / 49.38 / 16.7149.14 / 26.47 / 22.6767.80 / 49.29 / 18.51
LLaVA adapted [1]85.29 / 49.18 / 36.1180.69 / 51.79 / 28.9075.06 / 34.83 / 40.2398.02 / 49.29 / 48.73
VQAScore [1]87.75 / 48.95 / 38.8089.11 / 49.50 / 39.6168.87 / 27.94 / 40.9386.58 / 48.45 / 38.13
ADR-dense92.32 / 50.36 / 41.9697.52 / 51.24 / 46.2880.39 / 24.51 / 55.88100.00 / 51.41 / 48.59
ADR-π̃92.09 / 49.46 / 42.6396.53 / 52.48 / 44.0574.26 / 26.96 / 47.30100.00 / 50.85 / 49.15

Both ADR variants exceed both caption-likelihood baselines in gain on all subsets. Prompt choice affects LLaVA chat results. ADR-dense exceeds the best chat accuracy on all subsets; ADR-π̃ is slightly lower on WU-A. On WU-B, the best chat prompt's gain lies between ADR-dense and ADR-π̃.


We hope these responses address your concerns and are happy to clarify further.

Thank you again for your precise questions. As the public discussion ends on November 18, we would like to know whether our answers resolve them. We will upload the revised PDF shortly, with the changes described in our responses. If our answers resolve your concerns, we would be grateful if you would take them into account in your assessment. We are glad to answer any further questions before the discussion closes.

Reviewer_tPGBOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper analysis a common failure in CLIP like vision language models’ in distinguishing directional information in images, namely readout blindness, and propose Antisymmetric Displacement Readout (ADR) as a better readout for directional information. The paper provides elaborated theoretical analysis in readout blindness to arrive at the solution. The paper provides prior-controlled empirical study with prior deflation by computing the gain over a null setup where the image is deterministic random replacement with unrelated image. The empirical study shows better results of the proposed method over various baselines in readout of directional information. The claim is that the pre-trained encoder retains sufficient information to recover spatial direction, while the deployed scores fail to expose.

Critical strengths

S1. The paper provides a useful perspective analyzing the failure pattern in CLIP like vision language models on failure in understanding directional information.

S2. The paper provides established mathematically sound proofs under their assumption within the scope of readout blindness.

S3. The paper provides extensive experiments proving their points. The results provide evidence for advantage of the proposed ADR method in recovering directional information from pre-trained model’s embeddings.

Critical weaknesses

W1. The paper claims that “Notably, ADR succeeds without additional training or learned parameters, thereby demonstrating that directional information remains in the frozen encoder.”. However, ADR computes the directional information from estimated centroid coordinates, this establish the that the frozen feature support localization of objects, but does not directly indicate that directional information is embedded, as such localization could be done without resolving directional ambiguity. As object/subject localization could be done by general understanding of the single concept in the image and not necessarily through understanding the directional information.

W2. The paper lacks clarity on how their theoretical analysis relates to how a CLIP like model works. Line 128 starts with assuming content word is encoded independently, while as modern CLIP models employs attention mechanism where the caption content are mapped to tokens in a contextualized way.

W3. The demonstrated applicability concentrates on directional relations that could be evaluated through projected centroid displacements. This limits the conclusions’ support on relational information in frozen encoders.

Questions for authors

Q1. Could the author clarify on how the theoretical analysis relates to CLIP model’s behavior, while it is evident that the original CLIP model fails (deployed scores), could the method be used to analyze why this failure occurs.

Q2. Can the authors clarify whether ADR method only applies to directional information that’s recoverable through object localization? Or it could be apply to other relational information, such as spatial relation or action relation.

3 comments
AuthorsOpenReview ↗

Official Response from Authors of Submission 4198 [1/2]

Dear Reviewer tPGB,

Thank you for your thoughtful review and for recognizing our perspective on the directional failure of CLIP-like models, the soundness of our proofs, and the extensive experiments supporting ADR. Below, we clarify what our results establish and how our analysis relates to contextual CLIP scores.


W1: ADR shows that frozen features support object localization, but not necessarily that directional information is embedded in the encoder.

  • Scope of our claim. A directional judgment combines visual features, text features and a readout. ADR aligns the subject and object text features with image patch features to estimate their locations, then compares these locations along the relation axis. For image-plane relations such as left/right and above/below, the ordered locations are sufficient to determine direction. Our claim is frozen representations provide sufficient localization information for this judgment when combined with patch coordinates, rather than that the vision encoder alone explicitly represents direction.

  • Recovery depends on the image. The mismatched-image experiments confirm this. Across 20 same-source-excluded derangements, ADR maintains a positive mean grounded gain, showing that the frozen features provide useful, image-dependent localization information.

    Table 1. ADR-π̃ on VG-Relation left/right. Null accuracy and gain report mean ± sample std over 20 derangements.

    EncoderAcc. (%)Null (%)Gain (pp)
    CLIP ViT-B/1690.4550.03 ± 0.9640.42 ± 0.96
    SigLIP292.0950.17 ± 0.9841.92 ± 0.98

  • Revision. We will state the claim as recovery of direction from frozen encoders through suitable local feature extraction and readout.

W2 & Q1: How the theoretical analysis relates to contextual CLIP models, and whether it explains the failure of deployed scores.

  • Scope of the theorems. The independent word assumption describes the local scoring rules analyzed by our theorems, not CLIP's full-caption encoder. Nouns are encoded separately when constructing the patch–word alignment. The theorems explain why certain unary and symmetric-channel relational scores cannot distinguish direction under these assumptions.

  • Contextual full-caption cosine. We analyze this score separately in "Beyond the theorems." Appendix A.1 confirms that swapped captions have different text embeddings and produce no global-score ties among the 3,918 evaluated pairs. The poor directional performance of deployed scores therefore cannot be explained simply by identical text representations.

  • Role of feature extraction. We further test how feature extraction affects recovery, keeping the alignment and centroid scoring procedures fixed.

    Table 2. Feature extraction ablation with ADR-π̃ on CLIP ViT-B/16. Accuracy (%).

    Feature extractionVG left/rightCOCOWhat'sUp-AWhat'sUp-B
    Standard final-layer tokens23.3324.2621.8124.29
    Value-path features90.4592.3331.6293.22

This comparison shows that feature extraction is an important part of direction recovery. Together, our theoretical and empirical analyses identify the limits of specific scoring rules and the importance of local feature access. They do not determine whether pooling removes directional information or whether standard features require a different readout. Our central result is that direction can be recovered without changing the encoder weights.


AuthorsOpenReview ↗

Official Response from Authors of Submission 4198 [2/2]

W3 & Q2: Whether ADR applies beyond directional relations recoverable through object localization.

  • Why image-plane direction. ADR's centroid-displacement score targets spatial relations determined by object locations. Image-plane direction provides a clearly defined starting point for studying relational readout. Action relations, by contrast, often carry strong text priors that can mask the contribution of image–text alignment. For example, "the player is holding the racket" is more plausible than "the racket is holding the player" even without an image. Our prior-deflation results illustrate this: LLaVA achieves 93.26% accuracy on VALSE actant-swap and still retains 88.51% after image replacement, so high raw accuracy alone provides limited evidence of visual grounding.

  • Beyond spatial direction. Our conclusion identifies broader multimodal alignment tasks and relations beyond noun pairs as future work. We have also obtained positive results beyond spatial direction with the fixed-coupling framework. On SugarCrepe object-swap and attribute-swap, which test object and attribute assignments within a caption, we use a structural readout with dependency and local image-affinity kernels.

    Table 3. Fixed-coupling structural readout on SugarCrepe. Accuracy (%) and gain (pp).

    MetricObject swapAttribute swapCombined
    Acc.61.6371.7068.99
    Null50.2055.3353.95
    Gain11.4316.3715.04
    Samples245666911

The positive grounded gains support extending the fixed-coupling framework to non-spatial composition through suitable relation kernels.

  • Action relations. We also tested action relations in VG-Relation, including the holding example above and "the man is jumping off the ground" vs. "the ground is jumping off the man." These tests did not show comparable recovery after accounting for the mismatched-image baseline. Object locations alone generally do not determine action roles, which motivates future work on relation kernels that capture interactions.

We hope these clarifications address your concerns, and we would be happy to provide further details.

Thank you again for your precise questions. As the public discussion ends on November 18, we would like to know whether our answers resolve them. We will upload the revised PDF shortly, with the changes described in our responses. If our answers resolve your concerns, we would be grateful if you would take them into account in your assessment. We are glad to answer any further questions before the discussion closes.

Reviewer_YpEfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the gap between spatial information recoverable using frozen vision-language encoders and the performance of their deployed image-text scores. It proves directional invariance for context-free unary scores and certain relational scores with symmetric image kernels, and proposes Antisymmetric Displacement Readout (ADR): localize subject and object words through patch-word alignment, compute their spatial centroids, and score their signed displacement along a prescribed relation axis. The default alignment uses excess Sinkhorn mass; a second variant uses independent softmax weights. The paper also evaluates the difference between accuracy with the correct image and an unrelated image ("grounded gain"). Experiments across nine encoder families and five main spatial subsets show large improvements over global scores on most subsets, with localization, confidence-ranking, and computational-cost analyses.

Critical strengths
  1. Substantial and broad empirical improvements without additional weight training. On VG-Relation left/right, CLIP accuracy increases from 49.64% to 90.45% with the default ADR, and SigLIP2 increases from 49.95% to 92.09% (Table 2). The coverage of multiple encoder families and compositional fine-tunes makes this a useful demonstration that frozen weights can support considerably better spatial decisions through local information and explicit geometry. The localization maps and centroids also make the decisions inspectable.

  2. Useful evaluation controls and disclosure of limitations. Reporting both real-image and mismatched-image accuracy exposes cases where high raw accuracy provides limited evidence of visual grounding: LLaVA's VALSE actant-swap accuracy remains 88.51% after image mismatch, compared with 93.26% on the original pairs (Table 1). The paper also reports the What'sUp-A failures, the depth proxy, and reference-frame limitations. These controls and qualifications add practical value even though grounded gain should not be interpreted as a complete separation of priors from visual evidence.

Critical weaknesses
  1. The methodological advance over the closest prior work is not established. REGROUND (Bukkapatnam, ALVR 2026, Section 4, pp. 289-290, Eqs. 1-5) already uses frozen spatial visual tokens, separate subject/object softmax attention, attention-weighted spatial centroids, and a hand-specified relation vector without training. Its spatial term is the cosine between the centroid displacement and the relation vector, whereas ADR-dense uses the unnormalized projection. With matched features, noun representations, alignment temperature, and parsing, the same nonzero displacement, and a positive relation weight, these terms induce the same direction choice for a single-relation argument swap; REGROUND's two unary terms are unchanged by that swap. This does not establish equivalence of the complete systems, but it makes the overlap central to assessing novelty. The submission cites Bukkapatnam and critiques its theoretical argument, yet does not discuss this methodological overlap or directly compare against REGROUND. The remaining OT/excess-mass difference is not shown to provide a consistent advantage over the simpler dense variant (Tables 2 and 6). A matched comparison is needed to establish what is newly contributed beyond existing training-free geometric readouts. Reference: https://aclanthology.org/2026.alvr-main.28/

  2. The evidence does not sufficiently isolate the proposed explanation of deployed-score failure. The invariance theorems concern restricted scoring rules, rather than the contextual full-caption cosine used by CLIP and SigLIP. Appendix A.1/Table 4 explicitly finds no identical text features or global-score ties among the 3,918 evaluated pairs, and leaves pooling loss versus global alignment failure unresolved. Moreover, Appendix D.1 extracts CLIP patch features by bypassing the final attention mixing, residual connections, and feed-forward network; ADR then adds patch coordinates and prescribed relation axes. The results support useful spatial decisions from frozen weights under this extraction and geometric readout, but do not isolate the effects of feature extraction, local-versus-pooled access, and the scoring rule itself. A vanilla-patch versus D.1 extraction ablation under the same geometric readout is therefore consequential to the paper's central diagnosis. The explicitly disclosed use of vertical image position as a front/behind proxy (D.5) further limits the interpretation as general spatial understanding.

Questions for authors
  1. Novelty clarification and decisive comparison: How does ADR-dense differ from REGROUND's Section 4 beyond displacement normalization and implementation choices? Under matched feature extraction, noun representations, parsing, and temperature, what additional benefit comes from ADR's projection and from OT/excess-mass weighting? A direct comparison would materially affect the contribution assessment.

  2. Diagnostic evidence: How much of the gain remains with patch features from the standard forward path, using the same alignment and centroid readout? Can the current evidence distinguish information lost during pooling from information accessible only after the D.1 extraction modification?

  3. Evaluation clarification: How stable are the main grounded-gain comparisons across multiple same-source-excluded derangements? For VSR, how do the comparisons change when real and null evaluations share one threshold selected independently of the test labels, rather than separately optimized oracle thresholds (E.3)?

These questions concern the current contribution and its evidence, rather than requests to expand the paper to unrelated tasks.

4 comments
AuthorsOpenReview ↗

Official Response from Authors of Submission 4198 [1/2]

Dear Reviewer YpEf,

Thank you for your careful and constructive review. We appreciate your recognition of the substantial and broad gains of ADR across encoder families without any additional training, as well as the value of our evaluation controls and our transparent discussion of limitations. Below, we address your concerns point by point.


W1 & Q1: Methodological overlap with REGROUND and the need for a matched comparison.

  • Direct comparison. We identified and cited REGROUND during our literature review and we carefully implemented REGROUND following its published description (no public code available ) and compared it with ADR on the same CLIP ViT-L/14-336 backbone.

    Table 1. REGROUND vs. ADR. Real / null accuracy (%).

    MethodVG-Relation left/rightCOCO-spatialWhat'sUp-AWhat'sUp-B
    REGROUND44.88 / 50.8844.31 / 47.4027.82 / 25.9846.89 / 47.32
    ADR-dense90.30 / 49.2395.79 / 50.7461.52 / 24.26100.00 / 47.46
    ADR-π̃90.35 / 49.0694.06 / 50.9933.33 / 23.53100.00 / 47.46

Both ADR variants achieve higher real-image accuracy and grounded gain than REGROUND.

  • Source of the gap. The two pipelines differ in two components. (i) Feature extraction: REGROUND uses standard final-layer tokens, whereas ADR uses value-path features (Appendix D.1). (ii) Alignment and scoring: REGROUND uses independent noun attention and cosine displacement, whereas ADR uses signed displacement projection; the default ADR further uses a shared OT coupling with excess-mass localization. To isolate (i), we fix ADR's alignment and scoring and vary only the feature extraction.

    Table 2. Feature extraction ablation on CLIP ViT-B/16, with alignment and scoring fixed. Accuracy (%).

    Feature extractionVG-Relation left/rightCOCO-spatialWhat'sUp-AWhat'sUp-B
    Standard final-layer tokens23.3324.2621.8124.29
    D.1 value-path features90.4592.3331.6293.22

With standard final-layer tokens, accuracy falls below chance; with value-path features, it exceeds 90% on three of the four subsets. Feature extraction is therefore the main source of the gap in Table 1.

  • Matched scoring rules. Once feature extraction is matched, the comparison isolates the scoring rules rather than the complete methods. Consistent with your analysis, the signed projection and the cosine yield the same direction decision for a single-relation argument swap; the projection's benefit lies in confidence ranking rather than direction accuracy. OT/excess-mass weighting provides joint subject–object alignment and lower temperature sensitivity in the tested settings (Appendix C.4), rather than a consistent accuracy gain over ADR-dense.

  • Our contribution beyond REGROUND. (1) A theoretical account that characterizes when existing scores are direction-invariant and derives the directional readout from the antisymmetric channel of an explicit coupling; (2) the finding that feature extraction is decisive for recovering direction from frozen encoders (Tables 1–2); and (3) a grounded-gain evaluation across nine encoder families. We will add this discussion and the comparison above to the revision.


W2 & Q2: Isolating the explanation of deployed-score failure.

Our theorems explain why certain existing scoring methods cannot distinguish relation direction under the stated assumptions. Contextual full-caption cosine falls outside these assumptions, so we study it separately in “Beyond the theorems” via a score decomposition and mismatched-image controls. This shows that different text embeddings alone do not ensure correct direction judgments.

Table 2 fixes the alignment and centroid scoring. Standard CLIP patch features are far less accurate than value-path features, showing that feature extraction is key to recovering direction. The local features provide object localization, which ADR combines with patch coordinates and a specified relation axis to judge direction.

Information loss during pooling does not imply that directional information is lost. Our central claim concerns whether direction is recoverable. Under our settings, ADR recovers direction from frozen encoders better than deployed scores via local feature extraction and geometric scoring. Together with the gains over mismatched-image controls, this shows that frozen encoders provide localization information sufficient for these directional judgments given known patch coordinates. This does not determine whether pooling removes directional information or whether standard features need a different readout.

For SigLIP/SigLIP2, our extraction keeps all Transformer blocks, so their gains do not rely on bypassing the final block. The What’sUp-B front/behind results use vertical position as a proxy and should be read separately from image-plane direction results.

AuthorsOpenReview ↗

Official Response from Authors of Submission 4198 [2/2]

Q3: Stability of grounded gain and the VSR threshold protocol.

  • Multiple derangements. Across 20 same-source-excluded derangements, ADR retains a positive mean grounded gain on every evaluated subset. Closely performing local readouts can exchange ranks, but the broad advantage over deployed scores persists.

    Table 3. ADR-π̃ across 20 same-source-excluded derangements. Accuracy and null accuracy in %, gain in pp; null and gain report mean ± sample std.

    EncoderMetricVG left/rightCOCOWhat'sUp-AWhat'sUp-B
    CLIPAcc.90.4592.3331.6293.22
    CLIPNull50.03 ± 0.9653.08 ± 2.6424.56 ± 1.1549.75 ± 2.38
    CLIPGain40.42 ± 0.9639.25 ± 2.647.06 ± 1.1543.47 ± 2.38
    SigLIP2Acc.92.0996.5374.26100.00
    SigLIP2Null50.17 ± 0.9852.31 ± 3.3225.24 ± 1.7249.21 ± 2.03
    SigLIP2Gain41.92 ± 0.9844.22 ± 3.3249.02 ± 1.7250.79 ± 2.03

    Since real-image accuracy is fixed across derangements, null accuracy and grounded gain share the same standard deviation.

  • VSR with a shared, label-independent threshold. For deployed scores, we now select the threshold on training examples (with evaluation sources excluded) and apply the same threshold to both real and null images. ADR keeps its fixed zero threshold. The advantage remains on the common held-out cohort.

    Table 4. VSR horizontal on 377 held-out examples, with one shared threshold per readout. Real / null accuracy (%) and grounded gain (pp).

    ReadoutRealNullGain
    CLIP deployed52.5246.685.84
    CLIP ADR-π̃75.0746.1528.91
    SigLIP2 deployed55.7046.958.75
    SigLIP2 ADR-π̃77.4547.7529.71

    This table reports the fixed-threshold control rather than the 20-derangement average.


We hope these responses address your concerns, and we would be happy to provide further clarification.

Thank you for the additional experiments and clarifications, which substantially address my other concerns. However, the substantive methodological advance over REGROUND remains unclear, particularly when feature extraction is matched. I therefore lean toward maintaining my current rating.

AuthorsOpenReview ↗

Response to the follow-up comment: REGROUND as published and our contributions

Dear Reviewer YpEf,

Thank you for the follow-up. Our method is ADR-π̃, and ADR-dense is a softmax ablation of its weights. Feature extraction is a component of ADR (Appendix D.1) and is absent from REGROUND, so a comparison with matched features cannot evaluates REGROUND as published. Below, we assess REGROUND as its paper defines it and state the contributions of our work. Tables 1–4 refer to our first response.

1. The diagnosis does not depend on the features. With the value-path tokens fixed, every score without an antisymmetric term stays at the null level (Table 5), as our theorems predict. After a horizontal flip of the image, the deployed score keeps its decision on 94.69% to 98.42% of the left/right items across both encoders and five subsets.

Table 5. Readouts on CLIP ViT-B/16. Real / null accuracy (%). All rows except the global cosine use value-path tokens.

ReadoutVG-Relation left/rightCOCO-spatialWhat'sUp-AWhat'sUp-B
Global cosine49.64 / 49.6945.30 / 47.2826.96 / 25.9853.67 / 52.82
Mean-pool cosine49.82 / 50.5148.02 / 50.9925.25 / 25.9851.13 / 49.44
Unary OT50.00 / 50.0052.97 / 48.7624.75 / 25.9848.87 / 48.59
ADR-π̃90.45 / 49.0392.33 / 51.7331.62 / 25.7493.22 / 52.26

2. REGROUND as published. REGROUND has no released code. Its paper leaves the normalization of the inner products and all but one direction vector unspecified, and its stated resolution does not match its stated token count. We implemented it strictly from the paper without tuning. It stays at the null level as written (Table 1) and falls to between 16.34% and 25.14% when unit-normalized. Its phrase terms tie on at least 96% of the items, as our theorem on unary terms predicts. Its accuracy bound also assumes equal caption scores for one image, which does not follow from its pooling analysis (Related Work of our paper).

3. The tokens are part of the method. REGROUND's equations read direction only with dense tokens, which its paper does not use. With standard tokens, ADR-π̃ falls from 90.45 to 23.33 on VG-Relation with CLIP (Table 2) and to the null level with SigLIP2 (Table 6), where Appendix D.1 derives the tokens from the pooling head.

Table 6. ADR-π̃ on SigLIP2 with two sets of visual tokens. Real / null accuracy (%).

TokensVG-Relation left/rightCOCO-spatialWhat'sUp-AWhat'sUp-B
Standard final-layer50.61 / 50.2350.50 / 47.5222.79 / 25.4949.72 / 50.28
Per-token pooling head (Appendix D.1)92.09 / 49.4696.53 / 52.4874.26 / 26.96100.00 / 50.85

4. Differences from REGROUND. ADR-dense shares the per-word softmax and the centroid difference with REGROUND, and differs in the tokens and in the score. Its signed projection has a zero threshold and keeps the length of the displacement, which REGROUND's cosine discards. On the same tokens and weights, the projection ranks correct above wrong decisions better in six of the seven encoder and subset pairs, with an AUROC of 87.96% against 73.77% on VG-Relation with CLIP. ADR-π̃ further replaces the softmax by the excess mass of one OT coupling. For ε from 0.02 to 0.2, its accuracy varies by 0.10 points on VG-Relation and 2.94 on What'sUp-A, against 1.58 and 28.67 for the softmax weights (Appendix C.4), so it needs no temperature chosen per dataset.

5. Prior deflation. LLaVA keeps 88.51% on VALSE actant swap after the image is replaced, so accuracy alone cannot compare readouts. Our null is stable, with a standard deviation between 0.96 and 3.32 points over 20 derangements (Table 3). Under it, REGROUND as published shows no grounded gain, and ADR-π̃ has a larger grounded gain than every MLLM readout we tested on all four subsets (Table 7).

Table 7. Grounded gain (points) of MLLM readouts and of ADR-π̃ on identical items.

ReadoutVG-Relation left/rightCOCO-spatialWhat'sUp-AWhat'sUp-B
LLaVA-1.5 caption likelihood16.3625.006.1310.73
Qwen3.6 caption likelihood25.4739.1143.3828.53
LLaVA-1.5 chat, original prompt15.6414.8522.9213.70
LLaVA-1.5 chat, paraphrase17.9416.7122.6718.51
LLaVA-1.5 chat, prompt of Li et al. (2025)36.1128.9040.2348.73
VQAScore of Li et al. (2025)38.8039.6140.9338.13
ADR-π̃ (SigLIP2)42.6344.0547.3049.15

Summary. The paper explains why scores miss direction, identifies the tokens and the OT-based readout that recover it, and provides a protocol that separates grounding from priors. ADR-π̃ is one of these three contributions and the evidence for the first.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised paper uploaded: summary of changes and contributions

We thank all reviewers for the discussion. The revised paper is uploaded with all additions marked in blue.

  1. Claim wording (Abstract, Sec. 1, Sec. 8; tPGB). We now use a more precise statement, namely that direction can be recovered from the frozen encoder with dense patch features and an antisymmetric readout.
  2. Related work (Sec. 1, Sec. 2, App. D.1; FDKS, YpEf). We discuss Li et al. (2025), Koishigarina et al. (2026) and Test-Time Matching, credit MaskCLIP and ClearCLIP for the patch features, and position ADR against REGROUND. Five references are added.
  3. Scope of the theorems (Sec. 4, App. F.3, Tab. 21; YpEf, tPGB). The theorems concern scores on context-free word features. For the full-caption global score we add a mirror test, in which the score keeps its decision on more than 94% of the left/right items after a horizontal flip.
  4. Method (Sec. 5; YpEf). Dense patch features are stated as a component of ADR, and ADR-πd\pi_d as a softmax ablation of the OT weights of ADR-π~\tilde{\pi}.
  5. Ablations (new Sec. 7.4, Tab. 4, Tab. 18; YpEf, FDKS). With the readout fixed, the output of the last block does not support ADR, while the value path, query-query attention and the per-token pooling head do. With the patch features fixed, only the antisymmetric readout recovers direction, and the signed displacement ranks decisions better than the cosine of the displacement.
  6. Comparison with REGROUND (App. F.1, Tab. 17; YpEf). REGROUND as specified in its paper stays at or below the null level on all five subsets.
  7. Evaluation protocol (Sec. 7.4, App. F.2 and F.4, Tabs. 19, 20, 22; YpEf, FDKS). The derangement null is stable over 20 derangements. The VSR result holds with one threshold fitted on training items. The comparison with MLLMs holds across three prompts, including the question of Li et al. (2025) and VQAScore.

Broader alignment tasks and relations beyond noun pairs are stated as future work in Sec. 8 (tPGB).

On the contributions. ADR is one of three contributions. First, we show by theorem and by experiment which scoring rules lose direction and why, a failure that remains on dense patch features (Tab. 4b). Second, ADR-π~\tilde{\pi} recovers direction without training, and both of its components are required (Tab. 4). Its OT weights localize the words jointly, which keeps it stable across temperatures without tuning, unlike the softmax weights of ADR-πd\pi_d (Tab. 11). Third, prior deflation compares readouts by what they read from the image. The comparison with any single baseline is one part of this evidence, not the claim of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.