- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether the vision encoder already contain the directional information despite that vision-language matching scores fail on directed spatial relations. It introduces Antisymmetric Displacement Readout (ADR), which aligns words with image patches and scores signed displacements between object centroids. To solve the accuracy inflation problem brought by text and world priors, it further introduces prior deflation which measures the benefit of the image–caption pairing over a null. Experiments across nine encoder families show substantial recovery on several spatial subsets without weight updates, alongside comparisons with matching heads and MLLM likelihood/chat readouts. The results support the effectiveness of the readout design.
Critical strengths
-
A constructive connection between analysis and method. The symmetry analysis identifies explicit conditions under which direction is lost, and the antisymmetric construction reduces to an interpretable centroid-displacement score.
-
Substantial gains with frozen weights across encoder families. On VG-Relation left/right, CLIP improves from 49.64% accuracy with its deployed score to 90.45% with ADR's excess-coupling variant. The broader comparisons include multiple encoder families, while reporting the weaker results on What'sUp-A rather than implying uniform recovery. In addition, comparisons with half-crop, cross-encoder, and MLLM readouts help assess alternatives to the proposed approach.
Critical weaknesses
- Insufficient discussion of closely related work. The introduction frames whether frozen encoders retain information that their scores fail to expose as unresolved (lines 45–53), but [1, 2] already show that CLIP's failures can arise from downstream extraction or cross-modal alignment rather than missing information. MaskCLIP and ClearCLIP [3, 4] provide training-free dense-feature extraction relevant to Appendix D.1, while Test-Time Matching [5] studies capability obscured by evaluation procedures. The paper should discuss these connections explicitly and clarify the specific advances of its analysis, ADR, and prior-deflation protocol.
- The contributions of feature extraction and scoring are not isolated. ADR changes both how patch features are extracted (Appendix D.1) and how spatial relations are scored. The experiments evaluate these changes together. Holding the scoring rule fixed while comparing standard patch features with the modified extraction would clarify how much the extraction procedure contributes to the gains.
- Prompt robustness of the MLLM baselines is unclear. Appendix E.4 uses fixed prompt templates, whereas prior work [1] documents prompt sensitivity and reports substantially higher LLaVA-1.5 performance under related evaluation protocols. The authors should evaluate a small, predefined set of prompt variants, including relevant published templates, on the same items with fixed model and decoding settings. Reporting both real and null accuracy across these prompts would clarify whether the comparison with ADR is robust to prompting choices rather than specific to the selected templates.
[1] Siting Li, Pang Wei Koh, and Simon Shaolei Du. Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder. ACL, 2025.
[2] Darina Koishigarina, Arnas Uselis, and Seong Joon Oh. CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally. ICLR, 2026.
[3] Chong Zhou, Chen Change Loy, and Bo Dai. Extract Free Dense Labels from CLIP. ECCV, 2022.
[4] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference. ECCV, 2024.
[5] Yinglun Zhu, Jiancheng Zhang, and Fuzhi Tang. Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models. ICLR, 2026.
Questions for authors
- What are the specific advances over [1–5] in the diagnosis, readout design, and evaluation? How does Appendix D.1's feature extraction relate to MaskCLIP and ClearCLIP?
- How does ADR perform with standard patch features versus the modified extraction in Appendix D.1, keeping the word–patch alignment and directional scoring procedures fixed?
- Are the grounded-gain rankings and the identification of direction-balanced subsets stable across multiple derangements and relation/object-stratified null evaluations?
- How do LLaVA-1.5's real accuracy and grounded gain vary across a small, predefined set of prompts? Does its performance relative to ADR remain consistent?
5 comments
Official Response from Authors of Submission 4198 [2/2]
Q3: Stability across derangements and stratified nulls.
- Derangements. Over 20 same-source-excluded derangements, ADR-π̃ keeps positive gain and a clear lead over global cosine on all subsets. Null SDs are small relative to the gaps.
Table 3. CLIP ViT-B/16, 20 derangements. Acc. (%) / gain (pp, mean ± SD).
| Readout | VG L/R | COCO | WU-A | WU-B |
|---|---|---|---|---|
| Global cosine | 49.64 / −0.02±0.56 | 45.30 / −3.57±2.37 | 26.96 / 2.08±1.13 | 53.67 / 3.40±1.33 |
| ADR-π̃ | 90.45 / 40.42±0.96 | 92.33 / 39.25±2.64 | 31.62 / 7.06±1.15 | 93.22 / 43.47±2.38 |
- Stratified nulls. The null depends on replacement selection. On WU-B, the ADR-π̃ null is 55.82 ± 2.83% over five relation-stratified offsets, as same-relation images may keep the original direction. Under object-stratified replacement it is 6.78%. That control has one replacement per example (354 examples, 177 groups), so reseeding gives no independent stability test.
- Direction-balanced subsets. Over 20 derangements, LLaVA-1.5-7B and Qwen3.6-27B caption-likelihood nulls stay near chance on VG L/R (49.96 ± 0.63, 49.93 ± 0.70) and What'sUp. LLaVA is higher on COCO (54.65 ± 2.02). This does not show balance across all prompts and stratified controls, so we will state the readouts and null protocol defining these subsets.
W3 & Q4: Prompt robustness of MLLM baselines.
With fixed LLaVA-1.5-7B weights and decoding, we tested the original prompt, a paraphrase, and relation questions adapted from [1] (Table 8). Adapted questions keep our candidate sets and average both answer orders. We also ran the official VQAScore of [1] with its original checkpoint, system prompt, preprocessing, and scoring.
Table 4. Real (%) / null (%) / gain (pp) on identical items. ADR uses SigLIP2. VQAScore is a separate scoring protocol, not a chat prompt.
| Readout | VG L/R | COCO | WU-A | WU-B |
|---|---|---|---|---|
| LLaVA caption lik. | 65.80 / 49.44 / 16.36 | 79.21 / 54.21 / 25.00 | 30.39 / 24.26 / 6.13 | 63.56 / 52.82 / 10.73 |
| Qwen caption lik. | 74.76 / 49.29 / 25.47 | 88.12 / 49.01 / 39.11 | 67.89 / 24.51 / 43.38 | 79.10 / 50.56 / 28.53 |
| LLaVA original chat | 65.74 / 50.10 / 15.64 | 65.59 / 50.74 / 14.85 | 44.49 / 21.57 / 22.92 | 65.11 / 51.41 / 13.70 |
| LLaVA paraphrase | 68.21 / 50.27 / 17.94 | 66.09 / 49.38 / 16.71 | 49.14 / 26.47 / 22.67 | 67.80 / 49.29 / 18.51 |
| LLaVA adapted [1] | 85.29 / 49.18 / 36.11 | 80.69 / 51.79 / 28.90 | 75.06 / 34.83 / 40.23 | 98.02 / 49.29 / 48.73 |
| VQAScore [1] | 87.75 / 48.95 / 38.80 | 89.11 / 49.50 / 39.61 | 68.87 / 27.94 / 40.93 | 86.58 / 48.45 / 38.13 |
| ADR-dense | 92.32 / 50.36 / 41.96 | 97.52 / 51.24 / 46.28 | 80.39 / 24.51 / 55.88 | 100.00 / 51.41 / 48.59 |
| ADR-π̃ | 92.09 / 49.46 / 42.63 | 96.53 / 52.48 / 44.05 | 74.26 / 26.96 / 47.30 | 100.00 / 50.85 / 49.15 |
Both ADR variants exceed both caption-likelihood baselines in gain on all subsets. Prompt choice affects LLaVA chat results. ADR-dense exceeds the best chat accuracy on all subsets; ADR-π̃ is slightly lower on WU-A. On WU-B, the best chat prompt's gain lies between ADR-dense and ADR-π̃.
We hope these responses address your concerns and are happy to clarify further.
Thank you again for your precise questions. As the public discussion ends on November 18, we would like to know whether our answers resolve them. We will upload the revised PDF shortly, with the changes described in our responses. If our answers resolve your concerns, we would be grateful if you would take them into account in your assessment. We are glad to answer any further questions before the discussion closes.
Official Response from Authors of Submission 4198 [1/2]
Dear Reviewer FDKS,
Thank you for your expert review. We appreciate your recognition of the constructive link between our symmetry analysis and ADR and the substantial gains with frozen weights across encoder families. We address your concerns below.
W1 & Q1: Advances over [1–5] and relation of D.1 to MaskCLIP/ClearCLIP.
We will cite and discuss all five works.
W2 & Q2: Separating feature extraction and scoring.
Table 1 fixes alignment and centroid scoring and varies only feature extraction. Both value-path and ClearCLIP features far outperform standard tokens. Recovery thus does not depend on the value path specifically.
Table 1. ADR-π̃ on CLIP ViT-B/16 with different features. Accuracy (%). WU = What'sUp.
Table 2 fixes the value-path features and varies the readout. Mean-pooling cosine and unary OT stay near the null, while ADR recovers direction. Both extraction and readout therefore contribute. This does not establish pooling as the cause of failure.
Table 2. Readouts on CLIP ViT-B/16. Real / null accuracy (%). All rows use value-path features except global cosine.