- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the internal mechanisms of relative-position reasoning in vision-language models (VLMs) and language models (LLMs). The authors first introduce two paired tests, object-location swapping and target/reference role reversal, to evaluate spatial consistency. They then use activation patching to trace spatial information from source tokens through query tokens to the final answer. Location IDs are extracted and causally tested through targeted interventions.
The paper further identifies a target/reference role direction by contrasting hidden states of the same object under different query roles. Destructive interventions and activation steering are used to examine its causal effects. Experiments across multiple models and input modalities show that role-direction steering can improve accuracy and relational consistency on natural-image benchmarks without updating model parameters.
Critical strengths
1. Systematic experimental design with causal interventions.
- The paper combines two controlled perturbations with activation patching and targeted steering to examine different components of relative-position reasoning.
- The interventions provide stronger causal evidence than linear probing alone. Evaluations across multiple model architectures and input modalities further support the consistency of the observations.
2. Identification and transfer of role-related representations.
- The paper identifies a relatively stable target/reference role direction and demonstrates its causal relevance through destructive interventions.
- Role directions extracted from synthetic scenes improve relational consistency on several natural-image benchmarks without model retraining, suggesting potential practical value.
Critical weaknesses
1. Limited novelty beyond existing mechanistic studies of spatial reasoning.
Several central findings substantially overlap with prior work:
- Spatial representations: Kang et al. [1] already identified object-token spatial IDs and demonstrated their causal effects through interventions.
- Spatial information localization: Cui et al. [2] investigated spatial information in visual representations and complementary mechanisms in the language backbone.
- Information flow: Salazar et al. [3] already identified visual-to-query-to-answer information pathways using causal patching, including their flexibility under interventions.
Consequently, the location-ID analysis and layer-wise information-flow results largely extend existing findings rather than establish new mechanisms.
The main novel component is the target/reference role direction. However, its extraction relies on standard hidden-state contrasts, and its computational function remains insufficiently characterized. Moreover, the mechanistic experiments focus primarily on simple two- and three-object layouts and basic directional relations, limiting the generality of the conclusions.
2. Insufficient evidence for the proposed mechanistic interpretation.
Several gaps remain between the intervention results and the claimed mechanisms:
- Causal mediation: Source patching changes both query-side location information and final predictions, but this does not establish query tokens as necessary causal mediators. Alternative information pathways remain possible.
- Role disentanglement: The extracted role direction may also capture syntactic or contextual differences between queries. The two query templates do not fully eliminate these confounds. The paper also lacks independent cross-interventions on location and role representations to establish their functional separability.
- Consistency evaluation: Paired consistency requires both answers to be correct. Its changes may therefore partly reflect changes in ordinary accuracy rather than improvements in role binding itself.
These limitations weaken the evidence that the proposed representations constitute distinct computational mechanisms underlying relative-position reasoning.
Questions for authors
1. What new causal mechanism does the source-to-query analysis establish beyond Salazar et al. [3]?
Could the authors use path blocking or conditional activation patching to determine whether query-side location representations are necessary mediators of source-to-answer information flow, rather than one of several possible pathways?
2. Does the extracted role direction represent target/reference binding independently of linguistic context and spatial location?
Could the authors evaluate unseen query templates and independently intervene on location and role representations to demonstrate their functional separability?
3. Do the improvements in paired consistency specifically reflect better role binding?
Since paired consistency depends on both answers being correct, could the authors report relation-flip consistency independently of correctness, or evaluate reversal accuracy conditioned on the original answer being correct? This would help distinguish improved role consistency from general accuracy gains.
References
[1] Kang, R., Chen, H., Gkioxari, G., & Perona, P. (2026). Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models. International Conference on Learning Representations (ICLR 2026). https://arxiv.org/abs/2601.12626
[2] Cui, K., Prakash, N., Raina, A., Bau, D., Torralba, A., & Rott Shaham, T. (2026). The Dual Mechanisms of Spatial Reasoning in Vision–Language Models. The First Workshop on Efficient Spatial Reasoning. https://openreview.net/forum?id=Ueb38N1aF3
[3] Salazar, I., Frank, S., Oneata, D., Elliott, D., & Fierro, C. (2026). Pathways of Visual Information Flow in Vision-Language Models. arXiv preprint arXiv:2607.03358. https://arxiv.org/abs/2607.03358
5 comments
Response to Reviewer k3kw (2/2)
Role disentanglement from syntax and location (W2, Q2). We use a mean-contrast estimator to test whether target/reference roles correspond to a causally relevant and stable direction. We jointly contrast the same object in matched role-reversal questions, and our target-first and reference-first templates mitigate confounds with object identity and mention order. The held-out alignment in Figure 6 supports the stability of these joint contrasts. Additionally, destructive interventions reduce correct-versus-reversed answer margins much more than norm-matched orthogonal controls for both mention orders (Figures 7 and 15), providing evidence for a causal contribution to role-sensitive prediction.
To verify that the extracted role direction represents target/reference binding independently of linguistic context and spatial location, we conducted two new sets of experiments (Global Response, Tables R2–R3):
- Transfer to unseen query templates (Table R2a/b): Keeping role vectors, layers and strengths fixed, we tested four unseen templates on 128 held-out scene families. These templates differ in clause structure, token position and grammatical roles in both target-first and reference-first orders. Role steering has a larger flipped-answer margin shift than norm-matched random controls and both orientations of the surface mention-order direction across all 9 model and input settings.
- Independent cross-interventions on role and location (Table R3): Separately intervening in role and location on matched examples and token spans in LLaVA-1.6/Mistral, we report in Table R3 that in all six settings each intervention alters its probe readout (role→role 0.35–0.91; location→location 0.39–0.64) much more than the other readout (role→location 0.017–0.056; location→role 0.008–0.012). In addition, each intervention moves the answer margin over matched random controls in 5 of 6 settings and thus they do so in common for 4 of 6.
These results support role information distinguishable from mention order, tested syntax and location IDs, with causal effects on answers. Since roles are inherently tied to query syntax, we do not claim complete independence from it.
Consistency evaluation (W2, Q3). We agree that both-correct consistency alone cannot distinguish role binding. Using the predictions stored in Table 2, we report in Table R4 relation-flip agreement (whether the model predicts inverse relations, i.e. relations that can be reversed in object order, regardless of whether they are correct) and reversal accuracy conditioned on the original answer being correct (and therefore provides less conditioning than correctness alone), as well as the same quantity on the fixed subset of examples for which the original answer was correct at baseline (so that the same examples are compared before and after steering). We see that relation-flip agreement improves in 8/9 cases and reversal accuracy improves in 7/9 cases on the run's own conditioning set and in 8/9 cases on the fixed subset. The 95% bootstrap confidence intervals on the change (paired where appropriate) do not contain zero in 6/9 cases for relation-flip agreement and 5/9 for each of the two reversal accuracy metrics, and all of these changes are improvements. These analyses show that role reversal consistency is improved in most settings by steering, more than both-correct consistency alone can reveal.
Table R4. Consistency metrics that do not require both answers to be correct (baseline → steered). Bold marks changes whose paired 95% bootstrap confidence interval excludes zero.
| Model | Condition | Relation-flip agreement (%) | Rev. acc. | own original correct | Rev. acc. | baseline-correct (fixed) |
|---|---|---|---|---|
| LLaVA-1.6 | VLM+image | 76.7 → 79.7 (+3.1) | 254/333 → 266/334 (+3.4) | 254/333 → 266/333 (+3.6) |
| LLaVA-1.6 | VLM+text | 71.8 → 79.7 (+7.8) | 213/265 → 224/265 (+4.2) | 213/265 → 217/265 (+1.5) |
| LLaVA-1.6 | LLM+text | 49.2 → 53.3 (+4.2) | 139/229 → 158/235 (+6.5) | 139/229 → 152/229 (+5.7) |
| InternVL3.5 | VLM+image | 83.6 → 83.3 (−0.3) | 291/346 → 290/346 (−0.3) | 291/346 → 290/346 (−0.3) |
| InternVL3.5 | VLM+text | 85.3 → 86.4 (+1.1) | 288/306 → 295/314 (−0.2) | 288/306 → 291/306 (+1.0) |
| InternVL3.5 | LLM+text | 52.2 → 54.2 (+1.9) | 169/238 → 181/246 (+2.6) | 169/238 → 175/238 (+2.5) |
| Pixtral | VLM+image | 71.3 → 74.5 (+3.2) | 282/346 → 295/355 (+1.6) | 282/346 → 288/346 (+1.7) |
| Pixtral | VLM+text | 61.8 → 67.6 (+5.9) | 236/313 → 268/337 (+4.1) | 236/313 → 247/313 (+3.5) |
| Pixtral | LLM+text | 51.0 → 60.3 (+9.3) | 204/301 → 240/313 (+8.9) | 204/301 → 227/301 (+7.6) |
Thank you for the response and additional experiments. I have two remaining concerns regarding the new results:
-
Evaluation consistency (Table R4). The reported success counts, combined with the paired-consistency scores in Table 2, suggest total sample counts of either 360 or 408 across different settings, although Table 2 is described as evaluating What'sUp-A. Could the authors clarify the evaluation subsets, sample counts, and TF/RF aggregation protocol?
-
Functional disentanglement (Table R3). The intervention directions and probe readouts appear closely related by construction. To what extent does the observed selectivity reflect geometric separation between the extracted directions rather than functional disentanglement? What evidence supports the latter interpretation?
These points require clarification to properly assess the validity and implications of the additional experiments.
Response to Reviewer k3kw (1/2)
We thank Reviewer k3kw for the constructive review and for recognizing our target/reference role direction as a novel mechanistic component. Following your suggestions, we added bidirectional mediation, unseen-syntax transfer, role–location cross-interventions, and accuracy-decoupled consistency metrics.
Comparing to prior work (W1, Q1). Our study relates to Kang et al. [1], Cui et al. [2] and concurrent work by Salazar et al. [3], but is not a collection of existing observations. As discussed in §§1–2, we address fundamental gap that none of these studies can answer, i.e., knowing where objects are located is insufficient for relative spatial reasoning without knowing which object is the target and which is the reference. Below, we clarify our specific mechanistic contributions:
Causal mediation via path-blocking (W2, Q1). Source patching only shows that query-side location information and the answer change together, not that one mediates the other. Following the reviewer's suggestion, we performed bidirectional conditional activation patching: we keep the source patch and restore downstream query-object states to those of the unpatched run. In the VLM+image conditions, restoring the full query-object states removes 62.4–91.3% of the source-induced margin shift, compared to 2.1–26.7% for an equal number of non-query tokens. Restoring only the location-ID subspace removes 23.1–47.5%, against 1.9–4.5% for norm-matched random subspaces (Table R1). This experiment causally establishes that the source patch's location information impacts the final answer through the query object's location representation and the location-ID subspace.
Scope (W1). We use controlled synthetic scenes with 2 and 3 objects to enable counterfactual pairing and centering on attributes of objects that require orthogonal variation of object identities and locations and the absence of confounding natural backgrounds. Moreover, zero-shot steering experiments with synthetic estimated role directions on What'sUp, COCO-Spatial and GQA-Spatial (Tables 2, 10 and 11) broaden the study to real-world objects and natural scene backgrounds. These assessments concern transferability beyond the appearance of synthetic objects in pairwise directional tasks. In the Limitations section, we narrow the scope of our mechanistic conclusions to avoid claiming generalization beyond these capabilities.