acceptodds
Under review as a conference paper at ICLR 2027

Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning

Abstract

High instance-level accuracy can mask inconsistencies in spatial reasoning when objects exchange positions or their roles are reversed in the query. The internal representations supporting relative-position reasoning remain poorly understood. We investigate two complementary components of this process: tracking object locations in the input and representing their query roles. Across three VLMs with visual or textual inputs and their language-model backbones, activation patching reveals a staged progression from early-layer source representations through intermediate-layer query-object representations to late-layer answer states. Targeted interventions further establish causal links along this progression: manipulating source-side representations shifts location information at query-object mentions and ultimately alters relation predictions. Beyond object-location information, we also identify a stable query-side direction associated with the roles of the two objects in the comparison. Steering along directions estimated on synthetic scenes generalizes to natural-image benchmarks, improving accuracy and both forms of paired consistency in most settings without retraining. Our findings reveal complementary components of relational reasoning across visual and textual settings and show how targeted interventions can improve the consistency of models' behavior.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, spatial reasoning, vision-language models, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_k3kwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the internal mechanisms of relative-position reasoning in vision-language models (VLMs) and language models (LLMs). The authors first introduce two paired tests, object-location swapping and target/reference role reversal, to evaluate spatial consistency. They then use activation patching to trace spatial information from source tokens through query tokens to the final answer. Location IDs are extracted and causally tested through targeted interventions.

The paper further identifies a target/reference role direction by contrasting hidden states of the same object under different query roles. Destructive interventions and activation steering are used to examine its causal effects. Experiments across multiple models and input modalities show that role-direction steering can improve accuracy and relational consistency on natural-image benchmarks without updating model parameters.

Critical strengths

1. Systematic experimental design with causal interventions.

  • The paper combines two controlled perturbations with activation patching and targeted steering to examine different components of relative-position reasoning.
  • The interventions provide stronger causal evidence than linear probing alone. Evaluations across multiple model architectures and input modalities further support the consistency of the observations.

2. Identification and transfer of role-related representations.

  • The paper identifies a relatively stable target/reference role direction and demonstrates its causal relevance through destructive interventions.
  • Role directions extracted from synthetic scenes improve relational consistency on several natural-image benchmarks without model retraining, suggesting potential practical value.
Critical weaknesses

1. Limited novelty beyond existing mechanistic studies of spatial reasoning.

Several central findings substantially overlap with prior work:

  • Spatial representations: Kang et al. [1] already identified object-token spatial IDs and demonstrated their causal effects through interventions.
  • Spatial information localization: Cui et al. [2] investigated spatial information in visual representations and complementary mechanisms in the language backbone.
  • Information flow: Salazar et al. [3] already identified visual-to-query-to-answer information pathways using causal patching, including their flexibility under interventions.

Consequently, the location-ID analysis and layer-wise information-flow results largely extend existing findings rather than establish new mechanisms.

The main novel component is the target/reference role direction. However, its extraction relies on standard hidden-state contrasts, and its computational function remains insufficiently characterized. Moreover, the mechanistic experiments focus primarily on simple two- and three-object layouts and basic directional relations, limiting the generality of the conclusions.

2. Insufficient evidence for the proposed mechanistic interpretation.

Several gaps remain between the intervention results and the claimed mechanisms:

  • Causal mediation: Source patching changes both query-side location information and final predictions, but this does not establish query tokens as necessary causal mediators. Alternative information pathways remain possible.
  • Role disentanglement: The extracted role direction may also capture syntactic or contextual differences between queries. The two query templates do not fully eliminate these confounds. The paper also lacks independent cross-interventions on location and role representations to establish their functional separability.
  • Consistency evaluation: Paired consistency requires both answers to be correct. Its changes may therefore partly reflect changes in ordinary accuracy rather than improvements in role binding itself.

These limitations weaken the evidence that the proposed representations constitute distinct computational mechanisms underlying relative-position reasoning.

Questions for authors

1. What new causal mechanism does the source-to-query analysis establish beyond Salazar et al. [3]?

Could the authors use path blocking or conditional activation patching to determine whether query-side location representations are necessary mediators of source-to-answer information flow, rather than one of several possible pathways?

2. Does the extracted role direction represent target/reference binding independently of linguistic context and spatial location?

Could the authors evaluate unseen query templates and independently intervene on location and role representations to demonstrate their functional separability?

3. Do the improvements in paired consistency specifically reflect better role binding?

Since paired consistency depends on both answers being correct, could the authors report relation-flip consistency independently of correctness, or evaluate reversal accuracy conditioned on the original answer being correct? This would help distinguish improved role consistency from general accuracy gains.

References

[1] Kang, R., Chen, H., Gkioxari, G., & Perona, P. (2026). Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models. International Conference on Learning Representations (ICLR 2026). https://arxiv.org/abs/2601.12626

[2] Cui, K., Prakash, N., Raina, A., Bau, D., Torralba, A., & Rott Shaham, T. (2026). The Dual Mechanisms of Spatial Reasoning in Vision–Language Models. The First Workshop on Efficient Spatial Reasoning. https://openreview.net/forum?id=Ueb38N1aF3

[3] Salazar, I., Frank, S., Oneata, D., Elliott, D., & Fierro, C. (2026). Pathways of Visual Information Flow in Vision-Language Models. arXiv preprint arXiv:2607.03358. https://arxiv.org/abs/2607.03358

5 comments
AuthorsOpenReview ↗

Response to Reviewer k3kw (1/2)

We thank Reviewer k3kw for the constructive review and for recognizing our target/reference role direction as a novel mechanistic component. Following your suggestions, we added bidirectional mediation, unseen-syntax transfer, role–location cross-interventions, and accuracy-decoupled consistency metrics.

Comparing to prior work (W1, Q1). Our study relates to Kang et al. [1], Cui et al. [2] and concurrent work by Salazar et al. [3], but is not a collection of existing observations. As discussed in §§1–2, we address fundamental gap that none of these studies can answer, i.e., knowing where objects are located is insufficient for relative spatial reasoning without knowing which object is the target and which is the reference. Below, we clarify our specific mechanistic contributions:

  • Direct causal link between source-side states and query-side location IDs (§5, Table 1, Global response, Table R1): Kang et al. analyze spatial IDs only at query-object tokens, Cui et al. [2] localize spatial signals in the visual and language backbones, and Salazar et al. patch three broad groups: all image tokens, all query-text tokens and the final token. We intervene on semantically defined groups on both source and query sides: object bounding-box and row/column-strip tokens for visual sources, object and location spans for textual sources, and target/reference mentions and relation-option words in the query. In the source-to-query chain-patching experiment (Table 1,Global response, Table R1), patching corrupted source states into a clean run shifts the query-side relational projection score along the spatial axis and flips the final prediction.
  • Role binding vs. spatial location (§§3–4, §6). Prior studies evaluate location representations under fixed roles. By comparing activation-patching patterns under object-location swaps and target/reference reversals, we characterize the layer-wise contributions of source-to-query location transfer and query-side role representations (Figure 3), and we identify a stable query-side role direction (§6).
  • Comparing visual and textual spatial representations across VLMs and LLM Backbones: In [1, 2, 3] the scene input is always an image. Note that the "textual activations" of [1] and the "language backbone" of [2] are LM-side states under image input. By evaluating matched VLM+image, VLM+text and LLM+text conditions, we are the first to show that geometry-preserving Location IDs emerge even from purely linguistic descriptions in both VLMs and text-only LLM backbones (Figure 4, Figure 13).

Causal mediation via path-blocking (W2, Q1). Source patching only shows that query-side location information and the answer change together, not that one mediates the other. Following the reviewer's suggestion, we performed bidirectional conditional activation patching: we keep the source patch and restore downstream query-object states to those of the unpatched run. In the VLM+image conditions, restoring the full query-object states removes 62.4–91.3% of the source-induced margin shift, compared to 2.1–26.7% for an equal number of non-query tokens. Restoring only the location-ID subspace removes 23.1–47.5%, against 1.9–4.5% for norm-matched random subspaces (Table R1). This experiment causally establishes that the source patch's location information impacts the final answer through the query object's location representation and the location-ID subspace.

Scope (W1). We use controlled synthetic scenes with 2 and 3 objects to enable counterfactual pairing and centering on attributes of objects that require orthogonal variation of object identities and locations and the absence of confounding natural backgrounds. Moreover, zero-shot steering experiments with synthetic estimated role directions on What'sUp, COCO-Spatial and GQA-Spatial (Tables 2, 10 and 11) broaden the study to real-world objects and natural scene backgrounds. These assessments concern transferability beyond the appearance of synthetic objects in pairwise directional tasks. In the Limitations section, we narrow the scope of our mechanistic conclusions to avoid claiming generalization beyond these capabilities.

AuthorsOpenReview ↗

Response to Reviewer k3kw (2/2)

Role disentanglement from syntax and location (W2, Q2). We use a mean-contrast estimator to test whether target/reference roles correspond to a causally relevant and stable direction. We jointly contrast the same object in matched role-reversal questions, and our target-first and reference-first templates mitigate confounds with object identity and mention order. The held-out alignment in Figure 6 supports the stability of these joint contrasts. Additionally, destructive interventions reduce correct-versus-reversed answer margins much more than norm-matched orthogonal controls for both mention orders (Figures 7 and 15), providing evidence for a causal contribution to role-sensitive prediction.

To verify that the extracted role direction represents target/reference binding independently of linguistic context and spatial location, we conducted two new sets of experiments (Global Response, Tables R2–R3):

  • Transfer to unseen query templates (Table R2a/b): Keeping role vectors, layers and strengths fixed, we tested four unseen templates on 128 held-out scene families. These templates differ in clause structure, token position and grammatical roles in both target-first and reference-first orders. Role steering has a larger flipped-answer margin shift than norm-matched random controls and both orientations of the surface mention-order direction across all 9 model and input settings.
  • Independent cross-interventions on role and location (Table R3): Separately intervening in role and location on matched examples and token spans in LLaVA-1.6/Mistral, we report in Table R3 that in all six settings each intervention alters its probe readout (role→role 0.35–0.91; location→location 0.39–0.64) much more than the other readout (role→location 0.017–0.056; location→role 0.008–0.012). In addition, each intervention moves the answer margin over matched random controls in 5 of 6 settings and thus they do so in common for 4 of 6.

These results support role information distinguishable from mention order, tested syntax and location IDs, with causal effects on answers. Since roles are inherently tied to query syntax, we do not claim complete independence from it.

Consistency ‌evaluation ‌(W2, ‌Q3). We agree that both-correct consistency alone cannot distinguish role binding. Using the predictions stored in Table 2, we report in Table R4 relation-flip agreement (whether the model predicts inverse relations, i.e. relations that can be reversed in object order, regardless of whether they are correct) and reversal accuracy conditioned on the original answer being correct (and therefore provides less conditioning than correctness alone), as well as the same quantity on the fixed subset of examples for which the original answer was correct at baseline (so that the same examples are compared before and after steering). We see that relation-flip agreement improves in 8/9 cases and reversal accuracy improves in 7/9 cases on the run's own conditioning set and in 8/9 cases on the fixed subset. The 95% bootstrap confidence intervals on the change (paired where appropriate) do not contain zero in 6/9 cases for relation-flip agreement and 5/9 for each of the two reversal accuracy metrics, and all of these changes are improvements. These analyses show that role reversal consistency is improved in most settings by steering, more than both-correct consistency alone can reveal.

Table R4. Consistency metrics that do not require both answers to be correct (baseline → steered). Bold marks changes whose paired 95% bootstrap confidence interval excludes zero.

ModelConditionRelation-flip agreement (%)Rev. acc. | own original correctRev. acc. | baseline-correct (fixed)
LLaVA-1.6VLM+image76.7 → 79.7 (+3.1)254/333 → 266/334 (+3.4)254/333 → 266/333 (+3.6)
LLaVA-1.6VLM+text71.8 → 79.7 (+7.8)213/265 → 224/265 (+4.2)213/265 → 217/265 (+1.5)
LLaVA-1.6LLM+text49.2 → 53.3 (+4.2)139/229 → 158/235 (+6.5)139/229 → 152/229 (+5.7)
InternVL3.5VLM+image83.6 → 83.3 (−0.3)291/346 → 290/346 (−0.3)291/346 → 290/346 (−0.3)
InternVL3.5VLM+text85.3 → 86.4 (+1.1)288/306 → 295/314 (−0.2)288/306 → 291/306 (+1.0)
InternVL3.5LLM+text52.2 → 54.2 (+1.9)169/238 → 181/246 (+2.6)169/238 → 175/238 (+2.5)
PixtralVLM+image71.3 → 74.5 (+3.2)282/346 → 295/355 (+1.6)282/346 → 288/346 (+1.7)
PixtralVLM+text61.8 → 67.6 (+5.9)236/313 → 268/337 (+4.1)236/313 → 247/313 (+3.5)
PixtralLLM+text51.0 → 60.3 (+9.3)204/301 → 240/313 (+8.9)204/301 → 227/301 (+7.6)

Thank you for the response and additional experiments. I have two remaining concerns regarding the new results:

  1. Evaluation consistency (Table R4). The reported success counts, combined with the paired-consistency scores in Table 2, suggest total sample counts of either 360 or 408 across different settings, although Table 2 is described as evaluating What'sUp-A. Could the authors clarify the evaluation subsets, sample counts, and TF/RF aggregation protocol?

  2. Functional disentanglement (Table R3). The intervention directions and probe readouts appear closely related by construction. To what extent does the observed selectivity reflect geometric separation between the extracted directions rather than functional disentanglement? What evidence supports the latter interpretation?

These points require clarification to properly assess the validity and implications of the additional experiments.

Reviewer_wWdDOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the internal mechanisms of spatial relational reasoning in VLMs, focusing on object-location tracking and query-role representations. Through activation patching and targeted interventions, the authors identify a staged progression of spatial information across model layers and establish causal links between these representations and relation predictions. They further identify a query-role direction that enables training-free steering, improving spatial reasoning accuracy and consistency across synthetic and natural-image settings.

Critical strengths
  • The paper introduces a synthetic dataset designed to reveal limitations in the relative-position reasoning capabilities of existing VLMs.

  • The paper provides comprehensive experimental evaluations across multiple benchmarks, offering valuable insights into the internal mechanisms of spatial relational reasoning.

Critical weaknesses
  1. Unclear motivation for text-only experiments. Since the paper primarily investigates VLMs, the motivation for studying whether location representations also emerge from text-only inputs in both VLMs and their LLM backbones (Line 87) could be better explained. Some VLMs may exhibit different behavior or degraded performance without visual inputs, potentially complicating the interpretation of these comparisons.

  2. Insufficient analysis of the underlying failure mechanisms. While the paper identifies object-location tracking and query-role assignment as two key components, it remains unclear what causes failures in these processes, such as language-model biases, hallucinations, or attention drift. Further discussion of these underlying mechanisms and how activation patching helps identify or address them would strengthen the analysis.

  3. Limited explanation of activation patching. The paper would benefit from a clearer and more detailed description of how activation patching is implemented, including the construction of interventions and the selection of activation states. This would make the methodology easier to understand and reproduce.

  4. Insufficient dataset details in the main text. As the paper focuses on analyzing limitations in existing VLMs, the synthetic dataset is an important component of its contribution. Including representative examples, construction procedures, and dataset statistics in the main text would help readers better understand the identified reasoning failures.

  5. Limited evaluation on frontier VLMs. The evaluation could be extended to more recent and larger-scale VLMs, including proprietary models such as GPT, Gemini, and Claude, as well as larger open-source models such as Qwen3-VL-32B. Such experiments would help determine whether the identified limitations persist in more capable models.

Questions for authors

Please refer to the weaknesses for details.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wWdD

We thank Reviewer wWdD for feedback on clarity, presentation and benchmarking coverage.

First, we would like to clarify the central objective of our study: to characterize how object-location information and target/reference roles are represented and used in relative-position reasoning, and to examine their causal contributions across visual and textual input conditions.

Motivation for the text-only experiments (W1). We conduct matched VLM+text and LLM+text conditions not to examine whether VLMs fail without images, but to address a fundamental mechanistic question that previous image-only studies[1, 2, 3] cannot answer: do geometry-preserving Location IDs and role-binding representations also emerge from purely linguistic descriptions in the LM backbone? Comparing VLM+image, VLM+text and LLM+text allows us to isolate modality-specific encoding from backbone-level relational computation. This comparison indeed reveals that continuous Location-ID geometry is present even under purely textual scene descriptions in both VLMs and text-only LLMs (Figs. 4, 13). We discussed this mechanistic motivation in §1 and §3.

Failure mechanisms (W2). We appreciate the reviewer's interest in the origins of spatial-reasoning failures. Our study investigates object-location information and target/reference roles as components of the computation underlying relative-position judgments. Controlled location swaps and role reversals vary spatial evidence and query roles separately, while activation patching tests which internal states causally influence the answer.

Additional experiments support this analysis: bidirectional conditional patching shows that downstream query-object states, including Location-ID components, contribute to the source-induced answer effect (Global Response, Table R1); frozen role directions transfer to unseen query phrasings (Table R2); and separate role/location interventions selectively affect their respective readouts in LLaVA/Mistral (Table R3).

Diagnosing errors due to “language-model biases, hallucinations or attention drift” is a related but distinct research question. Our interventions do not identify these phenomena as the origins of the observed errors. We will clarify the research focus and be more explicit about how our analysis tests the contributions of spatial and role information to the computation.

Activation-patching implementation (W3). The core procedure is specified in the paper but spread across the main text and appendices. Specifically, §3 describes task definition and counterfactual construction, with image and text examples of query-fixed location swaps and source-fixed role reversals. §4 specifies residual-stream interventions, semantic token groups, pair screening, patching operations and metrics. For localization, we cache paired runs and replace corrupt-run residual states with clean-run states at a specified layer. We will strengthen references to Table 7 (Appendix C), listing token groups, and Appendix F, detailing PyTorch hooks during prefill, the intervention operation, precision and answer scoring. We will also clarify description-token groups, screening counts and inclusion criteria, and distinguish clean-to-corrupt localization (§4) from corrupt-to-clean source-to-query patching (§5).

Dataset statistics (W4). In the original submission, figs. 1A–B already show representative image/text examples and both paired transformations; §3 describes task and counterfactual construction; Figure 8 gives further examples; Appendix A details rendering, description generation and train/val/test statistics (Table 3); Appendix B (Table 5) reports behavioral evaluation counts.We agree a concise main-text summary would help. We will add a “Synthetic data” paragraph to §3 specifying the generation process.

Evaluation on frontier VLMs (W5). We added behavioral evaluations of Qwen3-VL-32B-Instruct (Qwen3-32B backbone) and GPT-6.1-Sol (Table R5).

Table R5. Additional behavioral evaluations.

ModelConditionSynthetic Acc.Rev.Swap.What'sUp Acc.Rev.Swap.
Qwen3-VL-32B-InstructVLM+image100.00100.00100.0099.2498.7598.75
Qwen3-VL-32B-InstructVLM+text98.2896.5696.6292.2584.7784.51
Qwen3-32BLLM+text91.3082.6082.4188.3476.6876.81
GPT-6.1-Solimage100.00100.00100.0099.4499.3199.17
GPT-6.1-Soltext100.00100.00100.00100.00100.00100.00

Our central contribution is a mechanistic analysis of how spatial evidence and target/reference roles support relative-position reasoning using controlled interventions. We analyze three open 7–12B models from different families across the three input conditions. Larger models require substantially more computation and activation storage, while closed-model APIs lack the internal activation access needed for these interventions. These new evaluations complement our mechanistic work by extending behavioral coverage.

Reviewer_wWdDOpenReview ↗

Official Comment by wWdD

I thank the authors for their detailed rebuttal and experiments, which address most of my concerns.

However, my concern regarding the failure mechanisms is only partially addressed. While the additional activation patching experiments provide stronger evidence for the causal contributions of location and role representations, they do not fully explain why these components fail during spatial reasoning, and I believe they are important limitation.

Given this detailed rebuttal and unsolved issue, I'd like to raise to 2.

We thank the reviewer for the continued discussion and for acknowledging the additional causal evidence.

We agree that understanding why models sometimes produce incorrect or inconsistent predictions remains an important open question. However, we would like to distinguish identifying the computational components that support relative-position reasoning from explaining why these components sometimes fail to support correct predictions. Our findings provide causal evidence for the former: activation patching and targeted interventions demonstrate that object-location and target/reference role representations influence relation predictions. Moreover, steering the role representations improves accuracy and relational consistency, linking the identified mechanisms to naturally occurring errors. Understanding the underlying causes of these failures remains an important direction for future research.

As already acknowledged in the Limitations section (p. 10) of the original submission, our activation-patching analyses primarily characterize correct predictions, and attributing individual errors to specific components is left for future work. We will make this distinction clearer in the revised submission.

Reviewer_LxE1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the internal mechanisms underlying relative-position reasoning in vision-language models (VLMs) and their language-model backbones, focusing on object-location representations and target/reference role binding. The authors introduce two paired consistency tests, object-location swaps and target/reference role reversals, and combine activation patching, spatial-direction extraction, and activation interventions to analyze the computational process from scene inputs to query representations and final answers. Experiments cover three input conditions, multiple models, and both synthetic and natural-image benchmarks. The results suggest that source-side location information influences query-side representations and that role-direction interventions improve accuracy and paired consistency in most settings.

Critical strengths
  1. The paper separates object-location representations from target/reference role binding and investigates both through activation patching and targeted interventions. Compared with evaluations based solely on accuracy, this framework provides a more informative perspective on the internal processes underlying relative-position reasoning.

  2. The authors compare VLM+image, VLM+text, and LLM+text settings and evaluate the transfer of role-direction steering to benchmarks such as What'sUp and COCO-Spatial. The improvements in consistency across most settings demonstrate the empirical potential of the proposed intervention.

Critical weaknesses
  1. The linear decodability of location directions, prediction changes induced by activation patching, and effects of role-direction interventions support the causal relevance of the associated representations, but do not establish that the model follows the specific staged computational mechanism proposed by the authors. In particular, the role direction may encode token-position information, sentence structure, or other query-related factors. The current controls do not fully rule out these confounds.

  2. The mechanistic analyses primarily cover three models in the 7B–12B parameter range and rely on fixed-grid scenes and templated queries. Improvements on natural images are not uniform across all metrics and settings; for example, Table 2 reports a 1.2-percentage-point decrease in accuracy for LLaVA under VLM+text. The current results therefore do not establish that the method generalizes broadly to more complex or open-ended spatial reasoning tasks. The incremental contribution over existing work on spatial representations also needs to be more clearly established.

Questions for authors

Source-side activation patching may introduce out-of-distribution representations. Can the authors provide more direct mediation analyses or bidirectional interventions to verify that source-side location information affects the final answer through query-side location representations?

Can the authors report the absolute sample sizes for the key results, along with additional evaluations on more complex spatial layouts or open-ended queries, to better characterize the method's applicability and limitations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer LxE1

We thank reviewer LxE1 for their constructive feedback and for recognizing our joint analysis of object-location and role representations, cross-modal comparisons and natural-image steering results. We address the remaining points below.

Staged mechanism and causal mediation (W1, Q1). We propose a computational staged mechanism (Fig1D). Additional bidirectional conditional activation patching on 216 held-out three-object scenes provide support for this hypothesis. With fixed source patching, restoring downstream query-object states removes 62.4–91.3% of source-induced answer-margin shift in VLM+image, versus 2.1–26.7% for matched non-query controls. Restoring only the location-ID subspace removes 23.1–47.5%, versus 1.9–4.5% for random subspaces (Table R1). These results strengthen evidence that source location information causally impacts the final answer representation via query-object location representations.

Role-related confounds (W1). To examine whether role directions are confounded by surface token-position and sentence structure factors, we tested frozen role directions on four unseen query templates (Table R2a), varying clause structure, phrasing and token positions, for both target-first and reference-first mention orders. For instance, the target is not the grammatical subject in U2/U4. On average, role interventions produce larger answer-margin changes than matched random controls and both mention-order controls in all 9 settings (Table R2b). Separate role and location interventions in LLaVA/Mistral selectively affect their respective readouts (Table R3). These findings support functional role information beyond template-specific syntax, mention order and location information. Since target/reference roles are inherently tied to query syntax, it’s difficult to fully disentangle role information from syntax. We therefore claim that the role directions encode target/reference role information separable from object attributes, mention order, tested syntactic variations and the location-ID information, and causally affect answers.

Natural-Image Consistency (W2). We recognized that benefits of steering vary across settings. For LLaVA VLM+text result, the 1.23% accuracy decrease is not statistically significant (paired 95% CI: [−2.80, +0.36%]), while target/reference reversal consistency improves by 2.7%. We preserve the negative accuracy result and note that better paired consistency does not imply better question-level accuracy. As noted in §6, natural-image steering benefits are condition-specific.

Incremental Contribution (W2). We address a major gap: encoding object locations is not enough for relative spatial reasoning without binding objects to target and reference roles. We distinguished our contributions from Kang et al. [1], Cui et al. [2] and concurrent work by Salazar et al. [3] in the original submission, and clarify the differences here:

  • Role binding vs. spatial location (§§3–4, §6): Previous work examines location representations under fixed roles. By using location swaps with target/reference reversals, we isolate a stable query-side role direction that is functionally dissociable from location IDs (Table R3), transfers to unseen query syntax (Table R2), and improves natural-image consistency without retraining (Tables 2, 10, 11).

  • Source-to-query causal mediation (§5): Previous studies analyze query-word spatial IDs (Kang et al.), global visual/backbone signals (Cui et al.), or coarse query-group flows (Salazar et al.) separately. We connect source and query sites and show through source-to-query patching (Table 1) and bidirectional path-blocking (Table R1) that source information contributes to downstream query-object location representations and final answers.

  • Visual vs. purely linguistic inputs: Previous studies measure LM states only under image inputs. With matched VLM+image, VLM+text and LLM+text conditions, we show geometry-preserving location IDs also appear in purely textual scene descriptions in both VLMs and text-only LLMs (Figs. 4, 13).

Generalization to complex/open-ended scenes (W2, Q2). Our goal is to characterize how location information and role information support relative-position judgments. Controlled scenes allow targeted interventions, while zero-shot steering on natural images tests transfer beyond synthetic scenes. As stated in Limitations, generalization to complex or open-ended scenes remains future work. We hope this work provides a starting point for broader mechanistic investigations of spatial reasoning.

Absolute sample sizes (Q2). Absolute sample sizes are reported in Appendix A (Table 3) and Appendix B (Table 5). Image, scene family, query and paired evaluation counts are reported in Table 5 for Figure 2 and Table 6. Annotation and image counts for COCO-Spatial and GQA-Spatial are reported in Appendix E. We will strengthen cross-references and add experiment-specific mechanistic counts in updated PDF.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response (2/2): Role Generalization and Role–Location Cross-Interventions

2. Frozen role directions on unseen query templates

To examine whether the role direction transfers beyond the query wording/syntax used in the original TF and RF templates, we evaluated frozen role directions (keeping direction vectors, layers, and intervention strengths fixed) on four unseen query templates across 128 held-out evaluation scene families.

Table R2a. Unseen query templates (for vertical queries, above/below replaces left/right).

Unseen templateMention orderHorizontal query template
U1Target firstWhere is T with respect to R: left or right?
U2Target firstDetermine the position of T relative to R. Choose left or right.
U3Reference firstTaking R as the reference point, is T to the left or right?
U4Reference firstWith R as the reference, the position of T is left or right.

We measure the additional incorrect-minus-correct answer-margin shift from role steering relative to controls. A positive contrast indicates a greater shift toward the incorrect answer. Role − random mean uses five norm-matched random interventions; Role − order (+) and Role − order (−) use the mention-order direction and its opposite. Positive values for both order contrasts indicate that the role intervention has a larger effect on this margin than either orientation of the tested mention-order control.

Table R2b. Additional flipped-answer margin shift from role steering vs. controls (averaged over U1–U4).

Model familyInputRole − random meanRole − order (+)Role − order (−)
LLaVAVLM+image0.4630.3860.505
LLaVAVLM+text0.2990.2610.327
LLaVALLM+text2.8572.9323.126
InternVL3.5VLM+image0.9540.7431.016
InternVL3.5VLM+text2.1511.7762.236
InternVL3.5LLM+text2.8652.8882.914
PixtralVLM+image2.3341.9452.474
PixtralVLM+text1.9141.8511.894
PixtralLLM+text0.6760.6550.626

These results support the out-of-distribution transfer of role-intervention effects: the mean margin shift exceeds the random and both mention-order controls across all four unseen query templates.

3. Separate cross-interventions on location and role

To test whether target/reference roles and spatial locations are functionally distinguishable, we conducted additional cross-intervention experiments in LLaVA-1.6 (and its Mistral LLM backbone) across 2-object and 3-object scenes (128 held-out families per dataset across all six templates: TF, RF, U1–U4). On matched examples and token spans, we separately perturbed either the Role component (subtracting/adding the role direction to target/reference tokens) or the Location component (shifting each queried object along the difference vector toward the other's location prototype).

In A → B, A is the intervention and B the probed variable. Role → role and Loc. → loc. measure signed changes toward the intended labels; Role → loc. and Loc. → role measure absolute cross-attribute changes. Margin vs. random measures the additional change in the reversed-minus-correct answer-score margin relative to matched random controls. Δ reversed-answer rate measures changes in the generated reversed-answer frequency from baseline for role/location interventions, respectively.

For the six displayed settings in Table R3, each intervention produces larger normalized changes in its intended probe readout and smaller changes in the other probe readout. Role and location interventions each produce positive margin contrasts against matched random controls in 5/6 settings, with both supported in the same 4/6 settings. Generated reversed-answer rates also increase with positive confidence intervals in 5/6 settings for each intervention, with both supported in the same 4/6 settings.

Table R3. Selective effects of role and location interventions in LLaVA-1.6 / Mistral.

ConditionObjectsRole → roleRole → loc.Loc. → loc.Loc. → roleMargin vs. random: roleMargin vs. random: loc.Δ reversed-answer rate (pp): role / loc.
VLM + image20.410.0190.540.010+0.26+0.35+3.3 / +3.4
VLM + image30.410.0260.420.008+0.24+0.22+4.7 / +4.5
LLM + text20.350.0170.640.011+0.04+0.36−0.2 / +0.7
LLM + text30.910.0560.390.009+2.37+0.28+8.4 / +0.7
VLM + text20.610.0350.450.012+0.03+0.0007+0.6 / −0.3
VLM + text30.580.0290.420.009+0.17+0.20+5.6 / +5.6

These experiments assess the selectivity and behavioral effects of separately applied role and location steerings. In our experiments, role steering selectively affected role-probe predictions with comparatively small changes in location-probe predictions, and location steering did the opposite. Both interventions moved answer margins away from matched random samples and increased the rate of generated reversed answers in 4/6 cases. Our findings demonstrate selective effects on role and location probe readouts and show that steering along either direction can influence final answers.

AuthorsOpenReview ↗

Global Response (1/2): Contributions, New Experiments Overview and Causal Mediation

We sincerely thank the reviewers for their constructive and insightful comments, which has helped us improve the paper. We are encouraged that the reviewers recognized the strengths of our work, including (1) the distinction between object-location representations and target/reference role binding (LxE1); (2) the systematic experimental design combining controlled perturbations with causal interventions (k3kw); (3) the identification of causally relevant role-related representations and their transfer from synthetic to natural-image benchmarks (LxE1, k3kw); and (4) the comparison across visual and textual input conditions (LxE1) and comprehensive evaluations across multiple benchmarks (wWdD).

Novelty and contributions. Our original submission already situates our work relative to Kang et al. [1] and Cui et al. [2] in §1 and §5, and further discusses these studies with concurrent work by Salazar et al. [3] in §2. In particular, we highlight three contributions beyond prior studies: (1) tracing how source-side spatial information causally influences object-specific query-side location representations and final predictions (§5); (2) identifying and causally testing target/reference role representations that complement object-location information (§6); and (3) extending the analysis of location-ID geometry to purely textual scene descriptions in both VLMs and their LLM backbones (§3-6). We will make these differences more explicit in §§1-2, and expand our comparisons to related work, in particular our object-specific causal analysis and the functional contribution of role binding. For detailed comparisons with specific works, see our responses to reviewers LxE1 and k3kw.

New experiments. Following the reviewers’ feedback, we added three mechanistic experiments to strengthen our argument:

  1. Mediation analysis (Table R1): We patch source states in both directions and restore the downstream query-object states to the unpatched run. This substantially attenuates source-induced answer changes, with selective effects from restoring their location-ID components.
  2. Unseen query templates (Table R2): Frozen role directions retain effects beyond random and mention-order controls across four new query templates and nine model/input settings.
  3. Role–location cross-interventions (Table R3): Separate interventions selectively affect their respective representations and influence final answer preferences in LLaVA/Mistral.

We additionally evaluated correctness-independent reversal consistency (Table R4) and extended behavioral benchmarking to larger models (Table R5). We will incorporate these analyses and clarifications into the submission PDF, with detailed results and experimental procedures provided there.

1. Bidirectional conditional activation patching (path-blocking)

To directly test whether source-induced answer changes are causally mediated by downstream query-object states and their Location-ID subspace (while avoiding unidirectional OOD artifacts), we ran bidirectional conditional patching (clean corrupt, 432 directed cases across 216 held-out 3-object scenes). Keeping the source patch at layer l, we restore downstream states (l'>l) to unperturbed baseline values. Restoring full query-object spans blocks 62.4–91.3% of the source-patching effect in VLM+image (vs. 2.1–26.7% for equal-length non-query controls), and restoring only the low-dimensional Location-ID subspace blocks 23.1–47.5% (vs. 1.9–4.5% for norm-matched random subspaces).

Table R1. Percentage of the source-patching effect removed by downstream restoration.

Model and inputSource LayerSource effectQuery-object spans (%)Location-ID subspace (%)Random control (%)Non-query control (%)
LLaVA / VLM+image84.67762.447.54.119.4
LLaVA / VLM+text80.37292.246.2-0.216.8
LLaVA / LLM+text81.44296.2103.2-1.818.5
InternVL3.5 / VLM+image917.41669.423.11.926.7
InternVL3.5 / VLM+text90.095142.4150.0-13.9-5.0
InternVL3.5 / LLM+text90.46589.029.52.96.2
Pixtral / VLM+image1018.54891.346.74.52.1
Pixtral/ VLM+text100.84194.278.60.1-1.9
Pixtral / LLM+text100.20385.770.9-0.711.9

These results provide causal evidence that query-object states and their location-ID components contribute to transmitting the source-patching effect to the final answer. We will update these new experimental details and results in section 5 in our submitted pdf.

Reference:

[1] Kang et al. (2026). Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models. ICLR 2026. https://arxiv.org/abs/2601.12626

[2] Cui et al. (2026). The Dual Mechanisms of Spatial Reasoning in Vision–Language Models. Workshop on Efficient Spatial Reasoning. https://openreview.net/forum?id=Ueb38N1aF3

[3] Salazar et al. (2026). Pathways of Visual Information Flow in Vision-Language Models. arXiv:2607.03358. https://arxiv.org/abs/2607.03358

AuthorsOpenReview ↗

End-of-Discussion Summary

We ‌thank ‌the ‌reviewers for their constructive feedback to help us improve the paper. We especially appreciate the recognition of our systematic causal interventions, cross-modal comparisons and identification of target/reference role representations.

Our main finding was a mechanistic account of how object-location information and role/mention representations of target and reference roles support relative position reasoning: (1) we traced the causal effect of source-side spatial evidence on object-specific query representations and final predictions; (2) we discovered and causally intervened on role-related representations that provide information complementary to location; (3) we compared these mechanisms for visual and textual inputs, showing that role direction steering estimated on synthetic scenes can improve consistency on natural image benchmarks without retraining.

In the rebuttal and discussion, we addressed all substantial reviewer concerns with extensive additional experiments, analyses and clarifications. The main new evidence is summarized below.

Reviewer concernAdditional experimentsKey findings
Causal mediation(LxE1 W1/Q1; k3kw W2/Q1)Bidirectional path-blocking on 216 held-out three-object scenes (Table R1)In VLM+image, restoring query-object states removes 62.4–91.3% of source-induced effects (non-query controls: 2.1–26.7%). Restoring only Location-ID components removes 23.1–47.5% (random controls: 1.9–4.5%).
Role generalization beyond syntax and mention order(LxE1 W1; k3kw W2/Q2)Frozen role directions on four unseen query templates, evaluated on 128 held-out scene families (Table R2)Role steering produces larger answer-margin shifts than random and both mention-order controls across all 9 model/input settings.
Functional differentiation of role and location(k3kw W2/Q2 and follow-up)Independent cross-interventions (Table R3), joint steering, and non-spatial size comparisonsOn-target readout effects (0.35–0.91) substantially exceed cross-effects (0.008–0.056). Joint steering yields a smaller reversal-rate increase than either intervention alone (+2.47 vs. +3.32/+3.39 pp). Role steering also changes size judgments more frequently than location steering (5.46% vs. 2.21%).
Consistency gains beyond accuracy(k3kw W2/Q3)Correctness-independent relation-flip agreement and reversal accuracy on fixed baseline-correct subsets (Table R4)Relation-flip agreement improves in 8/9 settings (significant in 6/9); reversal accuracy on the fixed subset improves in 8/9 settings(significant in 5/9).
Evaluation on more capable models (wWdD W5)Behavioral benchmarking of Qwen3-VL-32B-Instruct and GPT-6.1-Sol (Table R5)Extends the behavioral evaluation to larger and proprietary models.

Further clarifications. We further clarified the motivation for text-only comparisons, experimental procedures, dataset statistics, evaluation protocols and distinction from previous work. We also clarified that our interventions diagnose causally relevant components, and explaining why these components fail generalizes remains future work as noted in the original Limitations section (p. 10).

These additions greatly strengthen the causal evidence for the complementary roles of object-location information and target/reference role representations in relative-position reasoning. We thank the reviewers for their engagement and hope our new results and clarifications help the final evaluation of our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.