acceptodds
Withdrawn from ICLR 2027

A Mechanistic View of Spatial Reasoning in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Spatial reasoning is an emerging capability of Vision Language Models (VLMs). As VLMs see broader deployment in real-world settings requiring reliability and interpretability, understanding spatial reasoning mechanisms is critical. Through a mechanistic view, we find evidence that spatial reasoning emerges from a coordinated division of labor in VLMs defined by vision encoder (VE) components that produce tokens consumed by large language model (LLM) components. Specifically, we show that spatial reasoning in VLMs depends strongly on positional encodings in the VE and on interactions between background tokens, while the LLM selectively integrates this information from background image tokens and foreground image tokens through a sparse set of functionally specialized attention heads. Despite our probing task’s focus on queried objects, we find that background tokens also play a critical role by providing spatial context that is actively leveraged by the model during spatial reasoning. Based on these findings, we show that mechanistic interpretability can guide visual token compression. The performance on real-world datasets and diverse spatial tasks suggests that the discovered pathways extend beyond the original analysis setting, enabling more efficient visual token processing without sacrificing reasoning performance. Overall, our results improve understanding of the mechanisms underlying spatial reasoning in VLMs, and we demonstrate the resulting insights for inference efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Mechanistic Interpretability, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_T3SqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper seeks to extend prior works in uncovering the mechanisms behind spatial reasoning in vision language models and provide a more detailed account for how spatial/positional information is processed throughout the network. In particular, they explore how the vision encoder processes background image tokens for LLM decoding. The authors show that a subset of attention heads are responsible for the spatial reasoning pathway, and implement KV token caching with this insight.

Critical strengths
  • Expanding the community's understanding on the mechanisms behind spatial reasoning in VLMs is a timely and important topic, and the finding that background tokens play an important role in spatial reasoning is useful.

  • Extending mechanistic insights to efficient KV caching is an interesting way to make mechanistic insights useful and seems effective for Qwen3-VL-8B.

Critical weaknesses

Major Weaknesses:

  • All benchmarks considered are near-toy examples with low scene complexity. Does efficient KV caching improve or retain full model performance on more complicated spatial reasoning benchmarks, like 3DSRBench? And does it hurt performance on general VQA?

  • All VLMs explored in this paper use a Qwen LLM backbone (even InternVL), so it is not clear whether mechanistic findings generalize. There are many small academic sized model checkpoints from Gemma, LLaVA, LLaMA-VL, etc. Further, while the Appendix may include more than 2 models, the main text largely hinges on Qwen3-VL-8B. The paper would be stronger if the main text included a figure or table showcasing the generality of their findings across model families. Right now, it is more of a case study for Qwen.

  • On one hand, the background tokens' role in spatial reasoning feels insightful. But on the other hand, it is not particularly surprising given that the vision encoders explored in the paper do not have explicit attention sinks. As written, the paper treats background tokens as a monolith, so it is unclear whether tokens that were close to the objects happened to absorb spatially relevant information, or some select background tokens regularly act as attention sinks, etc. Without explicit understanding of why the background tokens carry spatially relevant information, the paper's findings feel surface level.

Minor weaknesses:

  • There are some AI-isms. For example, the SpinBench citation refers to the wrong paper (this could be an honest mistake), and some sentence structures are repeated (e.g. "Together, these ..." is said 6 times). I DON'T think the paper is fully AI-generated, but the style would improve if authors would do a more thorough pass to make the writing more natural. (I am NOT flagging it for AI.)

  • The highlighting in Table 2 is misleading. It looks like for Qwen's data-specific run, whether the heads had access to Bg or Tg, the accuracy remained unchanged at 98.28. So both entries should be highlighted, not just one.

  • Stylistic note: Figure 1 is not actually discussed until page 5 yet appears on page 2, which made the early sections confusing to read.

Questions for authors
  • Why were foreground tokens capped at 8? Was some ablation performed to ascertain this was an adequate hyperparameter for general tasks, beyond the benchmarks evaluated in the paper?
Reviewer_KMsTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses mechanistic interpretability to study how vision-language models (VLMs) perform spatial reasoning. It finds that spatial information is built in the vision encoder through positional encodings and interactions among background tokens, not just the queried objects. The LLM then reads this out through a sparse set of specialized attention heads (about 2–16% of heads), some tuned to background tokens and others to target-object tokens. Building on this, the authors keep full visual KV for the background heads and compress the rest, retaining only ~14–17% of visual KV entries while matching full-model accuracy and cutting decoding latency by 15–24%. Results are shown on CLEVR-Spatial, What’s Up, and SpinBench, though the analysis covers mainly Qwen-family models and InternVL3.

Critical strengths
  • The experiment described in Section 3.2, which shows that positional encodings directly affect the spatial reasoning and binding capabilities of VLMs, is insightful. To my knowledge, this has not yet been directly demonstrated for LLMs either.
  • The experiments are conducted across multiple models and datasets.
Critical weaknesses
  • The primary issue I see with this submission is the lack of novel insights. Apart from the finding described in Section 3.2, the other empirical insights are either intuitive or have largely been demonstrated in previous works.

    • Section 3.3 investigates whether spatial relations emerge through interactions between queried objects or globally within the vision encoder using attention-knockout experiments. However, the vision encoder has no information about which objects are being queried, since the queries are provided as text tokens only to the LM backbone. Therefore, the vision encoder cannot generate spatial-relation signals specifically for the queried objects. Additionally, previous work [1] has already demonstrated the importance of background tokens for spatial reasoning.
    • In Section 4, the authors identify attention heads in the LM backbone that retrieve important information from visual tokens. Both the method used to identify these heads and the finding that a sparse set of attention heads retrieves information from visual tokens have been demonstrated in previous literature [2].
    • Finally, the authors use the identified attention heads to improve computational efficiency. I do not find this particularly insightful in itself. Multiple previous works have shown that model computation can be localized to a small set of components, often referred to as a circuit, that can preserve the performance of the full model while the remaining components are ablated.
  • Lines 78–80 state that the VLM performs spatial reasoning by dividing the labor between the vision encoder and the LM backbone. This has already been demonstrated in [1]. At the very least, the authors should cite this work.

  • Line 309 states, “We probe the LLM component …”. I would suggest avoiding the word “probe,” as it refers to a specific method in mechanistic interpretability that the authors are not using. Its use here could therefore confuse readers.

  • Several works in the literature identify model components by learning a mask. The authors should cite relevant work in this area [3, 4, inter alia].

  • Lines 333–334 state that the method can “preserve performance using background or target information.” However, attention heads that attend to either background or target-object tokens can still retrieve information about both sources, since the vision encoder is bidirectional and each token most likely contains information about all visual patches. Lines 374–375 also seem to suggest the same.

  • The takeaway in Section 4 states that “target heads encode local semantic evidence tied to object identity, while background heads support information from the spatial visual context.” I do not understand how the experiments in Section 4 support this conclusion, especially given the concern raised in the previous point.

  • I found the procedure used to identify the visual-token set (F), described in Section 5.1, difficult to understand.

  • The takeaway in Section 5 mentions “path-level interventions,” a term with a specific meaning in mechanistic interpretability: intervening on the edges of the computation graph that describes the model. This does not correspond to the experiment conducted in this section. The authors should rephrase this statement.

[1] - Cui et al., "The Dual Mechanisms of Spatial Reasoning in Vision–Language Models" - https://openreview.net/forum?id=Ueb38N1aF3

[2] - Rim et al., "Visual symbolic mechanisms: Emergent symbol processing in vision language models" - https://openreview.net/forum?id=3RQ863cRbx

[3] - Davies et al., "Discovering Variable Binding Circuitry with Desiderata" - https://openreview.net/forum?id=emYjUPbx0K

[4] - Cao et al., "Sparse Interventions in Language Models with Differentiable Masking" - https://aclanthology.org/2022.blackboxnlp-1.2/

Questions for authors
  • At which token position is the attention-head mask learned?
  • To restrict information flow from visual tokens through non-selected heads, the authors perform zero ablation on the residual vector. This could have unintended effects. For instance, this would cause the key and value vectors to be zero at all token positions, rather than only at the last token, which is presumably where the authors intend to restrict information flow.
  • Please describe the procedure for identifying the visual-token set (F) in more detail.
Reviewer_1kXFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the source of emerging spatial reasoning capability in modern Vision-Language Models (VLMs). Through intervention analysis using VLMs (e.g., Qwen-3) on CLEVR-spatial and What’s Up datasets, this paper uncovers three findings: (i) positional encoding plays an important role for spatial reasoning, (ii) non-target relevant background tokens have much meaningful information for spatial perception, (iii) there exist small subsets of attention heads that specialize in reasoning about spatial relations. Together, they transform these empirical findings into a token pruning methodology that matches the performance of previous pruning approaches.

Critical strengths
  • Interesting findings. This paper discovers that blocking attention flows between background tokens surprisingly degrades accuracy on spatial reasoning. Other blocking settings (e.g., fg-bg, fg-fg) mostly preserve accuracy. These results seem counterintuitive, but interesting.

  • Findings to practical benefits. This paper exploits attention-head specialization for efficient inference of spatial reasoning. The design is well motivated by empirical evidence and shows it does not rely on dataset-specific attention-head search for pruning (CLEVR-spatial suffices for this purpose). While it does not consistently outperform comparable methods, this paper demonstrates some practical benefits of the findings.

Critical weaknesses
  • Insufficient evidence for the claimed spatial reasoning mechanisms. (i) The paper finds that blocking background-to-background attention causes larger performance drops than blocking direct interactions between target objects. However, these interventions may differ substantially in the number of affected tokens and attention connections. It is unclear whether the larger degradation reflects the importance of background interactions for spatial reasoning or simply a more disruptive intervention. (ii) This paper defines background tokens as complement patches that do not contain query-specified objects. Based on the construction procedure of CLEVR-spatial, there are 4-8 objects in each image (L184-186) with varying sizes. This may indicate that if we specify two different objects in the query, the remaining 2-6 objects naturally have the required information for determining the specified objects’ positions. To decouple this case from the observed findings, it is encouraged to examine whether the region comprised of no-object region also contains meaningful information for spatial reasoning, or to conduct similar experiments on fewer-object scenes.

  • The proposed compression method achieves promising results, but it remains unclear whether the improvements actually come from the discovered spatial heads. A key missing comparison is randomly selected heads or attention-based head selection under the same KV budget and token selection strategy. Without such controls, it is difficult to determine whether the mechanistic analysis provides meaningful guidance beyond a general non-uniform KV allocation strategy. This is particularly important because connecting mechanistic findings to practical compression is one of the paper's main contributions.

Questions for authors

I find this paper interesting to some extent. I look forward to the authors' responses to the weaknesses and questions. I'm open to increasing my score if the rebuttal adequately addresses my concerns.

  • Relation to the attention sinks/register tokens. “background token gains much higher attention or norm” is a well-known phenomenon in the field of SSL [a,b]. These papers also qualitative/quantitively demonstrates its uniqueness compared with other foreground tokens. It is suggested to add some experiments or analysis on the relationship between such phenomena.

  • Beyond Qwen3-VL-8B and InternVL3-8B. While observed findings are consistent across these models, it is suggested to investigate whether these findings generalize to larger ones (Qwen3-VL-30B-A3B) or different series of VLMs (e.g., Llama-3.2-V, LLaVA-1.5).

  • Background tokens may also contain other non-target objects. Have the authors considered separating pure background tokens (without any objects) from other object tokens? Do pure background tokens also contribute to spatial reasoning?

[a] Vision Transformers Need Registers, ICLR 2024.

[b] See What You Are Told: Visual Attention Sink in Large Multimodal Models, ICLR 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.