- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper seeks to extend prior works in uncovering the mechanisms behind spatial reasoning in vision language models and provide a more detailed account for how spatial/positional information is processed throughout the network. In particular, they explore how the vision encoder processes background image tokens for LLM decoding. The authors show that a subset of attention heads are responsible for the spatial reasoning pathway, and implement KV token caching with this insight.
Critical strengths
-
Expanding the community's understanding on the mechanisms behind spatial reasoning in VLMs is a timely and important topic, and the finding that background tokens play an important role in spatial reasoning is useful.
-
Extending mechanistic insights to efficient KV caching is an interesting way to make mechanistic insights useful and seems effective for Qwen3-VL-8B.
Critical weaknesses
Major Weaknesses:
-
All benchmarks considered are near-toy examples with low scene complexity. Does efficient KV caching improve or retain full model performance on more complicated spatial reasoning benchmarks, like 3DSRBench? And does it hurt performance on general VQA?
-
All VLMs explored in this paper use a Qwen LLM backbone (even InternVL), so it is not clear whether mechanistic findings generalize. There are many small academic sized model checkpoints from Gemma, LLaVA, LLaMA-VL, etc. Further, while the Appendix may include more than 2 models, the main text largely hinges on Qwen3-VL-8B. The paper would be stronger if the main text included a figure or table showcasing the generality of their findings across model families. Right now, it is more of a case study for Qwen.
-
On one hand, the background tokens' role in spatial reasoning feels insightful. But on the other hand, it is not particularly surprising given that the vision encoders explored in the paper do not have explicit attention sinks. As written, the paper treats background tokens as a monolith, so it is unclear whether tokens that were close to the objects happened to absorb spatially relevant information, or some select background tokens regularly act as attention sinks, etc. Without explicit understanding of why the background tokens carry spatially relevant information, the paper's findings feel surface level.
Minor weaknesses:
-
There are some AI-isms. For example, the SpinBench citation refers to the wrong paper (this could be an honest mistake), and some sentence structures are repeated (e.g. "Together, these ..." is said 6 times). I DON'T think the paper is fully AI-generated, but the style would improve if authors would do a more thorough pass to make the writing more natural. (I am NOT flagging it for AI.)
-
The highlighting in Table 2 is misleading. It looks like for Qwen's data-specific run, whether the heads had access to Bg or Tg, the accuracy remained unchanged at 98.28. So both entries should be highlighted, not just one.
-
Stylistic note: Figure 1 is not actually discussed until page 5 yet appears on page 2, which made the early sections confusing to read.
Questions for authors
- Why were foreground tokens capped at 8? Was some ablation performed to ascertain this was an adequate hyperparameter for general tasks, beyond the benchmarks evaluated in the paper?