- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work aims to uncover the mechanisms driving spatial reasoning in VLMs, and address why a model might derive accurate spatial relationships in its latent space but fail to accurately answer questions about it.
Critical strengths
-
Identifying the underlying mechanisms behind spatial reasoning in Vision Language Models is a timely and important problem.
-
The related works exploration is thorough, and background on the line of research is well covered.
Critical weaknesses
-
I have some novelty concerns with this manuscript. The introduction states that the paper contains 3 novel contributions. Contribution 1, the identification of the encoding-decision gap, was already stated and explored in at least two papers cited by this submission: "Linear Mechanisms for Spatiotemporal Reasoning in VLMs" [1] and "Knowing Isn’t Always Saying: When Do Spatial Encodings Reach Answers in Vision-Language Models?" [2]. ([2] is recent, but if the work was concurrent, it should clearly be acknowledged as such.) Contribution 2 mainly just restates Contribution 1, with a small mechanistic explanation addition. Therefore, half of the novel contributions of this work (as stated in the introduction) appear to not be novel, and the authors seem to be aware of the preexisting works that tackle this.
-
It also appears that Contribution 2 and 3 contradict each other. Contribution 2 states that much of the causal mechanism behind spatial reasoning lies outside of the "relation-defined spatial subspace". However, the recovery method in S3.4 relies on these latent space directions for steering. So it seems like the submission is arguing that the latent space steering is a poor method for control, but all results use that exact approach to steer output.
-
The paper claims that a few text latents form the causal bottleneck behind the encoding-decision gap, but doesn't show that this is a particular problem for wrong answers, while not occurring for correct answers. Without such evidence, it appears that this is just an emergent characteristic of such models, and does not serve as the explanation behind the gap.
Minor Weakness:
- Fig 4 and 5b has overlapping text. Also, Fig 6 appears corrupted somehow?
[1] https://arxiv.org/abs/2601.12626 [2] https://arxiv.org/pdf/2608.22916
Questions for authors
-
The proposed method for increasing spatial QA accuracy is upper bounded by readout accuracy, and seems to loop until readout-level performance is reached. Why wouldn't one just use the readout as the answer?
-
Were different data splits chosen to analyze adequate steering layers vs. performing the steering, or were they the same?
2 comments
W3: Interpretation of the causal bottleneck.
We thank the reviewer for raising this concern. We apologize if our original description caused confusion. We do not claim that the causal bottleneck is specific to incorrect predictions or that the encoding–decision gap is the cause of prediction errors.
Our encoding–decision gap describes a mechanistic phenomenon: spatial representations and decision-related signals are not fully aligned, and spatial information undergoes changes as it contributes to the final decision. Our goal is to understand this process and the internal mechanisms involved, not to attribute incorrect predictions to the gap itself.
Our causal bottleneck analysis addresses another part of this process. Through causal interventions and Block/Keep experiments, we identify a small number of downstream text states that play important roles in transmitting the influence of spatial information to the final decision. This pathway can exist in both correct and incorrect predictions.
Prediction errors may have different causes, including problems at the initial stage. Therefore, the existence of an encoding–decision gap or a causal bottleneck does not by itself explain why a particular prediction is incorrect. We will clarify these distinctions and add the relevant analysis of initial-stage errors to the revised manuscript.
Q1: Why not use the intermediate readout directly?
We thank the reviewer for this question. Our goal is to understand how spatial representations influence the model's final generation, not to develop a method for improving spatial QA accuracy.
A direct readout extracts an answer from intermediate representations, but skips the subsequent decision process that we want to study. Our steering experiments examine whether interventions on these representations can change the model's own generation.
The readout tells us what spatial information is available in the model, and steering helps us test how this information affects its final output. We will make this distinction clearer in the revised manuscript.
Q2: Selection of intervention layers.
We thank the reviewer for this question. We selected the intervention layers based on our layer-wise mechanistic analysis, not by searching for the layers that achieved the highest accuracy.
We conducted this analysis across multiple datasets and found that the same model consistently showed the same important layers, regardless of the dataset used. In other words, even if we select the intervention layers using a different dataset, we arrive at the same choice. This suggests that the selected layers reflect a consistent property of the model's internal mechanisms and are not specific to the evaluation dataset.
We will clarify the layer-selection procedure and report this cross-dataset consistency in the revised manuscript.
Thanks for pointing the clarity issue in the introduction, particularly in distinguishing our contributions from recent related work. We thank the reviewer for the constructive feedback and address each concern below.
W1: Novelty and distinction from prior work.
We agree that previous studies have made important progress in understanding spatial representations and their causal roles in VLMs. We also acknowledge that the gap between correct intermediate spatial representations and incorrect final predictions is not a new observation. Our work builds on these findings and takes a closer look at the internal mechanisms that connect spatial encoding to final decisions.
Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models investigates how spatial information is represented and used in VLMs. It identifies spatial IDs associated with object-word representations and explains their role in spatial predictions through a linear mechanism.
Knowing Isn't Always Saying focuses on when and where spatial information reaches the final answer. It examines how this process depends on layers, token positions, and prompt formats, and uses causal interventions to study the conditions under which spatial representations affect model outputs.
Our work asks a different set of questions. We want to understand what happens inside the model between spatial encoding and the final decision, and which internal states are responsible for carrying the causal influence of spatial information. Although previous studies have shown that spatial representations can influence model outputs, this does not fully explain how spatial information contributes to decision-related signals or how its influence passes through downstream states.
We investigate these questions through two mechanistic analyses. First, we study the spatial-to-decision transition by comparing intermediate image-conditioned spatial representations with late decision-related signals. This allows us to examine how the role of spatial information changes across different stages of the model, beyond simply observing the gap between intermediate readouts and final predictions. Second, we investigate the internal causal pathway through which spatial information affects the final decision. Through targeted interventions and Block/Keep experiments, we identify the downstream text states that mediate the effects of upstream spatial interventions.
These two analyses address different parts of the mechanism. The first examines the relationship between spatial representations and decision-related signals. The second identifies the downstream states through which spatial information exerts its causal influence. We also use steering to test whether the identified spatial representations can influence the model's own generation, which provides functional validation of our mechanistic findings.
We will revise the introduction and contribution statements to explain these differences more clearly.
W2: Spatial subspace and decision-related signals.
We thank the reviewer for raising this concern. We believe there may be a misunderstanding of our findings. We do not claim that latent space steering is ineffective or that spatial representations have little causal influence on model outputs. Our steering experiments actually demonstrate that interventions on spatial representations can change the model's final generation.
Our subspace analysis examines the relationship between spatial representations and decision-related signals. We find that the decision-related gradients are not strongly aligned with the spatial subspace. This distinction is part of the encoding–decision gap we investigate: the spatial information encoded within the model is not necessarily reflected in the same way in its decision-related signals.
This does not contradict our steering results. The gradient overlap measures the local sensitivity of a specific decision-related signal, whereas steering tests the effect of spatial interventions on the model's actual generation. The two experiments examine different aspects of the model's internal mechanisms, and a low overlap does not mean that spatial interventions cannot affect the final output.