acceptodds
Withdrawn from ICLR 2027

Look Once: A mid-depth commitment window explains Spatial Instability in VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision–language models (VLMs) perform inconsistently in spatial reasoning where some lateral relations are near ceiling, while real-world height and orientation remain at or below chance. Their predictions also change under geometric transformations that preserve the correct response. What remains unclear is where the decision forms and whether richer visual inputs, spatial post-training, and explicit reasoning alter this process or improve its reliability. We combine residual-state patching across token regions and depth with an answer-preserving scene reversal that jointly reverses the spatial relation and option order, changing the scene while keeping the correct answer slot fixed. This separates scene-dependent information transfer from an already resolved verdict. Across three architectures, we identify a narrow commitment window where option-token states become verdict-like as causal influence from the vision block declines. Restoring early visual states produces substantial scene-specific transfer before this boundary but little afterward, supporting reduced downstream use rather than content loss alone. Moreover, 88-98% of the most influential attention heads and MLPs lie downstream of commitment. Spatial post-training preserves the broad relay organization, while explicit reasoning improves accuracy by gaining 4.1%, yet prediction flips under answer-preserving reflection increase from 12.3% to 15.0%. These findings shift the design target from supplying more visual evidence to controlling how that becomes a verdict, the commitment window marks where spatial decisions take shape, while most influential downstream computation operates on the verdict already formed.

Primary area:
interpretability and explainable AI
Keywords:
Vision Language Models, Interpretability, Spatial Understanding, Attention, Causal Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_1pYHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when VLMs turn visual information into spatial decisions. The authors use activation patching to trace how information moves from image tokens to option tokens and eventually to the final answer. Their main intervention reverses both the scene and the option order while keeping the correct answer unchanged. By comparing this intervention with scene-only and order-only changes, they identify a mid-depth commitment window where option representations become verdict-like and visual tokens lose their influence on the answer.

The authors find similar patterns across three VLM architectures and spatially fine-tuned variants. They also show that explicit reasoning can improve accuracy without making spatial predictions more stable.

Critical strengths
  1. Well-designed counterfactual intervention. The answer-preserving reversal separates changes in visual content from changes in the correct answer. Together with the additional controls, it provides convincing evidence that option representations become verdict-like at intermediate layers.

  2. Careful mechanistic analysis. The paper uses early-state restoration to test whether later layers stop using visual information or whether that information has disappeared. The results support reduced downstream use of visual tokens.

  3. Broad experimental evaluation. The paper reproduces the main findings across different architectures and examines how spatial fine-tuning and explicit reasoning affect the process. The orientation results also show that visual handoff and commitment do not always coincide.

Critical weaknesses
  1. The paper does not establish whether commitment explains spatial instability. The experiments show when option representations become verdict-like, but they do not show why spatial predictions become unstable. In particular, the authors do not test whether earlier commitment leads to greater instability or whether changing the commitment process makes predictions more robust. Without this connection, the results characterize how models form decisions rather than explain why those decisions are unstable. This distinction matters because spatial instability motivates the paper's central claim.

  2. The contribution needs clearer positioning against prior work. Previous studies have shown that VLMs transfer visual information to text representations in early and intermediate layers and rely less on image tokens afterward (e.g., Kaduri et al., 2024; Basu et al., 2024; Zhang et al., 2025). This paper goes further by distinguishing visual content from the resulting decision. The authors should make this distinction central to their novelty claim and discuss the closest prior work more thoroughly.

  3. The component analysis offers limited independent evidence for the commitment window. The authors report that 88–98% of influential components lie after commitment. However, knockout effects generally increase with depth, so many influential components would fall after any boundary near the middle of the network. The additional analysis in Appendix E.3 finds no distinct change in mediation at the commitment window after accounting for depth. These results are compatible with the authors' interpretation, but they do not provide independent evidence that commitment marks a distinct change in downstream computation.

  4. The reasoning results need stronger statistical evidence. Table 4 shows modest improvements in accuracy alongside increases in prediction flips. However, the authors provide neither confidence intervals nor paired statistical tests for these differences. They should also report the number of eligible examples in the KV-cache experiments. These details are necessary to judge how reliably reasoning affects spatial stability.

  5. The manuscript needs further editing and clarification. Table 2 reports 320 examples for InternVL2.5, compared with 446 for other models, without explaining the difference. Table 1 refers to three spatial relations, while Section 4.2 refers to four, and the paper does not define "Mirror error." The authors also report the direct-to-total ratio differently in the abstract (2–3%) and Section 4.3 (1.6–3.1%). Table 4's caption has a grammatical error ("is includes"). Appendix B contains an unfinished editorial note ("must be pasted from the released scripts before submission"), and the Reflection-V system prompt is missing. Several experimental details also need clarification. Several grammatical errors, awkward sentences, and inconsistent numerical formats (e.g., 0.848 vs .848) make the paper harder to read. A careful revision would improve both clarity and reproducibility.

Questions for authors
  1. For the answer-preserving donor, recovery is normalized by the image-only denominator even though the expected effect is zero. How many items have |LD_image − LD_ref| near the exclusion threshold, and does the 0.03 median change if normalized by the order-only denominator instead?
  2. Is the drop in R(O, L; both) at the window driven by option states no longer reading vision, or by the option states themselves changing representation? Patching only the vision→option attention edges at the window would separate these. Reflection-V describes the swapped scene in its trace but answers from the prior verdict. Is the final-answer preference at the end of generation still readable from the option-caption states, i.e., does the verdict-like state persist through the trace?
  3. On orientation, commitment lags handoff. Does recovery from the vision block show a second, later peak on these items, or is the gap filled by text-side computation alone?
  4. Please report the per-relation breakdown of the answer-preserving flip rates for the reasoning models; the aggregate hides whether the added instability is concentrated in orientation.
Reviewer_xnqKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how spatial evidence reaches binary-choice answers in VLMs. Its central intervention jointly reverses a scene and the option order, preserving the correct answer letter. Residual-state patching identifies a transition at which option-state effects become invariant to this joint reversal while visual-position effects decline. The paper compares several architectures, spatially specialized checkpoints, and reasoning models, and finds that improved accuracy or renewed visual access does not necessarily improve transformation stability.

Critical strengths
  1. The control design that preserves answers unchanged is interesting, and the comparison across three base models further enhances the credibility of the results.
  2. I appreciate the relational word correspondence in the appendix, the hierarchical scanning, and the controls used to restore early visual states. These experiments make the paper more compelling than mere observation of attention.
Critical weaknesses
  1. My primary concern is whether the experiment truly demonstrates that the model has already selected the answer (Section 2.2, Fig. 3). For instance, an intermediate state could simultaneously retain “the cup is to the left of the book” and “Option A describes the left side,” yet defer comparison of the two statements until later layers. If both the image and the options are jointly flipped, such a model would still output the same letter and reproduce the results reported in the paper. Thus, co-locating the information required for answering does not equate to having already made the selection. Prior work has documented information transfer across intermediate layers and the emergence of spatial representations and answers at distinct layers [1][2][3]; the novelty of this paper therefore hinges significantly on whether it can further substantiate that “the answer has already been formed.”

  2. The title claims that this window explains spatial instability, yet the two experimental components remain disconnected (Table 1, Section 4.1–4.5). The paper does not show how the window differs between questions whose answers flip due to image transformations and those whose answers remain stable—within the same model. Moreover, the high accuracy and stability observed for on/under judgments suggest that this process may also occur during normal answering.

  3. The recovery of early visual state requires question-wise re-computation (Appendix D, Table 9). The authors clarify that the net effect reported in the table arises from subtracting aggregated values across two groups, rather than first computing the difference per question. For example, values of 0.269 and 0.271 being close does not imply that the two substitutions exert similar effects on each individual question; similarly, identical medians across groups do not preclude highly divergent per-question outcomes.

[1] Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models

[2] The Dual Mechanisms of Spatial Reasoning in Vision–Language Models

[3] Cross-modal Information Flow in Multimodal Large Language Models

Questions for authors

Please see weaknesses.

Reviewer_Y8ysOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors start by showing that models that achieve high spatial accuracy on different datasets can still exhibit sensitivity to spatial perturbations, implying their fragility at these tasks. To understand this, the authors conduct patching-based experiments, claiming to find a commitment window at the model’s middle layers where it already commits to an answer. They then apply the same analysis to existing spatially post-trained and reasoning models, finding that the commitment window persists in these models as well.

Critical strengths

The paper tackles a difficult question, which is to understand how information propagates through the model, and does so with interesting and complex experiments. First, the answer-preserving donor is a good idea, enabling the separation between different hypotheses of what information the option states carry. Second, conducting the same experiments on existing spatially post-trained and reasoning variants of the same backbones is an interesting and not often-conducted evaluation.

Critical weaknesses

In general, it seems to me that the central claims the paper makes just aren’t supported by the results. The evidence is consistent with the authors’ claims, but it does not distinguish them from simpler alternative explanations.

  • No link between the identified mechanism and the fragility: in the paper, the authors present the models’ fragility when evaluated with simple perturbations. They then conduct experiments trying to uncover how information propagates through the model, concluding that the image information ceases to be evaluated at some middle layer. They then conclude that insufficient processing of the image is therefore to blame for the models’ fragilities, and that perhaps addressing this issue would improve their capabilities. But no experiment tests this link. Half of a model’s layers could be entirely sufficient for making the correct spatial calculations, or the entire model’s layers could be insufficient. An information bottleneck in the model isn’t necessarily related to the problem at hand, or is even a problem at all.
  • The patching experiments don’t convincingly back up the claims made by the authors: while the patching experiments conducted by the authors are informative and well crafted, their results don’t necessarily imply the conclusions made by the authors. For example, the headline result of the answer-preserving patching experiment implies that the hidden states at the options section contain information sufficient for arriving at the answer. That information does not, to the best of my understanding, necessarily contain the final answer itself. The hidden states could conversely contain the option + sufficient scene information to determine the answer. The paper doesn’t conduct experiments that rule out this interpretation, and therefore cannot rule out that relevant processing continues after the “commitment window”. Similarly, the relay route shown in figure 2 additionally relies mainly on “timing”, looking at where patching effects spike at different token positions. This experiment doesn’t preclude additional routes (e.g., from the vision block directly to the instruction span or answer position), which could be elucidated through attention knockouts or path patching techniques.
Questions for authors
  • How would you distinguish between the two competing interpretations I presented for the answer-preserving patching experiment? Can you conduct this?
  • How can you show the link between the “commitment window” and the models’ fragilities?
Reviewer_18R3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors ask at which layer a VLM turns the image into its answer on two-choice spatial questions from What'sUp and 3DSRBench (e.g., "A. apple on armchair / B. apple under armchair"). Their method copies internal states from a modified input (the "donor") into the original run and checks whether the answer changes. The key donor flips both the photo (apple under instead of on) and the order of the two captions, so the correct letter stays the same. The authors report that copying states from this donor stops having any effect at a mid-depth layer, and that the image tokens no longer influence the answer from about the same layer. They interpret this as the point where the caption tokens come to hold only the answer, and call it the "commitment window". They find the same pattern in Qwen2.5-VL, InternVL2.5, Qwen3-VL and their spatially post-trained and reasoning variants. They also report that 16–48% of predictions change under small crops, and that reasoning models gain accuracy but flip more.

Critical strengths
  • The key donor, which flips both the photo and the caption order, is a clever way to tell whether a state carries the scene or only the answer.

  • Interesting side findings: DeepStack carries about 3% of the scene effect, and Reflection-V's trace describes a swapped image while its answer ignores it.

  • The analysis is careful, replicated across three architectures and their variants, and the code is released.

Critical weaknesses

While the paper has merit (see above) I find that it is still immature. The key donor idea has the potential of leading the authors towards explanations and fixes, rather than purely observations.

  1. The title's claim is not tested. The mechanism is studied on high-accuracy What'sUp items with strong image dependence. The predictions that flip under crops are never patched, so the window describes how decisions form, not why they are unstable. No fix is proposed.

  2. Is the window more than the end of image-to-option transfer? The handoff and the option conversion coincide within a fraction of a layer (Table 8), and staged image-to-text flow is already known (Zhang et al., 2025). The new element is the photo-and-caption-flip donor. "Commitment" holds only within prefill: Qwen3-VL-Thinking follows a swapped vision cache 90.9% of the time.

  3. Don't you think that the component results is predetermined? Components are knocked out only at the answer token, which receives information last, so early layers there have little to act on. The 88–98% should also be compared with the 44–64% of components that lie after the window anyway, and knockouts should be repeated at the caption tokens.

  4. The instability mentioned in the title is thinly documented.

    • The abstract names two problems, uneven accuracy across relations and flips under transformations, but only flips are measured.
    • Table 1 gives pooled flip rates only, with no breakdown by relation or shift size, and no examples.
    • The What'sUp crops are not validated.
    • There is no non-spatial perturbation baseline, and the margins of flipped items are not reported, although the code computes them.
    • There are no confidence intervals.
    • Logit-scored and generated answers are compared directly.
  5. Scope. The tasks are binary forced choice on a few hundred items, with 7–8B models only. The "verdict at the option tokens" depends on this format.

  6. Presentation. The text is dense and ungrammatical in places. I had to read the abstract multiple times to understand it. There are various glitches here and there.

Questions for authors
  1. At what depth does a flip arise when patching between shifted and unshifted runs of flipped items?
  2. Does an intervention at the window, e.g., averaging aligned visual states from shifted views, reduce flips more when applied before it than after it, or more than ordinary test-time augmentation?
  3. What do knockouts at the caption tokens show, and how does the 88–98% compare with the share of components that lie after the window anyway?
  4. What are the flip rates by relation and shift size, under non-spatial perturbations, and the margins of flipped vs. stable items?
  5. How many items are excluded by the thresholds on the whole-image effect, and are they disproportionately unstable?
  6. Does the window appear with free-form answers or more than two options?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.