acceptodds
Under review as a conference paper at ICLR 2027

The Dual Mechanisms of Spatial Variable Binding in Vision–Language Models

11 views

Abstract

Many multimodal tasks, such as image captioning and visual question answering, require vision–language models (VLMs) to bind objects with their properties and spatial relations. Yet it remains unclear where and how such associations are computed within VLMs. In this work, we show that VLMs rely on two concurrent mechanisms to represent spatial variable binding. In the language model backbone, intermediate layers represent content-independent spatial relations on top of visual tokens corresponding to objects. However, this mechanism plays only a secondary role in shaping model predictions. Instead, the dominant source of spatial information originates in the vision encoder, whose representations encode the layout of objects and are directly exploited by the language model backbone. Notably, this spatial signal is distributed globally across visual tokens, extending beyond object regions into surrounding background areas. We validate the generalization of our findings to complex natural images from the COCO dataset, where globally amplifying the vision-derived spatial representations across all image tokens corrects spatial variable binding failures across models of various sizes. Together, our results clarify how spatial variable binding is computed within VLMs and highlight the central role of vision encoders in enabling it.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, mechanistic interpretability, spatial reasoning, variable binding, vision encoders, activation patching, multimodal models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1vjFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates how vision–language models obtain the spatial ordering information needed to associate objects with their properties. The main idea is to combine linear probing with activation interventions that transfer representations between examples. The authors conduct experiments to identify spatial information distributed across visual object and background tokens, as well as ordering representations formed within the language backbone. They then use probe-derived directions to modify natural-image representations and test whether these interventions repair spatial-reasoning errors. These analyses partly explain how visual and language components contribute to spatial variable binding and how the identified representations can influence model behavior.

Critical strengths
  1. Interesting problem setup. I think this setting is indeed worth a deeper study as our understanding regarding the proposed issue (spatial variable binding) was mostly treating it as a blackbox problem. In this light this study has its worthwhile points.
  2. The anaylsis in my opionin is of sufficient depth.
Critical weaknesses
  1. The main conclusion overgeneralizes across architectures. The paper presents the vision encoder as the primary source of ordering information. However, Appendix C.3 attributes Pixtral’s ordering signal largely to the language backbone. Hence, in my opinion the findings support architecture-dependent mechanisms but not a consistent hierarchy. I encourage the authors to reflect this distinction in the abstract and main results.
  2. The interventions do not fully isolate spatial information. Strip-level patching changes both the number of replaced tokens and their spatial and contextual information. Following this, the behavioral changes cannot be attributed specifically to spatial ordering, which somewhat weakens the paper’s central mechanistic interpretation. I believe the authors should add equal-token-count controls and interventions restricted to the identified spatial component.
  3. The headline repair gains overstate net accuracy improvements. The main results use per-example best coefficients and retest only previous failures, yielding an upper bound on repair performance. However, in contrast full-set evaluation with a fixed coefficient improves Qwen from 60.0% to 70.9%, but Gemma only from 52.7% to 53.6%. Given this I encourage the authors to foreground full-set results with coefficients selected on a separate validation set, accounting for both repaired and newly introduced errors.
Questions for authors
  1. How should the central account of vision-encoder dominance be revised to accommodate Pixtral’s different mechanism?
  2. Do the intervention effects persist with equal-token-count controls and patching restricted to the identified spatial component?

These are mainly derived from weakness, please refer to weakness for full info.

Reviewer_Q5KyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study proposes that spatial variable binding in VLMs incorporates a primary ordering signal from visual representations with a secondary supporting signal formed in the language backbone. The authors localize image regions and model layers involved in spatial information flow with causal mech interp tools, report spatial information in background tokens, and introduce a probe-based intervention that corrects some spatial prediction errors on natural images.

Critical strengths
  • The paper studies an important question: where spatial binding information comes from in VLMs and how the vision encoder and language backbone contribute to it.

  • Combining probing with counterfactual activation patching provides useful evidence for causal localization. The comparison between object-region and strip patching also motivates looking beyond object regions when studying spatial information.

  • The experiments cover five models and different levels of visual complexity, from controlled synthetic scenes to naturalistic images. This gives the study sufficient experimental breadth.

  • The appendix includes comprehensive evaluations and helpful controls.

Critical weaknesses
  • Probing does not cleanly isolate ordinal binding. Fixed locations are a confounding factor: ordinal labels completely overlap with absolute coordinates, allowing probes to use positional embeddings as a shortcut. The shifted-layout and causal controls in Appendix C.6 help, but the main synthetic probing and patching experiments still use fixed, unshifted layouts. Therefore, these controls do not fully resolve the confounder in the main experiments or the interpretation of background strips as globally distributed ordinal binding.

  • The general claims and causal conclusions need more precise wording. The experiments mainly focus on localization across model components, layers, and spatial regions, so the claims should emphasize causal localization instead of broadly describing the mechanism. Conclusions from causal experiments also should be precise. For example, when transferring a single-layer residual state does not change the prediction, this does not show that the patched tokens have no causal role. Perhaps transferring embeddings of multiple layers could change the output of the patched context. The authors should distinguish sufficiency vs necessity more clearly.

  • Vision-encoder dominance is overstated across models and settings. Appendix C.3 identifies Pixtral as relying mainly on ordering information constructed in the language backbone, and Appendix C.6.2 reports an exception for Gemma's vertical arrangements. Could the authors extend the analysis to Qwen3.5-9B and InternVL3.5-14B? Gemma 4 12B could also test the architectural assumption, since it lacks a conventional vision encoder.

  • The steering method and headline results need clarification. It is unclear how probe directions are normalized and combined, particularly when different directions are applied together, or why this strengthens ordinal information. The exact update and controls for alternative explanations are needed. Steering with coefficients in [1, 15] also raises a concern about an out-of-distribution activation regime, which could weaken the mechanistic interpretation. Reporting perturbation norms relative to normal activations would help assess this risk.

  • Section 5.1 has limited independent novelty and is less directly related to the paper's central message. Its pointer/selector pattern follows prior lookback and retrieval research [1, 2] and VLM position-ID research [3]. It is a useful prerequisite, but should be positioned as such, with the main novelty assessed through Sections 5.2–5.3.

[1] Prakash et al., "Language Models Use Lookbacks to Track Beliefs", ICLR 2026.

[2] Sharma et al., "LLMs Process Lists With General Filter Heads", ICLR 2026.

[3] Assouel et al., "Visual Symbolic Mechanisms: Emergent Symbol Processing in Vision Language Models", ICLR 2026.

Questions for authors

1- Could you explain the activation-patching protocol for the visual-token interventions in Figure 5? Up to which token are the clean and counterfactual forward passes run, and are KV caches used? After transferring the target visual embeddings, do you continue the forward pass from the patched layer and token, or continue from the prompt? Is the clean question processed again using the patched visual states? If the transferred visual tokens precede the question under causal attention, what role does the counterfactual question shown in Figure 5 play? Does it affect any transferred states, or is it only for illustration?

2- Could you clarify which probes are used in Section 5.4 and how their directions are normalized, combined, and applied? You state that all visual tokens and probe directions are modified without needing test-time object locations. Does every token receive the same combined update, or are directions selected based on the image, question, or region? Is any additional supervision used to select relevant probes for particular regions at test time?

Reviewer_Ei7oOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes spatial variable binding in VLMs. For instance, when asked “The color of the square to the left of the green square is ___”, the process of assigning the color feature to the square involves variable binding. Prior work shows that LLMs bind variables using content-independent ordering representations, and that VLMs also use ordering information, but not where the ordering information originates from. This paper asks if the ordering information originates from the vision encoder, the LLM backbone, or their interaction.

This paper studies 2D spatial ordinal relations (left/right and above/below) in synthetic settings (Squares, Shapes, Objects) and real world images from What’sUp. They test five open weight VLMs: Qwen2-VL-2B/7B, Qwen3-VL-8B, Gemma-3-4b, and Pixtral-12B. They use linear probes and interchange interventions, where the activations for a scene are swapped with the activations from different counterfactual inputs). The authors can then look at the changes in the VLM outputs given the interchanged interventions.

The authors find that the ordering information originates from two sources: a primary signal in the vision encoder that is distributed across visual tokens (including background tokens), and a secondary signal originating from intermediate layers in the LLM backbone on the object tokens. The methods and experiments that support this are:

  1. Final token position patching: one layer at a time, the authors swap the actual image’s final token residual streams with activations from a counterfactual scene which has different square colors and a different queried location. In this interchange intervention, the authors look at swapping different layers and find that the counterfactual’s position occurs earlier for Qwen (layers 20-22) whereas the counterfactual’s color changes in all later layers. That is, swapping the activations at layer 20-22 would result in the actual image/question being flipped to the counterfactual square’s ordinal position and swapping the activations after layer 22 flips the answer to the counterfactual square’s color. Swapping the earlier layers does not affect the final answer.
  2. Linear probes: the authors then train linear probes to predict the ordinal position of each object in the image. The authors find that linear probes trained on object tokens can decode the ordinal position nearly perfectly and that they also generalize to background image tokens
  3. Strip vs object patching: the authors swap the visual token embeddings of left and right squares in the original image with the right and left squares (respectively) from a counterfactual image while preserving the colors. Eg (red, green, blue) -> (blue, green, red). When the tokens corresponding to the actual square in the images are swapped (object patching), the output does not reliably change. However, when the tokens corresponding to the background strip which includes the actual square are swapped, then the output does change to the counterfactual side.
  4. LLM backbone swapping: to test if the LLM backbone is also generating its own ordering information, the authors remove the ordering information from vision embeddings by replacing each squares’ embedding with the embedding of the same colored square from the middle of the image. Then the authors also change the background embeddings to that of an empty image. The authors find that ordinal decoding accuracy drops but remains above chance. However, when replacing only the tokens corresponding to the square, the output only changes when swapping in the intermediate layers.

The contributions are:

  1. The authors find that there are two sources of ordering information for spatial variable binding with the vision encoder as the primary source.
  2. The authors show that the vision-derived ordering information is distributed across background tokens instead of just in the square-relevant tokens.
  3. Show generalization to natural images on COCO spatial.
Critical strengths

I think the last token position patching was interesting - my assumption is that given a similar enough image, you can just swap patches and strips of the image without affecting the attention too much. It is interesting to find that ordering information originates earlier than color attribution. In general, I am a fan of counterfactual experiments as they provide clean causal evidence.

Critical weaknesses

On the other hand, I was quite curious to see if the strip ordinal decoding was actually reflecting absolute position information. The appendix C.6.2 suggests that for Gemma-3-4B, the signal is largely absolute. In the shifted-layout test (Fig 38), Gemma seems to mostly give the answer predicted by absolute position rather than ordinal rank (0.57/0.61 vs 0.28/0.18). This suggests that Gemma’s vision encoder signal is largely absolute coordinates from SigLIP’s absolute position embeddings, not necessarily an ordinal representation.

Questions for authors
  1. I wonder how generalizable these results are. In the main text it seems most results are mainly about Qwen2-VL-2B/7B and Gemma-3-4B models. In the appendix A.1, it seems Pixtral prediction is quite low (0.55).
  2. I think discussing the absolute vs relative position more in the main text would be helpful as it is only mentioned once (Section 5.3.2). If the vision embeddings carry the absolute position, the LLM backbone can answer left/right by comparing the absolute positions rather than using ordinal positions. Could the authors repeat the last token position patching of Sec. 5.1 using both original and counterfactual images from different shifted layouts? The pairs should be chosen so that the counterfactual’s target is the overlapping square. This would show whether layers 20-22 carry ordinal or absolute position.

Comments on the paper

3 comments

Dear Reviewers,

Although the initial review scores for this submission are somewhat negative, it is important that we thoroughly discuss our perspectives before reaching a final recommendation.

Now that the authors' rebuttal is available, I kindly ask each of you to respond in the discussion and clarify whether your original concerns have been addressed or still remain.

Thank you in advance for your timely engagement before the discussion period ends.

Best, Your AC

Thank you for running the shifted-layout final token experiment and adding the two models. My main concerns have been addressed. Since my main concerns have been addressed, I am raising my score to 3.

Thank you for taking the time to reconsider our submission and for suggesting the shifted-layout experiment. We're glad that the new results and revisions addressed your concerns, and we appreciate your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.