acceptodds
Under review as a conference paper at ICLR 2027

THE READOUT DOES NOT ROTATE: REFERENCE-FRAME BINDING FAILURE IN VISION-LANGUAGE MODELS

Abstract

Vision-language models frequently err in spatial perception when direction is defined by a reference object's heading. To investigate the cause, we fix the prompt, the target, and the camera position in controlled synthetic scenes, reverse only the reference object's heading, and examine eight open-source models stage by stage across perception, transmission, and answering. We find that both the reference-object heading and the camera information are linearly decodable and are carried to the answer token, yet the heading information is not used, and the model preferentially relies on the camera information instead. We then attempt a targeted fine-tuning remedy, but find that the model only memorizes the third-party headings it has seen and does not generalize what it has learned. Taken together, we conclude that the failure of perception in a third-party coordinate frame occurs in the mapping from representations to answers, and that fine-tuning does not repair this defect. We propose a set of metrics as a quantitative standard for this ability, which can test whether a proposed remedy genuinely establishes reference-frame binding and thereby support the evaluation of future work.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, vision-language models, spatial reasoning, frames of reference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BxX9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper uses controlled reference-heading reversals in COMFORT scenes, together with probes, activation replacement, phase analysis, and targeted fine-tuning, to argue that VLMs encode but do not read out object-centered heading information and instead rely on camera-relative cues.

Critical strengths

The paired heading-reversal design is a clear behavioral diagnostic, and the paper combines it with probes, interventions, phase measurements, and fine-tuning splits supported by substantial implementation details and controls.

Critical weaknesses
  1. The heading probe is not shown to capture an abstract heading variable. Held-out-object heading decoding is only 0.481--0.537 (Table 16), and phase scrambling preserves part of the signal. The proposed representation may therefore reflect object- or rendering-specific correlates rather than semantic object heading.

  2. The camera-versus-heading patch comparison is not semantically matched. Camera reversal changes the entire scene projection, while heading reversal mainly changes one object. Equal residual norms do not make them interventions of equal visual scope or information content, so the result establishes sensitivity to these particular patches, not a clean relative readout preference.

  3. The fine-tuning conclusion is too broad. The experiments show that the tested fixed-heading LoRA/full-tuning recipes fail to extrapolate to a reversed heading. But they do not show that fine-tuning with rotation augmentation, object-coordinate supervision, or reference-frame-aware data cannot work.

  4. The novelty is inadequately positioned. The paper should distinguish its heading-specific analysis from recent work on perspective consistency, allocentric/egocentric disambiguation, and causal transport from spatial representations to answers. Without this distinction, the remaining contribution appears incremental for ICLR.

Questions for authors
  1. Given the held-out-object heading decoding and phase-scrambling result, what evidence shows that the probed signal is semantic reference-object heading rather than object- or rendering-specific correlates? Would decoding and intervention effects transfer across objects, textures, and renderers?

  2. How can full-scene camera-reversal replacement and mainly local heading-reversal replacement be treated as matched tests of reliance on camera versus heading? Can the authors provide a semantically matched counterfactual, or limit the claim to sensitivity to these particular interventions?

  3. Which conclusion remains after distinguishing failure of the tested fixed-heading recipes from failure of fine-tuning more broadly? How does the heading-specific causal analysis differ from prior causal-transport work?

Reviewer_CEfROpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates reference-frame binding failures in vision-language models (VLMs), focusing on their inability to correctly interpret spatial relationships from an object-centered perspective. Through controlled heading-reversal experiments, representation probing, and activation interventions. The paper also examines fine-tuning strategies and introduces several metrics for evaluating reference-frame binding.

Critical strengths

1.The paper investigates an important limitation of current VLMs, particularly their ability to reason about spatial relationships under different reference frames.

2.The experiments cover multiple VLM architectures and provide detailed analyses of internal representations and model behavior.

Critical weaknesses

1.Limited methodological novelty. The paper primarily combines existing diagnostic techniques, including linear probing, activation interventions, and fine-tuning experiments, to analyze a known limitation of VLMs. Although the empirical observations are interesting, the methodological contributions appear relatively incremental. It is unclear what fundamentally new insight or technique is introduced beyond existing studies of spatial reasoning and reference-frame understanding.

2.Lack of an effective solution. While the paper identifies potential causes of reference-frame binding failures, it does not propose an effective method to address them. The explored fine-tuning strategies also fail to generalize across unseen headings. Consequently, the practical contribution remains limited, and it is unclear how the findings can be translated into concrete improvements in VLM spatial reasoning.

Questions for authors

The authors argue that the primary failure occurs during the mapping from internal representations to final answers. However, the linear decodability of heading information does not necessarily imply that the model has learned a usable representation of the required coordinate transformation. More direct causal evidence would strengthen this conclusion.

Reviewer_zHg8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies reference-frame binding failures in VLMs, specifically it targets cases where models provide wrong answers to spatial perception questions when the direction is defined by a reference object's heading. The study conducts multiple behavioral, mechanistic (probes), and causal intervention experiments, eventually finding that 1) models are much better on egocentric than allocentric vision, 2) the reference object's heading is actually decodable from the representation but carries no weight on the outcome, 3) camera-relative information is also carried forward and has a much stronger influence (including redundant representations).

Critical strengths
  • This submission includes a controlled study on reference-frame binding under heading reversal on both synthetic and real data
  • The problem is studied from behavioral, mechanistic, and causal angles and multiple models
Critical weaknesses
  • This submission takes a rather simple question incredibly hard to follow. This paper is directly within my research expertise, and I still struggle to comprehend most of it due to the lack of visualizations, textual bridges, and the excessive experiments and results. As a reader, I am constantly bombarded with results that lack context. Even the abstract was unclear until I read the introduction. Perhaps there is a great contribution buried in the 58 pages and 55 tables that I missed, but this paper feels to me like a clear example of why "less is more".
  • There is a likely confound in the binary (yes/no) experiments: models are known to have a confirmation bias, and while it seems like the samples are balanced by construction, a better way would be to test alongside logical opposites (https://arxiv.org/abs/1511.05099). In Appendix F.6 the results even show that Gemma is answering yes 99% of the time.
  • All models are limited to 8B parameter sizes. I do not want to be the reviewer that asks for more experiments on larger models, but it is simply unclear if the results are due to the rather small model size, or generally an issue in VLMs.
  • Just because the heading is decodable does not mean that it is useful information. The strongest experiment here is the transfer to unseen objects, where accuracy falls to random chance. Similarly, non-decodable by linear probes does not mean that the information is absent; it might just require more complex, non-linear transformations or indicate overfitting of the probe.
Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.