acceptodds
Under review as a conference paper at ICLR 2027

Seeing Depth but Choosing Cues in Vision-Language Models

Abstract

Do vision-language models use depth information to judge which object is closer, or rely on image-plane cues? We investigate this question through controlled depth–cue conflicts, representation analysis, and interventions. Among the many image-plane cues relevant to depth perception, we focus on apparent size and vertical position. We compare depth judgments under cue-consistent and cue-conflict conditions in both rendered scenes and real-world photographs. Across model families and scales, depth judgments systematically shift toward misleading cues. Yet the correct depth order remains decodable from the queried objects' visual representations, including when the model answers incorrectly. Image and category controls support interpreting this signal as image-dependent depth information beyond the two target cues. Strengthening the model's own depth readout through object-level write-back can correct conflict decisions without updating model parameters or using ground-truth depth annotations. Interventions show that the effect of this correction depends on the direction and target tokens of the written signal, while its strength varies across architectures and domains. Together, these findings reveal a mismatch between a model's ability to represent depth and use this information when deciding the answer. VLMs can retain useful depth information while allowing competing image-plane cues to exert greater influence on their answers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, depth perception, mechanistic interpretability, spatial understanding, spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TaZHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the susceptibility of VLMs to apparent object size and vertical position when judging the depth order of two objects. While prior work showed that VLMs rely on vertical position as a shortcut, this study systematically tests model behavior under cue conflict conditions using synthetic scenes and real images where apparent size and vertical position are independently manipulated. The authors find that even when models are misled by these cues into making incorrect judgments, correct depth relations remain linearly decodable from the target objects' visual representations. Furthermore, an intervention that amplifies this internal depth signal can correct these errors under conflict conditions without modifying model weights. However, this corrective effect is largely confined to synthetic scenes and remains limited on real photographs.

Critical strengths

A key strength of this paper is its systematic evaluation of how apparent size and vertical position bias VLM depth judgments across both controlled synthetic scenes and real images. In particular, examining cases where apparent size conflicts with true depth on real photographs meaningfully extends prior work (Min et al. 2026).

Critical weaknesses

I have serious reservations regarding how the authors treat the information decoded from internal representations as if it were a true depth signal. Monocular depth estimation is an inherently ill-posed problem, and both humans and VLMs must combine multiple depth cues to make plausible inferences. Because the probe in this study is trained on a dataset where apparent size and vertical position are decorrelated from depth and thus rendered unreliable, it naturally learns to rely on whichever other depth cues remain informative. The authors emphasize that correct depth can still be decoded when the model answers incorrectly under conflict conditions, but this is hardly surprising if the probe is simply reading out other visual cues (such as contrast or brightness). By the same logic, the accuracy gains achieved through the intervention under conflict conditions may simply reflect an artificial boost to reliance on these alternative cues, which could conversely amplify biases toward those specific cues. Indeed, the intervention yields limited improvement on real photographs and even degrades accuracy under consistent conditions. At present, there is insufficient justification for the dichotomous framing of depth cues versus a true depth signal, a contrast reflected in the paper's title (Seeing Depth but Choosing Cues).

Questions for authors

Do the error patterns of VLMs under conflict conditions on real images resemble those of human observers, or do models fail even in cases where humans would clearly not make mistakes? Given that humans also rely heavily on these depth cues, I wonder whether these estimation errors should immediately be characterized as model malfunctions.

Reviewer_CGYsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether VLMs actually use depth when answering spatial questions, or instead rely on 2D proxies such as object size and vertical position in the image. To separate the two, the authors construct controlled cue-conflict scenes and real-image pairs in which depth and the 2D cues disagree, measure which cue the model follows, train linear depth probes on object-level representations to check whether depth is encoded at all, and then write the probe-derived depth direction back into the activations to test whether strengthening the depth signal changes the model's answer. The central finding is a gap between representation and behavior: depth is linearly decodable from the model's features, yet its predictions largely track size and position cues. Write-back improves results on some of the synthetic conflict cases, but the gains transfer less reliably to real images. The contributions are a behavioral and representation-level diagnosis of this depth-cue bias, and a proof-of-concept intervention showing that depth information already present in the model can be amplified to shift its predictions.

Critical strengths

The finding that VLM depth judgments are biased by apparent object size and vertical image position is meaningful.

Critical weaknesses
  1. Major weakness: the method is unnecessarily complicated, and its generalizability is unclear. Extracting object features from intermediate VLM layers and probing them does not appear fundamentally different from probing the corresponding vision-encoder output features. More importantly, the method first trains an in-domain feature-based depth head and then uses its prediction to modulate intermediate VLM features so that the model produces an answer consistent with the head. Why not use the feature-based head directly to answer the question? Similarly, training a head on in-domain data does not appear fundamentally different from fine-tuning the VLM on the same data, including with a LoRA adapter.

  2. The reported size and position biases may not be a problem for frontier VLMs. In addition, whether the model is allowed to think or reason during inference may affect the final results, but this factor is not examined.

  3. In Figure 3, the performance drops for size and vertical conflicts are much smaller on real images than on synthetic images. The synthetic performance drop may therefore be caused partly by the conflict data being too far out of distribution.

  4. In Section 4.2, the prompt asks the model to answer only A/B. Under this prompt, it is unsurprising that the answer-letter probe performs well.

Questions for authors

Could the authors report the performance of recent frontier multimodal models, such as GPT, Claude, and Gemini, on these cue-conflict settings, both with and without allowing extended thinking or reasoning?

Reviewer_9ZL3OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the phenomenon on VLMs being biased towards using 2D shortcuts when judging depth relationships on relative depth questions. The authors benchmarked 11 VLMs and showed that a conflict between 2D cue and 3D relationship is the main cause of this bias. They later showed that with a linear probe on the visual tokens can retrieve the true depth ordering, which lead to the inference time intervention that improved the performance on VLMs on the original samples that had this conflict.

Critical strengths
  1. The research question, the scope, and the exact plan to understand this problem is cleanly designed and well-explained. Experiments have clearly demonstrated the consistent/conflict phenomenon on 2D queue with 3D relationships, and the write-back intervention is a good way to strengthen the finding by steering VLM behaviors.
  2. The authors are honest about negative results, transparent on the cases where the write-back does not work, which furthers the trustfuness on the work.
Critical weaknesses

None that supports rejection. There is some limitation in the practicality of the write-back fix, such as the requirement of ground-truth object masks.

Questions for authors

Can object token sets OAO_A and OBO_B be selected automatically without ground-truth masks, e.g., via text-to-image attention from the object nouns in the prompt?

Comments on the paper

1 comment

Dear Reviewers,

As the initial review scores for this submission are somewhat divergent, it is important that we thoroughly discuss our perspectives before reaching a final recommendation.

Now that the authors' rebuttal is available, I kindly ask each of you to respond in the discussion and clarify whether your original concerns have been addressed or still remain.

Thank you in advance for your timely engagement before the discussion period ends.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.