- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the susceptibility of VLMs to apparent object size and vertical position when judging the depth order of two objects. While prior work showed that VLMs rely on vertical position as a shortcut, this study systematically tests model behavior under cue conflict conditions using synthetic scenes and real images where apparent size and vertical position are independently manipulated. The authors find that even when models are misled by these cues into making incorrect judgments, correct depth relations remain linearly decodable from the target objects' visual representations. Furthermore, an intervention that amplifies this internal depth signal can correct these errors under conflict conditions without modifying model weights. However, this corrective effect is largely confined to synthetic scenes and remains limited on real photographs.
Critical strengths
A key strength of this paper is its systematic evaluation of how apparent size and vertical position bias VLM depth judgments across both controlled synthetic scenes and real images. In particular, examining cases where apparent size conflicts with true depth on real photographs meaningfully extends prior work (Min et al. 2026).
Critical weaknesses
I have serious reservations regarding how the authors treat the information decoded from internal representations as if it were a true depth signal. Monocular depth estimation is an inherently ill-posed problem, and both humans and VLMs must combine multiple depth cues to make plausible inferences. Because the probe in this study is trained on a dataset where apparent size and vertical position are decorrelated from depth and thus rendered unreliable, it naturally learns to rely on whichever other depth cues remain informative. The authors emphasize that correct depth can still be decoded when the model answers incorrectly under conflict conditions, but this is hardly surprising if the probe is simply reading out other visual cues (such as contrast or brightness). By the same logic, the accuracy gains achieved through the intervention under conflict conditions may simply reflect an artificial boost to reliance on these alternative cues, which could conversely amplify biases toward those specific cues. Indeed, the intervention yields limited improvement on real photographs and even degrades accuracy under consistent conditions. At present, there is insufficient justification for the dichotomous framing of depth cues versus a true depth signal, a contrast reflected in the paper's title (Seeing Depth but Choosing Cues).
Questions for authors
Do the error patterns of VLMs under conflict conditions on real images resemble those of human observers, or do models fail even in cases where humans would clearly not make mistakes? Given that humans also rely heavily on these depth cues, I wonder whether these estimation errors should immediately be characterized as model malfunctions.
Dear Reviewers,
As the initial review scores for this submission are somewhat divergent, it is important that we thoroughly discuss our perspectives before reaching a final recommendation.
Now that the authors' rebuttal is available, I kindly ask each of you to respond in the discussion and clarify whether your original concerns have been addressed or still remain.
Thank you in advance for your timely engagement before the discussion period ends.
Best, Your AC