- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RLVR has become a standard approach to improve the spatial reasoning capabilities of VLMs. However, current research indicates that VLMs often provide answers directly without observing the image. The authors attempted to examine this shortcut by controlling variables, specifically by removing text coordinates that contain depth information. They proposed three methods: using only image labels, using both labels and coordinates, and using only coordinates. Under RLVR training, all experiments using text coordinates outperformed the pure text model, while all experiments using visual labels outperformed their corresponding pure text models. This approach forces the VLM to examine the image, resulting in a non-zero performance in the case of the 7B model on worst-case bucket accuracy, which is a common phenomenon in other baseline methods.
Critical strengths
-
The authors accurately located this kind of VLM shortcut problem after the perception process and reproduced it. A small probe model is able to decode depth information from the frozen representation, achieving an accuracy 2.2 times higher than the model's actual output accuracy. This means that the relevant information already exists in the model representation but is not reflected in the output.
-
The authors find that a single comprehensive score is insufficient to distinguish whether the model has learned the geometric relationships or just matched the answer's distribution. In the experiments that trained on text coordinate data, the pass@1 score reached 0.1888, however, the worst bucket accuracy is still 0.
-
Especially in depth estimation, to test whether depth information could be recovered from frozen visual representations, the authors trained a small probe. They found that the MLP probe achieved an accuracy of 0.3475 on the base model, compared to 0.1650 on the baseline. In contrast, the linear probe only provided an improvement of 0.075 to 0.085, indicating that relevant information already exists in the visual representation but cannot be directly obtained through a simple linear mapping.
Critical weaknesses
-
The high-contrast visual markers may allow the model to read their row position in the image and apply the same planar geometry exposed by the text coordinates, resulting in a similar shortcut problem that the author claimed.
-
Even without text coordinates, could the reward function itself cause the model to rely on the prediction of the median prediction? The tolerance band verifier may reward safe outputs.
-
It remains unknown which computational processes are changed by the marker training. The authors demonstrate that the model's visual representation already contains depth information, but the performance improvement bring about by the proposed visual label training may also come from the model's implicit coordinate understanding of other visual cues rather than geometric representations.
Questions for authors
-
The method proposed by the authors cannot be reproduced at all on the InternVL3.5-8B backbone network. Is this because of the different visual tower architecture? Does this indicate that the effectiveness of the proposed method is highly dependent on the architecture of the model itself?
-
For the applicability of the method, the core mechanism proposed in the paper is specific to scenes with a dominant ground plane and may not be applicable to aerial images. Is it still effective for these scenes?
Thank you. Each point led to a new experiment.
Does the marker just expose its row? (W1). Tested three ways (general response, part 2; App. Q Table 9). Grey image with the marker kept in place leaves exactly the information you describe; the marker models fall below the blind baseline (RLVR 0.254 → 0.089; SFT 0.291 → 0.043). A 180° rotation reverses the row cue; the marker models keep ranking by LiDAR depth (ρ = .39, .60) and anti-correlate with the row formula (every seed). On off-ground points the submitted marker RLVR model follows LiDAR depth rather than the formula (regression weights 0.38 vs 0.19). The same controls are reported for the
mkcarm: on a grey image the SFTmkcarm still orders answers by the stated row (ρ = .39), as the coordinate arm does (.51).Prompt or reward? (W2). Purely continuous reward, no tolerance band: coordinate arm still below its blind baseline (−0.026); marker arm ahead on coordinate-free prompts by +0.091 (p = 0.002, 3/3 seeds). The effect does not come from the band.
Probe (W3). Agreed; Sec. 4.1 now reads: the probe locates the bottleneck after perception; it does not show that the marker unlocks this same representation.
Why
mkc<coon coordinate-free prompts? A model trained with coordinates in the prompt learns to rely on them, marker or not, and loses its main cue when they are absent (SFTmkc−cothere: −0.041). This supports the decomposition: the active part is removing the coordinates (mk−mkc: +0.071 RLVR, +0.134 SFT, on identical images). Sec. 5 now says this.InternVL (Q1). Its frozen features are the most depth-decodable of the three backbones (probe δ<1.25 0.3275 vs 0.3075 / 0.2225) at the same size as Qwen3-VL-8B, so neither a weaker vision tower nor capacity explains the failure; it lies in turning available features into a metric answer. We do not have evidence to separate decoder from training dynamics, and state the formulation effect as established for one model family and not reproduced on this second one.
No ground plane (Q2). The protocol does not assume one: the blind baseline is fitted to whatever text the benchmark gives, so where the text carries no depth it is simply low and any margin must come from the image. Indoors it flags our driving-trained arm as having no transferable metric ability (Sec. 7). The 180° rotation is a controlled version inside our data: it removes the usable ground-plane relation without changing the scene, and the marker models keep ranking by true depth.