- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether RGB-D VLMs genuinely use depth inputs or mainly rely on RGB shortcuts when answering spatial questions. It introduces CF-Depth, a counterfactual benchmark that keeps the RGB image and question fixed while changing the depth input so that the correct answer changes. The authors combine depth ablations (removal and shuffling), linear probing, and activation patching to tell apart whether depth is merely represented in the model or causally used to produce the answer. They find that conventionally trained models can encode depth without using it to answer questions, while scale-preserving depth encoding and counterfactual training can produce depth-dependent readout. The paper also analyzes how models use global metric scale and local spatial structure, and argues that absolute metric scale is the key depth information missing from RGB for the task studied.
Critical strengths
- The paper proposes the CF-Depth benchmark to test whether VLMs use depth information.
- The paper shows that depth provides absolute metric scale that RGB lacks. This is intuitive and useful.
Critical weaknesses
-
The main weakness is whether counterfactual pairs are an effective and valid way to train VLMs to use depth maps. Although they can force the model to read depth and recover the counterfactual answer, they may also reduce its ability to use RGB as complementary evidence when answering depth-related questions. The paper trains and evaluates only a few relatively small VLMs, so the observed behavior may result from the particular training method or data rather than a general property of RGB-D VLMs. This limits the broader value of the proposed method.
-
The manuscript is not publication-ready. The prose is rough, and the figures, tables, and text need substantial revision.
-
Section 2 is too long and does not add much information.
-
Figure 1 does not provide much additional information.
Questions for authors
-
When RGB and depth disagree, is the model told to trust depth? How can the experiments distinguish not reading depth from choosing RGB? How much of the counterfactual-training gain comes from learning this priority rule?
-
How is Figure 2(b) different from Table 3? Are they redundant, or does Figure 2(b) provide additional results?
3 comments
Response to author's rebuttal
I sincerely appreciate the authors' detailed response and their revisions to the paper. The rebuttal has addressed my questions and helped me better understand this submission. That being said, my major concern remains. As the authors acknowledge in the rebuttal, the conclusions of this work are limited to a narrow scope, which may significantly reduce its value to the broader conference audience. I also share the other reviewers' concerns about the counterfactual training. I therefore believe the submission is not yet ready for publication. I will raise my score to reflect the improvements, but my overall recommendation remains unchanged.
Scope of the claims and counterfactual training
Thank you for reading the revision and for raising your score.
Scope. The limits we stated apply to our claims about training: the models we fine-tuned are 2–9B, trained with LoRA, mostly with a frozen vision encoder. The test itself can be run on any model that accepts a depth image. We ran it on three released depth-VLMs trained by their authors with other data and methods (SpatialBot-3B, SD-VLM-7B, and RoboRefer-2B/8B), and without fine-tuning they also show little or no use of the depth content (§4, Appendix K).
Counterfactual training. We use it for a narrower point than a training recipe: with the same architecture and depth encoding, a model can learn to answer from the depth image when training requires it, so the ordinary models' non-use of depth is not a limit of the architecture or the encoding. On unedited real objects, its overall accuracy matches the matched ordinary model in bin-logit, but the two differ by item: it is more accurate on the 766 objects that an RGB-only model answers incorrectly (+0.098 [+0.036, +0.160]) and less accurate on the other 1,301 (−0.053 [−0.088, −0.019]). This breakdown, also given in our reply to Reviewer Cd4B, is not reported in the paper. The main finding, that ordinary fine-tuning with a frozen vision encoder leaves depth encoded but unused for absolute distance when RGB is present, does not depend on counterfactual training.
Response to Reviewer EwS6
Thank you for the specific questions.
Is the model told to trust depth? Not explicitly. §3 now quotes both prompts. The binary prompt begins "Looking at the depth image, which area of the scene is closer to the camera: …"; the three-way prompt asks how far the object is and lists the bins. Neither says which input to trust when they disagree. Even though the binary prompt names the depth image, the ordinary model is at chance under conflict (0.493, chance 0.5).
Not reading depth vs. choosing RGB. The new Table 3 tests depth use without conflict (see the general response). For absolute distance, with RGB present, the ordinary model with a frozen vision encoder gains nothing from depth even on unedited objects that an RGB-only model gets wrong (−0.004 [−0.021, +0.013]), so choosing RGB under conflict does not explain the result there. With RGB replaced by gray, it reads some depth (+0.125 on real objects, +0.064 on edited items; counterfactual model +0.208, +0.358). On binary ordering, choosing RGB does play a part: the ordinary model uses depth in part but is at chance under conflict (0.493), while a model given the same depths as text answers 0.928 of the conflicting items correctly. Claim (a) is narrowed to absolute distance and a frozen vision encoder.
How much of the gain is a priority rule? We cannot separate it exactly. Part of it is reweighting: RGB-token patching changes 0.559 of the one-epoch ordinary model's answers and 0.206 of the counterfactual model's. Part of it is reading depth: with gray RGB the counterfactual model gains +0.358 against +0.064, and the local model answers two-cell swaps whose global depth histogram is identical, beating a logistic baseline on global depth statistics by +0.283 in bin-logit and +0.063 in free generation. A priority rule alone would not produce this cell-level read-out.
Validity of counterfactual training. We agree that it can weaken RGB as complementary evidence, mainly for the local models. On unedited real objects, the global model shows no cost in bin-logit, our primary scoring (+0.003 [−0.026, +0.032] against the matched ordinary model), and loses 0.032 in free generation. The local models lose 0.029–0.055 in bin-logit and 0.067–0.078 in free generation. §8 now states this. We present the method as evidence that a depth read-out can be learned; we have not evaluated it as a deployment recipe.
Small models. Our claims are limited to 2–9B models trained with LoRA, mostly with a frozen vision encoder (§8). Within that range, the result does not depend on one training run: ordinary training leaves depth unused in three seeds, with five copies of the data, and with the bins balanced within each object class (§4, Appendix D). In all four models we trained (Qwen3.5-2B and 9B, SmolVLM2-2.2B, InternVL3-2B), ordinary LoRA training shows little depth use and counterfactual training raises it; for InternVL3-2B the shuffle drop rises from 0.019 to 0.095 [0.078, 0.113], less than for SmolVLM2 (0.395; Appendix G). Released depth-VLMs, trained by their authors with other methods and data, also show little use of the depth content on CF-Depth (Appendix K). Fine-tuning the vision encoder as well, which we added, yields no depth use at learning rate 5×10⁻⁶ and partial use, mainly of the global depth level, at 2×10⁻⁵: 0.314–0.481 on items whose answer the edit changes, against 0.162 with a frozen encoder and 0.817 for counterfactual training (§4).
Presentation. The text, figures, and tables were revised throughout. Section 2 is now four short paragraphs. Figure 1 shows a binary pair as evaluated: RGB and question, sensor depth, the counterfactual, and the shuffle control. You are right that the submitted Figure 2(b) and Table 3 showed the same patching results. The main text now has only Figure 2, and the per-layer values are in Tables 8 and 9 (Appendix C).