acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Tests Reveal Limited Depth-Image Use in RGB-D VLMs: A Probing and Causal-Patching Diagnosis on Metric-Distance Queries

Abstract

Multimodal models are routinely given depth maps, but whether they condition on the depth input is unclear. We introduce CF-Depth, which holds the RGB image and question fixed while editing the depth map to flip the correct answer; RGB-only shortcuts then fail by design, so above-chance performance requires using depth. Across ordinary RGB-D fine-tuning and three off-the-shelf depth-VLMs, answers stay largely invariant to depth edits and shuffling. The representation is present but unused: linear probes recover metric depth from the depth-image tokens, yet activation patching shows these tokens causally affect answers only after counterfactual training, pointing to learned read-out, rather than perceptual encoding, as the bottleneck. Per-image depth normalization removes a non-redundant signal, absolute metric scale, and depth use then returns in stages. Fixed-scale depth with counterfactual training first elicits a global metric-scale read-out. A 2B VLM answers counterfactual distance queries well above chance (bin-logit seven-seed mean vs. a depth-blind floor, our primary readout; free-generation vs. RGB-only, chance ), while the controls stay at the depth-blind floor. A stricter local counterfactual that keeps the -bin depth histogram exactly fixed then gives evidence for a spatially localized read-out beyond global statistics: the local-CF model stays above chance, exceeding the global-statistics and RGB-only baselines, and cell-restricted patching supports a causal role for the edited cells. This local read-out is in-domain, while the coarser scale read-out transfers to held-out real sensor depth. For these metric-distance queries, a depth image is thus not sufficient by itself: depth use depends on scale-preserving encoding and a training signal that elicits read-out.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, RGB-D, depth, counterfactual evaluation, activation patching, probing, spatial reasoning, multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EwS6OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether RGB-D VLMs genuinely use depth inputs or mainly rely on RGB shortcuts when answering spatial questions. It introduces CF-Depth, a counterfactual benchmark that keeps the RGB image and question fixed while changing the depth input so that the correct answer changes. The authors combine depth ablations (removal and shuffling), linear probing, and activation patching to tell apart whether depth is merely represented in the model or causally used to produce the answer. They find that conventionally trained models can encode depth without using it to answer questions, while scale-preserving depth encoding and counterfactual training can produce depth-dependent readout. The paper also analyzes how models use global metric scale and local spatial structure, and argues that absolute metric scale is the key depth information missing from RGB for the task studied.

Critical strengths
  1. The paper proposes the CF-Depth benchmark to test whether VLMs use depth information.
  2. The paper shows that depth provides absolute metric scale that RGB lacks. This is intuitive and useful.
Critical weaknesses
  1. The main weakness is whether counterfactual pairs are an effective and valid way to train VLMs to use depth maps. Although they can force the model to read depth and recover the counterfactual answer, they may also reduce its ability to use RGB as complementary evidence when answering depth-related questions. The paper trains and evaluates only a few relatively small VLMs, so the observed behavior may result from the particular training method or data rather than a general property of RGB-D VLMs. This limits the broader value of the proposed method.

  2. The manuscript is not publication-ready. The prose is rough, and the figures, tables, and text need substantial revision.

  3. Section 2 is too long and does not add much information.

  4. Figure 1 does not provide much additional information.

Questions for authors
  1. When RGB and depth disagree, is the model told to trust depth? How can the experiments distinguish not reading depth from choosing RGB? How much of the counterfactual-training gain comes from learning this priority rule?

  2. How is Figure 2(b) different from Table 3? Are they redundant, or does Figure 2(b) provide additional results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer EwS6

Thank you for the specific questions.

Is the model told to trust depth? Not explicitly. §3 now quotes both prompts. The binary prompt begins "Looking at the depth image, which area of the scene is closer to the camera: …"; the three-way prompt asks how far the object is and lists the bins. Neither says which input to trust when they disagree. Even though the binary prompt names the depth image, the ordinary model is at chance under conflict (0.493, chance 0.5).

Not reading depth vs. choosing RGB. The new Table 3 tests depth use without conflict (see the general response). For absolute distance, with RGB present, the ordinary model with a frozen vision encoder gains nothing from depth even on unedited objects that an RGB-only model gets wrong (−0.004 [−0.021, +0.013]), so choosing RGB under conflict does not explain the result there. With RGB replaced by gray, it reads some depth (+0.125 on real objects, +0.064 on edited items; counterfactual model +0.208, +0.358). On binary ordering, choosing RGB does play a part: the ordinary model uses depth in part but is at chance under conflict (0.493), while a model given the same depths as text answers 0.928 of the conflicting items correctly. Claim (a) is narrowed to absolute distance and a frozen vision encoder.

How much of the gain is a priority rule? We cannot separate it exactly. Part of it is reweighting: RGB-token patching changes 0.559 of the one-epoch ordinary model's answers and 0.206 of the counterfactual model's. Part of it is reading depth: with gray RGB the counterfactual model gains +0.358 against +0.064, and the local model answers two-cell swaps whose global depth histogram is identical, beating a logistic baseline on global depth statistics by +0.283 in bin-logit and +0.063 in free generation. A priority rule alone would not produce this cell-level read-out.

Validity of counterfactual training. We agree that it can weaken RGB as complementary evidence, mainly for the local models. On unedited real objects, the global model shows no cost in bin-logit, our primary scoring (+0.003 [−0.026, +0.032] against the matched ordinary model), and loses 0.032 in free generation. The local models lose 0.029–0.055 in bin-logit and 0.067–0.078 in free generation. §8 now states this. We present the method as evidence that a depth read-out can be learned; we have not evaluated it as a deployment recipe.

Small models. Our claims are limited to 2–9B models trained with LoRA, mostly with a frozen vision encoder (§8). Within that range, the result does not depend on one training run: ordinary training leaves depth unused in three seeds, with five copies of the data, and with the bins balanced within each object class (§4, Appendix D). In all four models we trained (Qwen3.5-2B and 9B, SmolVLM2-2.2B, InternVL3-2B), ordinary LoRA training shows little depth use and counterfactual training raises it; for InternVL3-2B the shuffle drop rises from 0.019 to 0.095 [0.078, 0.113], less than for SmolVLM2 (0.395; Appendix G). Released depth-VLMs, trained by their authors with other methods and data, also show little use of the depth content on CF-Depth (Appendix K). Fine-tuning the vision encoder as well, which we added, yields no depth use at learning rate 5×10⁻⁶ and partial use, mainly of the global depth level, at 2×10⁻⁵: 0.314–0.481 on items whose answer the edit changes, against 0.162 with a frozen encoder and 0.817 for counterfactual training (§4).

Presentation. The text, figures, and tables were revised throughout. Section 2 is now four short paragraphs. Figure 1 shows a binary pair as evaluated: RGB and question, sensor depth, the counterfactual, and the shuffle control. You are right that the submitted Figure 2(b) and Table 3 showed the same patching results. The main text now has only Figure 2, and the per-layer values are in Tables 8 and 9 (Appendix C).

Reviewer_EwS6OpenReview ↗

Response to author's rebuttal

I sincerely appreciate the authors' detailed response and their revisions to the paper. The rebuttal has addressed my questions and helped me better understand this submission. That being said, my major concern remains. As the authors acknowledge in the rebuttal, the conclusions of this work are limited to a narrow scope, which may significantly reduce its value to the broader conference audience. I also share the other reviewers' concerns about the counterfactual training. I therefore believe the submission is not yet ready for publication. I will raise my score to reflect the improvements, but my overall recommendation remains unchanged.

AuthorsOpenReview ↗

Scope of the claims and counterfactual training

Thank you for reading the revision and for raising your score.

Scope. The limits we stated apply to our claims about training: the models we fine-tuned are 2–9B, trained with LoRA, mostly with a frozen vision encoder. The test itself can be run on any model that accepts a depth image. We ran it on three released depth-VLMs trained by their authors with other data and methods (SpatialBot-3B, SD-VLM-7B, and RoboRefer-2B/8B), and without fine-tuning they also show little or no use of the depth content (§4, Appendix K).

Counterfactual training. We use it for a narrower point than a training recipe: with the same architecture and depth encoding, a model can learn to answer from the depth image when training requires it, so the ordinary models' non-use of depth is not a limit of the architecture or the encoding. On unedited real objects, its overall accuracy matches the matched ordinary model in bin-logit, but the two differ by item: it is more accurate on the 766 objects that an RGB-only model answers incorrectly (+0.098 [+0.036, +0.160]) and less accurate on the other 1,301 (−0.053 [−0.088, −0.019]). This breakdown, also given in our reply to Reviewer Cd4B, is not reported in the paper. The main finding, that ordinary fine-tuning with a frozen vision encoder leaves depth encoded but unused for absolute distance when RGB is present, does not depend on counterfactual training.

Reviewer_E2UzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CF-Depth, an empirical study showing that when answering spatial questions, existing VLMs are dominated by RGB images and do not use depth maps even when provided. The paper designs the CF-Depth evaluation so that RGB images and questions remain the same, but the depth map is edited so that the correct answer aligns with the depth map flips. Experiments show that most existing VLMs predict the same answer even when the depth map changes. Then, the paper studies that a linear probe can cover the depth information from the intermediate feature so it's encoded, but model fail to read-out. The paper shows that a scale-preserving encoding + counterfactual training can recover depth use.

Critical strengths

1. The experiment is carefully designed to show that VLMs mainly rely on RGB images, rather than depth images, for spatial questions, and probing experiments show that depth information is encoded.

I appreciate the author's effort in carefully designing CF-Depth to show this interesting finding.

  • First, the paper carefully designs the question such that, after editing the depth map, the answer associated with the depth map flips (e.g., which object is farther, or close/medium/far defined by absolute distance).

  • The most interesting part is to see that the depth information is recoverable via linear probing from the intermediate features, but the model fails to use them, neither the zero-shot model nor the fine-tuned ones. This provides further evidence that depth, even when informative, is usually ignored by the model.

Critical weaknesses

1. The current evaluation and analysis do not sufficiently distinguish (a) the failure to use depth from (b) RGB dominance under modality conflict.

My biggest concern is that the paper's central conclusion that "VLMs perceive depth information but fail to use it", may have an alternative explanation: when RGB and depth provide conflicting information, the model prioritizes RGB rather than depth. This concern affects the interpretation of CF-Depth and the subsequent causal analysis.

  • First, CF-Depth inherently introduces modality conflict. The benchmark keeps RGB unchanged while editing depth to flip the correct answer (Figure 3), creating inconsistencies between the two modalities. Since VLMs are predominantly trained on RGB images, models may prioritize RGB when the two disagree, even if they can use depth information. Therefore, low CF-combined accuracy may reflect modality preference rather than an inability to use depth.

  • Activation patching may suffer from the same issue. Replacing intermediate depth-token features between base and counterfactual examples can create inconsistencies with other multimodal representations. The patching experiment may therefore also reflect RGB dominance rather than the absence of a depth read-out.

  • Counterfactual training may primarily change modality preference. Improvements in CF-combined, cf-sensitivity, and activation-patching results could arise because the model learns to prioritize depth over RGB, rather than acquiring a previously missing depth read-out. Actually, Table 7 shows that after CF training, the absolute performance decreases when providing naturally aligned RGB-D inputs.

Overall, my concern is that the paper does not sufficiently distinguish the main hypothesis (a) "a missing depth read-out", from an alternative explanation (b) "RGB-dominant multimodal decision-making". Since the distinction between perception and read-out is a central contribution of the paper, I believe additional evidence is needed to support the core claim.

Several additional experiments are suggested to eliminate hypothesis-b.

To disentangle depth-use capability from modality preference, I suggest evaluating models on naturally aligned RGB-D samples where RGB alone is insufficient to answer the question, but depth provides the necessary information. For example, textureless surfaces; the same object at slightly different distances (chairs at 3 meters vs chairs at 5 meters); or objects with ambiguous physical size (a small toy car appears small but close to the camera). If depth still does not help here, I'm more convinced by the paper's claim that the bottleneck is in depth read-out rather than modality conflict.

2. Presentation issue: feels heavily AI-generated, which in term harms readability.

This point isn't the main reason for my final decision, but I want to point this out since there might be readability issues for general ICLR audiences:

  • Several abbreviations, such as CV, are used without being clearly introduced.
  • The heavy defensive writing, e.g., dense use of parentheses throughout the paper.

I suggest the author improve the writing and readability.


To summarize, showing more evidence of weakness 1 can make me lean more towards acceptance. Fixing readability does not impact my decision, but it's generally suggested due to the readability issue.

Questions for authors

Please see weakness-1 above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer E2Uz

Thank you for the clear statement of the alternative hypothesis. We now test it directly.

(a) missing read-out vs. (b) RGB preference. You wrote that if depth still does not help on naturally aligned RGB-D samples where RGB alone is insufficient, you would be more convinced that the bottleneck is the read-out. For absolute distance, this is what we find. On the 766 unedited real-object items that an RGB-only model answers incorrectly, the ordinary model with a frozen vision encoder gains nothing from real over shuffled depth: −0.004 [−0.021, +0.013]. Your example of the same object at different distances is closest to our second subset, objects whose true bin differs from the most frequent training bin of their class (913 items): −0.002 [−0.017, +0.013]. In bin-logit, our primary scoring, the two values are −0.003 and −0.004. The counterfactual model gains on both subsets (+0.056 [+0.021, +0.094] and +0.070 [+0.037, +0.104]), so these subsets can show depth use. On 2D-3D-S grid questions, which carry no object prior, removing depth changes the matched ordinary model by −0.007 [−0.020, +0.005] (Appendix E), and on all unedited NYU objects it scores 0.663, 0.667, and 0.662 with real, shuffled, and removed depth (Table 12). We did not build textureless or size-ambiguous sets; the two subsets are our closest substitute.

The read-out is not entirely absent, though. With the RGB image replaced by gray, the ordinary model reads some depth (+0.125 on real objects, +0.064 on edited items), less than the counterfactual model (+0.208, +0.358) and little of the absolute scale. For absolute distance we therefore describe a read-out that is weak for scale and goes unused when RGB is present, even where nothing conflicts. On binary ordering, RGB does compete with depth under conflict, as you suggested: the ordinary model uses depth in part (shuffle drop +0.107) but is at chance on conflicting items (0.493). The same depths given as text are followed even under conflict (0.928), so the competition concerns the depth image. Claim (a) is narrowed to absolute distance and a frozen vision encoder.

Activation patching. We agree that patching alone cannot separate (a) from (b) in the ordinary model: if RGB dominates, replacing depth tokens would also leave the answer unchanged. We therefore rely on Table 3 for that distinction. Patching still checks two things. First, the intervention can move answers: patching RGB tokens from another image changes 0.559 of the one-epoch ordinary model's answers, while depth-token patching changes at most 0.031 of the matched ordinary model's answers at any layer. Patching the text positions after the depth tokens changes at most 0.010 at layers 6 and 12, so the ordinary models do not read depth there either. Second, in the counterfactual model the effect follows the depth content: a donor from another image with the same answer bin changes 0.37 of the answers, and 0.80 of the patched answers stay in the correct bin; a donor with a different bin changes 0.87 (Appendix C).

Does counterfactual training only change priority? Not only, although it does change priority. Reliance on RGB drops: RGB-token patching changes 0.206 of the counterfactual model's answers (0.559 above). But the counterfactual model also reads depth when no RGB content competes with it: with a gray RGB image it gains +0.358 on edited items, against +0.064 for the ordinary model. And on unedited real objects, where depth and RGB show the same scene, it gains +0.056 [+0.021, +0.094] from real over shuffled depth on the items the RGB-only model gets wrong (+0.097 in bin-logit). We cannot split the gain exactly into the two parts.

Aligned RGB-D accuracy after counterfactual training (old Table 7, now Table 12). You are right for free generation, the scoring of old Table 7: on unedited real objects the counterfactual models are less accurate than the matched ordinary model (−0.032 [−0.060, −0.004] for the global model; −0.078 and −0.067 for the local ones). In bin-logit, our primary scoring, the global model shows no cost (0.761 against 0.758; +0.003 [−0.026, +0.032]); the two local models do (−0.029, −0.055). Free generation uses a repetition penalty, which lowers its accuracy because the answer words appear in the prompt (Appendix C). The submitted text did not state the cost; §8 now reports both scorings.

Presentation. CV is now written out as cross-validation, and we removed most parentheses.

Reviewer_E2UzOpenReview ↗

Reviewer's response

I thank the author for their detailed response and revision.

  • First, I agree with all other reviewers' concerns about the writing issues. The paper is difficult to read, with a heavily AI-generated writing style. For example, terms like "matched ordinary" are newly invented in the paper; the large number of such terms makes it hard to follow. To improve it, for example, the author could choose more straightforward wording, such as "normal RGB-D pair SFT model" instead of "matched ordinary model", which makes it hard to tell if the model is being fine-tuned or what "matched ordinary" actually means here.

  • Second, I thank the authors for their effort in conducting new experiments and narrowing down the claim. I think the experiment supports my hypothesis that the model can read some depth, but this read-out is suppressed when an RGB image is provided. However, this indicates that the bottleneck is RGB dominance (my hypothesis b) rather than a missing depth read-out, which weakens the novelty of the central “encoded but unused” finding, as it largely reduces to the known modality-competition effect.

  • Third, I understand the other reviewers’ concern about the CF training. However, from my reading, the paper frames CF training as a diagnostic tool rather than a deployable approach, so I do not regard it as a major concern as long as it serves the analysis.

Overall, I keep my score (weak accept): the diagnostic findings are useful, and the experiments are carefully designed, but the response to Weakness 1 limits the novelty of the central claim.

AuthorsOpenReview ↗

Read-out vs. RGB dominance, and wording

Thank you for the follow-up and for the concrete wording suggestion.

Wording. The revised PDF uploaded today follows your suggestion and uses descriptive names. The "matched ordinary model" is now the "SFT baseline": the RGB-D model fine-tuned on unedited question–answer pairs from the same training images, for the same three epochs as the counterfactual model. The other ordinary models are now "standard SFT" models, and "ordinary items" are "unedited items". No number changed. Beyond the names, the only changes are the word "three-way" in the abstract, which names the task behind the absolute-distance claim, and a few clarified references to models. Our earlier responses use the old names.

Read-out vs. RGB dominance. We agree with your reading of the evidence: the SFT baseline reads some depth when the RGB image is gray, and does not use depth when the RGB image is present. Two results bear on how to describe this. First, no conflict is needed. On unedited objects that an RGB-only model answers incorrectly, where depth shows the same scene, real depth adds nothing over shuffled depth for the SFT baseline (−0.004 [−0.021, +0.013]), although the counterfactual model gains there (+0.056 [+0.021, +0.094]). Second, with a gray RGB image the read-out is weak for absolute scale (+0.064 on edited items, against +0.358 for the counterfactual model), so the read-out itself is weak, not only suppressed. As you say, this pattern fits modality competition during training, which §2 cites. What the paper adds is the measurement for RGB-D VLMs: a test that, with probes and patching, separates whether depth is encoded from whether it is used; the finding that the unused signal is absolute scale, which per-image normalization removes; and the result that released depth-VLMs also show little use of depth.

Counterfactual training. This is how we use it: to show that the same architecture and encoding can learn to use the depth image, not as a deployment recipe.

Reviewer_Cd4BOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper finds that the existing 3D VLMs do not often regard depth as the decisive information for spatial reasoning. The paper investigates this by flipping the depth maps in the input and finds that the answer stays the same despite the change in depth, which means the VLM perceives distance mainly from images. The paper then verifies that depth information is already encoded into the network by linear depth probing, and enhances the depth utilization by incorporating more conterfactual training pairs.

Critical strengths

The use of depth priors in spatial VLM is important, and this paper investigates this interesting and important question. The conclusion that current models usually bypass depth information during reasoning is informative.

Critical weaknesses
  1. The most critical question is that, due to the heavy involvement of AI drafting without proper human intervention or steering, this paper contains a lot of obscure statements, wordy expressions, and lengthy captions that are extremely hard to follow, despite doing something that seems to be useful. It is not a submission that is ready for human readers.

  2. In addition to the first critical issue, I do not strictly agree with involving conterfactual samples during training. The counterfactual samples will encourage the model to totally overlook the perspective cues in images. For many applications where GT depth is noisy and predicted depth is not reliable for distant regions, perspective information is an effective alternative cue.

  3. The experiments section does not show the comparison against other methods on the full split of the public benchmark. Additionally, evaluating more complex reasoning tasks, such as question answering, will provide a comprehensive evaluation of the proposed approach.

Questions for authors

Please see the Weakness part.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Cd4B

Thank you for the review.

Readability. We agree that the submitted text was hard to follow. We rewrote the paper throughout. Table 1 now lists each claim with its evidence, and captions are short.

Counterfactual training and perspective cues. The concern is fair, and the revision now measures the cost. Counterfactual training reduces reliance on RGB but does not remove it: patching RGB tokens from another image still changes 0.206 of the counterfactual model's answers (0.559 for the one-epoch ordinary model). On unedited real objects, where the object prior is often right, the global counterfactual model shows no cost against the matched ordinary model in bin-logit, our primary scoring (+0.003 [−0.026, +0.032]), and is lower in free generation (−0.032 [−0.060, −0.004]). The two local models are lower in both (−0.029 and −0.055 in bin-logit; −0.078 and −0.067 in free generation; §8 Limitations, Table 12). Mixing counterfactual and ordinary data (Appendix D) keeps general question answering at the level of ordinary training (token F1 0.55–0.56 vs. 0.55–0.57). We have not tested degraded sensor depth, and the Limitations say this. Our one set with estimated depth, built from SpatialRGPT-Bench, also changes the question type to the distance between two objects; no model gains reliably from depth there (Appendix J). We therefore present counterfactual training as evidence that a depth read-out can be learned, not as a deployment recipe, and the Conclusion says so.

Public benchmarks and more complex QA. The paper asks whether a model uses its depth input, which benchmark accuracy alone does not show. On SUN RGB-D object questions, for example, the counterfactual model and an RGB-only model score almost the same (0.538 and 0.529), yet shuffling the depth lowers the counterfactual model by 0.100 [0.060, 0.142] (Appendix E). We therefore compare released depth-VLMs on depth use: on CF-Depth, SpatialBot-3B gains +0.02 from real over removed depth, SD-VLM-7B gains +0.01 from real over shuffled depth, and RoboRefer-2B and 8B follow a depth swap on 0.003 and 0.000 of the pairs (Appendix K). Full-split accuracy comparisons and harder reasoning tasks would be needed to assess task performance; the paper does not provide them and does not claim task performance. For general question answering, we report token F1 on 1,214 held-out SpatialQA questions for the binary models (§4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reviews. The revised PDF is uploaded. The title and the section order are unchanged. The TL;DR field cannot be edited in the revision form; for the revised claims it would read: “In 2–9B VLMs with a frozen vision encoder, ordinary RGB-D fine-tuning leaves depth decodable but unused for absolute distance when RGB is present; fixed-scale depth with counterfactual training makes the model use the depth image.” Table 1 now lists the claims as (a)–(e) with their main evidence; three of them are narrower (see Corrections below).

Readability. We rewrote the text for clarity, shortened Section 2, and replaced Figure 1 with a CF-Depth pair as evaluated.

The shared question. Reviewers E2Uz and EwS6 asked whether ordinary models fail to read depth or prefer RGB when the two conflict. The new Table 3 tests depth use where there is no conflict. Here and below, the matched ordinary model is trained on ordinary questions about the same training images, for the same three epochs as the counterfactual model.

test (no RGB–depth conflict)matched ordinarycounterfactual
real objects the RGB-only model gets wrong (766): real minus shuffled depth, free generation−0.004 [−0.021, +0.013]+0.056 [+0.021, +0.094]
objects whose true bin differs from their class's usual bin (913): same−0.002 [−0.017, +0.013]+0.070 [+0.037, +0.104]
gray RGB, real objects: gain from depth, bin-logit+0.125 [+0.087, +0.163]+0.208 [+0.164, +0.253]
gray RGB, edited items: gain from depth, bin-logit+0.064 [+0.047, +0.081]+0.358 [+0.332, +0.383]

For absolute distance, with RGB present, the ordinary model gains nothing from depth in the first two rows, on objects where an RGB-only model or the object prior fails and nothing conflicts; in bin-logit, the primary scoring, these two values are −0.003 and −0.004. Without RGB, it reads some depth, but less than the counterfactual model and little of the absolute scale. Our answer is therefore in two parts. For absolute distance, ordinary training with a frozen vision encoder leaves a read-out that is weak for scale, and the model does not use it when RGB is present, even where nothing conflicts. In new runs with the vision encoder fine-tuned as well (learning rate 2×10⁻⁵), the ordinary model partly uses depth, mainly its global level (new paragraph in §4). On binary ordering, RGB competes with depth: the ordinary model uses depth in part (shuffle drop +0.107) but is at chance under conflict (0.493), while a model given the same depths as text answers 0.928 of the conflicting items correctly. Claim (a) is narrowed accordingly.

Corrections. In preparing the revision we re-ran all evaluations with per-item records and re-checked every number against its result file. Appendix A lists the 31 resulting corrections and their effect on each claim. Those that change what a reader should take away are: claims (a), (b), and (e) are narrower; the binary set was rebuilt from NYU and 2D-3D-S because its KITTI part had defective depth, and every row of Table 2B was re-run; the headline is now the three-epoch primary model (0.835 in bin-logit on the edited items), with six further seeds at 0.805–0.844, instead of a mean over checkpoints trained for different numbers of epochs; the two-cell logistic baseline scores 0.468, not 0.406 (0.406 was its accuracy on swap rows only), so the local model's margin in free generation is smaller (+0.063); and the SpatialRGPT-Bench appendix is now a negative result. One correction goes the other way: the five runs counted as non-converging had not loaded their counterfactual images, and all six correctly trained seeds converge. The updated supplementary material contains the number ledger and the result files, from which every number in the paper can be recomputed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.