Counterfactual Tests Reveal Limited Depth-Image Use in RGB-D VLMs: A Probing and Causal-Patching Diagnosis on Metric-Distance Queries

Multimodal models are routinely given depth maps, but whether they condition on the depth input is unclear. We introduce CF-Depth, which holds the RGB image and question fixed while editing the depth map to flip the correct answer; RGB-only shortcuts then fail by design, so above-chance performance requires using depth. Across ordinary RGB-D fine-tuning and three off-the-shelf depth-VLMs, answers stay largely invariant to depth edits and shuffling. The representation is present but unused: linear probes recover metric depth from the depth-image tokens, yet activation patching shows these tokens causally affect answers only after counterfactual training, pointing to learned read-out, rather than perceptual encoding, as the bottleneck. Per-image depth normalization removes a non-redundant signal, absolute metric scale, and depth use then returns in stages. Fixed-scale depth with counterfactual training first elicits a global metric-scale read-out. A 2B VLM answers counterfactual distance queries well above chance (bin-logit $0.822$ seven-seed mean vs. a $0.445$ depth-blind floor, our primary readout; free-generation $0.62$ vs. $0.40$ RGB-only, chance $0.33$), while the controls stay at the depth-blind floor. A stricter local counterfactual that keeps the $256$-bin depth histogram exactly fixed then gives evidence for a spatially localized read-out beyond global statistics: the local-CF model stays above chance, exceeding the global-statistics and RGB-only baselines, and cell-restricted patching supports a causal role for the edited cells. This local read-out is in-domain, while the coarser scale read-out transfers to held-out real sensor depth. For these metric-distance queries, a depth image is thus not sufficient by itself: depth use depends on scale-preserving encoding and a training signal that elicits read-out.