- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DepthVLM adds dense metric depth prediction to a pretrained vision-language model using a DPT-style decoder. The decoder combines intermediate features from the visual encoder with image-token representations from the LLM. Training has two stages: first, only the depth head is trained; then, the LLM and depth head are trained together. The visual encoder remains frozen throughout.
The evaluation tests depth prediction, retention of general multimodal capabilities, and spatial reasoning. It also includes ablations of the architecture and training strategy.
Critical strengths
- Practically useful capability extension: This work shows the capability of depth labels in VLMs for many additional spatial reasoning tasks.
- Straightforward design and broad evaluation: The approach uses an established decoder and staged training, making it easy to understand and potentially straightforward to integrate into existing VLMs. Tests on multiple depth datasets provide strong evidence that the full system works well. Evaluation over all valid pixels in the full image and relevant ablation studies further support these results.
Critical weaknesses
- Limited methodological differentiation; the evaluation should place greater emphasis on 3D reasoning
Youtu-VL, which the paper includes as a baseline, already shows that visual supervision can support both language interaction and dense depth prediction in a single VLM. Combining these abilities through visual supervision is therefore not a distinctive contribution on its own.
DepthVLM’s main technical additions appear to be a DPT-style continuous depth decoder and two-stage supervised adaptation. With the depth head attached, the system already has a complete dense prediction pipeline: it extracts visual features, combines features from multiple levels, and decodes them into depth. Even when methods use the same task-specific training data, DepthVLM may benefit from the LLM’s pretrained knowledge and additional trainable capacity. Better results than vision-only methods are therefore plausible, but they do not necessarily show a distinctive benefit from combining vision and language. The gains could instead come from greater capacity, more pretraining resources, or feature adaptation.
A potentially stronger contribution is whether geometric supervision improves the VLM’s 3D understanding and reasoning, rather than just its depth accuracy. The paper should therefore place more emphasis on the 3D reasoning tasks in Table 4 and Figure 4. Controlled experiments are needed to show whether these gains come from learned geometric representations, rather than extra training or broader data coverage. Establishing this link would take the contribution beyond adding depth prediction to a VLM. It would also offer clearer insight into how geometric perception supports multimodal reasoning.
- The substantive contribution of text and language modeling remains unclear
Training uses depth instructions and fixed text responses, but neither adds scene-specific geometric information. It is unclear whether these texts help improve depth accuracy or preserve the model’s existing capabilities, or simply provide a consistent interaction interface. Three factors should be distinguished: 1. Conditioning depth prediction on the current textual input; 2. Using a language modeling loss during adaptation; 3. Leveraging the LLM’s pretrained priors and additional trainable capacity;
- Comparisons with vision-only models do not adequately isolate methodology, capacity, and training conditions
The paper does not clearly specify the model sizes, checkpoints, or metric-depth configurations used for baselines such as DepthAnythingV3. This makes the results harder to reproduce and interpret. The VLMs and vision-only models also differ substantially in parameter count, pretraining resources, and supervised adaptation settings(Even DAv3 Giant only have ~1.4B params, far less than 4 or 8 B).
Larger models do not necessarily perform better at depth estimation, so comparisons across model sizes are not inherently invalid. However, these results mainly show that the complete system is effective. They do not, on their own, establish that the proposed design or the VLM-based approach is superior.
Questions for authors
- Does geometric supervision genuinely improve 3D reasoning? The 3D reasoning results in Table 4 and Figure 4 may represent the paper’s more distinctive contribution. Could the authors provide controls matched for training data and budget to distinguish improvements from depth-supervised geometric representations versus generic additional fine-tuning?
- Do the depth gains reflect LLM priors or additional trainable capacity? With the depth head attached, the system already has a complete dense prediction pipeline. Could the authors clarify which components are trainable in the no-LLM control? Could they also compare against a visual encoder with a similar depth head, using the same depth data and a comparable adaptation budget, while allowing the visual encoder to update? This would help show whether the LLM adds value beyond extra capacity and supervised adaptation.
- What are the exact baseline and cross-domain evaluation configurations? Could the authors specify the checkpoints, model sizes, and metric-depth configurations used for vision-only baselines such as DepthAnythingV3? Please also clarify what camera information is provided, how metric scale is recovered, and what preprocessing is applied. Could the results be reported separately for datasets from training sources and those from unseen sources? Reporting the full system’s parameter count and inference cost would also help clarify the trade-off between performance and resource use.
6 comments
Response to Reviewer PVQH (2/4)
W2.1: Conditioning depth prediction on the current textual input.
Thank you for encouraging us to distinguish these roles more clearly. Our depth branch predicts metric geometry from intermediate visual features and the LLM’s image-token representations. In the current implementation, image tokens precede the task instruction under standard causal attention. These representations therefore incorporate preceding context but are not conditioned on the subsequent task instruction.
The depth instructions and fixed responses establish a consistent task interface; they do not provide additional scene-specific geometric information. Geometric supervision comes from the ground-truth depth maps. The language modeling objective serves a separate training-time role, whose contribution to capability retention we examine through the controlled ablations in W2.2. Likewise, the contribution of pretrained LLM representations and trainable capacity is distinct from inference-time instruction conditioning and is addressed separately in W2.3 & Q2.
We will revise the manuscript to make these distinctions explicit and clarify that our contribution concerns adapting shared VLM representations for dense geometry and multimodal understanding, without implying that the depth output is conditioned on subsequent task instructions.
W2.2: Using a language modeling loss during adaptation.
The language modeling loss serves a different purpose from inference-time text conditioning: it is included during adaptation to help preserve the VLM’s existing general multimodal understanding capabilities. To examine its contribution, we compare text-only, depth-only, and joint adaptation, starting from the same Stage-1-aligned checkpoint and using the same training images, training steps, and optimization settings.
Table 2. Effect of training objectives on depth estimation, general multimodal understanding, and spatial reasoning.
| Training Setting | Text Loss | Depth Loss | δ₁ ↑ | RMSE (m) ↓ | MMBench-EN ↑ | MMStar ↑ | BLINK ↑ | VSI-Bench ↑ | CV-Bench 3D ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Text-only Adaptation | ✓ | ✗ | 0.715 | 0.620 | 83.3 | 60.3 | 63.6 | 50.1 | 88.7 |
| Depth-only Adaptation | ✗ | ✓ | 0.850 | 0.405 | 80.7 | 58.6 | 61.2 | 55.8 | 93.3 |
| Joint Adaptation | ✓ | ✓ | 0.868 | 0.390 | 82.9 | 60.4 | 63.3 | 56.2 | 93.8 |
As shown in Table 2, text-only adaptation maintains stronger general multimodal performance than depth-only adaptation but does not provide the benefits of further geometric supervision. Compared with text-only adaptation, joint adaptation substantially improves depth accuracy and spatial reasoning while achieving comparable general multimodal performance. Compared with depth-only adaptation, adding the language modeling loss improves all three general multimodal benchmarks, alongside smaller gains in depth accuracy and spatial reasoning. These results support the role of language modeling supervision in helping retain general multimodal capabilities during geometric adaptation, with joint training providing the best overall balance across the evaluated tasks.
W2.3 & Q2: Leveraging the LLM’s pretrained priors and additional trainable capacity.
In the original no-LLM control, we remove the LLM from Qwen3-VL-4B and attach the same DPT-style depth head directly to the vision encoder. The vision encoder remains frozen, and only the depth head is trained. Following your valuable suggestion, we add a stronger vision-only control in which both the vision encoder and depth head are trainable, using the same depth training data and a matched adaptation budget.
Table 3. Depth estimation accuracy (δ₁ ↑) under different adaptation settings.
| Method | ViT | LLM | Depth head | nuScenes | Waymo | ETH3D | iBims-1 | SUN RGB-D | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| ViT + DPT | Frozen | Absent | Trainable | 0.712 | 0.758 | 0.507 | 0.777 | 0.798 | 0.710 |
| ViT + DPT | Trainable | Absent | Trainable | 0.750 | 0.805 | 0.720 | 0.816 | 0.840 | 0.786 |
| DepthVLM-4B | Frozen | Trainable | Trainable | 0.821 | 0.879 | 0.924 | 0.912 | 0.882 | 0.884 |
As shown in Table 3, updating the vision encoder improves average δ₁ from 0.710 to 0.786, while DepthVLM-4B achieves 0.884 and consistently outperforms it across all five datasets under matched depth data and adaptation settings. These results demonstrate that the gains extend beyond the frozen-encoder setting, highlighting the benefits of retaining and adapting the pretrained LLM, complementary to our separate analyses of inference-time text conditioning and training-time language modeling.
Response to Reviewer PVQH (3/4)
W3 & Q3.1: Baseline checkpoints, model sizes, and metric-depth configurations.
As suggested, we summarize the checkpoints, parameter counts, reported training resources, and metric-depth configurations of the vision-only baselines below. We will include these details in the revised paper to improve the transparency and reproducibility of our baseline and cross-domain evaluations.
Table 4. Checkpoints, parameter counts, reported training resources, and metric-depth inference configurations of the vision-only baselines used in our evaluation.
| Baseline | Checkpoint | Parameters | Reported Training Resources | Metric-depth Configuration |
|---|---|---|---|---|
| Depth Pro | depth_pro.pt | 0.95B | DINOv2 backbone; 21 depth datasets. | Official inference with dataset-provided focal length. |
| Metric3Dv2 | metric_depth_vit_giant2_800k.pth | 1.38B | DINOv2 backbone; 18 datasets, >16M images. | Canonical depth scaled by resized focal length / 1,000. |
| UniDepthV2 | unidepth-v2-vitl14 | 0.35B | DINOv2 backbone; 23 datasets, ~16M images. | Dataset-provided intrinsics; resolution_level=9. |
| DepthAnythingV3 | DA3METRIC-LARGE | 0.35B | DINOv2 backbone; 14 metric-depth datasets. | Canonical depth scaled by resized mean focal length / 300. |
Q3.2: Camera information, metric scale, and preprocessing.
Thank you for the suggestion. We clarify the settings for our quantitative and qualitative comparisons below.
Camera information: For quantitative evaluation, each method uses the camera information specified in the baseline configurations above. For point-cloud reconstruction, all methods use the dataset-provided ground-truth camera intrinsics to back-project their corresponding predicted depth maps.
Metric scale: Predictions are converted to meters according to each method’s metric-depth configuration. No additional scale or shift alignment against ground-truth depth is applied before evaluation or point-cloud reconstruction.
Preprocessing and visualization: For the indoor scenes in Figure 5, we retain only points with finite predicted depths satisfying (0.001 < D < 10) m. All methods share the same metric depth range and color normalization for depth-map visualization. Each reconstructed point receives the color of its corresponding RGB pixel, and all point clouds are rendered with identical viewing and visualization settings.
Q3.3: Separate results for training-source and unseen-source datasets.
Thank you for this suggestion. We reorganize the results below to distinguish evaluation on datasets whose training splits were used for DepthVLM’s depth adaptation (Seen) from evaluation on datasets excluded from that adaptation (Unseen) .
Table 5. Comparison with VLM baselines, grouped by DepthVLM’s training-source coverage.
| Method | Argoverse2 | Waymo | DDAD | nuScenes | ScanNet++ | Seen Avg. | ETH3D | SUN RGB-D | iBims-1 | NYUv2 | Unseen Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-4B | 0.033 | 0.006 | 0.086 | 0.040 | 0.208 | 0.075 | 0.085 | 0.220 | 0.037 | 0.155 | 0.124 |
| Youtu-VL-4B | 0.663 | 0.473 | 0.342 | 0.698 | 0.522 | 0.540 | 0.286 | 0.734 | 0.856 | 0.849 | 0.681 |
| DepthLM-12B | 0.761 | 0.588 | 0.654 | 0.736 | 0.756 | 0.699 | 0.666 | 0.785 | 0.754 | 0.866 | 0.768 |
| Ours-4B | 0.810 | 0.879 | 0.818 | 0.821 | 0.861 | 0.838 | 0.924 | 0.882 | 0.912 | 0.908 | 0.907 |
Table 6. Comparison with vision-only baselines using the same source-based grouping.
| Method | Waymo | nuScenes | Seen Avg. | ETH3D | SUN RGB-D | iBims-1 | Unseen Avg. |
|---|---|---|---|---|---|---|---|
| Depth Pro | 0.255 | 0.389 | 0.322 | 0.355 | 0.852 | 0.880 | 0.696 |
| Metric3Dv2 | 0.923 | 0.747 | 0.835 | 0.851 | 0.813 | 0.726 | 0.797 |
| UniDepthV2 | 0.730 | 0.872 | 0.801 | 0.657 | 0.911 | 0.941 | 0.836 |
| DepthAnythingV3 | 0.885 | 0.790 | 0.838 | 0.843 | 0.913 | 0.955 | 0.904 |
| Ours-4B | 0.879 | 0.821 | 0.850 | 0.924 | 0.882 | 0.912 | 0.906 |
DepthVLM-4B achieves the highest average δ₁ in both groups within each comparison, showing that its effectiveness extends to datasets excluded from its depth adaptation. We will incorporate this separation and explicitly define the source coverage in the revised paper so that readers can distinguish performance on familiar dataset sources from cross-dataset generalization.
Response to Reviewer PVQH (1/4)
We sincerely thank you for your careful review and positive feedback. We greatly appreciate your recognition of our method's practical value in extending VLM capabilities for spatial reasoning tasks, straightforward design, and comprehensive evaluation. We appreciate the opportunity to clarify these points and respond to your concerns and questions below.
Thank you for encouraging us to clarify the scope of our contribution. Our contribution centers on equipping an existing general-purpose VLM with continuous, pixel-level metric depth prediction while largely preserving its general multimodal capabilities and enhancing its 3D spatial understanding and reasoning. Youtu-VL predicts discretized depth bins through visual-token logits and spatial upsampling, within a framework developed through large-scale, multi-stage pretraining from scratch. In comparison, DepthVLM combines multi-scale visual features and the LLM’s final visual-token representations through a lightweight continuous depth decoder, and uses two-stage supervised adaptation without requiring access to the backbone’s original multimodal training data. The first stage initializes the decoder with the VLM frozen; the second jointly optimizes the decoder and LLM under depth and text supervision while keeping the vision encoder frozen. Our ablations examine how these design choices affect depth accuracy and multimodal capability retention.
Compared with the vision-only baselines in our evaluation, our method uses substantially less task-specific depth-training data and markedly improves the original pretrained VLM’s metric-depth performance. These results demonstrate the effectiveness and data efficiency of our proposed adaptation framework. As also noted by Reviewer
hdwU, our framework introduces dense geometric prediction while largely preserving the VLM’s existing capabilities, without training a new VLM from scratch or revisiting its original multimodal training corpus. Its applicability across VLM families and model sizes further supports the generality of this adaptation approach.As suggested, we will revise the manuscript to more clearly distinguish our adaptation framework from existing unified VLMs and precisely articulate the scope of our methodological contributions.
Thank you for this valuable suggestion. We fully agree that establishing how geometric supervision improves a VLM’s 3D understanding and reasoning is a particularly distinctive aspect of our contribution. Following your suggestion, we conduct controlled experiments on Qwen3-VL-4B with three adaptation objectives: text-only, depth-only, and joint supervision. All three settings start from the same checkpoint after Stage-1 depth-head alignment and use identical training images, training steps, optimizer settings, and learning rates. This isolates the contribution of the supervision objectives during Stage-2 under matched data exposure and optimization schedules.
Text-only adaptation uses only the language modeling loss, testing whether exposure to the same images and additional text-based fine-tuning can explain the reasoning improvements. Depth-only adaptation uses only the depth loss, testing whether geometric supervision improves spatial QA without a language modeling objective. Joint adaptation combines both losses and corresponds to our full method.
Table 1. Controlled experiments of geometric supervision on 3D spatial understanding and reasoning.
As shown in Table 1, text-only adaptation reduces performance on both benchmarks, indicating that additional image exposure and text-based fine-tuning alone do not explain the gains. This degradation is consistent with excessive specialization to the fixed depth-query response format. In contrast, depth-only adaptation improves VSI-Bench and CV-Bench-3D by 1.9 and 2.8 points over the original VLM, respectively, providing direct evidence that geometric supervision can benefit spatial reasoning without an additional language modeling loss. Joint adaptation achieves the strongest results, with improvements of 2.3 and 3.3 points, respectively. Together, these controlled results support the contribution of dense geometric supervision to multimodal spatial reasoning beyond depth prediction itself.
As suggested, we will incorporate these experiments and analyses into the main paper to clarify how geometric perception improves higher-level 3D understanding and reasoning, offering clearer insights that can inform future research.