acceptodds
Under review as a conference paper at ICLR 2027

Unlocking Dense Metric Depth Estimation in VLMs

Abstract

Vision–Language Models (VLMs) excel at 2D tasks such as grounding and captioning, yet remain limited in 3D understanding. A key limitation is their text-only supervision paradigm, which under-constrains fine-grained visual perception and prevents the recovery of dense geometry. Prior methods either distill geometry from external vision models, introducing error accumulation, or enable direct prediction with inefficient per-pixel query or coarse token-level outputs. In this paper, we propose DepthVLM, a simple yet effective framework that transforms a single VLM into a native dense geometry predictor while preserving its multimodal capability. By attaching a lightweight depth head to the LLM backbone and training under a unified vision–text supervision paradigm with a two-stage schedule, DepthVLM generates full-resolution depth maps alongside language outputs in a single forward pass. We further introduce a unified indoor–outdoor metric depth benchmark in a VLM-compatible format. Experiments show that DepthVLM significantly outperforms existing VLMs with higher inference efficiency, surpasses leading pure vision models, and improves complex 3D spatial reasoning, moving toward a truly unified foundation model. All code and checkpoints will be publicly released.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal foundation model, 3D spatial reasoning, 3D scene understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PVQHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DepthVLM adds dense metric depth prediction to a pretrained vision-language model using a DPT-style decoder. The decoder combines intermediate features from the visual encoder with image-token representations from the LLM. Training has two stages: first, only the depth head is trained; then, the LLM and depth head are trained together. The visual encoder remains frozen throughout.

The evaluation tests depth prediction, retention of general multimodal capabilities, and spatial reasoning. It also includes ablations of the architecture and training strategy.

Critical strengths
  1. Practically useful capability extension: This work shows the capability of depth labels in VLMs for many additional spatial reasoning tasks.
  2. Straightforward design and broad evaluation: The approach uses an established decoder and staged training, making it easy to understand and potentially straightforward to integrate into existing VLMs. Tests on multiple depth datasets provide strong evidence that the full system works well. Evaluation over all valid pixels in the full image and relevant ablation studies further support these results.
Critical weaknesses
  1. Limited methodological differentiation; the evaluation should place greater emphasis on 3D reasoning

Youtu-VL, which the paper includes as a baseline, already shows that visual supervision can support both language interaction and dense depth prediction in a single VLM. Combining these abilities through visual supervision is therefore not a distinctive contribution on its own.

DepthVLM’s main technical additions appear to be a DPT-style continuous depth decoder and two-stage supervised adaptation. With the depth head attached, the system already has a complete dense prediction pipeline: it extracts visual features, combines features from multiple levels, and decodes them into depth. Even when methods use the same task-specific training data, DepthVLM may benefit from the LLM’s pretrained knowledge and additional trainable capacity. Better results than vision-only methods are therefore plausible, but they do not necessarily show a distinctive benefit from combining vision and language. The gains could instead come from greater capacity, more pretraining resources, or feature adaptation.

A potentially stronger contribution is whether geometric supervision improves the VLM’s 3D understanding and reasoning, rather than just its depth accuracy. The paper should therefore place more emphasis on the 3D reasoning tasks in Table 4 and Figure 4. Controlled experiments are needed to show whether these gains come from learned geometric representations, rather than extra training or broader data coverage. Establishing this link would take the contribution beyond adding depth prediction to a VLM. It would also offer clearer insight into how geometric perception supports multimodal reasoning.

  1. The substantive contribution of text and language modeling remains unclear

Training uses depth instructions and fixed text responses, but neither adds scene-specific geometric information. It is unclear whether these texts help improve depth accuracy or preserve the model’s existing capabilities, or simply provide a consistent interaction interface. Three factors should be distinguished: 1. Conditioning depth prediction on the current textual input; 2. Using a language modeling loss during adaptation; 3. Leveraging the LLM’s pretrained priors and additional trainable capacity;

  1. Comparisons with vision-only models do not adequately isolate methodology, capacity, and training conditions

The paper does not clearly specify the model sizes, checkpoints, or metric-depth configurations used for baselines such as DepthAnythingV3. This makes the results harder to reproduce and interpret. The VLMs and vision-only models also differ substantially in parameter count, pretraining resources, and supervised adaptation settings(Even DAv3 Giant only have ~1.4B params, far less than 4 or 8 B).

Larger models do not necessarily perform better at depth estimation, so comparisons across model sizes are not inherently invalid. However, these results mainly show that the complete system is effective. They do not, on their own, establish that the proposed design or the VLM-based approach is superior.

Questions for authors
  1. Does geometric supervision genuinely improve 3D reasoning? The 3D reasoning results in Table 4 and Figure 4 may represent the paper’s more distinctive contribution. Could the authors provide controls matched for training data and budget to distinguish improvements from depth-supervised geometric representations versus generic additional fine-tuning?
  2. Do the depth gains reflect LLM priors or additional trainable capacity? With the depth head attached, the system already has a complete dense prediction pipeline. Could the authors clarify which components are trainable in the no-LLM control? Could they also compare against a visual encoder with a similar depth head, using the same depth data and a comparable adaptation budget, while allowing the visual encoder to update? This would help show whether the LLM adds value beyond extra capacity and supervised adaptation.
  3. What are the exact baseline and cross-domain evaluation configurations? Could the authors specify the checkpoints, model sizes, and metric-depth configurations used for vision-only baselines such as DepthAnythingV3? Please also clarify what camera information is provided, how metric scale is recovered, and what preprocessing is applied. Could the results be reported separately for datasets from training sources and those from unseen sources? Reporting the full system’s parameter count and inference cost would also help clarify the trade-off between performance and resource use.
6 comments
AuthorsOpenReview ↗

Response to Reviewer PVQH (1/4)

We sincerely thank you for your careful review and positive feedback. We greatly appreciate your recognition of our method's practical value in extending VLM capabilities for spatial reasoning tasks, straightforward design, and comprehensive evaluation. We appreciate the opportunity to clarify these points and respond to your concerns and questions below.


W1.1: Methodological differentiation and contribution beyond existing unified VLMs.

Thank you for encouraging us to clarify the scope of our contribution. Our contribution centers on equipping an existing general-purpose VLM with continuous, pixel-level metric depth prediction while largely preserving its general multimodal capabilities and enhancing its 3D spatial understanding and reasoning. Youtu-VL predicts discretized depth bins through visual-token logits and spatial upsampling, within a framework developed through large-scale, multi-stage pretraining from scratch. In comparison, DepthVLM combines multi-scale visual features and the LLM’s final visual-token representations through a lightweight continuous depth decoder, and uses two-stage supervised adaptation without requiring access to the backbone’s original multimodal training data. The first stage initializes the decoder with the VLM frozen; the second jointly optimizes the decoder and LLM under depth and text supervision while keeping the vision encoder frozen. Our ablations examine how these design choices affect depth accuracy and multimodal capability retention.

Compared with the vision-only baselines in our evaluation, our method uses substantially less task-specific depth-training data and markedly improves the original pretrained VLM’s metric-depth performance. These results demonstrate the effectiveness and data efficiency of our proposed adaptation framework. As also noted by Reviewer hdwU, our framework introduces dense geometric prediction while largely preserving the VLM’s existing capabilities, without training a new VLM from scratch or revisiting its original multimodal training corpus. Its applicability across VLM families and model sizes further supports the generality of this adaptation approach.

As suggested, we will revise the manuscript to more clearly distinguish our adaptation framework from existing unified VLMs and precisely articulate the scope of our methodological contributions.

W1.2 & Q1: Greater emphasis on geometric supervision for 3D understanding and reasoning.

Thank you for this valuable suggestion. We fully agree that establishing how geometric supervision improves a VLM’s 3D understanding and reasoning is a particularly distinctive aspect of our contribution. Following your suggestion, we conduct controlled experiments on Qwen3-VL-4B with three adaptation objectives: text-only, depth-only, and joint supervision. All three settings start from the same checkpoint after Stage-1 depth-head alignment and use identical training images, training steps, optimizer settings, and learning rates. This isolates the contribution of the supervision objectives during Stage-2 under matched data exposure and optimization schedules.

Text-only adaptation uses only the language modeling loss, testing whether exposure to the same images and additional text-based fine-tuning can explain the reasoning improvements. Depth-only adaptation uses only the depth loss, testing whether geometric supervision improves spatial QA without a language modeling objective. Joint adaptation combines both losses and corresponds to our full method.

Table 1. Controlled experiments of geometric supervision on 3D spatial understanding and reasoning.

SettingTraining ImagesText LossDepth LossVSI-Bench ↑CV-Bench 3D ↑
Original Qwen3-VL-4BNo additional training——53.990.5
Text-only AdaptationMatched✓✗50.188.7
Depth-only AdaptationMatched✗✓55.893.3
Joint Adaptation (DepthVLM)Matched✓✓56.293.8

As shown in Table 1, text-only adaptation reduces performance on both benchmarks, indicating that additional image exposure and text-based fine-tuning alone do not explain the gains. This degradation is consistent with excessive specialization to the fixed depth-query response format. In contrast, depth-only adaptation improves VSI-Bench and CV-Bench-3D by 1.9 and 2.8 points over the original VLM, respectively, providing direct evidence that geometric supervision can benefit spatial reasoning without an additional language modeling loss. Joint adaptation achieves the strongest results, with improvements of 2.3 and 3.3 points, respectively. Together, these controlled results support the contribution of dense geometric supervision to multimodal spatial reasoning beyond depth prediction itself.

As suggested, we will incorporate these experiments and analyses into the main paper to clarify how geometric perception improves higher-level 3D understanding and reasoning, offering clearer insights that can inform future research.

AuthorsOpenReview ↗

Response to Reviewer PVQH (2/4)

W2.1: Conditioning depth prediction on the current textual input.

Thank you for encouraging us to distinguish these roles more clearly. Our depth branch predicts metric geometry from intermediate visual features and the LLM’s image-token representations. In the current implementation, image tokens precede the task instruction under standard causal attention. These representations therefore incorporate preceding context but are not conditioned on the subsequent task instruction.

The depth instructions and fixed responses establish a consistent task interface; they do not provide additional scene-specific geometric information. Geometric supervision comes from the ground-truth depth maps. The language modeling objective serves a separate training-time role, whose contribution to capability retention we examine through the controlled ablations in W2.2. Likewise, the contribution of pretrained LLM representations and trainable capacity is distinct from inference-time instruction conditioning and is addressed separately in W2.3 & Q2.

We will revise the manuscript to make these distinctions explicit and clarify that our contribution concerns adapting shared VLM representations for dense geometry and multimodal understanding, without implying that the depth output is conditioned on subsequent task instructions.

W2.2: Using a language modeling loss during adaptation.

The language modeling loss serves a different purpose from inference-time text conditioning: it is included during adaptation to help preserve the VLM’s existing general multimodal understanding capabilities. To examine its contribution, we compare text-only, depth-only, and joint adaptation, starting from the same Stage-1-aligned checkpoint and using the same training images, training steps, and optimization settings.

Table 2. Effect of training objectives on depth estimation, general multimodal understanding, and spatial reasoning.

Training SettingText LossDepth Lossδ₁ ↑RMSE (m) ↓MMBench-EN ↑MMStar ↑BLINK ↑VSI-Bench ↑CV-Bench 3D ↑
Text-only Adaptation✓✗0.7150.62083.360.363.650.188.7
Depth-only Adaptation✗✓0.8500.40580.758.661.255.893.3
Joint Adaptation✓✓0.8680.39082.960.463.356.293.8

As shown in Table 2, text-only adaptation maintains stronger general multimodal performance than depth-only adaptation but does not provide the benefits of further geometric supervision. Compared with text-only adaptation, joint adaptation substantially improves depth accuracy and spatial reasoning while achieving comparable general multimodal performance. Compared with depth-only adaptation, adding the language modeling loss improves all three general multimodal benchmarks, alongside smaller gains in depth accuracy and spatial reasoning. These results support the role of language modeling supervision in helping retain general multimodal capabilities during geometric adaptation, with joint training providing the best overall balance across the evaluated tasks.

W2.3 & Q2: Leveraging the LLM’s pretrained priors and additional trainable capacity.

In the original no-LLM control, we remove the LLM from Qwen3-VL-4B and attach the same DPT-style depth head directly to the vision encoder. The vision encoder remains frozen, and only the depth head is trained. Following your valuable suggestion, we add a stronger vision-only control in which both the vision encoder and depth head are trainable, using the same depth training data and a matched adaptation budget.

Table 3. Depth estimation accuracy (δ₁ ↑) under different adaptation settings.

MethodViTLLMDepth headnuScenesWaymoETH3DiBims-1SUN RGB-DAvg.
ViT + DPTFrozenAbsentTrainable0.7120.7580.5070.7770.7980.710
ViT + DPTTrainableAbsentTrainable0.7500.8050.7200.8160.8400.786
DepthVLM-4BFrozenTrainableTrainable0.8210.8790.9240.9120.8820.884

As shown in Table 3, updating the vision encoder improves average δ₁ from 0.710 to 0.786, while DepthVLM-4B achieves 0.884 and consistently outperforms it across all five datasets under matched depth data and adaptation settings. These results demonstrate that the gains extend beyond the frozen-encoder setting, highlighting the benefits of retaining and adapting the pretrained LLM, complementary to our separate analyses of inference-time text conditioning and training-time language modeling.

AuthorsOpenReview ↗

Response to Reviewer PVQH (3/4)

W3 & Q3.1: Baseline checkpoints, model sizes, and metric-depth configurations.

As suggested, we summarize the checkpoints, parameter counts, reported training resources, and metric-depth configurations of the vision-only baselines below. We will include these details in the revised paper to improve the transparency and reproducibility of our baseline and cross-domain evaluations.

Table 4. Checkpoints, parameter counts, reported training resources, and metric-depth inference configurations of the vision-only baselines used in our evaluation.

BaselineCheckpointParametersReported Training ResourcesMetric-depth Configuration
Depth Prodepth_pro.pt0.95BDINOv2 backbone; 21 depth datasets.Official inference with dataset-provided focal length.
Metric3Dv2metric_depth_vit_giant2_800k.pth1.38BDINOv2 backbone; 18 datasets, >16M images.Canonical depth scaled by resized focal length / 1,000.
UniDepthV2unidepth-v2-vitl140.35BDINOv2 backbone; 23 datasets, ~16M images.Dataset-provided intrinsics; resolution_level=9.
DepthAnythingV3DA3METRIC-LARGE0.35BDINOv2 backbone; 14 metric-depth datasets.Canonical depth scaled by resized mean focal length / 300.

Q3.2: Camera information, metric scale, and preprocessing.

Thank you for the suggestion. We clarify the settings for our quantitative and qualitative comparisons below.

Camera information: For quantitative evaluation, each method uses the camera information specified in the baseline configurations above. For point-cloud reconstruction, all methods use the dataset-provided ground-truth camera intrinsics to back-project their corresponding predicted depth maps.

Metric scale: Predictions are converted to meters according to each method’s metric-depth configuration. No additional scale or shift alignment against ground-truth depth is applied before evaluation or point-cloud reconstruction.

Preprocessing and visualization: For the indoor scenes in Figure 5, we retain only points with finite predicted depths satisfying (0.001 < D < 10) m. All methods share the same metric depth range and color normalization for depth-map visualization. Each reconstructed point receives the color of its corresponding RGB pixel, and all point clouds are rendered with identical viewing and visualization settings.

Q3.3: Separate results for training-source and unseen-source datasets.

Thank you for this suggestion. We reorganize the results below to distinguish evaluation on datasets whose training splits were used for DepthVLM’s depth adaptation (Seen) from evaluation on datasets excluded from that adaptation (Unseen) .

Table 5. Comparison with VLM baselines, grouped by DepthVLM’s training-source coverage.

MethodArgoverse2WaymoDDADnuScenesScanNet++Seen Avg.ETH3DSUN RGB-DiBims-1NYUv2Unseen Avg.
Qwen3-VL-4B0.0330.0060.0860.0400.2080.0750.0850.2200.0370.1550.124
Youtu-VL-4B0.6630.4730.3420.6980.5220.5400.2860.7340.8560.8490.681
DepthLM-12B0.7610.5880.6540.7360.7560.6990.6660.7850.7540.8660.768
Ours-4B0.8100.8790.8180.8210.8610.8380.9240.8820.9120.9080.907

Table 6. Comparison with vision-only baselines using the same source-based grouping.

MethodWaymonuScenesSeen Avg.ETH3DSUN RGB-DiBims-1Unseen Avg.
Depth Pro0.2550.3890.3220.3550.8520.8800.696
Metric3Dv20.9230.7470.8350.8510.8130.7260.797
UniDepthV20.7300.8720.8010.6570.9110.9410.836
DepthAnythingV30.8850.7900.8380.8430.9130.9550.904
Ours-4B0.8790.8210.8500.9240.8820.9120.906

DepthVLM-4B achieves the highest average δ₁ in both groups within each comparison, showing that its effectiveness extends to datasets excluded from its depth adaptation. We will incorporate this separation and explicitly define the source coverage in the revised paper so that readers can distinguish performance on familiar dataset sources from cross-dataset generalization.

Reviewer_hdwUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work enables dense depth estimation by adding depth prediction head to VLMs. The main contribution is 1) a carefully tuned finetuning recipe for maintaining the original VLM capabilities. 2) inference speed for dense estimation.

Critical strengths

As mentioned in the summary, the main contribution of this work lies in 1) maintaining the original VLM capabilities for other tasks, which is not done by mix-data training, but by a 2 stage finetuning recipe with depth head first aligned with the freezed VLM and then in the second stage a small finetuning without touching the vision encoder. 2) speed when dense prediction is needed since a separate prediction head is used.

Though there are some critical issues with the claims and the experiments, I think the paper provides some interesting insights on how to maintain the original VLM capabilities without access to the original model's training data.

Critical weaknesses

I think this work needs either a more careful statement or a fairer comparison to SOTA.

First of all, it only compared to the numbers of DepthLM-12B without mentioning the 4B and 8B version with the same backbone, though I understand that probably the model weights of 4B and 8B are not released, but the reported numbers on most datasets should still be included, a footnote can be used to state the reason why datasets like Waymo cannot be evaluated for those models.

Secondly, the paper trains on the training split of all outdoor eval data, whereas competitors like DepthLM didnt use DDAD for training, this makes it hard to assess the zero-shot generalization capability of the model, and the numbers are not directly comparable. So a clear statement on the training and evaluation difference in the comparison table is needed so that the readers do not misunderstand.

Questions for authors

The paper writing is clear, I dont have any question.

5 comments
AuthorsOpenReview ↗

Response to Reviewer hdwU (1/2)

We sincerely thank you for recognizing the value of our carefully designed training recipe, which preserves the original VLM’s capabilities without access to its original training data while enabling dense depth estimation with fast inference. We appreciate the opportunity to clarify these aspects of our work and address your concerns and questions below.


W1: Comparison with additional DepthLM variants at comparable model sizes.

We sincerely thank you for this helpful suggestion. We appreciate that including the results of DepthLM’s smaller variants, despite the unavailability of their checkpoints, would provide a more comprehensive comparison.

To clarify, the smaller variants reported in the DepthLM paper are based on Qwen2.5-VL-3B and Qwen2.5-VL-7B, whereas our models use Qwen3-VL-4B and Qwen3-VL-8B. These are the closest reported DepthLM variants in terms of model family and parameter count. Currently, the official DepthLM release only provides the checkpoint fine-tuned from Pixtral-12B, while the 3B and 7B checkpoints remain unavailable. We therefore include their published results in Table 1 below and explicitly distinguish these numbers from results obtained under our evaluation protocol.

Table 1. Expanded comparison of metric depth estimation accuracy (δ1 ↑). Avg. (5) averages all five datasets, while Avg. (4) excludes Waymo to accommodate DepthLM variants with unavailable checkpoints.

MethodWaymoNuScenesETH3DSUN RGB-DiBims-1Avg. (5)Avg. (4)
Depth Pro0.2550.3890.3550.8520.8800.5460.619
Metric3Dv20.9230.7470.8510.8130.7260.8120.784
UniDepthV20.7300.8720.6570.9110.9410.8230.845
DepthAnythingV30.8850.7900.8430.9130.9550.8770.875
Youtu-VL-4B0.4730.6980.2860.7340.8560.6090.644
DepthLM-3B†—0.8700.7450.8500.890—0.839
DepthLM-7B†—0.8650.7180.8590.920—0.841
DepthLM-12B0.5880.7360.6660.7850.7540.7060.735
Ours-4B0.8790.8210.9240.8820.9120.884‾\underline{0.884}0.885‾\underline{0.885}
Ours-8B0.8650.8310.9280.8890.9360.8900.896

† Results are taken directly from the original DepthLM paper, as the Qwen2.5-VL-based 3B and 7B checkpoints have not been publicly released. We therefore cannot re-evaluate these variants under our unified protocol or obtain their Waymo results. Their published scores are provided for reference and may reflect differences in evaluation samples and settings. For these two variants, “—” indicates a result that is neither reported in the original paper nor obtainable without access to the checkpoints. Avg. (4) uses the same four dataset names for all methods.

We will incorporate this expanded table and its explanatory footnote into the main paper, and revise our state-of-the-art claims to reflect these dataset-specific results and the limitations of comparisons across different backbones and evaluation protocols.

AuthorsOpenReview ↗

Response to Reviewer hdwU (2/2)

W2: Clarifying training-data differences and evaluating the zero-shot generalization capability.

We sincerely thank you for highlighting the importance of training-data transparency when comparing with prior work. We fully agree that evaluation on held-out images from datasets used during training should be distinguished from evaluation on entirely unseen datasets, and that differences in training data should be made explicit when comparing with DepthLM and other methods.

Our model uses the training splits of four outdoor datasets—Argoverse2, Waymo, DDAD, and NuScenes—and four indoor datasets—ScanNet++, Taskonomy, HM3D, and Matterport3D. Although evaluation images are excluded from training, results on datasets included in this mixture measure within-dataset generalization rather than zero-shot cross-dataset generalization. We will revise the comparison tables and accompanying discussion to indicate, for each method, whether each evaluation dataset was used during depth training, based on the available training-data descriptions, and explicitly mark cases where this information is undisclosed. For example, DepthLM reports using Argoverse2, Waymo, and NuScenes for outdoor training, and ScanNet++, Taskonomy, HM3D, and Matterport3D for indoor training, but does not include DDAD. In contrast, ETH3D, SUN RGB-D, iBims-1, and NYUv2 are excluded from task-specific depth training for both our method and DepthLM.

To separately assess our models’ zero-shot generalization on cross-domain datasets, Table 2 reports performance on ETH3D, SUN RGB-D, iBims-1, and NYUv2, none of which is included in our mixed-source depth training. Our 8B and 4B models achieve the highest and second-highest average δ₁ scores among the compared methods, at 0.918 and 0.907, respectively. These results support their ability to generalize to datasets unseen during depth training.

Table 2. Metric depth estimation performance (δ₁ ↑) on four datasets unseen during our mixed-source depth training, assessing our models’ zero-shot cross-dataset generalization.

MethodETH3DSUN RGB-DiBims-1NYUv2Avg.
Depth Pro0.3550.8520.8800.8510.735
Metric3Dv20.8510.8130.7260.8650.814
UniDepthV20.6570.9110.9410.9040.853
DepthAnythingV30.8430.9130.9550.9100.905
Youtu-VL-4B0.2860.7340.8560.8490.681
DepthLM-3B†0.7450.8500.8900.8680.838
DepthLM-7B†0.7180.8590.9200.9150.853
DepthLM-12B0.6660.7850.7540.8660.768
Ours-4B0.9240.8820.9120.9080.907‾\underline{0.907}
Ours-8B0.9280.8890.9360.9200.918

† Results are quoted from the original DepthLM paper because the Qwen2.5-VL-based 3B and 7B checkpoints were unavailable for re-evaluation under our unified protocol. These scores are provided for reference and may reflect differences in evaluation samples and settings.

We will incorporate these results into the main paper, explicitly annotate each method’s training-data usage in the comparison tables, and revise the accompanying discussion and claims to clearly distinguish within-dataset performance from zero-shot cross-dataset generalization.


We believe that the expanded comparisons with additional DepthLM variants, explicit clarification of training-data and evaluation-protocol differences, and separate assessment of cross-dataset generalization will strengthen the paper and provide a more transparent basis for interpreting our results. We hope that these responses have addressed your concerns, and we sincerely thank you once again for your thoughtful and constructive feedback.

AuthorsOpenReview ↗

Look forward to your post-rebuttal feedback

Dear Reviewer hdwU,

Thank you again for your valuable time and effort in reviewing our submission. We sincerely hope that our responses have sufficiently addressed the concerns and points you raised. As the discussion stage draws to a close, we would like to respectfully follow up and would greatly appreciate any further feedback or suggestions you may have. We would be happy to provide additional clarifications if needed.

We sincerely appreciate your thoughtful feedback and continued engagement throughout the review and discussion process.

Best regards,

The Authors

Reviewer_7dPZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates dense metric depth estimation within VLMs, aiming to enable pixel-level geometric prediction while preserving general multimodal understanding capabilities. The authors propose DepthVLM, a framework that integrates a lightweight DPT-style depth head into a pretrained VLM, combining multi-scale ViT features with the LLM's final visual-token representations to predict dense metric depth maps in a single forward pass. A two-stage training strategy is adopted to balance depth estimation accuracy and general multimodal capabilities, while focal-length normalization is employed to facilitate training across heterogeneous datasets. Experiments demonstrate competitive metric depth estimation performance against specialized vision models, substantial efficiency improvements over existing depth-generative VLMs, and improvements in downstream 3D spatial reasoning tasks.

Critical strengths
  • The paper leverages VLMs to perform dense metric depth estimation while preserving their general multimodal understanding capabilities. Unifying low-level geometric perception and high-level language understanding is an interesting and practically valuable research direction.
  • The proposed DepthVLM achieves substantial improvements over existing VLM-based depth estimation methods and demonstrates competitive performance against specialized pure vision models.
Critical weaknesses
  1. Although the paper demonstrates promising results, its methodological contributions appear relatively incremental. The proposed framework primarily integrates established techniques, including a DPT-style depth decoder, multi-scale feature fusion, SILog supervision, focal-length normalization, and two-stage fine-tuning. Furthermore, existing works such as DepthLM and Youtu-VL have already explored metric depth estimation within VLMs. While the proposed integration is practically useful, the paper does not sufficiently establish a fundamentally new mechanism for dense geometry prediction or vision-language modeling. The authors should more clearly distinguish their methodological contributions from existing dense prediction architectures and geometry-generative VLMs.
  2. The proposed depth head takes intermediate ViT features and the LLM's final hidden states at image-token positions as inputs (Eq. 2). However, the paper does not investigate whether the predicted depth map actually depends on the input text prompt. In particular, if image tokens precede text tokens under standard causal attention, their hidden states cannot access subsequent text instructions. Consequently, the depth prediction branch may effectively operate as an image-only predictor built on a VLM backbone. A simple ablation using different depth-related, irrelevant, and empty prompts on the same images would help clarify the actual contribution of language conditioning.
  3. The proposed DepthVLM-Bench includes several datasets used for both training and evaluation, although the authors employ disjoint training and testing splits. Specifically, five of the nine evaluation datasets are also included in the training corpus. Consequently, the reported average performance combines in-domain and cross-dataset evaluations, while the compared methods have different training-data coverage. For example, on NuScenes, which is included in DepthVLM's training data, DepthVLM-8B achieves a δ1 of 0.831, below UniDepthV2's 0.872 (Table 2). The authors should clearly distinguish in-domain and unseen-dataset results, and explicitly report the training-data coverage of the compared methods to better substantiate their generalization claims.
  4. Despite proposing a dense, pixel-level depth predictor, the main quantitative comparisons rely primarily on only 10 randomly sampled valid pixels per image and report δ1 as the principal metric. Although this protocol accommodates query-based VLM baselines, sparse sampling may overlook local geometric errors, object boundaries, and fine-grained depth structures, while potentially introducing sampling variability. The authors do provide full-image evaluation in Appendix Table 12, which is appreciated. However, comprehensive dense evaluation should receive greater emphasis, ideally including standard metrics such as AbsRel and RMSE, as well as an analysis of sampling stability.
  5. Figure 5 does not clearly indicate whether all depth maps are visualized using a shared absolute depth range and identical color normalization. Moreover, some methods exhibit severely degraded-looking depth maps while their corresponding point clouds remain relatively recognizable, making the relationship between these visualizations difficult to interpret. The authors should clarify the depth visualization and point-cloud reconstruction protocols, including color scales, camera parameters, viewing configurations, and any normalization applied. Additionally, Table 2 should consistently highlight the best and second-best results for each dataset to improve readability.
Questions for authors
  1. Could the authors clarify the actual token ordering and attention mechanism used to extract the LLM image-token features? Can these features attend to the text prompt? Does changing or removing the prompt affect the predicted depth map?
  2. Are the depth maps in Figure 5 rendered using identical metric depth ranges? Are the point clouds reconstructed from the corresponding predicted depths without additional scale alignment or normalization, and are identical visualization settings used across methods?
10 comments
AuthorsOpenReview ↗

Response to Reviewer 7dPZ (1/5)

We sincerely thank you for your careful review and positive feedback. We appreciate your recognition of the practical value of unifying dense geometric perception and high-level language understanding while preserving VLMs’ general multimodal capabilities, as well as our method’s competitive depth estimation performance. We appreciate the opportunity to clarify these aspects of our work and address your concerns and questions below.


W1: Clarifying the methodological contributions and distinctions from prior work.

Thank you for this valuable suggestion. We fully agree that the manuscript should more clearly distinguish the established components we adopt from the methodological contribution of DepthVLM. Our contribution centers on a lightweight adaptation framework that equips a pretrained VLM with native dense metric depth prediction while retaining its general multimodal capabilities and improving downstream 3D spatial reasoning. We clarify the distinctions below.

Compared with specialized dense prediction architectures, models such as UniDepth, DepthAnything, and Metric3D series primarily target geometric prediction and do not themselves provide general language interaction or visual question answering. DepthVLM addresses the additional challenge of integrating dense geometry into an existing multimodal model. Specifically, it combines multi-scale ViT features with the LLM’s final visual-token representations to decode a full-resolution metric depth map while retaining the original language-generation pathway. Our evaluations demonstrate that this adaptation largely preserves the backbone’s general multimodal performance and improves downstream 3D spatial reasoning.

Compared with geometry-generative VLMs, DepthLM formulates depth estimation as textual responses to individual pixel queries without changing the VLM architecture. Recovering a full depth map therefore requires repeated queries, whereas our method predicts all pixels jointly in a single forward pass. In our evaluation of the released DepthLM-12B checkpoint, we also observed depth-specific responses to general VQA prompts, highlighting the importance of explicitly evaluating capability retention. Youtu-VL supports full-image prediction through discretized depth-bin logits on the visual-token grid, followed by spatial upsampling, which requires a large-scale and multi-stage pretraining pipeline from scratch. DepthVLM instead adapts an existing pretrained VLM through continuous, pixel-level depth regression, without requiring a new visual vocabulary or repeating multimodal foundation pretraining.

Our central methodological contribution lies in the adaptation and training design that makes these capabilities coexist. We deliberately keep architectural changes minimal and jointly optimize dense depth supervision and autoregressive text supervision through a shared VLM backbone. The two-stage schedule first initializes the depth head with the backbone frozen, then jointly optimizes the head and LLM while keeping the vision encoder frozen. This design limits disruption from the randomly initialized decoder and subsequently allows geometric supervision to shape the LLM representations. Our feature-fusion and training-strategy ablations support these choices, showing their contributions to depth accuracy and the balance between geometric prediction and multimodal capability retention.

We will revise the manuscript to make these distinctions explicit, clearly acknowledge the established decoder, loss, and normalization components, and substantiate our contribution through the adaptation design and its empirical benefits.

AuthorsOpenReview ↗

Response to Reviewer 7dPZ (2/5)

W2 & Q1: Clarifying token ordering, prompt dependence, and the roles of geometric and language supervision.

Thank you for highlighting this important distinction. We agree that the manuscript should more clearly explain whether language instructions condition the depth predictions and distinguish this from the role of language supervision during training and inference.

1. Token ordering and attention.

To maintain compatibility with existing VLM backbones and facilitate adaptation across model families and sizes, we retain the backbone’s standard causal attention mechanism and arrange the input as a fixed system prompt followed by image tokens and the task instruction. The LLM hidden states at image-token positions can therefore attend to preceding context, including the system prompt, but cannot attend to the subsequent task instruction. Consequently, the depth branch is image-conditioned under fixed preceding context and does not depend on the subsequent task instruction.

To examine prompt dependence, we conduct an ablation on DepthVLM-Bench, holding the images and preceding system prompt fixed while varying only the subsequent task instruction:

  • Default: “Given this image, estimate the metric depth (in meters, rounded to two decimal places) for every pixel of the image.”
  • Depth-related: “Predict a dense metric depth map for this image. Give the depth of every pixel in meters.” and “Estimate the distance along the camera optical axis for each pixel in this image, measured in meters.”
  • Unrelated: “What is the capital of France?” and “Write a short poem about spring.”
  • Empty: An empty task string ("").

Table 1. Effect of inference-time task prompts on DepthVLM-Bench. Images and preceding system context are held fixed.

Inference Promptδ₁ ↑AbsRel ↓RMSE (m) ↓
Default depth prompt0.8680.1240.390
Other depth-related prompts0.8680.1240.390
Unrelated prompts0.8680.1240.390
Empty prompt0.8680.1240.390

Both other depth-related prompts and both unrelated prompts yield the scores shown above. The metrics remain unchanged at the reported precision, consistent with the causal attention structure. This finding concerns only the subsequent task instruction; the preceding system context is held fixed throughout the experiment.

2. Geometric supervision and spatial reasoning.

The absence of conditioning on subsequent instructions does not prevent geometric supervision from shaping the shared LLM representations. During Stage 2, the LLM is trainable, and the depth loss updates its parameters through the image-token hidden states:

∂Ldepth∂θLLM=∂Ldepth∂Himg∂Himg∂θLLM.\frac{\partial \mathcal{L}_{\mathrm{depth}}}{\partial \theta_{\mathrm{LLM}}}=\frac{\partial \mathcal{L}_{\mathrm{depth}}}{\partial H_{\mathrm{img}}}\frac{\partial H_{\mathrm{img}}}{\partial \theta_{\mathrm{LLM}}}.

This gradient path does not require image tokens to attend to subsequent instructions. Meanwhile, subsequent language-generation tokens can attend to the image representations when answering spatial questions. Geometric supervision can therefore benefit spatial reasoning without making the depth output instruction-dependent.

3. The complementary role of language supervision.

The language modeling loss is included during adaptation to help retain general multimodal capabilities. To examine the respective contributions of the two objectives, we compare text-only, depth-only, and joint adaptation using the Qwen3-VL-4B backbone. All variants start from the same Stage-1-aligned checkpoint, with training images, trainable parameters, training steps, and optimization settings matched.

Table 2. Controlled ablation of Stage-2 objectives on depth estimation, general multimodal understanding, and spatial reasoning.

Training SettingText LossDepth Lossδ₁ ↑RMSE (m) ↓MMBench-EN ↑MMStar ↑BLINK ↑VSI-Bench ↑CV-Bench 3D ↑
Text-only Adaptation✓✗0.7150.62083.360.363.650.188.7
Depth-only Adaptation✗✓0.8500.40580.758.661.255.893.3
Joint Adaptation✓✓0.8680.39082.960.463.356.293.8

The language modeling loss helps retain general multimodal capabilities during geometric adaptation. Compared with depth-only adaptation, joint training improves MMBench-EN from 80.7 to 82.9, MMStar from 58.6 to 60.4, and BLINK from 61.2 to 63.3, while also improving depth accuracy and spatial reasoning. These results support the complementary role of language supervision in balancing capability retention with geometric learning.

We will revise the manuscript to explicitly describe the token ordering and attention constraints, include these ablations, and clearly distinguish depth prediction without conditioning on subsequent task instructions from the benefits of geometric and language supervision during adaptation.

AuthorsOpenReview ↗

Response to Reviewer 7dPZ (3/5)

W3: Distinguishing in-domain and unseen-dataset evaluation and clarifying training-data coverage.

Thank you for this valuable suggestion. We agree that the original nine-dataset average combines within-dataset and cross-dataset evaluation and should not be interpreted as a measure of zero-shot generalization alone. We also agree that differences in training-data coverage must be made explicit when comparing methods.

1. Separating within-dataset and cross-dataset results

We reorganize the results into two groups according to DepthVLM’s task-specific depth-training coverage. ScanNet++, Argoverse2, Waymo, DDAD, and NuScenes are included in our training mixture, with disjoint evaluation images. Results on these datasets therefore measure within-dataset generalization. SUN RGB-D, iBims-1, NYUv2, and ETH3D are excluded from our depth training and assess zero-shot cross-dataset generalization. These group assignments describe our model’s training exposure; the corresponding status can differ for each baseline.

Table 3. Metric depth estimation performance (δ₁ ↑) on five datasets included in DepthVLM’s depth-training mixture, evaluated on held-out images.

MethodScanNet++Argoverse2WaymoDDADNuScenesAvg.
GPT-5.50.4320.3780.3680.3040.2760.352
Cambrian-S-7B0.1450.0060.0190.0380.0330.048
Qwen3-VL-32B0.4340.0290.0170.1380.0490.133
InternVL3.5-38B0.4000.1310.1290.2190.1100.198
Youtu-VL-4B0.5220.6630.4730.3420.6980.540
DepthLM-12B0.7560.7610.5880.6540.7360.699
Ours-4B0.8610.8100.8790.8180.8210.838‾\underline{0.838}
Ours-8B0.9010.7980.8650.8130.8310.842

Table 4. Metric depth estimation performance (δ₁ ↑) on four datasets excluded from DepthVLM’s depth training.

MethodSUN RGB-DiBims-1NYUv2ETH3DAvg.
Depth Pro0.8520.8800.8510.3550.735
Metric3Dv20.8130.7260.8650.8510.814
UniDepthV20.9110.9410.9040.6570.853
DepthAnythingV30.9130.9550.9100.8430.905
Youtu-VL-4B0.7340.8560.8490.2860.681
DepthLM-12B0.7850.7540.8660.6660.768
Ours-4B0.8820.9120.9080.9240.907‾\underline{0.907}
Ours-8B0.8890.9360.9200.9280.918

Our 8B and 4B models achieve average δ₁ scores of 0.918 and 0.907, respectively, on the four datasets excluded from our depth training. These results support cross-dataset generalization, while the individual dataset results show that no method consistently performs best everywhere.

2. Explicitly reporting training-data coverage.

To make the comparisons more transparent, we summarize the publicly reported training coverage below. We distinguish datasets explicitly included in training from those absent from published training lists, and mark cases where coverage cannot be established for the evaluated checkpoint.

Table 5. Publicly reported depth-training coverage of the compared methods, with ScanNet and ScanNet++ listed separately.

Method / Reported ConfigurationScanNetScanNet++Argoverse2WaymoDDADNuScenesSUN RGB-DiBims-1NYUv2ETH3D
Depth ProTNNNNNNNNN
Metric3DV2TNTTTNNNNN
UniDepthV2TTTTNNNNNN
DepthAnythingV3NTTTTNNNNN
Youtu-VL-4B†??????????
DepthLM-12BNTTTNTNNNN

T: explicitly included in the reported depth-training mixture. N: not listed in the published training mixture for the stated configuration. ?: insufficient checkpoint-specific information to determine coverage.

† Youtu-VL describes dataset-specific depth training and evaluation involving NYUv2 and DDAD, but its complete dataset-to-checkpoint training coverage is not sufficiently specified to assign definitive labels across datasets. We therefore retain “?” rather than assume zero-shot evaluation.

We will incorporate the separated results and training-coverage annotations into the revised manuscript, specify the evaluated checkpoint variants, and qualify our claims accordingly. In particular, the overall nine-dataset average will be described as aggregate benchmark performance, while our zero-shot generalization claims will be supported specifically by results on datasets excluded from our depth training.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful comments and constructive feedback throughout the review process. Below, we briefly summarize the key outcomes of the author–reviewer discussions.


We appreciate all reviewers’ recognition of the practical value of our work. DepthVLM enables existing VLMs to predict dense metric depth in a single forward pass while largely preserving their multimodal capabilities and enhancing spatial reasoning, without access to the original model's pretraining data. Its competitive depth accuracy and efficient inference establish a practical foundation for integrating geometric perception into general-purpose multimodal models through lightweight geometric adaptation.

Overall, we are encouraged by the reviewers’ positive feedback, which highlights:

  • Capability preservation and efficiency: A carefully designed two-stage fine-tuning strategy preserves existing VLM capabilities while enabling efficient, single-pass dense depth prediction (Reviewers 7dPZ, hdwU).
  • Improved spatial understanding and reasoning: Geometric supervision enhances the VLM’s 3D understanding and spatial reasoning beyond depth estimation itself (Reviewer PVQH).
  • Practical design and competitive performance: The framework effectively unifies low-level dense geometric perception with high-level language understanding while achieving competitive depth accuracy (Reviewers 7dPZ, PVQH).

To address the reviewers’ concerns, we conducted additional experiments and analyses:

  • Controlled supervision ablations under matched training conditions, supporting the contributions of geometric supervision for spatial reasoning and language supervision for capability retention (Reviewer PVQH).
  • Prompt ablations and a trainable vision-only control, clarifying inference-time prompt independence and evaluating the benefit of retaining the LLM (Reviewer 7dPZ).
  • Expanded dense evaluation and sampling stability analysis, reporting additional metrics (AbsRel and RMSE) beyond δ₁ (Reviewer 7dPZ).
  • More transparent comparisons, distinguishing within-dataset and unseen-dataset results and clarifying training data coverage (Reviewers hdwU).

Summary of Author–Reviewer Discussion for the AC

Following the rebuttal, both Reviewers hdwU and PVQH explicitly supported acceptance:

  • Reviewer hdwU stated, “My concerns are mostly addressed. Therefore I maintain my positive score,” retaining 3 (Weak Accept).
  • Reviewer PVQH acknowledged that most concerns had been addressed, stated, “I find the overall contribution convincing,” and raised the score to 3 (Weak Accept).
  • Reviewer 7dPZ likewise acknowledged that “most of my concerns are addressed” and recognized “the empirical finding” and “practical value” of our work, while maintaining a reservation about research novelty.

Our Follow-up Response directly addresses this remaining concern by further clarifying the methodological and conceptual contributions of DepthVLM, as briefly summarized below.

Methodologically, DepthVLM integrates dense geometric learning into the shared representations of an existing VLM. The depth decoder combines multi-scale visual features with contextualized LLM image-token features, allowing geometric supervision to shape representations shared by depth prediction and language generation. Our two-stage training strategy first aligns the decoder with the frozen backbone, then jointly adapts the decoder and LLM under geometric and language supervision while keeping the visual encoder frozen. This coordinated design enables single-pass, full-resolution metric depth prediction while preserving multimodal language capabilities, without revisiting the original pretraining corpus.

Conceptually, our controlled experiments provide insights into how geometric and language capabilities interact during adaptation. Under matched data exposure and optimization settings, geometric supervision enhances spatial reasoning, while language supervision helps preserve general multimodal performance. Together with validation across VLM sizes, these findings provide empirical insights into the interplay between geometric learning and multimodal capabilities, offering practical guidance for extending pretrained VLMs with dense geometric perception.

As of the discussion deadline, Reviewer 7dPZ had not provided further feedback on our follow-up response. We respectfully hope that the additional clarifications and experimental evidence presented during the discussion will help inform the AC’s final assessment of our contributions.


We sincerely thank all reviewers and the Area Chair for their valuable time, thoughtful feedback, and careful consideration throughout the review and discussion process.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.