- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Depth Pro 2, a monocular metric depth model that achieves improvements over the recent affine literature, in particular regarding sharpness and temporal consistency (if a sequence of frames is provided). The authors claim that achieving sharpness and temporal consistency at the same time is an open problem, where prior work focuses only on one of the two problems (Fig. 2).
The core idea behind this paper is an engineering combination of factors that provide a depth that is both sharp and time-consistent under a unified ViT architecture family. In particular, the paper proposes the following methodology:
- A camera focal length estimator model to convert normalized inverse depth into metric depth;
- A frame-permutation-invariant DINOv3 ViT backbone with initial self-attention (first 33% of layers) and alternating self- and cross-attention for the remaining layers for multi-frame consistency;
- A novel aspect-ratio-aware spatial position encoding for image tokens;
- A modified DPT head with transposed conv for feature upsampling;
- Training using a large amount of synthetic and real datasets (>50M samples) and a large computing cluster (i.e., 128 NVIDIA B200s);
- A resolution-preserving augmentation that avoids interpolation to maintain GT sharpness;
- A multi-view self-consistency loss based on optical flow;
- Evaluating at full resolution to maintain the details of the original image, thus increasing sharpness.
The authors compared the proposal to several single-image depth estimation works (Tab. 1) and time-consistent depth estimation works (Tab. 2), comparing both accuracy (Tables 1-2) and sharpness (Tables 3.a-3.b). The benchmarks show that Depth Pro 2 achieves the best average rank on all tables.
Critical strengths
Major
- State-of-the-art results for sharpness and time-consistency at the same time. The tables speak for themselves: the proposed Depth Pro 2 achieves the best average rank on all tables of the main paper.
- Interesting engineering choices. Given the frame-permutation-invariant and aspect ratio-preserving spatial position encoding, the self- and cross-attention alternating strategy, the native resolution processing, the DPT with transposed convolutions, and the multi-view self-consistency, the authors managed to create a very powerful depth estimation model.
Minor
- Easy to follow. The paper is easy to read and well structured, with clear explanations of the method and the experiments. The qualitative results are impressive and show the sharpness improvements. The anonymous website is also very interesting and shows the capabilities in video.
Critical weaknesses
Major
-
Novelty is limited. A lot of the engineering strategies used in the paper are already studied in the literature. For example, the alternating self- and cross-attention for the remaining layers is already presented in DepthAnythingV3, the focal-invariant normalization and FoV estimator model are already presented in DepthPro, and the self-consistency loss is adapted from Luo et al.
-
Native Resolution claim needs more clarification. The claim itself sounds. The problems are related to the experimental/pratical part: 1) It is not clear if increasing resolution for other baselines increases sharpness as well as for DepthPro 2 (Tab. 14). The authors should investigate this with a similar table as Tab. 14 but using baselines. If the additional experiments confirm this, other baselines in Tab. 3 should be re-evaluated using native resolution for a fairer comparison. Tab. 17 suggests that this is not always the case: the Bokeh performances degrade for DAv3 if using higher res. 2) Processing at native resolution is expensive (for high-res inputs), thus requiring hardware (e.g., NVIDIA B200) not available to the majority of people. The authors could add an additional table (similar to Tab. 7) with runtime on lower-tier hardware and/or additional tables (similar to Tab.7) but with other baselines.
Questions for authors
- Please consider adding additional tables as requested in the weaknesses section, in particular regarding the native resolution.
- Row 128: "Depth Pro predicts focal-length-normalized inverse depth together with focal length,[...]" -> "Depth Pro 2 predicts focal-length-normalized inverse depth together with focal length,[...]" ?
- Arkitscenes is cited two times.
- Row 1722: "The camera camera" -> "The camera model"?
3 comments
Dear Authors,
Thanks for the additional PDF file with the required experiments. I'm satisfied with the authors' response to the "Native Resolution claim needs more clarification" weakness: even when running competitors at native resolution, the proposal still ranks first. Also, the larger model can run on lower-tier hardware up to 1080p. Thus, I'm going to raise my score.
Best regards,
Reviewer QWyL
Dear QWyL, Thank you very much!
Thank you for the thoughtful review and the concrete suggestions. We are glad you find Depth Pro 2 "a very powerful depth estimation model" and that its "qualitative results are impressive". We ran each experiment you suggested. The supplementary
QWyL.pdfcontains the full tables.W1: novelty. Please see our general response.
W2.1: higher resolution for baselines.
We thank you for the suggestion. Following the suggestion, we directly re-evaluated Tables 1-4 of the main paper with all methods running at native resolution (except Depth Pro and MetricAnything, which only operate at a fixed resolution). Tables Q1-Q4 report results of the baselines evaluated at native resolution and show the performance delta relative to the results at their default resolution. At native resolution, the baselines produce slightly sharper boundaries but less accurate metric depth. For all tables, Depth Pro 2 either retains or improves its average rank. It remains the best-performing method with the lowest average rank on all tables.
Place of Depth Pro 2-H+ among all methods, ranked by average rank (its average rank in parentheses). Full tables: Tables Q1–Q4 in
QWyL.pdf.W2.2: runtime.
We agree that processing long clips at native resolution can exceed the capabilities of lower-tier hardware, which is why we present a family of models. The family allows selecting an optimal model for specific hardware and task requirements.
As suggested, we benchmark the throughput of our model family on three generations of GPUs. For comparison, Table Q5 reports results on A100 (40GB), H100, and B200 GPUs.
Excerpt of Table Q5: throughput and peak memory of our models on three GPUs, with
torch.compileenabled (full table inQWyL.pdf):We also measured the throughput and peak memory of all baselines operating at native resolution on a B200 GPU (Table Q6). Video models, ours included, process a 32-frame clip in one call; single-image models process its frames one at a time. Depth Pro 2-S achieves a higher throughput than all baselines (except VeloDepth at 4K). At native 1080p, Depth Pro 2-H+ also needs less memory than 8 of the 12 video baselines.
Excerpt of Table Q6, native 1080p, not compiled: the five fastest baselines and ours (full table in
QWyL.pdf). ˢ: single-image method, memory for one frame.Minor. Thank you for catching the typos. We have fixed them in the revision.