acceptodds
Withdrawn from ICLR 2027

Foundation-Agnostic Metric Depth Refinement via Sparse Semantic Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advancements in depth estimation rely heavily on powerful foundation models. While these massive networks capture robust scene representations, their real-world applications still require sparse depth guidance to resolve absolute scale and local precision. However, integrating this guidance remains highly problematic because existing refinement paradigms suffer from severe architectural entanglement. They are tightly coupled to specific network designs and require costly retraining for any new foundation architecture. To break this barrier, we present FreeDepth, the first universally decoupled framework for foundation-agnostic metric depth refinement. Our method achieves true decoupling by mapping sparse inputs into a universal semantic space through anchor-relative log-depth representations. This unified signal then seamlessly modulates the shared spatial topology of any heterogeneous network without relying on specific architectural priors. To guarantee absolute robustness against localized feed-forward errors, this decoupled pipeline is complemented by test-time optimization gated by gradient compatibility, while keeping all primary weights strictly frozen. Extensive experiments across five datasets demonstrate that FreeDepth establishes a new state-of-the-art performance. Remarkably, despite being trained solely via frozen interfaces of DA3 and InfiniDepth, the identical corrector demonstrates strong zero-shot transferability to unseen foundation models, such as and VGGT. All code and models will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
metric depth estimation, depth refinement, sparse depth guidance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gdGLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a single shared depth refiner for various foundation models. The authors also apply test-time optimization to difficult depth samples.

Critical strengths

The attempt to refine depth predictions from diverse foundation models using a single shared refiner.

Critical weaknesses
  1. CKA analysis does not sufficiently support compatibility across features from various foundation models. The grid artifacts visible in Fig. 4 suggest that a single corrector trained on other foundation models may not be sufficient.
  2. It is unclear whether the gradient-compatibility gate is a reliable indicator of refinement difficulty. If the refinement is successful, the structure may still change substantially. Therefore, stronger justification is required.
  3. The qualitative changes for DA3 in Fig. 4 appear relatively small compared with the large quantitative gains reported in Table 2.
  4. The results in Fig. 3 show that PriorDA estimates the pole structure more precisely than FreeDepth.
Questions for authors

Can you provide an ablation in which the projected foundation features are zeroed out while retaining the aligned anchor and sparse prompt?

Reviewer_uAc9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FreeDepth, a foundation-agnostic framework for sparse-guided metric depth refinement. Given RGB, sparse depth, and a frozen depth or visual-geometry foundation model, it fits a per-image scale and shift to align the foundation prediction, encodes the remaining discrepancy as an anchor-relative log-depth prompt, and uses one shared corrector to predict a bounded multiplicative depth refinement. An optional LoRA test-time optimization handles difficult samples. The method is trained on Hypersim and VKITTI2 through DA3 and InfiniDepth, and evaluated on five datasets with a synthetic 16-beam sparse pattern. It reports the best average rank (1.3) and transfers to held-out π3 and VGGT without retraining.

Critical strengths
  1. The decoupling idea is well motivated. The shared corrector with a minimal prediction-and-feature interface is a sensible response to base-specific refinement pipelines.

  2. The anchor-relative log-depth prompt and the bounded multiplicative edit are simple and well justified.

  3. The reuse experiments (held-out foundations, mixed feature hierarchies, channel widths) demonstrate the zero-shot transferability to unseen foundation models.

Critical weaknesses
  1. More comparisons. It would be better to compare with more recent depth completion baselines, like LingBot-Depth [1], LDCM [2], and Any2Full [3].

  2. Sparse patterns. It would be better to test other sparse patterns, such as SIFT/ORB keypoints (as in OMNI-DC) and different densities, instead of only the 16-beam scanline pattern.

  3. TTO robustness. Since TTO supervises with sparse depth, it would be better to test it with noisy or outlier sparse points and with very sparse inputs, and to report whether it still improves over the feed-forward output.

References:

[1] LingBot-Depth: Masked Depth Modeling for Spatial Perception. ECCV, 2026.

[2] Large Depth Completion Model from Sparse Observations. ICLR, 2026.

[3] Any to Full: Prompting Depth Anything for Depth Completion in One Stage. ECCV, 2026.

Questions for authors

Please see weaknesses above.

Reviewer_b2gvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FreeDepth, a framework for reusing a shared sparse-guided metric depth corrector across different frozen depth foundation models. It first aligns the foundation prediction to sparse metric observations, then combines anchor-relative log-depth residuals with projected multiscale features through a FiLM-conditioned decoder to predict bounded multiplicative corrections. An optional gradient-compatibility gate selects samples for LoRA-based test-time optimization. The main contribution is enabling replacement of the active foundation without retraining the corrector or its learned projections. The corrector is trained on Hypersim and VKITTI2 using DA3 and InfiniDepth, then transferred directly to held-out π³ and VGGT models. Validation includes comparisons across five datasets, held-out foundation transfer, mixed-feature and channel-width experiments, component ablations, and accuracy–runtime analyses of test-time optimization. The reported results demonstrate competitive feed-forward accuracy and improvements over sparse-aligned predictions on unseen foundations.

Critical strengths
  1. Demonstrated reuse across held-out foundation interfaces. The same corrector and learned projections, trained using DA3 and InfiniDepth, improve predictions from held-out π³ and VGGT without further updates. Table 3 reports substantial improvements, including ScanNet AbsRel decreasing from 2.29% to 1.07% for π³. This directly supports the practical value of reusing a learned correction mechanism across changing foundation representations.
  2. Competitive feed-forward accuracy supported by complementary evaluations. With its own test-time optimization disabled, FreeDepth achieves seven first-place and three second-place results across the ten AbsRel/RMSE entries in Table 2. Component ablations and the image-level statistics in Table 13 further support gains beyond global sparse alignment. This strengthens the contribution by showing that reuse can coexist with effective refinement.
Critical weaknesses
  1. The incremental contribution of the shared feature interface remains incompletely isolated. PriorDA already permits replacing the frozen depth predictor, while PSD combines foundation features with sparse propagation and learned residual correction. FreeDepth’s distinctive contribution therefore lies in reusing the active foundation’s internal features through fixed learned projections and a shared corrector. The transfer and mixing experiments support feasibility, but do not directly quantify the feature interface’s marginal benefit over an anchor-and-prompt-only corrector. A comparison using the same prediction source and sparse inputs would also clarify its advantage over output-conditioned refinement.
  2. The breadth of the universality claim exceeds the demonstrated scope. The paper provides useful evaluations across datasets, held-out foundations, mixed features, and channel widths. However, the interface still assumes four ordered spatial feature stages, and sparse guidance is evaluated primarily under one simulated LiDAR-16-style protocol. Numerical sampling and perturbation settings are not fully specified, and robustness across observation densities, patterns, and outliers remains insufficiently characterized. These limitations constrain the applicability of the broad foundation-agnostic and robustness claims.
Questions for authors
  1. Clarification on the comparison with PriorDA: Which frozen depth predictor and checkpoint were used for PriorDA in Table 2? Do PriorDA and FreeDepth receive the same dense depth prior and sparse observations?
  2. Targeted validation of the feature interface: Do the authors have a controlled comparison against a corrector trained using only the aligned anchor and sparse prompt, with comparable capacity and training budget?
  3. Clarification on the sparse observation protocol: Please specify the horizontal sampling stride, spatial and depth perturbation distributions, dropout probability, and resulting valid-point counts for each dataset. Were these settings fixed across training and evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.