acceptodds
Under review as a conference paper at ICLR 2027

D3SS: Depth Recovery Covering Diverse 3D Scene Scales

Abstract

Existing depth perception methods overemphasize 2D features of depth maps, introducing diverse 3D distortions caused by the scale problem, creating the demand for depth recovery in 3D scenes. Conventional depth recovery approaches only simulate raw depth at fixed scales in limited scenes, causing a significant simulation-to-reality gap that hinders the generalization of depth recovery models. To address the problem, we propose a diffusion-based raw depth generation pipeline for simulating diverse scene-scale distortions in both global and local patterns. To enhance the generalization capability in diverse RGB conditions, we first integrate the frozen depth foundation model (DFM), which is trained on large-scale RGB-D data, into the proposed framework. We further propose loss functions based on different views of 3D scenes to refine the 3D structure. On the one hand, our loss functions enforce the model to learn 3D features; on the other hand, our losses mitigate overfitting of the 3D distortions caused by DFM. Extensive studies on the recovery of different conditions in multiple datasets demonstrate the effectiveness and progressiveness of the proposed method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
depth recovery, monocular depth, depth scale, diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CuWwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes D3SS, a depth recovery framework designed to improve the generalization of depth refinement across diverse 3D scene scales and degradation patterns.

The method consists of three main components. First, a diffusion-based raw-depth simulation pipeline generates distorted depth maps by varying diffusion sampling steps, initial noise, and interpolation with ground-truth depth. Second, the framework incorporates predictions from a frozen Depth Anything V2 model as additional depth priors. Third, the authors introduce novel-view consistency losses, including a direct constraint against ground-truth geometry and a reliability-aware supervision strategy that encourages agreement with trustworthy foundation-model predictions while discouraging agreement in unreliable regions.

Experiments on Middlebury 2014, NYUv2, KITTI, and RGBDD demonstrate improvements over several depth recovery and super-resolution baselines. Additional experiments investigate different data simulation strategies, foundation-model integration, and geometric supervision.

Critical strengths
  • The paper addresses a practically relevant problem: improving depth recovery under diverse scales, local distortions, and structural errors.
  • The method substantially improves depth maps from MiDaS, LeReS, and Marigold across multiple benchmarks, with smaller additional gains for stronger models such as MoGe and MoGe-2.
  • Combining diffusion-based degradation synthesis, a frozen depth foundation model, and alternative-view geometric supervision is a promising strategy for reducing dependence on paired training data.
  • Ablations cover simulation components, foundation-model integration, geometric losses, and architecture choices, while RGBDD experiments extend evaluation to real-world depth inputs.
Critical weaknesses
  • The theoretical justification for diffusion-based scale diversification contains substantial gaps. Solver convergence does not imply monotonic convergence to metric depth, and the Lyapunov/Jacobian argument does not establish amplification of arbitrary local perturbations. Moreover, the ground-truth scale-and-shift alignment in Equation (2) may remove the intended global variations; empirical measurements of the final distortion distributions are needed.

  • The novel-view geometric loss lacks a precise mathematical and implementation description. Algorithm 1 involves depth sorting, discrete indexing, and pixel assignment without clearly explaining gradient propagation, while the virtual-camera construction is ambiguous. The authors should also justify why Equation (7) maximizes disagreement with unreliable DAV2 predictions instead of masking or downweighting them.

  • The contributions of diffusion simulation and the frozen DAV2 prior are insufficiently disentangled. In Table 4, interpolation alone matches the full simulation strategy on Middlebury AbsRel (0.038), while Table 5 suggests that test-time DAV2 conditioning already approaches the full method. Comparisons with simple interpolation, affine perturbations, and spatially varying noise under matched settings are needed.

  • Existing comparisons do not isolate the proposed method’s benefits because it jointly changes the inputs, architecture, and training procedure. A matched ConvNeXt baseline using the same RGB, raw depth, and frozen DAV2 inputs with standard regression and simple augmentation would better assess the value of the proposed simulation and geometric supervision. Scale-and-shift alignment protocols should also be clarified.

  • Real-world generalization is less convincing than synthetic performance. On RGBDD, the method matches SGN in AbsRel (0.021) and has slightly worse RMSE than the best baseline (0.060 versus 0.059), while the heavy-misalignment results are inconsistent across metrics. Evaluation on additional sensors and controlled real-world degradation types is needed.

  • Statistical analysis and related-work positioning are incomplete. A fixed seed does not establish robustness across training initializations or generated degradations, so multi-seed results and uncertainty estimates are needed. The discussion should include DORNet (CVPR 2025), DEPTHOR (ICCV 2025), and Distilling Monocular Foundation Model for Fine-grained Depth Completion (CVPR 2025).

Questions for authors
  • Could the authors empirically verify that varying diffusion steps systematically changes global depth scale, particularly after the ground-truth scale-and-shift alignment in Equation (2)? The monotonic scale-convergence and noise-amplification claims in Appendix A.1 should also be reconsidered.

  • What differentiable rasterization or point-splatting implementation is used for novel-view consistency, and how do gradients propagate through pixel assignment, depth sorting, and occlusion in Algorithm 1? Please also clarify the construction of camera translation and the orthonormal rotation basis.

  • Why does Equation (7) maximize disagreement with unreliable foundation-model predictions instead of masking or downweighting them? An ablation against confidence-weighted and masked supervision should assess geometry quality and training stability.

  • How does diffusion-based simulation compare with affine depth transformations, spatially correlated noise, and interpolation under identical training budgets? Please also quantify how closely the simulated degradation distributions match real raw-depth inputs.

  • Could the authors provide a controlled ConvNeXt baseline using identical RGB, raw-depth, and frozen DAV2 inputs with standard depth regression? This would isolate the gains from diffusion augmentation and novel-view consistency.

  • Could the method be evaluated on additional real-world sensors and degradation types? Multi-seed results and confidence intervals are also needed, particularly for RGBDD and comparisons with strong depth foundation models.

  • How are scale and shift ambiguities handled for MiDaS, LeReS, Marigold, and DAV2, and is ground-truth-dependent alignment used at test time? Please also report end-to-end latency and memory, including both frozen DAV2 inference and the recovery network.

Reviewer_qsgmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author introduced the problem of depth recovery and pointed out the scale distortion in the raw depth is the key issue, then proposed a diffusion based pipeline to generate diverse scale-distorted depth to train a depth recovery model.

Critical strengths

None

Critical weaknesses

First of all, the author never clearly defined what depth recovery is. The introduction starts with the scale ambiguity problem, but it is not related to scale distortion. It mentions poor 3D performance of sota depth estimation models, which indicates that the scale distortion refers to this affine ambiguity that a depth estimation model may have. However, all the baseline methods the paper compared with take noisy sensor depth as input. This is completely different. Also, a lot of places in the paper suggest about a "local" distortion. This terminology does not make sense at all because in this way all the errors can be classified as "local" distortion and this is saying nothing.

As all the baseline methods are designed to take sensor depth as input, the proposed method outperformed them easily. And as for the improvement on top of those scale-invariant models, this might be more about the effect of aggregating the original model with the DAV2 components.

If the depth recovery is to be understood as removing the affine ambiguity. Then only an affine perturbation needs to be applied to the depth in the training data. There is no need for a diffusion model at all.

Questions for authors

The author should clarify the definition of the problem and compare with suitable baselines.

Reviewer_ibn5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes D3SS to recover depth maps affected by different 3D scale distortions. During training, it uses a pretrained diffusion model, varied sampling conditions, and interpolation with ground-truth depth to generate simulated raw depth. The recovery network takes an RGB image, the raw depth, and a frozen Depth Anything V2 prediction as inputs. It is trained with GT depth supervision, a novel-view consistency loss, and two auxiliary losses based on whether the DFM prediction agrees with GT. The paper evaluates recovery from several depth prediction models and from real low-resolution depth.

Critical strengths
  1. The paper addresses an important generalization problem in depth recovery: a model trained with one type of simulated degradation may not work well across different input depths and scene scales. Varying diffusion steps, initial noise, and interpolation is a concrete way to generate more diverse training inputs. The method also uses a frozen depth foundation model and a novel-view constraint to address 3D structure beyond pixel-wise depth supervision. These choices are relevant to the paper’s stated problem, and the ablations in Tables 4, 5, 7, and 8 provide evidence for their usefulness.

  2. The evaluation covers several settings relevant to the central generalization claim. Table 1 reports strong results for inputs from MiDaS, LeReS, and Marigold across Middlebury 2014, NYUv2, and KITTI; Table 2 tests MoGe and MoGe-2; and Table 3 includes captured low-resolution depth from RGB-D-D. The real-data evaluation is particularly valuable because it tests whether the method transfers beyond simulated inputs. D3SS does not lead on every metric, but the breadth of the evaluation and its results are substantial strengths.

Critical weaknesses
  1. The experimental evidence does not fully establish the contribution of the proposed losses or position the method against recent related work.

    (a) Table 6 reports results with only LpL_p, only LnL_n, and both losses, but omits a baseline that retains the DFM input and the GT-supervised losses LdL_d and LcL_c while removing both LpL_p and LnL_n. Without this baseline, I cannot determine the joint benefit of the two auxiliary losses beyond the existing GT supervision. I suggest adding this comparison.

    (b) The depth recovery methods compared in Tables 1 and 3 extend only to 2025. Prior Depth Anything (ICLR 2026) also combines existing depth with monocular depth prediction and evaluates on RGB-D-D, which overlaps with Table 3, but is neither cited nor discussed. A comparison under a common evaluation protocol, or an explanation of the specific differences that prevent one, would help assess the contribution relative to recent work.

  2. Parts of the mathematical description and the novel-view algorithm need correction or clarification.

    (a) On page 4, the text refers to Eq. (9) immediately after presenting Eq. (1). On page 14, the appendix calls Eq. (2) an ordinary differential equation, although Eq. (2) is the depth interpolation formula. These appear to be incorrect equation references.

    (b) Page 4 describes sampling from Z0Z_0 to Z−tZ_{-t}, while Eq. (1) uses the index t−1t-1 and defines the cumulative noise coefficient using a product from i=1i=1 to tt. Please clarify how these expressions are used under the negative-index convention.

    (c) I cannot reconcile Eq. (15) with the chain rule for the complete reverse update. It multiplies the Jacobians of the two update components separately and then adds them. Since each step combines these components, multiplying the Jacobian of each complete update step would also produce cross terms. Please correct the derivation or explain how Eq. (15) is obtained.

    (d) Algorithm 1 defines EE as the eye position in line 2, sets T=−ET=-E in line 5, and transforms points using [R,T][R,T] in line 6. Under the stated transformation, the eye position maps to RE−ERE-E, which is generally not the camera origin. If EE is the camera center, the translation would normally be −RE-RE. Please clarify the coordinate convention and confirm whether the written algorithm matches the implementation.

Questions for authors
  1. With the DFM prediction still provided as input and Ld+LcL_d+L_c unchanged, what are the results when both LpL_p and LnL_n are removed? How much improvement does adding both losses provide over this baseline?

  2. How does the method compare with Prior Depth Anything (ICLR 2026) under the RGB-D-D evaluation protocol used in Table 3? If a direct comparison is not possible, which specific differences in the input or evaluation protocol prevent it?

  3. In Algorithm 1, is the translation used in the implementation T=−ET=-E, as written, or T=−RET=-RE? Please clarify the coordinate convention and how the eye position is mapped to the camera origin.

  4. How is Eq. (15) derived from the complete reverse-update step? The expression shown appears to omit cross terms when the Jacobians are multiplied across steps. Does the claimed local sensitivity depend on this expression?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.