acceptodds
Under review as a conference paper at ICLR 2027

Refine Any Depth

Abstract

Depth maps are essential to applications such as novel view synthesis and robotics, but the depth maps produced by sensors and neural networks are frequently flawed: low resolution, noisy, incomplete, spatially inaccurate, or temporally unstable. Many existing fixes are task-specific, tailored to a single problem such as completion or super-resolution, and do not generalize beyond that setting. We introduce a unified model to refine a broad class of depth inputs from real sensor data to the outputs of state-of-the-art depth models. We formulate depth refinement as residual regression. Given an input video and depth map, we fine-tune a video diffusion model to perform a single-step residual update to repair the input depth. Training on a curriculum of depth corruptions enables generalization and strong performance across depth modalities and error types. We present a simple recipe that converges quickly with a modest amount of training data. Our model achieves state-of-the-art performance in depth super-resolution and depth completion, and improves boundary sharpness and temporal stability of state-of-the-art depth models, including lifting single-frame estimates to coherent video depth.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Depth Estimation, Depth Refinement, Depth Completion, Depth Super-resolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_M1qZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RAD, one model to refine depth maps from different sources: depth networks, RGB-D sensors and low resolution dToF. The authors fine-tune Wan 2.1, a video diffusion model. Given the RGB video, the input depth and a confidence map, the model predicts in one step a residual on the latent of the input depth. Training uses only synthetic data with four kinds of input: VGGT predictions, perframe Depth pro predictions, simulated sensor depth with holes and noise, and simulated low resolution dToF. The output is affine invariant disparity, then aligned to the input depth with scale and shift. Experiments show better boundaries and temporal consistency for several depth models, also models not seen in training. They also show state of the art results for depth completion and depth super-resolution.

Critical strengths
  • One single model improves depth from many sources and for different problems. The gains on boundaries and on temporal consistency hold also for predictors never seen in training. This is the most convincing result of the paper and has practical value.
  • The recipe is simple and the evaluation is wide, with real sensor data.
  • The evaluation is exhaustive and the appendix is very complete: results for a lot of predictors, many qualitative examples on video, completion and SR, ablations and runtime
Critical weaknesses
  • RAD is very close to DVD: same training data, same one-step regression, same gradient loss. The main new parts are the input depth and the corruptions. But the input depth is used two times: as condition, and for the final scale/shift alignment. One important control is missing: DVD (or RAD without input depth) aligned to the input depth in the same way. Without this it is difficult to know how much the conditioning helps.
  • Table 8 shows that each task works only when its own corruption is in training. So the "universal" model is multi-task training, not generalization across tasks. This is fine, but then the paper should compare single-task models against the joint model with the same compute.
  • The confidence channel is set by hand for each source: 0.45 for Depth Pro, 0.55 for dToF, 0.65 for real sensors. Correct and corrupted pixels get the same value. How were these values chosen, and what happens with a wrong value?
  • There is no passive stereo (RGB-RGB) input: no SGM, no stereo networks, no COLMAP/MVS. The "stereo" of figure 1 is the active sensor. Stereo matching errors are very different from VGGT or DepthPro errors: outliers with high confidence in textureless or reflective areas, streaking, fattening at depth discontinuities. TartanAir and vKITTI have stereo pairs, so it is easy to run SGM or stereo networks and use their outputs in training. Please also evaluate the refinement of stereo methods, including filtered semi-dense maps, on Middlebury, ETH3D or KITTI.
  • A real per-pixel confidence (learned stereo confidence, left-right consistency check, native uncertainty of the model) would make the confidence channel meaningful, instead of a constant.
  • The simulated sensor noise is not compared with real sensor statistics, even if SCRREAM and HAMMER give both sensor and GT. multipath and material-dependent holes are missing.
  • bF1/siF1 measure where the edge is and not how sharp the transition is. L2 loss an 8x8 VAE and a gradient loss computed in latent space can still give oversmooth transitions, which mean flying pixels in the pcl. The paper shows no point clouds. Please show point clouds around depth discontinuities and measure the pixels with intermediate depth near ground truth edges (check SEE metric -- soft edge error). Also in tab. 1 the refined models do not beat methods detailed on boundaries ( such as InfiniDepth, MoGe-3, Depth Pro).
  • Please test with GT (or near-gt) depth as input, and report the error only on observed sensor pixels. In training the model never sees a clean input and the confidence never reaches 1. Training also on points x_t of your own flow path (t between 0 and 1) could teach the model to not over-correct.

Some considerations about the eval protocol:

  • For completion RAD uses 45-frame clips, while all baselines are single frame. On HAMMER (single images) RAD wins only 6/12 metrics. Please report RAD single-frame on SCRREAM.
  • a small error in the shift becomes a big error at far range. All sensor benchmarks are indoor, and on KITTI VGG absrel gets worse. Please give error vs depth and explain how holes and sky are handled (depth 0 has no inverse)
  • super reoslution: the test degradation is the same simulation used in training. DORNet and DuCos are worse than bicubic, so they are out of their setting. The sota claim holds only for this protocol. NYU or real low-res data (eg. ARKitScenes) would make it stronger.
  • NVDS is applied to VGGT which is already temporally consistent; it should be applied to a per-frame model like Depth Pro. PriorDA receives dense affine depth. DA3 metric on KITTI looks wrong.
  • The inference resolution on image benchmarks (eg., Middlebury) is not given. Part of the boundary gain can come only from upsampling.
  • Some claims are too strong. "Largely preserving accuracy" -> VGGT gets worse on 4 of 6 video datasets, and Depth Pro on DyDToF has a similar problem. TAE gets worse for altready strong video models. Table 4 and the HAMMER results should be in the main paper and discussed.

Related work / comparisons issues:

  • Refinement of depth from any source is older than the paper says. Some examples: DDR (Gidaris & Komodakis, CVPR 2017), learned stereo refinement (Knöbelreiter & Pock), Neural Disparity Refinement (Aleotti et al., 3DV 2021/ Tosi et al., TPAMI 2024), which refines with one model the output of SGM, stereo networks and COLMAP, at arbitrary output resolution. These should be discussed, at least. Some of them (Aleotti et al) has code and weights available for comparisomn.
  • Simple RGB-guided baselines (joint bilateral upsampling for example) are missing. They are needed to show that a 1.4B prior is necessary for the boundary gains.
  • The fair comparison (method vs method+RAD) is given "only" for 8 predictors. What if the refinement is applied to good predictors like InfiniDepth, MoGe-3 and Marigold-v2? I suggest to put the table in the main paper.
  • For "lifting single-frame to video": NVDS on Depth Pro, Video Depth Anything and RollingDepth aligned in the same way.

Some minor points:

  • how the latents are concatenated (channels or tokens), init of the patch embedding, text prompt, why t=0.5, how the depth is normalized (per clip or per frame, min/max or percentiles). Eq. 8 says x_Draw + r, but the text says the DiT predicts x̂ directly: which one?
  • No ablation on residual vs direct prediction, one-step vs multi-step, video vs image backbone. The claim that the properties "emerge from video pretraining" is not shown.
  • Eq1 and the one-step flow matching view are equal to direct regression. "Curriculum" in the abstract, but the sources are sampled uniformly.
  • † in Table 2 is not explained. Runtime is not compared with baselines.
  • Radar normalization is not given in fig4.
  • some typos: "fixed at during training" (value missing), "for for", "affine-invarient"..
Questions for authors
  • DVD, or RAD without input depth, aligned to the input with your protocol, on at least one estimation, one completion and one SR benchmark.
  • How were the confidence constants and the per sensor alignment choices (OLS vs trimmed LS) selected, and on which data? Results with a wrong or a realistic confidence.
  • Refinement of passive stereo (SGM and one stereo network, also filtered) on Middlebury, kitti or ETH3D.
  • Point clouds near discontinuities and a flying-pixel measure (?) gt as input (does RAD keep it?).
  • RAD single-frame on SCRREAM, to separate the effect of temporal context.
  • Inference resolution on image benchmarks / normalization of depth / handling of holes and sky.
  • Were Hypersim, TartanAir or vKITTI in the training data of VGGT or Depth Pro? If yes, their errors on your training set can be smaller than at test time.
  • Do you plan to release code and weights?

My current score is weak accept, mainly for the strong and wide results. But this score depends on the rebuttal. I also expect that many of the weaknesses indicated are handled somehow in the rebuttal (if possible)

6 comments
AuthorsOpenReview ↗

Response to Reviewer M1qZ (1/3)

Thank you for the thoughtful and comprehensive review and for recognizing the simplicity and practical value of RAD. We appreciate your observation that a single framework performs well across depth estimation, completion, and super-resolution.

We address each of your questions below. For easier reading, we also provide a formatted version of our response to your review.

Question1: How does RAD compare with DVD using the same input-depth alignment on estimation, completion, and super-resolution?

RAD outperforms DVD across all three tasks when both methods use the same alignment protocol for each task. Table 1 of the paper reports the depth estimation comparison. We extended this comparison to depth completion and depth super-resolution by evaluating DVD with the same input-depth alignment used for RAD. As shown in Tables R7 and R8, RAD achieves lower RMSE and AbsRel on both SCRREAM sensors and at every evaluated super-resolution scale on DyDToF and DynamicReplica.

Table R7. RAD versus DVD on depth completion.

MethodD435 RMSE↓D435 AbsRel↓D435 δ1↑iToF RMSE↓iToF AbsRel↓iToF δ1↑
DVD32.78.8197.7100.812.6798.1
RAD15.47.0298.78.323.1898.7

Table R8. RAD versus DVD on depth super-resolution.

MethodDyDToF ×4 RMSE↓DyDToF ×4 AbsRel↓DyDToF ×8 RMSE↓DyDToF ×8 AbsRel↓DyDToF ×16 RMSE↓DyDToF ×16 AbsRel↓DynamicReplica ×4 RMSE↓DynamicReplica ×4 AbsRel↓DynamicReplica ×8 RMSE↓DynamicReplica ×8 AbsRel↓DynamicReplica ×16 RMSE↓DynamicReplica ×16 AbsRel↓
DVD122.165.00134.665.11258.996.2013.923.9613.953.9713.933.96
RAD15.500.7017.980.7620.580.785.150.765.520.766.020.76

Question2: How were confidence and alignment selected?

See the general response on confidence.

For alignment, we use ordinary least squares (OLS) by default. Helios iToF measurements are noisier, so we use trimmed least squares for this sensor to reduce the influence of outliers. We fixed the trimming fraction at 20% without tuning it.

AuthorsOpenReview ↗

Response to Reviewer M1qZ (2/3)

Question3: Can RAD refine SGM and stereo-network outputs?

We designed RAD to refine monocular depth predictions and sensor readings, and stereo depth falls outside the original scope of our study. We evaluated RAD on depth maps produced by OpenCV's semi-global block matching (SGBM) algorithm to show our method's generalizability to stereo depth. We leave stereo-specific refinement to future work.

We first applied a left–right consistency filter to the SGBM matches. We then gave OMNI-DC and RAD's sensor input the same filtered depth maps, leaving unmatched pixels unfilled. We construct a dense SGBM baseline by inpainting unmatched pixel with the value of its nearest valid neighbor. As shown in Table R9, RAD achieves the lowest AbsRel on both Middlebury and ETH3D, showing the generalizability to stereo depth inputs.

Table R9. Dense stereo refinement on 15 Middlebury scenes and 25 rectified ETH3D pairs from five scenes.

MethodMiddlebury AbsRel (%) ↓ETH3D AbsRel (%) ↓
SGBM (inpainted)5.2615.85
OMNI-DC + SGBM4.9817.48
RAD + SGBM1.786.30

Question4: Does RAD reduce flying pixels near depth discontinuities?

Yes. When refining VGGT-Ω predictions, RAD reduces the flying-pixel rate from 26.6% to 22.7% on Middlebury and from 41.0% to 39.0% on iBims-1, as shown in Table R10.

To measure the sharpness of depth transitions, we examine pixels within 3 pixels of a ground-truth depth jump greater than 10%. We count a prediction as a flying pixel if its depth lies strictly between the local foreground and background depths and differs from each by more than 5%.

Following your suggestion, we also report a relative-error variant of soft edge error (SEE). We average this error over ground-truth edge pixels, using 3×3 and 5×5 neighborhoods to allow for localization offsets of up to one and two pixels, respectively. RAD reduces relative SEE on both datasets for both neighborhood sizes. Figure S3 (supplement) shows the 3D points from the narrow horizontal band marked by the red line in the RGB image, and foreground and background surfaces are more clearly separated after RAD refinement.

Table R10. Flying-pixel rate and relative soft edge error (SEE) for VGGT-Ω before and after RAD.

DatasetFlying pixels↓ InputFlying pixels↓ + RADRelative SEE (k=3)↓ InputRelative SEE (k=3)↓ + RADRelative SEE (k=5)↓ InputRelative SEE (k=5)↓ + RAD
Middlebury26.622.710.363.629.112.68
iBims-141.039.013.218.8111.016.40

Figure S3. Point-cloud cross-sections on Middlebury. The red line marks a narrow horizontal band in the RGB image. The remaining panels show the corresponding 3D points.

AuthorsOpenReview ↗

Response to Reviewer M1qZ (3/3)

Question5: How does single-frame RAD perform on SCRREAM without temporal context?

With a single input frame, RAD still outperforms every baseline in the paper’s Table 2 on both SCRREAM sensors in All-GT RMSE, as shown in Table R11. Temporal context provides some additional gains, but RAD’s overall advantage already holds in the single-frame setting.

Table R11. Single-frame and 45-frame RAD versus CDMs and LingBot-Depth on SCRREAM. Best results are bold; second-best results are italicized.

MethodD435 (All GT) RMSE↓D435 (All GT) AbsRel↓D435 (All GT) δ1↑iToF (All GT) RMSE↓iToF (All GT) AbsRel↓iToF (All GT) δ1↑
CDMs17.77.8497.756.84.2197.6
LingBot-Depth22.48.9096.015.83.7796.3
RAD (1 frame)14.857.0898.99.173.5698.4
RAD (45 frames)15.397.0298.78.323.1898.7

Question6: How are inference resolution, depth normalization, holes, and sky handled?

Depth Estimation benchmarks run at each dataset's native resolution rounded up to a multiple of 16. Depth is normalised per clip in disparity by its 2nd/98th percentiles. We mask missing sensor or ground-truth depths and sky pixels before computing inverse depth or estimating the percentiles. We encode these locations at the far end of the normalized disparity range (-1), avoiding division by zero. Missing sensor measurements also receive zero confidence, so RAD can distinguish a hole from an observed distant surface.

Question7: What is the round-trip error on ground-truth depth?

Encoding and decoding ground-truth depth through the VAE gives 0.81% AbsRel on 7-Scenes, 1.21% on TUM-dynamics and 1.95% AbsRel on NYUv2. This is a known limitation as discussed in main paper Appendix D and the general response on refinement accuracy.

Question 8: Does overlap with the training data of VGGT-Ω and Depth Pro make RAD's training inputs more accurate than its test inputs?

VGGT-Ω and Depth Pro may produce more accurate predictions on datasets they saw during training. To assess RAD’s generalization, we conducted a comprehensive zero-shot evaluation across depth estimation benchmarks. The results show that RAD can generalize to new datasets and predictors even if its training inputs are more accurate.

Question9: Will the code and model weights be released?

We will release our code and weights.

Reviewer_qNeZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RAD, a shared model for refining image and video depth from existing predictions or sensor measurements. A pretrained video diffusion model is adapted to predict residual corrections from RGB and imperfect depth. Training on a mixture of prediction errors and sensor degradations allows the same model to address blurred boundaries, temporal inconsistency, missing measurements, and low resolution.

Critical strengths
  1. Unified Framework and Strong Results: Extending the refinement approach in BetterDepth to video and different depth sources with one shared model is useful. The source ablations in Table 8 support this training design. The completion and super-resolution results are strong: RAD leads on 10 of 12 SCRREAM completion metrics and obtains the lowest RMSE in all six super-resolution settings (Tables 2 and 3).

  2. Writing: I found the paper easy to follow. Figure 3 clearly explains how the different inputs enter the shared refiner. The appendix also reports results for individual predictors and datasets, making it possible to assess where refinement helps and where it fails.

Critical weaknesses
  1. Loss of Geometric Accuracy: My main concern is that refinement can make an accurate prediction worse. In Table 4, VGGT-Ω's AbsRel increases from 3.21% to 5.25% on TUM-dynamics and from 2.63% to 3.58% on KITTI. Table 7 also shows worse temporal consistency for VGGT-Ω on 7-Scenes. These losses matter for a method intended to refine existing depth: sharper boundaries do not compensate for incorrect geometry. The authors acknowledge possible VAE reconstruction errors, but do not explain when users should expect refinement to preserve or damage the input geometry. The aggregate gains in Figure 4 do not resolve this concern.

  2. Effective Operating Range: This paper needs to specify its effective operating range. The experiments show useful transfer to unseen predictors and real indoor sensors. However, the super-resolution tests use a degradation family and ×4/×8/×16 factors included in training, and the sensor tests do not isolate sensitivity to missing-depth patterns, measurement density, or noise severity. Generalization to another predictor does not establish robustness to these changes. The paper should limit its claims to the input conditions supported by the evidence.

The shared framework is useful, and its completion and super-resolution results are strong. However, the loss of geometric accuracy on some predictions and the unclear operating range raised my concern. I am leaning toward a Weak Reject.

Questions for authors
  1. Could the authors analyze the failures in Table 4, particularly whether initially accurate regions are changed? I would like to understand when refinement helps and when it damages reliable geometry.
3 comments
AuthorsOpenReview ↗

Response to Reviewer qNeZ (1/3)

Thank you for the detailed feedback and for recognizing that RAD achieves strong performance across depth estimation, completion, and super-resolution within a unified framework and single set of weights.

We address each of your concerns below. For easier reading, we also provide a formatted version of our response to your review.

Concern1: Refinement can worsen an initially accurate geometric prediction.

See the general response on refinement accuracy.

Concern2: the effective operating range is unclear.

We interpret your question about RAD’s effective operating range as robustness toward varying degrees and patterns of depth degradation. In the main paper, we prioritized settings relevant to real-world applications and followed standard evaluation protocols for depth completion and depth super-resolution in literature. RAD achieved state-of-the-art results on both tasks.

For depth completion, RAD achieved state-of-the-art results on both the D435 and Helios iToF sensors in SCRREAM and on the L515 sensor in HAMMER. Because these benchmarks use real sensor readings, the results provide direct evidence of RAD’s robustness to the degradation patterns relevant in downstream real-world applications. For depth super-resolution, we followed the standard protocol for generating low-resolution inputs by simulating the loss of spatial detail when a single direct time-of-flight (dToF) measurement spans multiple surfaces (Appendix B.3). RAD outperformed all evaluated baselines under this protocol.

To further address your concern, we additionally evaluated RAD on depth completion under a wide range of simulated sensor corruptions and on depth super-resolution at scales unseen during training. RAD remains robust over these evaluations.

AuthorsOpenReview ↗

Response to Reviewer qNeZ (2/3)

Depth super-resolution. We evaluated RAD on DynamicReplica at two depth super-resolution scales unseen during training, ×12 and ×20. As shown in Table R5, RAD consistently outperforms both DVSR and bicubic interpolation across all four scales.

Table R5. Depth super-resolution on DynamicReplica. We report RMSE (cm) and AbsRel (%).

Method×8 RMSE↓×8 AbsRel↓×12 (unseen) RMSE↓×12 (unseen) AbsRel↓×16 RMSE↓×16 AbsRel↓×20 (unseen) RMSE↓×20 (unseen) AbsRel↓
Bicubic12.221.6615.401.9818.102.3320.422.67
DVSR6.730.979.361.157.671.197.941.17
RAD5.520.765.890.866.020.767.811.06
AuthorsOpenReview ↗

Response to Reviewer qNeZ (3/3)

Depth completion. We simulated sensor corruptions by degrading the ground-truth depth in SCRREAM D435 clips. Following your suggestion, we varied one factor at a time across three axes: missing-depth pattern, input depth density, and sensor noise severity.

For missing-depth patterns, we evaluated seven conditions. In each case, we added 1% noise to the retained depth values.

  • No holes. We retained all valid ground-truth depth pixels.
  • Random dropout. We randomly removed 5% of valid depth pixels in each frame. This dropout pattern was unseen during training.
  • Measured D435 holes. We removed ground-truth depth wherever the D435 sensor returned no valid reading in the corresponding frame.
  • Border band. We removed a slanted band of pixels along the image boundary.
  • Occlusion shadows. We masked regions near foreground boundaries to simulate missing stereo depth caused by occlusion.
  • Low-texture blobs. We removed contiguous patches to simulate missing depth on surfaces with little texture.
  • Range falloff. We preferentially removed depth measurements at greater distances from the sensor.

As shown in Table R6, RAD's performance remains consistent across the six missing-depth patterns despite differences in the shape and fraction of missing pixels. Relative to the no-hole condition, RMSE increases by at most 0.87 cm and AbsRel by at most 0.38%, proving the robustness of RAD to different missing-depth patterns.

To isolate the effects of input depth density and sensor noise severity, we used the same synthetic corruption model as in training and varied each factor independently.

  • For depth density, we test RAD's robustness by keeping valid fraction at 90%, 75% and 50%.
  • For sensor noise, we tested severity levels of 0, 0.25, 0.5, 1, and 1.5, where 1 denotes the severity used during training.

RAD remains robust over these ranges, with RMSE slightly rising from 4.24 to 6.16 cm as depth retention falls from 90% to 50%, and from 4.83 to 5.32 cm as noise severity increases to 1.5 times the training maximum.

Table R6. RAD sensitivity on SCRREAM D435 with synthetic inputs. We report RMSE (cm) and AbsRel (%).

(a) Missing-depth pattern

MethodNo holes RMSENo holes AbsRelRandom dropout RMSERandom dropout AbsRelMeasured D435 holes RMSEMeasured D435 holes AbsRelBorder band RMSEBorder band AbsRelOcclusion shadows RMSEOcclusion shadows AbsRelLow-texture blobs RMSELow-texture blobs AbsRelRange falloff RMSERange falloff AbsRel
RAD4.211.244.821.555.081.624.311.274.871.324.501.335.031.40

(b) Depth density

Method90% RMSE90% AbsRel75% RMSE75% AbsRel50% RMSE50% AbsRel
RAD4.241.224.611.346.161.62

(c) Sensor noise severity

Method×0 RMSE×0 AbsRel×0.25 RMSE×0.25 AbsRel×0.5 RMSE×0.5 AbsRel×1 RMSE×1 AbsRel×1.5 RMSE×1.5 AbsRel
RAD3.931.104.201.214.371.324.831.635.321.92

Reviewer_QW1SOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RAD adapts a pretrained video diffusion backbone to refine depth using RGB, an initial depth estimate, and a confidence map. It performs a single deterministic prediction and is trained on several sources of imperfect depth. The same model is evaluated on depth-predictor refinement, sensor-depth completion, and depth super-resolution.

Critical strengths
  • The main appeal is the breadth of the method. A single set of weights handles inputs with quite different error characteristics and also transfers to predictors not used during training. This is useful in practice, where the available depth source can vary across applications.

  • The completion and super-resolution results are strong. RAD achieves the lowest RMSE in all six dataset/scale comparisons in Table 3 and performs well on SCRREAM. There are also substantial temporal improvements for several predictors: for example, UniDepth v2's 7-Scenes TAE drops from 1.54 to 0.693.

Critical weaknesses
  • The improvement is not uniform across inputs and metrics. For VGGT-Ω, TUM-dyn AbsRel increases from 3.21% to 5.25%, while π3's 7-Scenes TAE increases from 0.521 to 0.654. Several boundary scores on Bonn and SUN RGB-D also decline. These results do not negate the aggregate gains, but they are important for deciding when to apply the refiner. The main text should make these trade-offs more visible and discuss whether confidence information can help identify inputs that should be left largely unchanged.

  • A few inference details need clarification. The confidence values used for unseen predictors are not specified. For the sensor tasks, RAD fits scale and shift to the input measurements. This is a valid operation, but the corresponding treatment of the baselines in Tables 2 and 3 should be stated explicitly; the description of unaligned baselines in the HAMMER appendix does not establish the protocol for those tables.

Questions for authors
  • What confidence values are used for unseen predictors, and have you observed a practical way to identify cases where refinement reduces accuracy?
1 comment
AuthorsOpenReview ↗

Response to Reviewer QW1S

Thank you for the detailed feedback and for recognizing that our methods generalize across tasks and perform strongly on completion and super-resolution.

We address each of your concerns and questions below. For easier reading, we also provide a formatted version of our response to your review.

Concern: Refinement can degrade accuracy, the trade-offs should be discussed.

See the general response on refinement accuracy.

Question1: What confidence is used for unseen predictors?

See the general response on confidence.

Question2: Are sensor-task baselines aligned using the same protocol as RAD?

In Tables 2 and 3 of the paper, RAD recovers metric depth by fitting scale and shift to the available input measurements in each frame. It uses observed sensor depth for completion and low-resolution depth for super-resolution. No oracle ground-truth depth is used in fitting. The baselines predict metric depth directly, so we evaluated their original outputs without alignment.

To test whether this difference explains RAD's advantage, we compared RAD with the strongest baseline for each task under matched alignment. As shown in Tables R3 and R4, the metric-depth baselines gain little from alignment to the input depth maps, and RAD retains its overall lead. RAD's lead therefore does not come from the alignment. We will clarify these protocols in the paper.

Table R3. Depth completion on SCRREAM (paper Table 2) with LingBot-Depth given RAD's own alignment. The alignment fits scale and shift in disparity per frame to the observed sensor pixels (least squares for D435, trimmed least squares for iToF). GT is never used. We report RMSE (cm) and AbsRel (%).

MethodD435 (All GT) RMSE↓D435 (All GT) AbsRel↓D435 (All GT) δ1↑D435 (Holes) RMSE↓D435 (Holes) AbsRel↓D435 (Holes) δ1↑iToF (All GT) RMSE↓iToF (All GT) AbsRel↓iToF (All GT) δ1↑iToF (Holes) RMSE↓iToF (Holes) AbsRel↓iToF (Holes) δ1↑
LingBot-Depth22.48.9096.032.112.0686.115.83.7796.344.318.1371.0
Lingbot-Depth + alignment22.28.8395.932.012.0186.015.83.7796.344.318.1271.0
RAD15.47.0298.718.17.7197.58.33.1898.725.311.4385.8

Table R4. Depth super-resolution (paper Table 3) with DVSR given RAD's alignment to the low-resolution input.

MethodDyDToF ×4 RMSE↓DyDToF ×4 AbsRel↓DyDToF ×8 RMSE↓DyDToF ×8 AbsRel↓DyDToF ×16 RMSE↓DyDToF ×16 AbsRel↓DynamicReplica ×4 RMSE↓DynamicReplica ×4 AbsRel↓DynamicReplica ×8 RMSE↓DynamicReplica ×8 AbsRel↓DynamicReplica ×16 RMSE↓DynamicReplica ×16 AbsRel↓
DVSR22.190.7723.040.7424.990.776.491.006.730.977.671.19
DVSR + alignment22.270.8623.240.8325.130.866.801.287.011.257.761.31
RAD15.500.7017.980.7620.580.785.150.765.520.766.020.76

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all reviewers for their detailed and constructive feedback and for recognizing that RAD achieves strong performance across multiple depth refinement tasks within a simple, unified framework.

Below, we address the common concerns raised by reviewers. For easier reading, we also provide a formatted version of our rebuttal. All figures are also included in the updated supplementary material.

Concern: Refinement can worsen an initially accurate geometric prediction. (QW1S, qNeZ)

We thank the reviewer for recognizing the aggregate gain from refining depth predictions with RAD. Below, we examine cases in which refinement increases AbsRel and discuss where these accuracy costs arise. We found that real world datasets tend to underrepresent challenging area due to GT limitation, and AbsRel can rise because it captures reconstruction errors from VAE’s loss while overlooking the geometry RAD recovers in unannotated regions. We will include this analysis in the revised paper.

We found regression in AbsRel metrics consistently happening in real-world datasets where ground truth is incomplete. As suggested by reviewers, we studied examples in TUM-dynamics and 7-Scenes. Qualitatively, we found RAD achieves finer details (in 2D visualization) and fewer flyers (in 3D point clouds), see Figure S1 (supplement). Due to the limitation of real-world sensors, the GT annotation can underrepresent challenging scenarios like thin structures and large depth discontinuities, and those refinements are masked out and not reflected in the metrics.

These examples show that RAD can improve scene geometry even when AbsRel increases. Results on benchmarks with high-quality reference depth are consistent with this interpretation. When refining VGGT-Ω predictions, RAD reduces AbsRel on iBims-1, Middlebury, and Sintel.

To understand the actual gain and potential regression, we studied synthetic datasets, namely DyDToF and Sintel, where all pixels have valid annotations (Table R1). With perfect input, refinement in a diffusion latent is not lossless: encoding and decoding ground truth through the VAE already gives 0.50% (DyDToF) and 0.93% (Sintel) AbsRel in surface interiors, and conditioning RAD on ground truth raises this to 1.34% and 2.40%. As noted in Appendix D, our method also inherits the reconstruction limitations of the pretrained VAE, as residual regression is performed in its latent space. With real VGGT-Ω input, RAD reduces AbsRel at depth edges by 0.83% on DyDToF and 0.21% on Sintel (Figure S2 (supplement)). The input has its largest errors at depth edges. In surface interiors, AbsRel rises by 0.69% on both datasets. The input is already accurate there. This cost is less than the 0.84% and 1.47% that conditioning adds even with perfect input. Future work could lower it with VAE-free models, depth-specific autoencoders, or a final refinement step in pixel space.

Table R1. Regional AbsRel (%) on fully annotated synthetic depth (first 45-frame window). The depth edge band lies within 5 pixels of a GT depth step exceeding 5%. The surface interior excludes the edge band and the farthest 15% of GT depth. Perfect input: ground truth encoded and decoded by the VAE (VAE), and RAD conditioned on ground truth (VAE + conditioning), showing the error that refinement in a diffusion latent adds even to exact depth. Real input: VGGT-Ω before and after RAD. Δ is the absolute change.

DatasetRegionPerfect input (GT) VAEPerfect input (GT) VAE + conditioningReal input (VGGT-Ω) InputReal input (VGGT-Ω) + RADReal input (VGGT-Ω) Δ
DyDToFDepth edge4.875.417.436.60−0.83
DyDToFSurface interior0.501.340.971.66+0.69
SintelDepth edge6.497.4712.4212.21−0.21
SintelSurface interior0.932.403.063.75+0.69
AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

Question: How were confidence selected? (QW1S, qNeZ, M1qZ)

We set each source’s confidence based on how closely its depth estimates match the training ground truth. This gives confidence values of 0.45 for Depth Pro, 0.55 for low-resolution depth, and 0.65 for sensor depth. We vary these values slightly during training. For unseen predictors, we reuse Depth Pro’s value of 0.45 without additional tuning. For VGGT-Ω, we use the model’s own per-pixel confidence estimates and rescale them to the same confidence scale. We will clarify these in the revised paper.

RAD is not sensitive to the choice of confidence. We test different inference-time confidence constant c from 0.25 to 0.85, shown in Table R2. On Sintel, AbsRel varied by only 0.23% and 0.14% for unseen predictor π3 and DA3-Giant, respectively.

Table R2. Effect of inference-time confidence c when RAD refines unseen predictor π3 and DA3-Giant.

PredictorInput AbsRel↓Input siF1↑c=0.25 AbsRel↓c=0.25 siF1↑c=0.45 (default) AbsRel↓c=0.45 (default) siF1↑c=0.65 AbsRel↓c=0.65 siF1↑c=0.85 AbsRel↓c=0.85 siF1↑
π313.2811.812.8428.412.8128.412.6828.912.6128.9
DA3-Giant16.079.614.3129.714.3029.514.1730.314.1730.7

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.