- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RAD, one model to refine depth maps from different sources: depth networks, RGB-D sensors and low resolution dToF. The authors fine-tune Wan 2.1, a video diffusion model. Given the RGB video, the input depth and a confidence map, the model predicts in one step a residual on the latent of the input depth. Training uses only synthetic data with four kinds of input: VGGT predictions, perframe Depth pro predictions, simulated sensor depth with holes and noise, and simulated low resolution dToF. The output is affine invariant disparity, then aligned to the input depth with scale and shift. Experiments show better boundaries and temporal consistency for several depth models, also models not seen in training. They also show state of the art results for depth completion and depth super-resolution.
Critical strengths
- One single model improves depth from many sources and for different problems. The gains on boundaries and on temporal consistency hold also for predictors never seen in training. This is the most convincing result of the paper and has practical value.
- The recipe is simple and the evaluation is wide, with real sensor data.
- The evaluation is exhaustive and the appendix is very complete: results for a lot of predictors, many qualitative examples on video, completion and SR, ablations and runtime
Critical weaknesses
- RAD is very close to DVD: same training data, same one-step regression, same gradient loss. The main new parts are the input depth and the corruptions. But the input depth is used two times: as condition, and for the final scale/shift alignment. One important control is missing: DVD (or RAD without input depth) aligned to the input depth in the same way. Without this it is difficult to know how much the conditioning helps.
- Table 8 shows that each task works only when its own corruption is in training. So the "universal" model is multi-task training, not generalization across tasks. This is fine, but then the paper should compare single-task models against the joint model with the same compute.
- The confidence channel is set by hand for each source: 0.45 for Depth Pro, 0.55 for dToF, 0.65 for real sensors. Correct and corrupted pixels get the same value. How were these values chosen, and what happens with a wrong value?
- There is no passive stereo (RGB-RGB) input: no SGM, no stereo networks, no COLMAP/MVS. The "stereo" of figure 1 is the active sensor. Stereo matching errors are very different from VGGT or DepthPro errors: outliers with high confidence in textureless or reflective areas, streaking, fattening at depth discontinuities. TartanAir and vKITTI have stereo pairs, so it is easy to run SGM or stereo networks and use their outputs in training. Please also evaluate the refinement of stereo methods, including filtered semi-dense maps, on Middlebury, ETH3D or KITTI.
- A real per-pixel confidence (learned stereo confidence, left-right consistency check, native uncertainty of the model) would make the confidence channel meaningful, instead of a constant.
- The simulated sensor noise is not compared with real sensor statistics, even if SCRREAM and HAMMER give both sensor and GT. multipath and material-dependent holes are missing.
- bF1/siF1 measure where the edge is and not how sharp the transition is. L2 loss an 8x8 VAE and a gradient loss computed in latent space can still give oversmooth transitions, which mean flying pixels in the pcl. The paper shows no point clouds. Please show point clouds around depth discontinuities and measure the pixels with intermediate depth near ground truth edges (check SEE metric -- soft edge error). Also in tab. 1 the refined models do not beat methods detailed on boundaries ( such as InfiniDepth, MoGe-3, Depth Pro).
- Please test with GT (or near-gt) depth as input, and report the error only on observed sensor pixels. In training the model never sees a clean input and the confidence never reaches 1. Training also on points x_t of your own flow path (t between 0 and 1) could teach the model to not over-correct.
Some considerations about the eval protocol:
- For completion RAD uses 45-frame clips, while all baselines are single frame. On HAMMER (single images) RAD wins only 6/12 metrics. Please report RAD single-frame on SCRREAM.
- a small error in the shift becomes a big error at far range. All sensor benchmarks are indoor, and on KITTI VGG absrel gets worse. Please give error vs depth and explain how holes and sky are handled (depth 0 has no inverse)
- super reoslution: the test degradation is the same simulation used in training. DORNet and DuCos are worse than bicubic, so they are out of their setting. The sota claim holds only for this protocol. NYU or real low-res data (eg. ARKitScenes) would make it stronger.
- NVDS is applied to VGGT which is already temporally consistent; it should be applied to a per-frame model like Depth Pro. PriorDA receives dense affine depth. DA3 metric on KITTI looks wrong.
- The inference resolution on image benchmarks (eg., Middlebury) is not given. Part of the boundary gain can come only from upsampling.
- Some claims are too strong. "Largely preserving accuracy" -> VGGT gets worse on 4 of 6 video datasets, and Depth Pro on DyDToF has a similar problem. TAE gets worse for altready strong video models. Table 4 and the HAMMER results should be in the main paper and discussed.
Related work / comparisons issues:
- Refinement of depth from any source is older than the paper says. Some examples: DDR (Gidaris & Komodakis, CVPR 2017), learned stereo refinement (Knöbelreiter & Pock), Neural Disparity Refinement (Aleotti et al., 3DV 2021/ Tosi et al., TPAMI 2024), which refines with one model the output of SGM, stereo networks and COLMAP, at arbitrary output resolution. These should be discussed, at least. Some of them (Aleotti et al) has code and weights available for comparisomn.
- Simple RGB-guided baselines (joint bilateral upsampling for example) are missing. They are needed to show that a 1.4B prior is necessary for the boundary gains.
- The fair comparison (method vs method+RAD) is given "only" for 8 predictors. What if the refinement is applied to good predictors like InfiniDepth, MoGe-3 and Marigold-v2? I suggest to put the table in the main paper.
- For "lifting single-frame to video": NVDS on Depth Pro, Video Depth Anything and RollingDepth aligned in the same way.
Some minor points:
- how the latents are concatenated (channels or tokens), init of the patch embedding, text prompt, why t=0.5, how the depth is normalized (per clip or per frame, min/max or percentiles). Eq. 8 says x_Draw + r, but the text says the DiT predicts x̂ directly: which one?
- No ablation on residual vs direct prediction, one-step vs multi-step, video vs image backbone. The claim that the properties "emerge from video pretraining" is not shown.
- Eq1 and the one-step flow matching view are equal to direct regression. "Curriculum" in the abstract, but the sources are sampled uniformly.
- † in Table 2 is not explained. Runtime is not compared with baselines.
- Radar normalization is not given in fig4.
- some typos: "fixed at during training" (value missing), "for for", "affine-invarient"..
Questions for authors
- DVD, or RAD without input depth, aligned to the input with your protocol, on at least one estimation, one completion and one SR benchmark.
- How were the confidence constants and the per sensor alignment choices (OLS vs trimmed LS) selected, and on which data? Results with a wrong or a realistic confidence.
- Refinement of passive stereo (SGM and one stereo network, also filtered) on Middlebury, kitti or ETH3D.
- Point clouds near discontinuities and a flying-pixel measure (?) gt as input (does RAD keep it?).
- RAD single-frame on SCRREAM, to separate the effect of temporal context.
- Inference resolution on image benchmarks / normalization of depth / handling of holes and sky.
- Were Hypersim, TartanAir or vKITTI in the training data of VGGT or Depth Pro? If yes, their errors on your training set can be smaller than at test time.
- Do you plan to release code and weights?
My current score is weak accept, mainly for the strong and wide results. But this score depends on the rebuttal. I also expect that many of the weaknesses indicated are handled somehow in the rebuttal (if possible)
6 comments
Response to Reviewer M1qZ (2/3)
Question3: Can RAD refine SGM and stereo-network outputs?
We designed RAD to refine monocular depth predictions and sensor readings, and stereo depth falls outside the original scope of our study. We evaluated RAD on depth maps produced by OpenCV's semi-global block matching (SGBM) algorithm to show our method's generalizability to stereo depth. We leave stereo-specific refinement to future work.
We first applied a left–right consistency filter to the SGBM matches. We then gave OMNI-DC and RAD's sensor input the same filtered depth maps, leaving unmatched pixels unfilled. We construct a dense SGBM baseline by inpainting unmatched pixel with the value of its nearest valid neighbor. As shown in Table R9, RAD achieves the lowest AbsRel on both Middlebury and ETH3D, showing the generalizability to stereo depth inputs.
Table R9. Dense stereo refinement on 15 Middlebury scenes and 25 rectified ETH3D pairs from five scenes.
| Method | Middlebury AbsRel (%) ↓ | ETH3D AbsRel (%) ↓ |
|---|---|---|
| SGBM (inpainted) | 5.26 | 15.85 |
| OMNI-DC + SGBM | 4.98 | 17.48 |
| RAD + SGBM | 1.78 | 6.30 |
Question4: Does RAD reduce flying pixels near depth discontinuities?
Yes. When refining VGGT-Ω predictions, RAD reduces the flying-pixel rate from 26.6% to 22.7% on Middlebury and from 41.0% to 39.0% on iBims-1, as shown in Table R10.
To measure the sharpness of depth transitions, we examine pixels within 3 pixels of a ground-truth depth jump greater than 10%. We count a prediction as a flying pixel if its depth lies strictly between the local foreground and background depths and differs from each by more than 5%.
Following your suggestion, we also report a relative-error variant of soft edge error (SEE). We average this error over ground-truth edge pixels, using 3×3 and 5×5 neighborhoods to allow for localization offsets of up to one and two pixels, respectively. RAD reduces relative SEE on both datasets for both neighborhood sizes. Figure S3 (supplement) shows the 3D points from the narrow horizontal band marked by the red line in the RGB image, and foreground and background surfaces are more clearly separated after RAD refinement.
Table R10. Flying-pixel rate and relative soft edge error (SEE) for VGGT-Ω before and after RAD.
| Dataset | Flying pixels↓ Input | Flying pixels↓ + RAD | Relative SEE (k=3)↓ Input | Relative SEE (k=3)↓ + RAD | Relative SEE (k=5)↓ Input | Relative SEE (k=5)↓ + RAD |
|---|---|---|---|---|---|---|
| Middlebury | 26.6 | 22.7 | 10.36 | 3.62 | 9.11 | 2.68 |
| iBims-1 | 41.0 | 39.0 | 13.21 | 8.81 | 11.01 | 6.40 |
Figure S3. Point-cloud cross-sections on Middlebury. The red line marks a narrow horizontal band in the RGB image. The remaining panels show the corresponding 3D points.
Response to Reviewer M1qZ (3/3)
Question5: How does single-frame RAD perform on SCRREAM without temporal context?
With a single input frame, RAD still outperforms every baseline in the paper’s Table 2 on both SCRREAM sensors in All-GT RMSE, as shown in Table R11. Temporal context provides some additional gains, but RAD’s overall advantage already holds in the single-frame setting.
Table R11. Single-frame and 45-frame RAD versus CDMs and LingBot-Depth on SCRREAM. Best results are bold; second-best results are italicized.
| Method | D435 (All GT) RMSE↓ | D435 (All GT) AbsRel↓ | D435 (All GT) δ1↑ | iToF (All GT) RMSE↓ | iToF (All GT) AbsRel↓ | iToF (All GT) δ1↑ |
|---|---|---|---|---|---|---|
| CDMs | 17.7 | 7.84 | 97.7 | 56.8 | 4.21 | 97.6 |
| LingBot-Depth | 22.4 | 8.90 | 96.0 | 15.8 | 3.77 | 96.3 |
| RAD (1 frame) | 14.85 | 7.08 | 98.9 | 9.17 | 3.56 | 98.4 |
| RAD (45 frames) | 15.39 | 7.02 | 98.7 | 8.32 | 3.18 | 98.7 |
Question6: How are inference resolution, depth normalization, holes, and sky handled?
Depth Estimation benchmarks run at each dataset's native resolution rounded up to a multiple of 16. Depth is normalised per clip in disparity by its 2nd/98th percentiles. We mask missing sensor or ground-truth depths and sky pixels before computing inverse depth or estimating the percentiles. We encode these locations at the far end of the normalized disparity range (-1), avoiding division by zero. Missing sensor measurements also receive zero confidence, so RAD can distinguish a hole from an observed distant surface.
Question7: What is the round-trip error on ground-truth depth?
Encoding and decoding ground-truth depth through the VAE gives 0.81% AbsRel on 7-Scenes, 1.21% on TUM-dynamics and 1.95% AbsRel on NYUv2. This is a known limitation as discussed in main paper Appendix D and the general response on refinement accuracy.
Question 8: Does overlap with the training data of VGGT-Ω and Depth Pro make RAD's training inputs more accurate than its test inputs?
VGGT-Ω and Depth Pro may produce more accurate predictions on datasets they saw during training. To assess RAD’s generalization, we conducted a comprehensive zero-shot evaluation across depth estimation benchmarks. The results show that RAD can generalize to new datasets and predictors even if its training inputs are more accurate.
Question9: Will the code and model weights be released?
We will release our code and weights.
Response to Reviewer M1qZ (1/3)
Thank you for the thoughtful and comprehensive review and for recognizing the simplicity and practical value of RAD. We appreciate your observation that a single framework performs well across depth estimation, completion, and super-resolution.
We address each of your questions below. For easier reading, we also provide a formatted version of our response to your review.
Question1: How does RAD compare with DVD using the same input-depth alignment on estimation, completion, and super-resolution?
RAD outperforms DVD across all three tasks when both methods use the same alignment protocol for each task. Table 1 of the paper reports the depth estimation comparison. We extended this comparison to depth completion and depth super-resolution by evaluating DVD with the same input-depth alignment used for RAD. As shown in Tables R7 and R8, RAD achieves lower RMSE and AbsRel on both SCRREAM sensors and at every evaluated super-resolution scale on DyDToF and DynamicReplica.
Table R7. RAD versus DVD on depth completion.
Table R8. RAD versus DVD on depth super-resolution.
Question2: How were confidence and alignment selected?
See the general response on confidence.
For alignment, we use ordinary least squares (OLS) by default. Helios iToF measurements are noisier, so we use trimmed least squares for this sensor to reduce the influence of outliers. We fixed the trimming fraction at 20% without tuning it.