acceptodds
Under review as a conference paper at ICLR 2027

Selecting the Minimizer: Test-Time Depth Completion Beyond the Measured Pixels

Abstract

Recent test-time optimization for depth completion adapts a frozen monocular metric-depth model to the sparse measurements of a single image through a low-rank adapter, and so needs no completion training set. Eliminating its scale-and-shift alignment analytically puts the objective in closed form, which explains why the procedure is fragile and slow: its minimizers leave at least 95% of the parameter dimensions free, so the difficulty is not reaching a minimizer but choosing one, and its standard zero initialization is degenerate, with one factor's gradient and curvature vanishing identically. We add one component for each. Semiparametric residual propagation (SRP) chooses the minimizer explicitly, in function space, splitting the correction into a parametric term in the predicted depth and a field harmonic on a bandwidth-free affinity graph, with the measurements imposed as constraints. A spectral warm start (SWS) replaces the degenerate initialization with the best rank-r descent step, whose amplitude is computed rather than searched. Across five benchmarks, 10 adaptation steps—a quarter of the baseline's budget—match or beat the full 40-step baseline on every metric while lowering MAE by 27% on KITTI and 31% on DDAD. SRP alone needs no backward pass, runs 30–40× faster than that adaptation and lowers its MAE by 22% on KITTI and 26% on DDAD. Code is available at https://anonymous.4open.science/r/selecting-the-minimizer-test-time-depth-completion/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
depth completion, test-time adaptation, monocular depth estimation, low-rank adaptation, harmonic functions on graphs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pvbfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes low-rank test-time optimization for zero-shot depth completion after analytically eliminating scale-and-shift alignment. It shows that the objective is a variance-scaled correlation criterion, that standard zero initialization is degenerate, and that sparse measurements leave a large set of minimizers. Two complementary remedies follow: Semiparametric Residual Propagation selects a function-space correction through constrained graph-harmonic interpolation plus a parametric depth term, and Spectral Warm Start computes a rank-r descent initialization and amplitude. Across five benchmarks and two backbones, SRP without adaptation is 30-40x faster than 40-step LoRA-TTO, while SWS+SRP at 10 steps matches or improves the 40-step baseline on every reported metric and benchmark.

Critical strengths
  1. The analysis identifies precise structural causes of slow and unstable test-time adaptation and leads directly to two targeted algorithms. The paper then tests several falsifiable consequences, including negative results for optimization and parameter-space regularization interventions.
  2. The empirical study is unusually comprehensive and reproducible: five benchmarks, all 6,479 evaluation images, repeated stochastic baselines, paired statistics, consistent timing, transfer to a second backbone, strong ablations, and a substantial code supplement. The resulting accuracy-latency frontier is practically important.
Critical weaknesses
  1. The empirical comparison is centered on LoRA-TTO and TestPromptDC; diffusion-based and supervised completion methods are not run under the same protocol. This limits claims about the broader state of the art, although it does not undermine the main within-family conclusions.
  2. The method still relies on the affinity graph and measurement pattern. The paper documents weaker behavior for the adaptation-free point on RMSE and for biased VOID measurements, so deployment outside the evaluated sparsity regimes remains uncertain.
Questions for authors
  1. How sensitive are SRP and the 10-step operating point to substantially different measurement densities and structured missingness beyond the five benchmark protocols? Please report or discuss failure thresholds.
  2. Can the authors clarify which conclusions require the locally submersive adapter assumption and whether any evaluated samples materially violate it? A concise summary of diagnostic failures would strengthen the theoretical-to-empirical link.
4 comments
AuthorsOpenReview ↗

Response to Reviewer pvbf

Thank you for the careful review and for recognizing the falsifiable predictions and the reproducibility effort. Changes are in Revision 1 (blue).

W1. Other method families. Revision 1 scopes our claims to training-free completion (Sec. 1, 5.2, 5.3); supervised completion needs paired training data. For diffusion priors, Marigold-DC is running under our protocol on all 6,479 images; on a subset fixed in advance it already trails our 10-step configuration in all ten cells (table in our reply to htJo).

W2/Q1. Density and structured missingness. The weaker RMSE of the adaptation-free point is addressed in our reply to hzPS (W2); the 10-step point matches or beats the baseline on both metrics on every benchmark. On density, the evaluated range already spans 82 to 18,966 measured pixels, and over the core benchmarks the gain tracks the baseline's error (Spearman −0.50) more than density (+0.34) (App. D). To find failure thresholds, we fixed a sweep before seeing any result: on 100 NYUv2 and 100 KITTI held-out images, nested thinning to 50–2% of the measurements, KITTI scan lines down to one (as in Seo et al., 2026), and removal of the left half, a central box, or the farthest 25–50% of points. We will report error relative to the baseline against the measurement count, with the threshold where the 10-step configuration first falls behind (App. D, Revision 2).

Q2. Submersion hypothesis. Only the exact dimension count of Prop. A.5 uses it; the closed form, (C1)–(C3), SWS and SRP (Prop. A.6–A.8) do not. Without it, the "≥95% free" statement still holds to first order, since the Jacobian has rank at most m. No sampled block showed a rank drop: on the 50-image diagnostic subset, at three points of the run, 150 of 150 random 64-row blocks are full rank (App. E). Sec. 3.2 now states this.

AuthorsOpenReview ↗

Results: density and structured missingness (Q1)

Thank you again. The sweep we fixed in advance is complete on 100 NYUv2 and 100 KITTI held-out images. Values are MAE / RMSE relative to 40-step LoRA-TTO on the same images and thinned measurements (below 1 is better). In the sweep the baseline is a single seed (seed 0); on the full NYUv2 split its seed-to-seed standard deviation is 2.4% of MAE (Tab. 13). In short, the 10-step configuration stays within 4% of the baseline in every setting and never falls behind on KITTI in MAE, and SRP alone keeps its MAE lead on KITTI down to a tenth of the points.

Failure thresholds, 10-step configuration. On NYUv2 it first falls behind at 95 measured pixels (1.017 / 1.003), a gap smaller than the baseline's seed spread; it is never more than 2% behind at any density and leads at the sparsest setting (10 pixels: 0.960 / 0.971). On KITTI it never falls behind, down to 2% of the points (334) or a single scan line (377 points; 0.820 / 0.847). The 40-step configuration leads at every density and line count on both.

Failure thresholds, SRP alone. The adaptation-free point keeps its MAE lead on KITTI down to 10% of the points (1,665) or 16 lines, and falls behind at sparser settings except a single line; on NYUv2 it is level at full density (0.992) and falls behind as points thin (1.119 at 10 points). This is the regime the 10 adaptation steps cover.

Structured missingness (MAE ratio of the 10-step configuration):

Patternleft halfcentral boxfarthest 25%farthest 50%
NYUv21.0380.9971.0080.942
KITTI0.9620.9450.8140.780

On KITTI the RMSE ratio is above 1 only with the left half removed (1.004). On NYUv2 the left-half pattern leaves half the image with no measurement at all; there the 10-step configuration is 3.8% behind and the 40-step one 1.1%. SRP alone trails under all eight structured patterns.

Revision 2, which we will upload by November 13, adds the full table, with every density, pattern and configuration, to a new appendix, and a summary with one figure to Sec. 5.2.

W1. The Marigold-DC results on the full splits, and its 10-sample ensemble, are in our reply to htJo.

Thank you for your response. It has addressed my concerns. After some consideration, I decide to keep my original rating unchanged.

Thank you for reading our responses and for letting us know that they address your concerns. We appreciate your careful review.

Reviewer_hzPSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the pain points of existing unsupervised Test Time Depth Completion methods based on Low Rank Adaptation (LoRA TTO), which are slow and prone to instability, and deeply analyzes the mathematical essence of optimizing the objective function during testing. The author proves that the objective function is analytically equivalent to a correlation criterion scaled by the variance of the measured values, and draws three key conclusions: (1) the gradient has data units and cannot be generalized across benchmarks; (2) The standard zero initialization is in a degraded state, resulting in the disappearance of gradients and curvatures; (3) Extremely underdetermined, the Minimizer Set preserves at least 95% of the parameter dimensions, and the difficulty of optimization lies not in "finding a minimum value", but in "choosing which minimum value". Therefore, the paper proposes two core components: Semiparametric Residual Propagation (SRP): Explicitly selecting the minimum value in the function space. It splits the correction into a parameterized term for predicting depth (to expand the ability of long-range extrapolation) and a field harmonic based on a bandwidth free gradient rank affinity map, and uses sparse measurement values as hard constraints rather than penalty terms.
Spectral Warm Start (SWS): replaces traditional zero initialization with the optimal rank - $r $descent step, and its direction, scale, and amplitude can be calculated in a closed form without the need for hyperparameter grid search.

Extensive experiments on five real and synthetic benchmarks have shown that the zero step (no backpropagation) scheme with SRP is 30-40 times faster than the 40 step baseline and has lower MAE; The 10 step SWS+SRP scheme only requires a quarter of the baseline calculation budget, surpassing or equaling the 40 step LoRA TTO baseline on all benchmarks and indicators (reducing MAE by 27% on KITTI and 31% on DDAD).

Critical strengths
  1. The paper breaks away from the trap of blind parameter tuning and, through mathematical derivation of the objective function during testing, reveals for the first time the severe underqualification of the solution space in the low rank TTO method (preserving the $\ ge 95 % $dimension) and the geometric degradation root of zero initialization, explaining why traditional optimization is prone to implicit bias.

2.By designing sparse measurements as hard constraints and introducing a bandwidth free gradient rank affinity matrix, SRP completely eliminates the influence of data units on hyperparameters; SWS solves the problem of traditional zero initialization consuming early iteration steps through spectral decomposition and closed form amplitude calculation, achieving 30-40 times acceleration and significant accuracy improvement.

Critical weaknesses

1.As mentioned by the author in the limitations section, the VOID dataset exhibits severe feature driven sparsity in measurement values due to its point set being derived from corner and texture structures tracked by visual inertial odometry (VIO), resulting in poor performance of SRP affinity maps and even negative impacts in partial ablation.

2.Although MAE significantly decreased, SRP without adaptation stage performed slightly worse than the 40 step baseline in the RMSE index, and about 11% of images on some benchmarks (such as VOID) showed a decrease in RMSE after inclusion. This indicates that the penalty for long-range long tail errors of unmeasured pixels in function space harmonic propagation is still insufficient.

3.The parameterization term $\ Pi (\ hat {f})=[1, \ hat {f} ^ 2, \ log \ hat {f}] $in SRP is designed based on the physical intuition that depth error increases superlinear with distance. However, there is a lack of broader theoretical proof to determine whether this fixed polynomial and logarithmic basis function can be universally applicable to all unknown wild scenes.

  1. The framework heavily relies on the reasonable geometric priors provided by frozen forward large models such as UniDepthV2 or MoGe-2. If the underlying model produces global structural errors in certain specific reflective, translucent, or extreme dynamic scenarios, optimizing and constraining propagation during testing will be difficult to correct this fundamental prior bias.

5.To eliminate kernel startup overhead, the method captures the solving process as a static CUDA graph. Although this greatly optimizes the processing speed of a single image with a fixed resolution, frequent re capturing or adjusting of the image structure may introduce additional engineering overhead in practical application scenarios such as video stream processing or dynamic resolution changes.

Questions for authors

1.Regarding the improvement of feature driven sparsity: Is there a plan in the future to introduce adaptive or learning affinity weights for sparse measurement distributions such as VOID that are highly biased, focused on corners and textures, to avoid graph structure failure in weakly connected regions?

2.Regarding the extension of extremely long videos or streaming scenes: For video streams with high frame rates and strong temporal continuity between frames, can the current static CUDA graph solving strategy be extended to an online continuous propagation form that combines temporal consistency constraints?

4 comments
AuthorsOpenReview ↗

Response to Reviewer hzPS

Thank you for the detailed summary and questions. Below we answer each point with the evidence in the paper and note what Revision 1 adds (changes in blue).

W1. VOID. Even on VOID's tracker-placed points (App. J), the 10-step configuration matches or beats 40-step LoRA-TTO: MAE 0.134 / RMSE 0.504 against 0.136 / 0.505 (Tab. 1). Our density sweep also removes measurements from whole regions, which probes weakly connected areas (Revision 2).

W2. RMSE without adaptation. Correct: SRP alone is a separate, 30–40× faster operating point, and it trails the 40-step run on RMSE. The gap sits mostly 4–16 px from a measurement, and on the driving benchmarks also farther out, where the maximum principle binds (Tab. 16, Sec. 6). The 11% figure in App. D refers to SRP added to the 40-step run, where RMSE improves on 95.6% of images and on every benchmark's mean. The configuration we lead with, 10 steps, matches or beats the baseline on RMSE everywhere:

RMSE (m)iBims-1VOIDNYUv2KITTIDDAD
LoRA-TTO, 40 steps0.1460.5050.1251.3725.937
SWS + SRP, 10 steps0.1440.5040.1251.3015.494

W3. Parametric basis. The alignment leaves the residual orthogonal to span{1, f̂} on the measured pixels, so f̂² is the lowest polynomial order left, and log f̂ models an error that is roughly constant in relative terms, as a monocular backbone's is (Sec. 4.1). Three properties keep it safe in unfamiliar scenes: the three coefficients are refitted on every image; f̂ is clamped to the range it takes on the measured pixels, so the terms never extrapolate in depth (Tab. 5); and the measurements stay hard constraints. Richer bases generalize worse: from 3 to 14 terms, dense error rises from 93.4% to 95.8% of the baseline's (Tab. 18). Sec. 6 now states this rationale.

W4. Backbone prior. We agree this dependence matters; it is shared by every training-free method. Because measurements are hard constraints, a backbone error is corrected wherever it is measured, and the components carry over to MoGe-2 with nothing retuned (App. D).

W5. CUDA graph. One capture per input shape serves a whole fixed-resolution stream; a new shape costs 0.8–0.9 s once, or runs uncaptured at 179–184 ms with identical output (Sec. 5.3, App. D).

Q1/Q2. A learned affinity needs a training set, either paired depth or video for self-supervised training, which a per-image method does not use. An affinity that adapts to the measurement pattern without data-unit constants, and a solve warm-started across video frames, are natural next steps that we do not claim here.

AuthorsOpenReview ↗

Results: sparse measurements and region removal (W1)

Thank you again. The density sweep mentioned under W1 is complete on 100 NYUv2 and 100 KITTI held-out images; the full results are in our reply to pvbf (Q1). Values are MAE / RMSE relative to 40-step LoRA-TTO on the same images (below 1 is better).

Sparse measurements. Thinned to 10 measured pixels on NYUv2, far sparser than any VOID frame (82 at the least), the 10-step configuration still leads the baseline (0.960 / 0.971), while SRP alone falls behind (MAE 1.119).

Weakly connected areas. Removing whole regions leaves parts of the image with no nearby measurement. MAE ratio of the 10-step configuration:

Patternleft halfcentral boxfarthest 25%farthest 50%
NYUv21.0380.9971.0080.942
KITTI0.9620.9450.8140.780

It stays within 4% of the baseline under every pattern and leads under all four on KITTI; SRP alone trails under all eight. In both cases the 10 adaptation steps close the gap that SRP alone leaves.

Revision 2, which we will upload by November 13, adds these results to a new appendix. We would be glad to answer any further questions.

Thank you for the author's serious response to the review comments and supplementary experiments. The newly added baseline comparison, parameter analysis, and computational cost explanation help to further understand the effectiveness of the method, and some issues have also been clarified.

However, I believe there are still several key issues that have not been fully resolved. Firstly, regarding the feature driven non-uniform sparse observations in the VOID dataset, the newly added experiments mainly examine the measurement density and structural deficiencies, which are not yet sufficient to verify the robustness of the method to real measurement distribution bias. Secondly, SRP only still exhibits significant long tail errors in unobserved areas. Although optimizing with a small number of tests can improve the results, this limitation has not been eliminated. In addition, the method still relies heavily on pre trained depth priors, and there is currently a lack of sufficient validation for cases where priors have obvious structural errors.

Overall, I acknowledge the supplementary work done by the author, but the aforementioned issues still affect the reliability of the method in complex sparse observation scenarios. The current response is not sufficient to change my overall assessment of the paper, so I will maintain the original rating.

AuthorsOpenReview ↗

Response to the follow-up

Thank you for the follow-up and for acknowledging the added comparisons. We respond briefly to the three remaining points; the placement experiment under point 1 will go into the final version if the paper is accepted, and the breakdown under point 3 regroups the runs of Tab. 1.

1. Measurement-distribution bias. We agree that this deserves a direct test. On VOID, whose points a visual-inertial tracker places on corners and textured structure, the 10-step configuration matches or beats 40-step LoRA-TTO (MAE 0.134 vs 0.136, RMSE 0.504 vs 0.505; Tab. 1). To isolate placement on data with dense ground truth, we placed the measurements of the 100 held-out NYUv2 images only on image corners (Shi–Tomasi), at the protocol's count and at VOID's share of the image (154 points), against random placement at the same count. MAE / RMSE relative to 40-step LoRA-TTO (below 1 is better):

PlacementPointsSRP (0)SWS + SRP (10)SWS + SRP (40)
Random4750.992 / 1.1280.972 / 0.9780.896 / 0.969
Corners4751.078 / 1.0690.982 / 0.9590.925 / 0.966
Random1541.017 / 1.0601.008 / 0.9900.948 / 0.973
Corners1541.073 / 1.0311.015 / 0.9730.943 / 0.952

Moving the points onto corners changes the 10-step configuration's MAE ratio by at most one point and lowers its RMSE ratio; it stays within 1.5% of the baseline on MAE and below it on RMSE. SRP alone is affected most, 7–8% behind on MAE, and the 10 adaptation steps close most of that gap. We agree that affinities robust to clustered measurements remain an open problem (App. J).

2. Tail of SRP alone. We agree, and we state this limitation (Sec. 6, App. J). SRP alone is the adaptation-free operating point; with 10 steps, RMSE is at or below the baseline on all five benchmarks (Tab. 1).

3. Priors with structural errors. We agree that this is a real limitation, and one shared by test-time completion in general: every approach that adapts or guides a frozen prior at test time, including LoRA-TTO, prompt tuning, guided diffusion and ours, corrects the prior only as far as the measurements constrain it, and far from every measurement it inherits the prior's structure. Within that shared limit, what we can measure is how much of the prior's error a method recovers. We grouped each benchmark's images (all 6,479) into quarters by the frozen prediction's error after scale-and-shift alignment (AbsRel); change against 40-step LoRA-TTO in each quarter, mean over the five benchmarks, MAE / RMSE:

Quarter of prior errorSRP (0)SWS + SRP (10)SWS + SRP (40)
Q1 (smallest)−8.9% / +8.1%−10.9% / −4.0%−16.2% / −1.4%
Q2−7.0% / +10.6%−11.9% / −2.2%−18.4% / −1.8%
Q3−7.2% / +9.7%−11.6% / −1.8%−18.1% / −2.2%
Q4 (largest)−9.6% / +7.5%−14.1% / −3.7%−19.0% / −2.7%

Where the prior is most wrong, the 10-step configuration's gain is the largest, and it improves on the baseline on every benchmark and both metrics.

We understand and respect your assessment. Thank you again for your careful reading and constructive comments throughout the discussion; they have improved the paper.

Reviewer_htJoOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper takles depth completion; find the dense depth given a sparse set of depth mesurements and an RGB image (pixels aligned). The paper proposes to go beyond the traditional LoRA-TTO approaches of CAPA/Marigold-DC and proposes to contributions: (1) Semiparametric Residual propagation to find dense solution in fast and benchmark agnostic ways (2) Spectral Warm Start to improve the initialization of LoRA. The paper claims better accuracy at faster latencies.

Critical strengths
  • The paper is well written, with a extensive amount of experiments and mathematical reasoning.
  • The paper is tecoretically well motivated.
  • The speed gains of the method is substaintial.
Critical weaknesses
  • Marigold-dc (Viola et al., 2025) is absent from the benchmark tables, although it largely evaluates on the same datasets.
  • I would judge the novely of the SRP as small, several previous literature have already well explored this space. Particularly the folumulation of the harmonic interpolation has been around since decades (Zhu 2023 and Levin 2004)

-- Zhu, Ghahramani & Lafferty, “Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions” ICML 2003. Levin, Lischinski & Weiss, “Colorization Using Optimization” SIGGRAPH 2004.

Questions for authors

Q1. Several design choise seem to have been selected based on the public benchmark results. Could you please clarify which choices were made using the evaluation GT, and which haven been made based on validation sets? Q2. I think the paper could benefit from a discussion comparing your SRP module to classical edge-aware solvers like the Fast Bilater Solver (Barron 2016), since it is quite close in nature. Particulary how SRP conceptually or empirically differes from such a residual solver applied to sparse depth.


Barron & Poole, "The Fast Bilateral Solver", ECCV 2016

2 comments
AuthorsOpenReview ↗

Response to Reviewer htJo

Thank you for the positive assessment and precise questions. Changes are in Revision 1 (blue).

W1. Marigold-DC. We are running it under our protocol (same inputs, masks and metrics; official code, one sample by default, and the processing resolution of the authors' benchmark scripts; each method scored on its own output): one sample on all 6,479 images, and the 10-sample ensemble used for its headline results on a subset fixed in advance (all 95 iBims-1 images plus 40 from each other benchmark by a fixed seed, 255 in total). Preliminary results on that subset (one sample; MAE / RMSE in m):

iBims-1VOIDNYUv2KITTIDDAD
Marigold-DC, 1 sample0.063 / 0.2030.069 / 0.3670.058 / 0.1450.554 / 1.5372.878 / 7.549
SWS + SRP, 10 steps0.041 / 0.1440.052 / 0.3280.055 / 0.1310.305 / 1.2031.337 / 5.361

On the four benchmarks whose full splits have finished (iBims-1, VOID, NYUv2 and KITTI), one Marigold-DC sample likewise trails 40-step LoRA-TTO and every configuration of ours on both metrics; DDAD is still running. Its published headline figures use the 10-sample ensemble, so one sample can trail them; the ensemble run on the fixed subset will show by how much. Revision 2 will add the full-split results to Tab. 1, both settings with latency to Fig. 3, and seed variance to App. D.

W2. Novelty of SRP. We agree the propagation is classical and now say so (Sec. 2). Our contribution is the analysis showing why the choice of minimizer belongs in function space (Sec. 3.2), and three departures it forces, measured without adaptation: constraints instead of a penalty (a tuned penalty is 18% worse in MAE over the core benchmarks), an affinity with no length scale (a tuned bandwidth is 12% worse; Zhu et al. learn one per feature), and a parametric term in the predicted depth that escapes the maximum-principle ceiling (DDAD 16–64 px group: from 5.7× to 2.3× of adaptation's error, Tab. 16).

Q1. Which choices used evaluation ground truth? All the choices we compared did: they were compared on development subsets drawn from the evaluation splits, and no separate validation set was used. Nothing is tuned per benchmark or per image. The new Tab. 4 (App. C) lists every decision: the constraints and the rank affinity follow from (C1) and were only ablated, while the parametric term and its basis, the amplitude rule and the 10-step budget were selected by comparison. These choices hold up out of sample. On the 4,907 images outside every subset, the 10-step configuration still matches or beats the baseline in all eight cells (Tab. 7; every iBims-1 image entered some subset). With any one benchmark held out, the constraint, affinity, warm start and 10-step budget are re-selected unchanged, and every other choice stays within 0.6 points of its best alternative; at 40 steps all of them are re-selected unchanged (Tab. 8).

Q2. Fast bilateral solver. On our residual, FBS is the penalized form of Sec. 4.1 in a bilateral grid: its measurements are a weighted penalty (λ to tune), its bandwidths are data-unit thresholds, and its output is a weighted average of its targets, so it shares the maximum-principle ceiling our parametric term escapes. Tab. 14 already isolates the first two on our graph. We are also running FBS itself on the residual, tuned per benchmark on ground truth (App. F, Revision 2).

AuthorsOpenReview ↗

Results: Marigold-DC (W1) and the fast bilateral solver (Q2)

Thank you again for these questions. The results we promised are below; we are folding them into the paper and will upload Revision 2 by November 13.

W1. Marigold-DC on the full splits. One Marigold-DC sample on all 6,479 images under our protocol (same inputs, masks and metrics; official code and settings; scored on its own output), next to Tab. 1 (MAE / RMSE in m):

iBims-1VOIDNYUv2KITTIDDAD
LoRA-TTO, 40 steps0.041 / 0.1460.136 / 0.5050.054 / 0.1250.438 / 1.3722.002 / 5.937
Marigold-DC, 1 sample0.063 / 0.2030.156 / 0.5510.059 / 0.1500.576 / 1.6832.967 / 7.872
SWS + SRP, 10 steps0.041 / 0.1440.134 / 0.5040.054 / 0.1250.321 / 1.3011.386 / 5.494

As the four benchmarks finished at the time of our first reply indicated, one sample trails 40-step LoRA-TTO, and every configuration of ours, on every benchmark and metric. Under the timing protocol of Tab. 11 (one RTX 4090, VGA input), it takes 22.0 s per image at the 768-pixel processing resolution the authors use for NYUv2 (16.4 s at native resolution), against 1.73 s for our 10-step configuration and 4.34 s for 40-step LoRA-TTO. Even SRP on the frozen prediction, with no adaptation step (0.11 s), is more accurate than one Marigold-DC sample on every benchmark and metric.

W1. The 10-sample ensemble. Marigold-DC's best published numbers average 10 samples, at ten times the cost of one: 219 s per image at VGA, 127× the latency of our 10-step configuration. On the subset fixed in advance (all 95 iBims-1 images and 40 from each other benchmark; MAE / RMSE in m):

iBims-1VOIDNYUv2KITTIDDAD
LoRA-TTO, 40 steps0.041 / 0.1460.056 / 0.3360.055 / 0.1290.414 / 1.2601.877 / 5.671
Marigold-DC, 1 sample (mean of 5 seeds)0.063 / 0.2020.069 / 0.3650.058 / 0.1450.552 / 1.5502.876 / 7.534
Marigold-DC, 10 samples0.045 / 0.1660.063 / 0.3540.049 / 0.1270.423 / 1.3492.257 / 6.101
SWS + SRP, 10 steps0.041 / 0.1440.052 / 0.3280.055 / 0.1310.305 / 1.2031.337 / 5.361
SWS + SRP, 40 steps0.037 / 0.1460.049 / 0.3270.051 / 0.1270.292 / 1.2141.290 / 5.499

The ensemble has the lowest MAE on NYUv2 and ties our 40-step configuration on its RMSE; on the other four benchmarks it trails the baseline and both of our configurations on both metrics. Averaged over the five benchmarks it reaches 1.07 of the baseline's MAE and 1.06 of its RMSE, against 1.33 and 1.23 for one sample and 0.87 and 0.98 for our 10-step configuration. Across the five seeds, one sample's standard deviation is 0.3–2.7% of its mean on every benchmark.

Q2. The fast bilateral solver. We applied FBS where SRP acts, to the residual, and gave it an oracle advantage: its spatial and color bandwidths and λ were tuned per benchmark on ground truth (30 images per benchmark, 455 combinations, both bandwidths searched to their natural ends) and scored on the 895 images of Tab. 14, which include the tuning images. On the frozen prediction, tuned FBS reaches 1.03 of 40-step LoRA-TTO's MAE averaged over the five benchmarks, against 0.93 for SRP; with its reference parameters it reaches 3.14. It trails SRP by 12–24% on four benchmarks and leads by 1.7% on VOID, where our parametric term itself costs 1.2% (Tab. 14). After 40 steps the two are within 2.1% everywhere. Its best λ lies inside the range we searched, on the flat floor of its tuning curve, where the penalty acts almost as the constraint that (4) imposes exactly. FBS needs three constants chosen per benchmark on ground truth; SRP needs none.

In Revision 2. The full-split results go to Tab. 1, Tab. 11 and Fig. 3; the subset results with the seed variance, and the solver comparison, go to a new appendix, with one sentence each in Sec. 5.2 and Sec. 2.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of revisions (Revision 1)

We thank the reviewers for their careful reading. All three found the analysis sound; htJo and hzPS found the speed gains substantial, and pvbf found the study comprehensive and reproducible. Revision 1 is uploaded with this comment; changes are marked in blue:

ChangeWhereFor
How each design choice was made, with held-out checks (Tab. 4, 7, 8)App. C, D; Sec. 5.1htJo Q1
Classical propagation and the fast bilateral solverSec. 2htJo W2, Q2
Claims scoped to training-free completionSec. 1, 5.2, 5.3pvbf W1
Where the submersion hypothesis is usedSec. 3.2, App. Epvbf Q2
CUDA-graph capture cost per input shapeSec. 5.3, App. DhzPS W5
Rationale of the parametric basisSec. 6hzPS W3
Roadmap for Sec. 3 and a notation table (Tab. 2)Sec. 3, App. APresentation

Coming in Revision 2: Marigold-DC on all 6,479 images and its 10-sample ensemble on a fixed subset, with latency (preliminary numbers in our reply to htJo); a density and structured-missingness sweep (pvbf, hzPS); and the fast bilateral solver on our residual (htJo). We will post these results in the relevant threads within two days.

AuthorsOpenReview ↗

Results of the experiments announced with Revision 1

The experiments announced with Revision 1 are complete, and their results are posted in the relevant threads: Marigold-DC on all 6,479 images and its 10-sample ensemble, with latency, and the fast bilateral solver tuned per benchmark on ground truth (reply to htJo); and the density and structured-missingness sweep (reply to pvbf, with a note to hzPS). In short, one Marigold-DC sample trails the 40-step baseline on every benchmark and metric at 5.1× its latency, and its 10-sample ensemble, at ten times that cost, leads only on NYUv2 and trails the baseline on both metrics on the other four benchmarks; the 10-step configuration stays within 4% of the baseline under every density and structured pattern and leads in MAE under all of them on KITTI; and tuned FBS reaches 1.03 of the baseline's MAE without adaptation, against 0.93 for SRP, with three constants per benchmark where SRP needs none. We are folding these results into the paper and will upload Revision 2 by November 13.

AuthorsOpenReview ↗

Summary of revisions (Revision 2)

Revision 2 is uploaded with this comment. Revision 1 changes remain in blue, and Revision 2 additions are in purple. It adds the results we posted in each reviewer's thread:

Change in Revision 2WhereFor
Marigold-DC on all 6,479 images (one sample), with latencyTab. 1, Tab. 11, Fig. 3; Sec. 1, 5.1, 5.2; App. ChtJo W1, pvbf W1
Its 10-sample ensemble and seed variance on a fixed subsetApp. K.1 (Tab. 26)htJo W1
Density and structured-missingness sweepSec. 5.2, Fig. 4; App. K.2 (Tab. 27)pvbf Q1, W2; hzPS W1
Fast bilateral solver on the residual, tuned per benchmark on ground truthSec. 2; App. K.3 (Tab. 28, Fig. 13)htJo Q2

Numbering. Every table and section number cited in this discussion is unchanged; the new tables are Tab. 26–28, the new figures Fig. 4 and Fig. 13, and appendix figures now start at Fig. 5. To keep these numbers, three items sit elsewhere than our first replies said: the ensemble and the seed variance, measured on a fixed subset, are in App. K.1 rather than in Fig. 3, which averages all 6,479 images, or App. D; the density sweep is in App. K.2 rather than App. D; and the solver comparison is in App. K.3 rather than App. F.

We thank the reviewers again and would be glad to answer any further questions before the discussion closes on November 18.

AuthorsOpenReview ↗

Thank you, and a summary of the discussion

We are grateful to the Area Chairs, Program Chairs and reviewers who give their time and judgment, much of it unseen, to keep peer review working. Thank you for the attention you have given our submission.

As the discussion closes, we summarize how we answered the three questions that ran through it.

Is the comparison broad enough? We ran the diffusion prior Marigold-DC under our protocol on all 6,479 images: one sample trails the 40-step baseline on every benchmark and metric at 5.1× its latency, and its 10-sample ensemble, at ten times that cost, still trails it on four of the five benchmarks. We also applied the fast bilateral solver to the residual, with its constants tuned per benchmark on ground truth; without adaptation it reaches 1.03 of the baseline's MAE, against 0.93 for SRP, which has no constant to tune.

Were the design choices fitted to the evaluation data? We listed every decision and the data it was compared on (Tab. 4). On the 4,907 images outside every development subset, the 10-step configuration still matches or beats the baseline in every cell, and with any one benchmark held out, every choice is either re-selected or stays within 0.6 points of its best alternative.

How sensitive is the method to the measurement pattern? Under random thinning, fewer lidar lines, removal of whole regions and placement on image corners only, the 10-step configuration stays within 4% of the baseline's MAE in every setting. SRP alone is the most sensitive configuration, as the paper states, and errors the prior makes far from every measurement are a limit that test-time completion shares as a whole.

Frozen monocular models now predict metric depth across scenes, and test-time completion lets one such model serve a lidar scan, a structured-light frame or a handful of visual-inertial points without any completion training data. What has held it back is cost and instability. Our analysis traces both to the choice among minimizers rather than to the optimization, and turns that into a read-out that needs no backward pass and a 10-step configuration that matches or beats the full 40-step baseline on every benchmark and metric. We hope this helps make test-time completion a practical default rather than an expensive alternative.

Thank you again for a careful and constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.