Selecting the Minimizer: Test-Time Depth Completion Beyond the Measured Pixels

Recent test-time optimization for depth completion adapts a frozen monocular metric-depth model to the sparse measurements of a single image through a low-rank adapter, and so needs no completion training set. Eliminating its scale-and-shift alignment analytically puts the objective in closed form, which explains why the procedure is fragile and slow: its minimizers leave at least 95% of the parameter dimensions free, so the difficulty is not reaching a minimizer but choosing one, and its standard zero initialization is degenerate, with one factor's gradient and curvature vanishing identically. We add one component for each. Semiparametric residual propagation (SRP) chooses the minimizer explicitly, in function space, splitting the correction into a parametric term in the predicted depth and a field harmonic on a bandwidth-free affinity graph, with the measurements imposed as constraints. A spectral warm start (SWS) replaces the degenerate initialization with the best rank-r descent step, whose amplitude is computed rather than searched. Across five benchmarks, 10 adaptation steps—a quarter of the baseline's budget—match or beat the full 40-step baseline on every metric while lowering MAE by 27% on KITTI and 31% on DDAD. SRP alone needs no backward pass, runs 30–40× faster than that adaptation and lowers its MAE by 22% on KITTI and 26% on DDAD. Code is available at https://anonymous.4open.science/r/selecting-the-minimizer-test-time-depth-completion/.