acceptodds
Under review as a conference paper at ICLR 2027

Kalman Filter-based Temporal Prompting for Video Depth Completion

Abstract

3D depth sensors provide accurate measurements, yet recovering dense, temporally consistent depth from these sparse observations is crucial. Existing single-frame depth completion methods suffer from temporal inconsistencies due to frame-wise processing, whereas recent video depth models incur substantial computational overhead, limiting their practical deployment. To address this limitation, we adopt a lightweight framework for adapting foundation models using visual prompts, avoiding the need for full retraining or computationally intensive temporal designs. To enhance temporal consistency across multiple frames, we propose a Kalman Filter-based Temporal Prompting (KFTP), which models temporal relationships by fusing the previous and current visual prompts. Furthermore, we design uncertainty-aware prompting with adaptive process noise that explicitly models temporal uncertainty to prevent error accumulation. This enables robust adaptation under dynamic motion, occlusions, and highly sparse supervision. Extensive experiments on indoor and outdoor benchmark datasets demonstrate that our approach achieves superior temporal stability and consistent spatial accuracy, significantly outperforming previous methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Depth completion, Video depth estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_991bOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends test-time visual prompt tuning for zero-shot depth completion (TestPromptDC; Jeong et al., 2025) from single frames to video. In TestPromptDC, an additive input-space prompt is optimized per frame so that a frozen UniDepthV2 matches sparse depth measurements. KFTP instead treats the prompt as a time-evolving state with an element-wise uncertainty map, updated by a Kalman-style predict–update recursion:

Prediction step: the previous prompt and its uncertainty are warped to the current frame with optical flow (MemFlow). A hand-designed process-noise term is added, combining a constant floor, inverse feature variance, forward–backward flow inconsistency and flow magnitude. Observation: the current-frame observation z_t is produced by 50 steps of test-time optimization initialized from the warped prompt. The loss combines sparse L1+L2, a second-order smoothness/"Poisson" term, and a KL temporal-consistency term on soft depth-bin histograms. Update step: the warped prior and z_t are fused with an element-wise Kalman gain, using an observation-noise map built from UniDepthV2's predictive uncertainty and the flow inconsistency. Experiments on KITTI, DDAD, vKITTI2, DyDToF, Sintel and NYUv2 report lower RMSE, MAE and a flow-based temporal error (TEPE) than frame-wise test-time adaptation baselines, with very large improvements on DDAD, DyDToF and Sintel. The appendix adds an iPhone ARKit evaluation, a sparsity study, cost analysis and comparisons with CAPA and supervised video depth completion.

Critical strengths

The idea is simple and sensible. Carrying the adaptation state (the prompt) forward in time, instead of re-solving from scratch each frame, is a natural way to add temporal consistency to test-time adaptation without retraining. The approach is a plug-in on top of TestPromptDC with almost no runtime overhead relative to it (Table 7: +1% time, +4% memory). The authors are honest about the Kalman framing. They do not overclaim: Sec. 4.4 and App. H state clearly that the update is the exact minimizer of a per-element weighted least-squares objective, that the uncertainty maps are designed rather than calibrated, and that no optimality of a probabilistic filter is claimed. App. K acknowledges prior Kalman-based depth fusion (Matthies et al., 1989). I appreciate this candour. The evaluation is broad. Six benchmarks spanning indoor/outdoor and real/synthetic data, plus real iPhone captures, a sparsity stress test (App. E), a cost comparison (App. F) and a comparison with the concurrent CAPA. The dataset-wise discussion around Table 3, which explains that gains are large where the frame-wise baseline is unstable, is a useful sanity check. The fusion-rule ablation is informative. Table 4 shows that the adaptive gain beats every fixed EMA weight on all three datasets, while fixed weights hurt on KITTI and NYUv2. The exploratory analysis supports the premise. App. J shows that frame-wise prompts become more correlated after flow alignment in the low- and mid-frequency bands, which motivates propagating the prompt.

Critical weaknesses

The only possible weakness I can think of is that the source of the large accuracy gains is not clearly identified: The headline improvements are very large: DDAD RMSE 5.65 → 2.98, DyDToF 2.31 → 0.94, Sintel 5.21 → 1.58. Plain temporal smoothing usually improves consistency, not per-frame accuracy, by factors of 2–3. Several mechanisms other than Kalman fusion could explain the gains such as: (a) Warm-starting. z_t is initialized from the warped previous prompt, so after t frames the prompt has effectively received about 50t optimization steps instead of 50. If frame-wise TestPromptDC with 50 steps is far from convergence, which its poor DDAD and Sintel numbers suggest (worse than raw UniDepthV2 on Sintel: 5.21 vs. 4.26), warm-starting alone could account for most of the gain. (b) Accumulating sparse measurements over time. If sparse points are re-sampled independently at each frame (e.g., 500 random points from dense GT on DyDToF, NYUv2 and Sintel), temporal propagation effectively aggregates many more depth measurements than a single-frame method sees. That would be an unfair advantage over frame-wise baselines, and it would also explain why gains are largest on datasets with synthetic sparse sampling and smallest on KITTI, whose LiDAR pattern is fixed. (c) The extra loss terms. L_poisson and L_temporal may contribute on their own.

The paper is missing an ablation of :

(i) TestPromptDC warm-started from the warped previous prompt, with no Kalman fusion and no extra losses; (ii) TestPromptDC with a matched total number of optimization steps; (iii) KFTP without L_temporal and without L_poisson; (iv) KFTP without each Q term and without R_Φ; (v) an explicit statement of whether sparse samples are fixed or re-sampled across frames, with results under a fixed pattern. with that ablation in place, coming back with clear results, I would consider a 4.

Minor problems:

In Table 2, DepthCrafter's Sintel row (RMSE 14.277, MAE 12.191, TEPE 1.112) is identical to its DDAD row in Table 1, which strongly suggests a copy-paste error. The Fig. 4 caption mentions only NYUv2 and KITTI, but the figure also shows DyDToF. Sec. 4.3 is titled "Prediction Step" but describes the observation/update step. There is a stray "d" at the end of the first sentence of Sec. 4.4.

Questions for authors
  1. Are the sparse depth samples fixed or re-sampled at each frame, for each dataset? If re-sampled, please report results with a fixed sampling pattern.
  2. What is the performance of TestPromptDC warm-started from the warped previous prompt, without Kalman fusion or extra losses? And with an equal total step budget?
  3. Please provide component ablations for L_temporal, L_poisson, each term of Q_t, and R_Φ.
  4. In Table 4, do the "None" and "EMA" rows use warm-start initialization and the same loss as KFTP?
  5. What are the values of the proportionality constants, Q_base, β, Σ_0 and K/σ for the soft binning, and how sensitive are results to them?
  6. How are UniDepthV2's depth-space uncertainties mapped into a prompt-space variance?
Reviewer_7toCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a test-time adaptation method for the video depth completion task based on tuning visual prompts for a frozen monocular image depth model. The method builds directly on TestPromptDC. The authors' main contribution is the temporal fusion mechanism of visual prompts based on a Kalman filter. The method is evaluated on both outdoor and indoor benchmarks and compared against different depth estimation and depth completion methods. Experimental results show improvements in spatial accuracy and temporal stability.

Critical strengths

The paper demonstrates significant improvements in spatial accuracy and temporal stability.

Critical weaknesses
  • The paper does not ablate the necessity of its pipeline's complexity. In particular, a naive baseline is missing: initializing the current frame's prompt with the warped or even non-warped prompt from the previous frame. Ideally each of the components of the method should be ablated separately;
  • The manuscript requires significant editorial work. In particular (non-exhaustive list):
    • Several terms in the formulas are loosely defined. For example, the observation z_t is never introduced explicitly;
    • The claim in the last sentence of "Temporal Coherence of Prompts" paragraph is unsupported and based on numbers whose origin is unclear. The paragraph itself also looks out of its place.
Questions for authors

None

Reviewer_NafrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Kalman Filter-based Temporal Prompting (KFTP) for zero-shot video depth completion. Given RGB frames and sparse depth measurements, it keeps a pre-trained depth foundation model frozen and optimizes a pixel-wise input prompt at test time. The previous prompt and its element-wise uncertainty are propagated with optical flow; a prompt optimized on the current frame acts as an observation, and an uncertainty-weighted Kalman-style update produces the final prompt. The optimization includes sparse-depth, spatial regularization, and temporal distribution-matching losses. Experiments on six indoor/outdoor benchmarks, additional iPhone sequences, and sparsity/efficiency analyses compare KFTP with frame-wise depth completion, RGB-only depth prediction, and several temporal methods.

Critical strengths
  1. KFTP provides a coherent causal formulation for adapting a frozen monocular depth model to streaming RGB plus sparse-depth inputs. Propagating a compact prompt state and adjusting the relative weight of a current optimized prompt offer an interpretable alternative to retraining a temporal backbone. The explicit distinction between prior and observation uncertainty is central to this design.
  2. The reported improvements over the directly inherited frame-wise baseline are substantial on several challenging datasets: RMSE decreases from 5.652 to 2.977 on DDAD and from 2.311 to 0.940 on DyDToF (Tables 1–2). Table 4 also shows that adaptive fusion improves DDAD RMSE from 3.242 for the best listed fixed-weight EMA to 2.977, providing direct evidence for a benefit beyond simply fixing the gain. The additional sparse-input and runtime results help characterize the operating regime.
Critical weaknesses
  1. The spatial evaluation protocol is ambiguous at a point that can materially change the meaning of the main results. Equations (8) and (18) define M_t as the validity mask of sparse depth used to optimize the prompt, while Equations (29)–(30) reuse M_t to compute RMSE/MAE against ground truth. If the masks are identical, the reported errors are evaluated only at depth locations that directly supervised the test-time optimization, rather than measuring completion at missing locations. The manuscript does not explicitly distinguish an input mask from an evaluation mask, nor does Appendix C specify their overlap or the default sparse-input sampling protocol for each dataset. This is an ambiguity requiring clarification, not evidence that the implementation necessarily reused the same mask.
  2. The contribution of the proposed uncertainty mechanism is not sufficiently isolated from the other temporal changes to TestPromptDC. Table 4 usefully compares constant and adaptive fusion gains, but fixed EMA alone already reduces DDAD RMSE from 5.652 to 3.242, whereas the adaptive rule reaches 2.977. The full method additionally changes initialization through the propagated prompt, adds temporal KL regularization, performs flow warping, and constructs process/observation uncertainty from several cues (Equations (11)–(24). Their individual effects and sensitivities are not separated. The closest sequence-level adaptation baseline, CAPA, is compared for RMSE/MAE but not temporal metrics in Table 3; the public March 2026 online video depth completion method StreamTTO is not discussed. These omissions limit the attribution and positioning of the claimed temporal advantage.
Questions for authors
  1. Are the validity masks used for sparse-depth supervision in Equations (8) and (18) different from those used for RMSE/MAE evaluation in Equations (29) and (30)? If the same mask is used, the depth measurements used to optimize the prompt would also be used for evaluation. How does this reflect the accuracy of depth completion? Are the reported RMSE and MAE also computed for the completed depth predictions at locations without input sparse-depth measurements?
  2. Table 4 demonstrates the effectiveness of temporal prompt fusion and adaptive Kalman gains. However, the adaptive process noise is also emphasized as a key contribution in the abstract and methodology. Could the authors provide an ablation comparing the proposed adaptive process noise with a constant process noise, while keeping the remaining components unchanged, to demonstrate its effectiveness?
  3. How should the TEPE metric in Equation (31) be interpreted when the true camera-frame depth of a point changes under ego-motion? The main goal of KFTP is to improve temporal consistency, yet it does not achieve the lowest TEPE on DDAD or vKITTIv2 in Table 1. Could the authors further clarify the evidence supporting its temporal consistency improvements on these datasets?
  4. Equation (11) uses backward flow F_{t→t−1} to warp the previous prompt into the current frame, whereas Equation (31) uses F_{t→t+1} to warp the depth at frame t. Could the authors clarify the warping convention used in Equation (31) and explain how occlusions and invalid correspondences are handled when computing TEPE?
  5. For reproducibility, please specify the first-frame initialization (μ_0, Σ_0), all scales/constants in Q_t and R_t (including β and the mapping from foundation-model depth uncertainty R_Φ to prompt-space weights), and the depth-bin count/range and Gaussian width in Equation (20). Section 4.3 describes a motion-adaptive temporal weight λ_t, whereas Appendix C reports λ_t = 0.1; how are the fixed base weight and spatially adaptive weighting combined?
  6. The main comparisons in Tables 1 and 2 focus largely on monocular depth estimation, video depth estimation, and single-frame depth completion methods. Could the authors provide a more focused comparison with methods specifically designed for video depth completion? Although RayFusion and SVDC are discussed as supervised methods, are there other relevant video depth completion approaches with comparable training or adaptation settings that could be included or discussed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.