acceptodds
Under review as a conference paper at ICLR 2027

StreamTTO: Efficient Online Test-Time Optimization for Video Depth Completion

Abstract

Monocular depth foundation models generalize across diverse scenes, but recovering accurate metric depth consistent with a target sensor remains challenging under sensor variation and domain shift. Test-time optimization (TTO) guided by sparse depth addresses this limitation, yet independent per-frame adaptation incurs substantial computational cost. We propose StreamTTO, a online test-time optimization framework for video depth completion that reuses visual features and adaptation state. A frozen feature extractor with causal temporal attention incorporates past visual context into features cached for repeated decoder optimization. Causal Sliding-Window Optimization directly updates a shared depth decoder using cached features and sparse observations from recent frames, requiring only two passes over the window per incoming frame after initialization. Reusing recent observations and the adapted decoder substantially reduces optimization steps. We also introduce MS-Depth, comprising approximately 1.45M synchronized RGB–LiDAR frames from 40 hours of recordings. Its uninterrupted sequences capture transitions among indoor, underground, and outdoor environments under daytime and nighttime conditions, enabling evaluation of adaptation to changes in illumination, scene structure, and depth range. Experiments on KITTI, Bonn, NYUv2 Raw, TUM RGB-D, DDAD, and MS-Depth demonstrate state-of-the-art depth completion performance. StreamTTO achieves average processing speeds of 15 FPS with VGGT and 25 FPS with MoGe-2 at an input resolution of , combining accurate metric depth with efficient online processing.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Depth Completion, Test-Time Optimization, Depth Foundation Models, Efficient Online Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Fm1eOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes StreamTTO, an online test-time optimization framework for video depth completion. A frozen feature extractor with causal temporal attention computes features once per frame and caches them, and only the shared depth decoder is updated using sparse observations. CSWO performs R = 2 window passes over a W = 3 causal window per incoming frame, and decoder parameters and optimizer state are carried forward. The paper also introduces MS-Depth, approximately 40 hours of synchronized RGB–LiDAR recordings with continuous transitions among indoor, underground, and outdoor environments. On KITTI, Bonn, NYUv2 Raw, TUM RGB-D, DDAD, and MS-Depth, StreamTTO achieves the lowest RMSE on most benchmarks, at approximately 15 FPS (VGGT) and 25 FPS (MoGe-2) on MS-Depth.

Critical strengths

On MS-Depth, StreamTTO achieves 15.73 (VGGT) and 25.90 (MoGe-2) FPS, compared with 0.426–1.652 FPS for Marigold-DC, TestPromptDC, and CAPA-LoRA. In all four conditions, the MoGe-2 and VGGT variants achieve the lowest and second-lowest RMSE, respectively. The compute-matched transition experiment (Table 4) and warm-start applied to other TTO methods (Table 13) support the design.

Critical weaknesses
  1. Retaining model parameters across frames, updating on current and recent frames in a sliding window, and performing few updates per frame were already proposed in Wang et al. (2025b), which also reports that one update per frame is sufficient and faster than reset with 20 updates per frame. The paper cites this work only in Appendix E, as adapting encoders through reconstruction objectives, without discussing these points. The main differences are the adaptation location (the depth decoder on a frozen feature extractor) and supervision from sparse observations, which enable reuse of cached features. This relation should be stated in the Related Work.

  2. The motivation stresses variations in measurement density, spatial distribution, and noise. Because decoder parameters and optimizer state are carried forward, errors from incorrect sparse observations may persist across frames. However, no experiment adds noise to the sparse observations or changes their spatial distribution on the same dataset; the only controlled robustness experiment reduces observation density on NYUv2 Raw (Appendix A.8). Robustness to sensor variation, a main motivation of the paper, therefore remains unverified.

Questions for authors
  1. How does StreamTTO perform when noise is added to the sparse observations, or when their spatial distribution changes within a sequence (e.g., from LiDAR rings to uniform sampling)? Since decoder parameters and optimizer state are carried forward, does accuracy recover after such frames?

  2. How were λ = 0.1, the clipping threshold of 1.0, and weight 2 for the current frame selected? In Table 7, equal weights at W = 3 and R = 2 give an unobserved RMSE of 0.8696 m, nearly the same as the default 0.8694 m.

  3. Appendix E states that some frames have fewer than 50 LiDAR points. The scale and shift in Eq. 3 are used both in the adaptation objective and in the final prediction (Eq. 2), and the updated decoder is carried forward. How does StreamTTO perform on these frames compared with the depth completion baselines, and is the alignment or the update handled differently for such frames?

Reviewer_HBQmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose streamTTO to address online video depth completion using RGB and sparse metric depth. The proposed framework utilizes a representation state and an adaptation state to avoid restarting an expensive test-time optimization for every frame. The paper also introduces MS-Depth with roughly 1.45M synchronized RGB-LiDAR frames covering about 40 hours of indoor, outdoor and underground transitions.

Critical strengths

The proposed framework is simple and the contribution is meaningful. The authors propose an efficient method to cast an expensive per-frame optimization problem to a streaming adaptation process by propagating representations and learned corrections temporally. The authors provide enough ablation studies and enough experiments to support their claims. The quantitative and qualitative results are strong. The dataset proposed in the paper is a significant contribution; it is a large dataset that includes various scenes under different environments and conditions.

Critical weaknesses
  1. Since streamTTO is a temporal method, studies about robustness to dynamic objects or a discussion about potential limitations and failure modes could have been included in the paper. MS-Depth explicitly excludes moving pedestrians, so a discussion on how the method would perform in dynamic scenes would have been beneficial.

  2. The reference filter that keeps lidar points only in cases that stereo correspondence and consistency succeed can potentially bias the evaluation towards easier regions. Especially in degraded areas or difficult environments that stereo is known to have a hard time, e.g night/textureless regions

Questions for authors

See above

Reviewer_Dit1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents StreamTTO, an online test-time optimization framework for video depth completion. The method exploits temporal continuity by reusing both visual representations and optimization states across frames. A frozen causal feature extractor caches historical features, while a depth decoder is continuously adapted using sparse depth observations from the current and recent frames through Causal Sliding-Window Optimization (CSWO). The paper also introduces MS-Depth, a large-scale RGB-LiDAR dataset containing long continuous sequences across indoor, outdoor, underground, and day/night conditions. Experiments on several public benchmarks and MS-Depth show that StreamTTO achieves strong depth-completion performance while substantially reducing the computation required by conventional per-frame test-time optimization.

Critical strengths

The paper addresses an important and practical problem, and the overall design is simple and well motivated. Reusing both cached visual representations and the previous optimization state is a natural way to reduce redundant computation in continuous video. The experimental evaluation is also relatively thorough. The authors provide ablations on warm-start optimization, temporal window size, optimization steps, and adaptation location. I particularly appreciate the compute-matched transition experiment, which shows that using a temporal window remains beneficial even when optimization FLOPs are controlled. The long-horizon experiments also compare continuous adaptation with periodic resetting and provide useful evidence that the method does not immediately suffer from accumulated drift. In addition, the same-backbone comparisons with both VGGT and MoGe-2 make it less likely that the reported improvements are simply due to a stronger underlying representation model.

Critical weaknesses
  1. The algorithmic novelty of StreamTTO is not sufficiently clear relative to existing streaming test-time adaptation methods.

A central idea of StreamTTO is to warm-start the current adaptation state from the previous frame and optimize over a short temporal window. However, warm-started online adaptation over recent video frames has already been explored in prior work on test-time training for video streams, while causal feature processing and KV-cache reuse are also established ideas in streaming visual geometry models. Therefore, it is currently difficult to determine which part of StreamTTO should be regarded as the main new algorithmic contribution rather than an effective integration of existing components.

The most interesting aspect to me is actually more specific: by restricting adaptation to the decoder while freezing the expensive feature extractor, StreamTTO can reuse both cached representations and persistent optimization states. The adaptation-location ablation provides useful evidence for this design choice. I would encourage the authors to position this contribution more precisely and clarify what StreamTTO introduces beyond existing online TTT, causal representation caching, and sparse-depth test-time optimization. A clearer conceptual and experimental comparison with the closest prior methods would significantly strengthen the originality of the paper.

  1. The construction of the MS-Depth reference depth may introduce a systematic evaluation bias, especially in difficult conditions.

MS-Depth constructs its reference depth from projected LiDAR points and further removes points that cannot be reliably verified through stereo correspondence. The paper itself notes that this filtering may reduce coverage more strongly in dark or distant regions. This is potentially important because these regions are also likely to be among the most difficult cases for depth completion. As a result, the reported metrics for night, distant, or low-texture scenes may primarily reflect the subset of pixels for which reliable stereo verification is already possible.

I do not think this invalidates MS-Depth, but the benchmark needs to characterize this effect more carefully. It would be helpful to report the valid reference-depth density and coverage after filtering, stratified by environment type, day/night condition, and depth range, together with the fraction of LiDAR points removed by the filtering process. These statistics are important for interpreting whether performance across different conditions is directly comparable.

Questions for authors
  1. What is the precise algorithmic contribution of StreamTTO beyond existing online video TTT with warm-started adaptation and recent-frame optimization, together with causal representation caching? In particular, is decoder-only adaptation enabling simultaneous representation and optimization-state reuse the main intended novelty?

  2. For MS-Depth, could the authors report the valid reference-depth coverage after stereo-based filtering, separated by environment type, day/night condition, and depth range? How many projected LiDAR points are removed by the filtering process?

Comments on the paper

3 comments

Dear Reviewers,

Thank you for taking the time to review our paper so thoughtfully. We know this is a busy time for everyone. If you have even ten minutes to spare, we’d really appreciate it if you could take a look at our rebuttal.

We wanted to leave a quick follow-up while the discussion is still open. If your schedule is tight, we’d be grateful for any feedback you’re able to share toward the end of the discussion period.

We have deep respect for the Area Chairs, Program Chairs, and reviewers who volunteer their time and expertise, often behind the scenes, to sustain peer review. Thank you for the care you have given our submission.

A central concern was whether LiDAR–stereo consistency filtering could make the benchmark easier. We applied this filter only during ground-truth generation. The model receives LiDAR points before filtering, so points excluded from the ground truth remain in the input. We clarified this distinction in the revised manuscript.

To obtain reliable ground truth in low light, we purchased industrial lenses suited to low-light imaging and designed, assembled, and calibrated a dedicated stereo–LiDAR rig. Supplementary Figure 8 shows that many LiDAR points remain in the ground truth after filtering, even where scene details are barely visible in the RGB image.

More RGB-only models can now produce visually plausible depth maps. Data for measuring their depth accuracy across the conditions humanoid robots encounter, including indoor and outdoor environments, day and night, underground spaces, and elevators, remain scarce. Our dataset addresses this timely need by providing reliable ground truth across these conditions.

I understand that today is the final day of the rebuttal period. I appreciate that the overlapping ICLR and CVPR deadlines have left many of you with limited time to engage in the discussion. Please feel free to leave any questions or comments at any time tonight. I will be monitoring the discussion throughout the night and will respond within five minutes. At this stage, further experiments are no longer feasible, but I would be happy to clarify our methods, discuss the existing results, or address any remaining concerns.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.