acceptodds
Under review as a conference paper at ICLR 2027

HybridDepth: Local Detail and Long-Horizon Consistency for Online Video Depth Estimation

Abstract

Online video depth estimation must preserve useful temporal evidence while processing frames causally under a bounded memory budget. Existing streaming methods typically rely on one of two memory strategies with complementary limitations: a short causal window preserves recent fine-grained information but discards older observations, whereas a fixed-size recurrent state carries information over longer horizons but compresses history and can lose frame-specific details. The key challenge is therefore not simply to retain more history, but to coordinate recent explicit evidence with compressed longer-range context under bounded memory. We propose HybridDepth, a coordinated dual-memory framework for online video depth estimation. HybridDepth maintains a local token window and a recurrent global state as separate memory paths, allowing the current frame to access both recent detail and longer-range context without growing temporal storage with sequence length. To coordinate these memories, we introduce an attention-evidence gating mechanism that modulates each branch at the token level using the model's own query-key relevance. We further introduce an orthogonal learning objective that discourages the local and global branches from producing redundant corrections. Together, these designs preserve distinct temporal roles while allowing their contributions to adapt to the current observation. Across NuScenes, KITTI, Bonn, and TUM-Dynamic, HybridDepth achieves the best average depth accuracy among the compared online methods at 50, 100, and 200 frames. On 50-frame Bonn sequences, it also achieves the lowest temporal alignment error, providing direct evidence of improved adjacent-frame geometric consistency. At 500 frames, HybridDepth remains competitive, although it is not the best average method, while retaining bounded temporal memory. It operates at 24.94 FPS on a single H200. These results demonstrate a useful accuracy, temporal consistency, and memory trade-off for causal video depth estimation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Depth Estimation, Online System, Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dToyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a method alongside CUT3R’s recurrent state tokens for dual-memory streaming depth estimation. The two memories, local and global, are read through parallel cross-attention branches. The method is evaluated on NuScenes, KITTI, Bonn, and TUM-Dyn at 50–500 frames.

Critical strengths

The problem is well motivated. The windowed and recurrent memory design is a clean framing. The overhead is modest: about 4% more parameters and roughly 25 FPS on an H200.

Critical weaknesses
  1. The major improvement of this paper seems to come from fine-tuning on datasets rather than the memory design. The baseline + gating in Table 2 beats other methods by over 20%, while the gain from the memory design is minimal, at merely 4%. The strong gain is from the gating, but the gating mechanism is weakly justified. Meanwhile, given the degraded numerical results on 500 frames, I doubt the usability of the local and global memory design.
  2. Missing critical baselines. STream3R, Point3R, and Online Video Depth Anything are all cited but not compared.
Questions for authors
  1. What is the baseline in Table 2? The fine-tuned CUT3R?
  2. A fairness concern about the evaluation protocol. FlashDepth targets 2K, while CUT3R and StreamVGGT run around 512. FlashDepth natively predicts relative disparity, while CUT3R-family models predict pointmaps. If every method is aligned in one space, some baselines are being evaluated outside their native output. So, how did the authors handle evaluation fairness?
  3. Figure 2 does not show an improvement of the proposed method, but raises a concern about the relative location of the balloon and the people. If a fair visualization protocol were used, the improvement would be mainly in temporal consistency, as shown in Table 4. The authors are encouraged to add results on TAE to both Tables 1 and 2.
Reviewer_MSmTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

HybridDepth estimates relative depth from video using only current and past frames. Building on CUT3R, it combines a local memory that retains recent frame details with a recurrent state that compresses longer-term history. Separate attention branches read the two memories, and gates derived from attention scores control their contributions to the current prediction. An orthogonality loss discourages redundant updates between the branches. The aim is to retain both recent detail and older context without increasing memory storage as the video grows.

Critical strengths
  1. Sound Design and Useful Ablations: Keeping recent observations alongside a compressed global state is a sensible way to preserve local detail with bounded memory. The matched ablations support the use of both memories: at 200 frames, the combined model improves AbsRel from 9.48 for global-only memory to 8.92 (Table 6). This shows that the local cache provides useful information beyond the recurrent state.

  2. Writing: I found the paper easy to follow. Figure 1 clearly separates memory reading from state updating, and the equations explain how the two branches contribute to each prediction. The authors also acknowledge the weaker long-sequence results and the limited scope of the temporal consistency evaluation.

Critical weaknesses
  1. Limited Gains on Long Videos: This is my main concern, since retaining useful history is a central motivation for the method. The gains over TTT3R at shorter horizons do not hold consistently on long videos. On the common KITTI/Bonn/TUM-Dyn subset in Table 5, average AbsRel at 500 frames is 10.12 versus TTT3R's 9.80, although HybridDepth still performs best on KITTI. Temporal consistency is evaluated only on 50-frame Bonn sequences. These results leave me unconvinced that the additional memory design offers a sufficiently strong advantage for long-video depth estimation.

  2. Limited Methodological Novelty: TTT3R already uses attention-derived per-token gating. HybridDepth applies it to memory reads rather than state updates, so the roles differ. The contribution is mainly the coordination of local and global memory with an orthogonality loss. I see this as a useful but incremental extension, which makes the limited gains on long videos more consequential.

The approach is reasonable and well presented, and the ablations show that the two memories work well together. However, the advantage over a strong existing method on long videos is limited. Combined with the incremental design, this leads me to a Weak Reject.

Questions for authors
  1. Using the existing long-sequence predictions, could you show how errors develop over time relative to TTT3R, especially on Bonn and TUM-Dyn? This would help explain where the shorter-sequence gains are lost.
Reviewer_vsqkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HybridDepth, a method for estimating depth from a video stream as frames arrive, without letting memory grow with the length of the video. It keeps two memories side by side: a sliding window of features from the last W frames, and a fixed-size global feature. The current frame reads each memory through its own cross-attention branch. Each branch’s update is then scaled by a token-level gate built from the average of that branch’s attention scores, which adds no parameters. A cosine-similarity loss pushes the two branches toward making different corrections, and training uses sequence-level scale-and-shift and relative-depth losses. Starting from CUT3R weights, the model is fine-tuned on about 40K sequences from six datasets. On NuScenes, KITTI, Bonn and TUM-Dynamic it reports the best average accuracy among the compared online methods at 50, 100 and 200 frames, with the error 10.4%, 5.5% and 2.8% below the next-best method. It also has the lowest temporal alignment error on Bonn (0.55), the lowest peak memory, and runs at 24.94 FPS on one H200. At 500 frames it loses to TTT3R and degrades faster with length, which the authors acknowledge.

Critical strengths

The paper’s most convincing design choice is to keep the raw tokens from the last W = 4 frames in a sliding-window KV cache, alongside the compressed recurrent state. Recurrent-only methods like CUT3R and TTT3R squeeze all of history into a fixed set of state tokens, so details from individual frames get diluted or overwritten. HybridDepth’s window keeps that recent evidence uncompressed and reads it through its own cross-attention branch, so the current frame can look up fine detail directly instead of rebuilding it from a lossy summary. Memory stays bounded at O(WK) rather than growing with the sequence.

Critical weaknesses

The baseline comparison is confounded. HybridDepth starts from CUT3R and is then fine-tuned on six more datasets. The paper’s own ablation baseline already beats CUT3R on NuScenes, KITTI, and Bonn before any of the proposed parts are added. So part of the headline gain may come from the extra training rather than the memory design. A fair test would fine-tune CUT3R and TTT3R on the same data. The ablation “Baseline” (row #1 in the ablation table) is also never defined: it isn’t clear whether it already has both memories with plain summed updates.

Questions for authors
  1. Since softmax is invariant to per-query additive shifts, the proposed gate, the mean pre-softmax score q·k̄/√d, depends only on the component of the attention logits that the attention itself discards. It is therefore unclear why it should reflect memory relevance. In particular, appearance shifts that are irrelevant to geometry, such as exposure or white-balance changes between the current frame and the cached frames, or systematic scale differences between image tokens and state tokens, may also change the gate without changing relevance. Could the authors (i) specify the activation function σ; (ii) report gate statistics and depth error under photometric perturbations, including exposure changes between frames; (iii) visualize the value of per-token gates with respect to the frame contents.
  2. The explicit local memory contains only the W most recent frames. For sequences that revisit earlier viewpoints, such as handheld indoor scenes like TUM and Bonn, the only route back to earlier observations is the compressed global state. Since the CUT3R backbone already predicts camera poses, have the authors considered filling the explicit memory with spatially retrieved keyframes, selected by predicted-pose frustum overlap or feature similarity, under the same memory budget? An ablation comparing recent, retrieved and mixed windows at fixed W would clarify whether the benefit of local memory comes from recency or from spatial relevance, and may address the degradation at 500 frames. A discussion of, or comparison with, spatial-memory methods such as Spann3R and Point3R would also strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.