- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Dandelion, a generative framework for streaming video depth estimation. To address challenges in inference latency, temporal consistency, and historical information utilization, the authors adapt several techniques from autoregressive video generation, including chunk-wise causal modeling, two-stage training, and multi-scale memory.
Overall, the paper achieves strong results in depth estimation accuracy, temporal consistency, and inference efficiency, demonstrating practical value.
However, my main concern is that most core components closely resemble existing autoregressive video generation methods, yet their relationships and technical differences receive little or no discussion.
My final rating will depend on the authors' response.
Critical strengths
-
The paper systematically analyzes several key challenges in streaming video depth estimation. The overall methodology is clearly presented.
-
The paper achieves strong depth estimation accuracy and temporal consistency while maintaining real-time inference efficiency, demonstrating its practical value.
Critical weaknesses
Most core components of this paper are closely related to existing autoregressive video generation methods. However, the authors do not adequately discuss these prior works.
Specifically:
-
Two-stage Training Strategy: The second stage uses model-generated depth predictions as autoregressive history, which is consistent with the underlying idea of Self Forcing [1], which addresses the train–inference gap through self-generated history. However, the paper lacks a discussion of their relationship and differences.
-
Multi-scale Depth Memory: This module closely resembles the history compression mechanism in FramePack [2], even adopting the same three-level compression configurations. However, FramePack is neither cited nor discussed in the manuscript.
-
Processing video in chunks with bidirectional attention within each chunk and causal attention across chunks is also a commonly adopted design in autoregressive video works.
While I acknowledge the practical value of this work, I believe that adequately discussing related work and clearly distinguishing technical contributions from existing methods are particularly important for a research conference like ICLR.
[1] Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, NeurIPS 2025
[2] Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models, NeurIPS 2025
Questions for authors
- How do the core components of this paper relate to and differ from the prior methods mentioned in the Weaknesses?
I also have the following specific questions:
-
In Sec. 3.4, historical depth latents are compressed at different spatiotemporal scales and concatenated with the current RGB tokens before being fed into the DiT. However, it is unclear how RoPE positional encoding is handled for tokens.
-
Fig. 5 presents qualitative comparisons with and without the two-stage training strategy. Since the second stage primarily aims to improve robustness to accumulated prediction errors during autoregressive inference, it would be helpful to clarify whether the visualized depth predictions come from the initial chunk or subsequent chunks.
-
In Fig. 9, Only Stage 2 eventually outperforms Only Stage 1, which is somewhat surprising and deserves further analysis. Additionally, the training procedure for Only Stage 2 needs clarification, particularly how it is implemented without initialization from Stage 1.
6 comments
Q4: Stage 2 Initialization and Performance
Only Stage 1 uses GT depth history throughout training, whereas the depth history in Only Stage 2 is generated by the current model from the outset. Both variants start from pretrained streaming video generation weights.
We interpret Fig. 9 as follows: GT history provides reliable conditioning for faster initial learning but leaves a mismatch at inference. Only Stage 2 is initialized directly from pretrained streaming video generation weights, without using a Stage 1 checkpoint. It already has some ability to extract information from temporal history. Although it faces more challenging conditioning initially, it is trained under history conditions that better match autoregressive inference, consistent with its eventual advantage over Only Stage 1.
Thank you for the clarification. Throughout my review, I have consistently acknowledged the motivation for adapting a FramePack-like mechanism to the current task. The more important thing is that the similarities between the proposed design and FramePack go far beyond the high-level idea. The implementation details, including the RoPE alignment strategy newly clarified in the rebuttal, are all highly consistent with FramePack.
Were these designs independently developed from scratch, or were they adapted from FramePack for the current task? If the latter, why was FramePack neither cited in the original manuscript nor explicitly acknowledged in the rebuttal, while these technical designs were presented as the authors' own contributions?
Overall:
We sincerely thank the reviewer for recognizing our
systematic analysis,strong depth estimation performance,temporal consistency, andinference efficiency. Below, we address each weakness (W) and question (Q) point by point. We will incorporate the clarifications and citations into the revised manuscript.W1 & Q1: Differences & Relationships from Self Forcing
The two methods address fundamentally distinct tasks: Dandelion focuses on accurate and efficient geometry estimation, whereas Self Forcing focuses on generating robust videos. The former has corresponding GT depth available for constructing historical geometric context, whereas the latter lacks a unique GT, as generation requires diversity. This difference leads to the following distinctions in geometric anchoring.
Stage 1 uses GT depth as depth history to establish a reliable geometric anchor. The model learns to use this anchor as a geometric reference to guide subsequent predictions. Stage 2 inherits the geometric anchoring prior learned in Stage 1 and continually updates the geometric reference with new depth predictions. In contrast, Self Forcing uses self-generated RGB history to guide video generation and optimizes distribution matching, without a corresponding geometric anchor. Moreover, we admit that using self-generated histories reduces the training–inference distribution gap, which is shared by both methods.
W2 & Q1: Differences from FramePack
The tasks differ fundamentally. Dandelion addresses a geometric prediction task, recovering accurate depth from observed RGB videos. FramePack [1] addresses a video generation task, synthesizing diverse RGB content. This distinction in task objectives fundamentally changes the purpose and motivation of multi-scale history compression.
Our three-level compression is motivated by the observed temporal propagation of depth history. Fig. 6(b) shows three levels of detail relevance across temporal distances. Recent history exhibits strong detail relevance, whereas distant history retains persistent scale similarity. These observations motivate preserving recent geometric details while encoding intermediate and distant history at progressively coarser resolutions to retain global geometric guidance. In contrast, FramePack studies context packing based on temporal proximity for video generation, unable to investigate these depth-specific characteristics, such as detail relevance and scale similarity.
[1] Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models, NeurIPS 2025
W3 & Q1: Differences from Chunk Design
Notably, to our knowledge, prior streaming video depth estimators operate frame by frame and have not explored the chunk-based mechanism. Our experiments in Sec. 4.2 show that frame-wise processing is suboptimal, motivating us to introduce streaming chunks for video depth estimation.
Then, the information processed within each chunk differs: Dandelion uses predicted depth history and current RGB observations, treating past geometry as a reference and limited future RGB observations as supplementary evidence. This chunk-based mechanism allows causal attention across chunks to convey past depth information to the current chunk, where bidirectional attention integrates this geometric reference with current RGB observations to predict the current depth. In contrast, autoregressive video diffusion generators condition on generated RGB content from past chunks and denoise target latents initialized from noise in the current chunk, using within-chunk interaction to synthesize new RGB content.
Finally, our core contribution here is the systematic analysis of chunk length for streaming depth estimation. This analysis is independent of studies in video generation and, to our knowledge, is the first systematic study of chunk length for streaming video depth estimation. Our experiments further identify an optimal chunk length tailored to depth estimation.
Q2: RoPE Alignment
We use a shared 3D RoPE coordinate system for historical depth and current RGB tokens. Within each rolling window, consecutive temporal indices are assigned to the history followed by the current chunk. To align RoPE with compressed history tokens, we average-pool its cosine/sine tensors on the base patch grid with factors of (1,1,1), (2,2,2), and (4,4,4) for short-, mid-, and long-range history, respectively, with corresponding boundary padding. The aligned positional encodings are concatenated in the same [long, mid, short, RGB] order as the tokens and applied to queries and keys before attention.
Q3: Chunk Selection
The examples in Fig. 5 are taken from subsequent chunks during autoregressive inference.