acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Streaming Video Diffusion Model: Context, Execution, and Training

Abstract

Understanding the design space of streaming video diffusion is essential to exploring its potential for generation quality and computational efficiency. We develop a unified analytical framework that relates model and sampler choices, historical conditioning, execution scheduling, and training strategies. The framework accommodates a broad family of causal context-selection policies and makes their computational dependencies and training–inference alignment explicit. Within this design space, we study three representative policies: clean, same-level, and progressive history. On the VBench evaluation set, the three policies use rank-64 fake-score LoRA, with same-level and progressive history achieving aggregate scores of 85.24 and 85.60, respectively, compared with 85.16 for clean history. Long-video comparisons further show improved subject consistency and more coherent motion with progressive history. By allowing multiple denoising nodes to be processed together, progressive-history pipelining achieves – steady-state DiT speedups under our evaluated conditions. We additionally find that LoRA adaptation of the DMD fake-score network improves generation quality under clean and progressive history using only 2.15% as many trainable fake-score parameters as full-parameter adaptation. Together, these findings show that fully denoised history is not a prerequisite for high-quality streaming generation and motivate the joint design of historical conditioning, execution, and training.

Primary area:
generative models
Keywords:
Streaming Video Generation, Video Diffusion Models, Context Selection, Pipelined Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t57iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the joint design of historical context, execution scheduling, and training in streaming video diffusion. It introduces a framework that distinguishes historical noise-state selection, attention visibility, context representations, training-state sources, and supervision objectives. Three representative policies—clean, same-level, and progressive history—are used to examine how historical conditioning determines dependencies between video chunks and the feasibility of pipelined denoising. The authors adapt a shared autoregressive video model to these policies through video-level distribution matching distillation. They also investigate LoRA adaptation of the auxiliary fake-score network, treating its parameterization as a separate training choice alongside the history policy.

Critical strengths
  1. A useful distinction between modeling choices and execution constraints. The framework clearly separates which historical noise states are selected from which token representations are visible to attention. It also distinguishes matching noise levels from matching the source and distribution of historical states. These distinctions are relevant to understanding training–inference consistency. The scheduling discussion correctly separates the number of sequential rounds from the total number of denoising updates and explicitly conditions the ideal pipeline schedule on sufficient concurrency and feature availability.

  2. Several meaningful experimental controls and transparent reporting. The three LoRA policy variants share initialization, training data, backbone, chunking, and a four-step sampling setup. Evaluation uses matched prompts and generation seeds, and all 16 VBench dimensions are reported. The authors acknowledge tradeoffs between motion, consistency, and appearance rather than claiming uniform improvement. They also explicitly delimit the timing measurements and include communication and cache operations.

  3. Useful findings about history selection. With fake-score LoRA, same-level history maintains aggregate quality comparable to clean history, while progressive history achieves the best aggregate quality and supports faster pipelined execution. Clean history performs better on several consistency metrics, revealing a practical tradeoff between motion dynamics, consistency, and efficiency.

  4. Promising results from fake-score LoRA adaptation. Under both clean and progressive history, LoRA adaptation yields better reported student generation quality than full-parameter fake-score updates while training substantially fewer auxiliary parameters. This suggests that limiting fake-score adaptation capacity can be beneficial in the evaluated setting.

Critical weaknesses
  1. Limited incremental contribution. The framework primarily organizes established ideas in noisy-history conditioning, distillation, and pipelined denoising. While this organization is useful, the paper does not establish a substantial methodological advance or sufficiently generalizable empirical insight. Its added value over closely related approaches, including Rolling Forcing and Causal-rCM, remains insufficiently demonstrated.

  2. Limited support for the central quality claims. The evaluation covers one backbone and a fixed sampling configuration, while long-video improvements rely on qualitative examples. Higher dynamic degree does not establish more coherent motion, and clean history remains stronger on several consistency metrics. Quantitative long-video evaluation and broader validation are needed to establish when the proposed policies are preferable.

  3. Underspecified construction of historical context. The training recipe uses random early exit, but it does not explain how progressive history obtains the required cleaner historical states when those denoising levels have not been visited. The corresponding KV construction and refresh rules are also unclear. These details are central to reproducing the method and verifying the framework's stated training–inference alignment.

Questions for authors
  1. What substantive new insight does the framework establish beyond existing noisy-history and pipelined methods? Can matched comparisons with Rolling Forcing and Causal-rCM clarify its added value?

  2. Do the claimed improvements in long-video consistency and motion coherence hold in quantitative or blinded human evaluations, and across different backbones or sampling configurations?

  3. How does random early exit interact with progressive history, particularly if requested intermediate noise levels are not visited? How are the corresponding historical states and KV representations constructed? Please provide pseudocode for state construction, cache refresh, and replay that clarifies how effective conditioning is aligned between training and inference.

Reviewer_pbRjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a framework for streaming video diffusion that separates historical-state selection, attention visibility, execution scheduling, state sources, and training objectives. The core insight is that fully denoised history is not required for competitive streaming-video quality and that context, execution, and training should be designed jointly.

Critical strengths
  1. In my opinion, the framework connects modeling and execution choices. Specifically, it separates the historical-state policy, attention mask, state source, supervision objective, and execution schedule. This is consequential because prior work often couples these choices in a specific implementation, making comparisons and extensions difficult.

  2. The result shows that partially denoised history can retain approximately clean-history quality. The progressive policy also permits substantially higher steady-state DiT throughput. This supports the paper’s central practical message even though the precise quality ranking is uncertain.

Critical weaknesses
  1. The quality advantage is not sufficiently established. The VBench gains over clean history are small, no uncertainty or multiple training runs are reported, and each policy selects its best checkpoint at a different iteration using the evaluation set. The long-video improvement is also supported only qualitatively.

  2. The efficiency evaluation is not sufficient. Table 2 compares clean-history serial execution with progressive-history pipelining, changing both policy and execution. It excludes pipeline fill/drain and VAE decoding, omits the pipeline-compatible same-level policy, and the 12-device speedup partly results from the clean baseline becoming slower. Thus, end-to-end streaming benefits remain unclear.

Questions for authors

Are the comparisons assessed by independent human raters or only by the authors?

Reviewer_7JvqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies which noise level of the history should be used as context in autoregressive streaming video diffusion. It expresses the choice as a policy over the current noise level and history distance, and compares three instances: clean history, same-level history, and progressive history where older chunks are at lower noise levels, which allows multiple chunks to be denoised in one pipelined forward pass. All three are trained from the same Causal-rCM Wan2.1-1.3B initialization with DMD. The paper reports VBench scores for the three policies, steady-state latency of pipelined versus serial execution on 4–12 devices, and a comparison of LoRA versus full-parameter fake-score networks in DMD.

Critical strengths
  • The question of which history noise level to condition on is an important research point for diffusion video generation, and the paper poses it cleanly through the policy.

  • The three policies are trained from the same initialization with the same DMD recipe, which makes the comparison controlled.

  • Hyper-parameters are reported in detail (Table 3).

Critical weaknesses
  • Lack of novelty. Clean history has been used in CausVid and Self Forcing, and progressive history has been explored in Rolling Diffusion and FIFO-Diffusion. The paper does not propose a new policy or mechanism on top of them.

  • The differences between the three policies on VBench (Table 1) are marginal, and no error bars or std are provided. Also, the checkpoints used for each policy were not taken at the same training step (Appendix B, L763), which further weakens the conclusion. In the per-dimension comparison (Table 4), progressive history leads in only 5 of 16 dimensions and is clearly behind clean history on scene (64.7 vs 70.3) and aesthetic quality, as well as on all four temporal-consistency dimensions.

  • I do not understand the numbers in Table 2. When the number of devices goes from 8 to 12, clean serial becomes slower than at 4 or 8 devices, while progressive pipeline stays at the same level as 8 devices. In addition, same-level history can also be pipelined according to Eq. 7, and would be a more informative baseline than clean serial.

  • The qualitative analysis is limited. Only two examples are provided in the whole paper (Fig. 3), and neither shows a clear difference between the policies.

Questions for authors

Please refer to the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.