- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper unifies five video tasks—T2V, V2V, first-frame edit propagation, reference-guided editing, and camera pose control—in a single autoregressive framework, with a dual-type conditioning interface that is a clean and practical design choice. The main methodological contribution is the VMM-plus-unrolling distillation recipe, which converts a bidirectional teacher to a few-step causal student without offline ODE pairs or teacher causalization, a genuine simplification over prior conversion pipelines. Results show competitive quality and 10.56 FPS throughput at 720p. Weaknesses include the absence of a unified quantitative metric across all five tasks (several are evaluated qualitatively), and no real-time playback at full resolution. The claim of generality across tasks is supported but could be strengthened with a more systematic cross-task ablation. Overall, a useful and well-engineered contribution.
Critical strengths
1.The paper precisely identifies the difficulty of converting a strong multi-task bidirectional video model into a few-step causal student, and convincingly argues that existing conversion pipelines (offline ODE-pair preparation, teacher causalization) are prohibitively costly across multiple tasks.
2.Matching conditional velocity moments in velocity space, combined with autoregressive unrolling, avoids the distribution-matching instability of DMD and the need for offline ODE pairs, aligning supervision more closely with the few-step inference trajectory.
3.It achieves 0.90 s time-to-first-output and 10.56 FPS throughput at 720p on a single H100, and Energy Annealing effectively mitigates over-saturation, demonstrating clear potential for interactive workflows.
Critical weaknesses
1.Lack of a unified quantitative metric across tasks and insufficient ablation. The five tasks are not evaluated under a single quantitative framework; several (e.g., first-frame propagation and egocentric editing) are only compared qualitatively, making it hard to systematically assess the overall benefit of the unified model. Independent contributions of the dual-type conditioning interface, task-mixing ratios, and Energy Annealing hyperparameters are not fully isolated, leaving the causal attribution of gains partly unestablished.
2.Full-resolution playback is at only 10.56 FPS under 720p, the system falls short of playback rate, so the real-time interactive value proposition still highly depends on further inference techniques like decoder distillation or quantization.
3.Long-horizon stability is under-evaluated. Despite supporting streaming generation, error accumulation and cross-chunk consistency over long sequences are only quantitatively characterized to a limited extent, and no systematic comparison against memory/cache-based long-horizon methods is provided.
Questions for authors
What is the performance of the model on long-horizon stability? How to prove the model performs better under all five tasks are united? Others please refer to the weaknesses.
5 comments
Q2: Inference Efficiency Thank you for raising inference efficiency. We benchmark several configurations using student checkpoint and report time to first frame (TTFF) and throughput.
| Configuration | TTFF (s) ↓ | FPS ↑ |
|---|---|---|
| Baseline: single GPU, original decoder | 0.90 | 10.56 |
| TinyVAE decoder only | 0.80 | 13.0 |
| 480p | 0.72 | 14.5 |
| Async 3-GPU pipeline (encode / DiT / decode) | 1.29 | 19.8 |
| Async 3-GPU + compiled DiT + TinyVAE decoder | 0.82 | 22.4 |
The asynchronous three-GPU pipeline with DiT torch.compile and TinyVAE achieves 22.4 FPS with 0.82 s TTFF, a 2.12× throughput improvement over the single-GPU/original-decoder baseline. Because it uses additional GPUs, this is a system-level throughput gain rather than a per-GPU efficiency gain.
Further kernel optimization, quantization, inference infrastructure improvements, and newer hardware such as B200 could further improve TTFF and FPS. These deployment optimizations are complementary to our method and beyond this paper's scope.
Q3:Long-horizon Stability We originally listed long-horizon generation as a limitation because this paper focuses on the distillation recipe, while long-horizon generation needs further study. Following the reviewer’s suggestion, we now include additional discussion of the inference-pipeline strategies we explored when the generation is beyond the training 5 seconds.
We add a 60 s evaluation of the four-step student (704×1280, 16 FPS). Beyond the 81-frame training length, it retains a 21-latent-frame KV window, 3 static sink frames, and a rolling sink that reintroduces early history. The eviction begins at about 5 s.
We extend 22 EditVerseBench videos (2/edit type) to 961 frames by forward/backward playback. The 160-frame cycle lets us measure drift on repeated source frames. SANA-Streaming uses its released long-streaming checkpoint, default settings, and identical inputs.
| Method | CLIP-T last ↑ | Δ | DINO src last ↑ | Δ | Sharp. last →1 | Cyc. PSNR last ↑ | Δ | Jumps/min ↓ |
|---|---|---|---|---|---|---|---|---|
| Ours (static + rolling sink) | 0.271 | -0.002 | 0.703 | -0.030 | 0.93 | 25.44 | +0.04 | 1.18 |
| Ours w/o rolling sink | 0.248 | -0.026 | 0.539 | -0.195 | 0.98 | 17.81 | -2.76 | 2.91 |
| Ours w/o static sink | 0.266 | -0.007 | 0.680 | -0.054 | 1.16 | 24.52 | +0.28 | 1.36 |
| Sliding-window KV | 0.158 | -0.115 | 0.071 | -0.662 | 23.65 | 8.88 | -7.62 | 9.76 |
| SANA-Streaming | 0.245 | -0.014 | 0.614 | -0.153 | 1.25 | 14.64 | -5.92 | 5.09 |
Scores are averaged over videos. “Last”: final 5 s; Δ: last minus first window (Cyc. PSNR: last minus first repeated cycle). CLIP-T: edit/text alignment; DINO src: source structural similarity; Sharp.: sharpness relative to the first window (1=unchanged; ≫1 indicates artifacts); Cyc. PSNR: repeated-frame output consistency; Jumps: motion-compensated discontinuities/min.
Our edits remain stable over 60 s: CLIP-T Δ=-0.002, DINO Δ=-0.030, and Cyc. PSNR=25.44 dB with Δ=+0.04 dB (95% CI includes zero). We observe no measurable cross-cycle drift after eviction on this test.
The rolling sink is the key: removing it lowers Cyc. PSNR to 17.81 dB, versus 24.52 without the static sink. Sliding-window KV collapses. SANA-Streaming's GDN memory shows more drift (Cyc. PSNR Δ=-5.92 dB; DINO Δ=-0.153) and discontinuities (5.09/min).
Rebuttal
Q1:Unified Tasks and Ablations We clarify that our claim is not that a unified model outperforms every single-task specialist. Rather, our conditioning interface enables joint training across tasks, followed by direct conversion into one autoregressive streaming model.
To strengthen quantitative coverage, we add a common assessment of editing propagation and egocentric editing. GPT6.1-Sol evaluates six evenly spaced frames using edit consistency (E), content preservation (P), temporal/motion consistency (T), and visual quality(Q), each in [0,5], with S=8E+5P+4T+3Q.
Editing propagation (GenPropBench)
Egocentric editing
On these released EgoEdit examples, EditStream is broadly comparable to EgoEdit and scores substantially above SANA-Streaming, which was not trained on egocentric cases.
For the conditioning interface, matched 2k-step runs give hybrid/full-token EditVerse scores of 6.68/6.64 and RefVIE scores of 2.91/2.95. Hybrid inference is 2.24×/2.36× faster at 480p/720p, while full-token conditioning better preserves backgrounds (2.63 vs. 2.45).
For distillation, paired ablations compare random exit/full rollout within Self-Forcing and VMM/DMD within our recipe, from the same warm-up checkpoint. Full rollout improves VBench from 84.13 to 84.39; VMM/DMD in our recipe obtain 84.42/84.27 and identical EditVerse scores of 7.35, supporting the combined recipe.
For Energy Annealing, Updated Table 5 and Figure 6 compares EA=0.7, no EA, and EA=0.2 over 946 generations. No EA increases color richness but also clipping/flicker; EA=0.2 reduces artifacts with flatter colors, while EA=0.7 balances both. The training-data mixture and sampling ratios are detailed in Appendix A.2.
These evaluations support our goal of unifying multiple generation/editing tasks and efficiently converting the joint model to streaming inference.