- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to use speculative decoding for few-step auto-regressive video generation. It uses a per-token verification to early stop denoising process. The method adaptively decides how many inference steps should be allocated to a token. According to its visualization results, some low-frequency areas adaptively get fewer inference steps, while other high-frequency areas still get more inference steps to maintain the quality. The paper shows about 1.5x speed up compared to standard 4-step ar model. Overall, I think this paper gets a good balance between quality and efficiency.
Critical strengths
- The overall Idea is reasonable and simple. It is natural that some low-frequency areas may need fewer denoising steps to converge, while others may need more denoising steps. Adaptively deciding per-token NFE makes sense.
- The ablation experiments are comprehensive. It describes the selection choice of verification standards.
Critical weaknesses
- The effectiveness largely depends on the ability of the selected model. The speculative decoding itself doesn't improve the two-step or one-step generation quality of AR models.
- The concern of model collapse. The training-free verifier compares the token with the most similar clean history token may encourage the model to stay static e.g. copying from first frame?
- The shown case are little bit easy that can’t fully shows method’s capability.
Questions for authors
- The per-token speculative decoding although reduced overall computation amount. The per-frame inference time is still bottllenecked by the last finished token, where is the 4 step inference time. Would this be a big problem that harms the generation speed?
3 comments
response to rebuttal
Thank you for your response. My concerns have been mostly addressed, thus I maintain my original score.
We sincerely thank the reviewer for taking the time to carefully consider our rebuttal and revised manuscript. We are glad that our responses have addressed your concerns, and we appreciate your constructive feedback, which has been valuable in improving our work. We would be happy to provide any further clarifications if needed.
Response to reviewer xj5a
All changes are highlighted in blue in the revised manuscript. Table, section, and appendix references in our responses refer to the revised version.
W1: Quality Preservation and Generalization Beyond Four-Step Sampling
We agree that Speculative Forcing does not intrinsically improve the quality of the underlying one-step generator. Rather, our objective is to avoid unnecessary refinement for tokens whose intermediate clean estimates have already converged, while preserving additional denoising for tokens that require it.
This distinction allows adaptive computation to preserve the quality of the full-step generator without uniformly applying all denoising steps. To further validate its applicability beyond the original 1.3B backbones, we additionally evaluated Krea Realtime 14B [1], with the results included in Section 4.2 (Table 4). Our learned verifier achieves 1.85× speedup over dense four-step sampling (8.86 vs. 4.78 FPS) with nearly unchanged VBench Total (84.85 vs. 84.88). Our framework is not inherently limited to four-step generators, and these results provide indirect evidence of its potential applicability to two-step generators, demonstrating that high generation quality can be maintained with fewer than two denoising steps on average.
References
[1] Millon, E. Krea Realtime 14B: Real-time Video Generation. GitHub repository, 2025. https://github.com/krea-ai/realtime-video.
W2: Risk of Static Generation or History Copying
We appreciate the reviewer's concern regarding potential static collapse. Our training-free verifier does not copy historical tokens or encourage predictions to resemble past observations. Instead, it uses history/reference latents to assess whether the current clean prediction has reached a sufficiently stable level of convergence. Upon acceptance, the model commits its own predicted clean estimate, without replacing it with any historical token.
Empirically, we observe no evidence of suppressed motion in either short or long video generation. As shown in Table 2, our training-free variant achieves a Dynamic Degree of 41.57, compared with 39.17 for uniform four-step sampling in long-video generation. This observation is further supported by Appendix C.3 (Tables C and E), where Dynamic Degree remains comparable to full-step sampling across short and long video settings, providing additional evidence against motion suppression or static collapse.
W3: Challenging Qualitative Examples
Thank you for this suggestion. Beyond the VBench prompt-based examples in the main paper, our supplementary qualitative results also include more challenging world-generation scenarios involving dynamic object interactions and camera movement. These examples further illustrate that Speculative Forcing maintains generation quality under diverse motion and viewpoint changes, beyond relatively simple scenes.
Q1: Last-Token Bottleneck and Practical Inference Speedup
We agree that the final active token can determine the number of denoising iterations executed for a chunk. However, the computational cost of each DiT forward is not constant under token-wise commitment. Accepted tokens are excluded from subsequent denoising computation, while their attention context remains available through cached KV states. Thus, even when a few tokens require all four steps, later forwards operate on progressively smaller active-token sets.
We additionally provide a wall-clock runtime breakdown in Appendix C.5 (Table G). The realized speedup does not scale linearly with the mean denoising steps, since each executed forward incurs fixed overhead, and an additional clean KV-cache refresh pass is required. Our wall-clock profiling on LongLive-30s shows that denoising-forward latency decreases from 410.7 to 235.8 ms/chunk for the learned verifier.
Furthermore, in our additional Krea Realtime 14B evaluation (Section 4.2, Table 4), the learned verifier achieves 1.85× measured DiT throughput over uniform four-step sampling (8.86 vs. 4.78 FPS), despite the last-token bottleneck. This demonstrates that token-wise commitment provides substantial practical acceleration even when some tokens require the full denoising schedule.