acceptodds
Under review as a conference paper at ICLR 2027

Speculative Forcing for Efficient Autoregressive Video Generation

12 views

Abstract

Few-step autoregressive video generators enable streaming generation, yet they run every token through the final denoising step, regardless of how much refinement it needs. We find this uniform computation is often unnecessary: many tokens already closely match their eventual full-step outputs in the initial steps, while others require refinement until the end. We introduce Speculative Forcing, a framework that treats intermediate clean estimates as drafts and predicts when individual tokens can be safely accepted early. Once accepted, a token is frozen as its final output and undergoes no further denoising, while its cached keys and values remain available as context for the remaining tokens. We explore two alternative verifier designs: a training-free one based on similarity to clean content and stability across steps, and a learned one that reads the frozen generator's intermediate features to estimate each token's remaining gap to its full-step output. Across short-video, long-video, and world generation, Speculative Forcing preserves the quality of full-step sampling while substantially reducing computation. On LongLive, it matches the four-step generator with  1.7 denoising steps per token, running over 1.5× faster at 41.7 and 37.8 fps. On Lingbot-World 2.0, where camera motion continually changes the visible content, it maintains quality with  2.5 denoising steps per token and runs 1.35× faster at 24.5 fps.

Primary area:
generative models
Keywords:
Efficient Video Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xj5aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to use speculative decoding for few-step auto-regressive video generation. It uses a per-token verification to early stop denoising process. The method adaptively decides how many inference steps should be allocated to a token. According to its visualization results, some low-frequency areas adaptively get fewer inference steps, while other high-frequency areas still get more inference steps to maintain the quality. The paper shows about 1.5x speed up compared to standard 4-step ar model. Overall, I think this paper gets a good balance between quality and efficiency.

Critical strengths
  • The overall Idea is reasonable and simple. It is natural that some low-frequency areas may need fewer denoising steps to converge, while others may need more denoising steps. Adaptively deciding per-token NFE makes sense.
  • The ablation experiments are comprehensive. It describes the selection choice of verification standards.
Critical weaknesses
  • The effectiveness largely depends on the ability of the selected model. The speculative decoding itself doesn't improve the two-step or one-step generation quality of AR models.
  • The concern of model collapse. The training-free verifier compares the token with the most similar clean history token may encourage the model to stay static e.g. copying from first frame?
  • The shown case are little bit easy that can’t fully shows method’s capability.
Questions for authors
  • The per-token speculative decoding although reduced overall computation amount. The per-frame inference time is still bottllenecked by the last finished token, where is the 4 step inference time. Would this be a big problem that harms the generation speed?
3 comments
AuthorsOpenReview ↗

Response to reviewer xj5a

All changes are highlighted in blue in the revised manuscript. Table, section, and appendix references in our responses refer to the revised version.

W1: Quality Preservation and Generalization Beyond Four-Step Sampling

We agree that Speculative Forcing does not intrinsically improve the quality of the underlying one-step generator. Rather, our objective is to avoid unnecessary refinement for tokens whose intermediate clean estimates have already converged, while preserving additional denoising for tokens that require it.

This distinction allows adaptive computation to preserve the quality of the full-step generator without uniformly applying all denoising steps. To further validate its applicability beyond the original 1.3B backbones, we additionally evaluated Krea Realtime 14B [1], with the results included in Section 4.2 (Table 4). Our learned verifier achieves 1.85× speedup over dense four-step sampling (8.86 vs. 4.78 FPS) with nearly unchanged VBench Total (84.85 vs. 84.88). Our framework is not inherently limited to four-step generators, and these results provide indirect evidence of its potential applicability to two-step generators, demonstrating that high generation quality can be maintained with fewer than two denoising steps on average.

References

[1] Millon, E. Krea Realtime 14B: Real-time Video Generation. GitHub repository, 2025. https://github.com/krea-ai/realtime-video.

W2: Risk of Static Generation or History Copying

We appreciate the reviewer's concern regarding potential static collapse. Our training-free verifier does not copy historical tokens or encourage predictions to resemble past observations. Instead, it uses history/reference latents to assess whether the current clean prediction has reached a sufficiently stable level of convergence. Upon acceptance, the model commits its own predicted clean estimate, without replacing it with any historical token.

Empirically, we observe no evidence of suppressed motion in either short or long video generation. As shown in Table 2, our training-free variant achieves a Dynamic Degree of 41.57, compared with 39.17 for uniform four-step sampling in long-video generation. This observation is further supported by Appendix C.3 (Tables C and E), where Dynamic Degree remains comparable to full-step sampling across short and long video settings, providing additional evidence against motion suppression or static collapse.

W3: Challenging Qualitative Examples

Thank you for this suggestion. Beyond the VBench prompt-based examples in the main paper, our supplementary qualitative results also include more challenging world-generation scenarios involving dynamic object interactions and camera movement. These examples further illustrate that Speculative Forcing maintains generation quality under diverse motion and viewpoint changes, beyond relatively simple scenes.

Q1: Last-Token Bottleneck and Practical Inference Speedup

We agree that the final active token can determine the number of denoising iterations executed for a chunk. However, the computational cost of each DiT forward is not constant under token-wise commitment. Accepted tokens are excluded from subsequent denoising computation, while their attention context remains available through cached KV states. Thus, even when a few tokens require all four steps, later forwards operate on progressively smaller active-token sets.

We additionally provide a wall-clock runtime breakdown in Appendix C.5 (Table G). The realized speedup does not scale linearly with the mean denoising steps, since each executed forward incurs fixed overhead, and an additional clean KV-cache refresh pass is required. Our wall-clock profiling on LongLive-30s shows that denoising-forward latency decreases from 410.7 to 235.8 ms/chunk for the learned verifier.

Furthermore, in our additional Krea Realtime 14B evaluation (Section 4.2, Table 4), the learned verifier achieves 1.85× measured DiT throughput over uniform four-step sampling (8.86 vs. 4.78 FPS), despite the last-token bottleneck. This demonstrates that token-wise commitment provides substantial practical acceleration even when some tokens require the full denoising schedule.

Reviewer_xj5aOpenReview ↗

response to rebuttal

Thank you for your response. My concerns have been mostly addressed, thus I maintain my original score.

We sincerely thank the reviewer for taking the time to carefully consider our rebuttal and revised manuscript. We are glad that our responses have addressed your concerns, and we appreciate your constructive feedback, which has been valuable in improving our work. We would be happy to provide any further clarifications if needed.

Reviewer_32jHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Speculative Forcing, an inference-time acceleration framework for few-step autoregressive video generation that allows individual tokens to stop denoising once they are predicted to have converged. The method retains cached keys and values of accepted tokens and introduces two verification strategies: a training-free verifier based on historical similarity and inter-step stability, and a learned verifier that predicts each token's remaining discrepancy from its full-step output. Experiments on LongLive and Lingbot-World 2.0 cover short-video, long-video, and world generation. The results show that the method achieves generation quality comparable to that of four-step sampling while reducing the average number of denoising steps and improving inference throughput.

Critical strengths
  1. The paper addresses the practical challenge of accelerating few-step autoregressive video generation without sacrificing generation quality. Speculative Forcing offers an interesting approach through token-wise early commitment, selectively avoiding redundant denoising computation while preserving attention context, without modifying the underlying generator's weights.

  2. The paper evaluates Speculative Forcing on both short- and long-video generation using LongLive, as well as controllable world generation using Lingbot-World 2.0. The experiments demonstrate 1.35–1.58× inference throughput improvements while maintaining comparable generation quality. Additional ablation studies examine verification strategies and token-wise versus chunk-wise commitment, providing useful evidence of the method's effectiveness.

Critical weaknesses
  1. The main experiments primarily compare Speculative Forcing against uniform-step sampling, which demonstrates its effectiveness but does not fully establish its advantages over existing adaptive methods. DSA [1] and Surprise Forcing [2] directly investigate adaptive denoising in few-step autoregressive video generation. However, DSA is not experimentally compared, while Surprise Forcing is only evaluated on LongLive in Appendix C.3, without comparisons under matched computational budgets. HSA [3] also explores token-wise heterogeneous computation and KV-cache preservation for video diffusion, although under a different sampling setting. Stronger comparisons under compatible settings would make the experimental evaluation more convincing.

  2. Although the paper provides ablation studies on verification strategies and acceptance granularity, some improvements over simpler alternatives are relatively modest. For example, Table 5 shows that token-wise commitment achieves only a slightly higher Quality Score than chunk-wise acceptance (82.65 vs. 82.54), with marginally lower throughput (24.5 vs. 25.2 FPS), although it improves Drift. Similarly, the learned verifier provides only small quality improvements over the training-free verifier. The paper does not report statistical uncertainty for these differences, making their reliability difficult to assess. Furthermore, the reported throughput excludes VAE encoding and decoding, leaving the end-to-end acceleration benefits insufficiently characterized. Additional analysis would strengthen the evidence supporting the proposed design choices.

References

[1] DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation. arXiv:2606.04432, 2026.

[2] Surprise Forcing: What to Remember, When to Skip in Long Video Generation. arXiv:2607.18436, 2026.

[3] Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation. arXiv:2605.06892, 2026.

Questions for authors

Please see the Critical Weaknesses above for my main concerns and related questions.

  1. Could the authors provide comparisons with DSA [1] and Surprise Forcing [2] under matched computational budgets, and clarify the differences from HSA [3] or whether comparable evaluations are feasible?

  2. Could the authors further clarify the benefits of token-wise commitment over chunk-wise acceptance, particularly given the relatively small Quality Score difference in Table 5?

  3. Could the authors provide statistical uncertainty estimates to establish the reliability of the reported quality improvements over the baselines?

  4. Could the authors report end-to-end inference speedups including VAE encoding and decoding, and clarify the computational overhead introduced by the verification and KV-cache management?

Overall, the paper is well written and clearly presented. Although the underlying idea of adaptive computation offers limited novelty, its application to autoregressive video generation addresses a practical problem.

5 comments
AuthorsOpenReview ↗

Response to reviewer 32jH (1/3)

All changes are highlighted in blue in the revised manuscript. Table, section, and appendix references in our responses refer to the revised version.

W1 & Q1: Comparison with HSA, DSA, and Surprise Forcing

We thank the reviewer for suggesting additional comparisons with existing adaptive methods. While direct comparisons with DSA and HSA are challenging due to differences in training requirements and sampling dynamics, we provide a controlled comparison with Surprise Forcing under matched denoising budgets. We also clarify the key differences between our method and HSA below.

DSA [1]. DSA jointly trains its confidence estimator and generator, whereas our method operates on a frozen generator. Without publicly available code and pretrained weights, a controlled comparison using identical generator checkpoints is not feasible. Nevertheless, DSA employs chunk-wise 1/4-step allocation with approximately 2.3 mean steps estimated from its reported NFE, whereas ours achieves 1.6–1.7 mean steps through finer-grained token-wise commitment. Although not directly comparable across different settings, this suggests greater potential for computational savings by avoiding unnecessary refinement within each chunk.

Surprise Forcing [2]. Surprise Forcing combines surprise-guided memory selection with chunk-wise adaptive denoising. Its reported end-to-end throughput is lower than that of the LongLive baseline (16.45 vs. 18.01 FPS for 5s), and its code is not publicly available, preventing direct reproduction under identical settings. To isolate the contribution of adaptive denoising from other system components, we adapted its surprise-based step-allocation mechanism to the same LongLive generator, as reported in Appendix C.3 (Tables C–F).

To address the reviewer's concern regarding unmatched computational budgets, we further extended its original 2/4-step policy to 1/4-step allocation, enabling comparison at nearly identical mean denoising steps on LongLive-30s.

MethodMean Steps ↓FPS ↑Total ↑Quality ↑Drift ↓
Surprise-based allocation (adapted 1/4-step)1.7344.3882.4082.952.055
Ours (H)1.7237.7882.8183.321.707

At nearly identical mean denoising steps, our method improves VBench Total by 0.41, Quality by 0.37, and reduces temporal Drift by 16.9%. Our results suggest that token-wise verification and commitment better preserve generation quality and temporal stability at comparable denoising budgets.

HSA [3]. HSA allocates token-wise denoising budgets based on velocity dynamics and advances skipped tokens using cached Euler updates. However, its strategy requires an initial observation phase, which consumes a substantial fraction of our four-step sampling budget. Moreover, our DMD-distilled generators employ stochastic re-noising between steps [4, 5], unlike the deterministic trajectory assumed by HSA's cached-velocity updates. Thus, directly applying HSA would require adapting both its observation strategy and velocity-reuse mechanism. In contrast, our method verifies clean latent estimates and commits converged tokens without modifying the generator's native sampling dynamics.

References

[1] DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation. CVPR Findings 2026; arXiv:2606.04432.

[2] Surprise Forcing: What to Remember, When to Skip in Long Video Generation. ECCV 2026; arXiv:2607.18436.

[3] HSA: Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation. arXiv:2605.06892, 2026.

[4] DMD2: Improved Distribution Matching Distillation for Fast Image Synthesis. NeurIPS 2024; arXiv:2405.14867.

[5] Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. NeurIPS 2025; arXiv:2506.08009.

AuthorsOpenReview ↗

Response to reviewer 32jH (2/3)

W2 & Q2: Benefits of Token-Wise Commitment over Chunk-Wise Acceptance

We thank the reviewer for this question. The fundamental limitation of chunk-wise acceptance is that convergence is not uniform across tokens within a chunk. While some tokens may have already converged, others may still require further refinement.

Importantly, our chunk-wise ablation uses the same convergence-based verification criterion as our token-wise method, differing only in the granularity of acceptance. Consequently, chunk-wise acceptance can still identify when a chunk is broadly stable and preserve competitive average Quality scores. However, as shown in Table 6, it exhibits higher temporal Drift than uniform four-step sampling, despite maintaining comparable overall Quality.

To further quantify the benefits of token-wise commitment, we conducted additional prompt-level paired bootstrap analyses (B = 20,000), reporting 95% confidence intervals for the differences against chunk-wise acceptance.

ComparisonΔ Quality (95% CI) ↑Δ Imaging Quality (95% CI) ↑Δ Drift (95% CI) ↓
Ours H − Chunk-wise+0.12 [+0.02, +0.22]+0.39 [+0.14, +0.65]−0.38 [−0.56, −0.20]

All reported confidence intervals exclude zero, providing statistical support for the measured improvements in generation quality and temporal stability. In particular, the significant reduction in Drift highlights the vulnerability of coarse chunk-wise acceptance to temporal error accumulation.

W2 & Q3: Statistical Uncertainty of Quality Improvements

We thank the reviewer for highlighting the importance of establishing the statistical reliability of our reported quality improvements. While aggregate metrics such as VBench do not fully capture perceptual quality in video generation, our results show measurable quality degradation under uniform low-step sampling, despite relatively small differences in aggregate scores. To quantify their statistical reliability, we conducted additional prompt-/scene-level paired bootstrap analyses (B = 2,000), recomputing the official VBench aggregation for each resample and reporting 95% confidence intervals against the relevant uniform-step baselines.

SettingComparisonΔ VBench (95% CI)Δ Drift (95% CI)
LongLive-5sOurs TF − 2-step+0.65 [+0.39, +0.92] (Total)—
LongLive-5sOurs H − 2-step+0.75 [+0.50, +1.01] (Total)—
LongLive-30sOurs TF − 2-step+0.44 [+0.08, +0.82] (Total)−0.31 [−0.38, −0.24]
LongLive-30sOurs H − 2-step+0.49 [+0.14, +0.88] (Total)−0.28 [−0.36, −0.22]
LingBot-World 2.0Ours TF − 3-step+0.21 [+0.09, +0.33] (Quality)−0.43 [−0.63, −0.25]
LingBot-World 2.0Ours H − 3-step+0.28 [+0.16, +0.39] (Quality)−0.52 [−0.72, −0.32]

All reported confidence intervals exclude zero, providing statistical support for the improvements achieved by both learned and training-free verifiers over the corresponding uniform-step baselines. The training-free verifier already achieves statistically supported improvements without additional training. The learned verifier consistently achieves slightly higher average quality scores across the evaluated settings and higher verification accuracy, as shown in Appendix Table A. These results suggest that learned verification offers further potential for improving adaptive refinement.

These results strengthen our findings that adaptive token-wise computation improves both generation quality and temporal stability over uniform sampling while preserving favorable throughput, particularly for long-horizon generation and camera-controlled world modeling.

AuthorsOpenReview ↗

Response to reviewer 32jH (3/3)

W2 & Q4: End-to-End Inference Speedup and Computational Overhead

We thank the reviewer for this suggestion. To provide a more comprehensive evaluation of practical inference efficiency, we additionally measured end-to-end throughput including VAE processing, alongside the DiT-only throughput reported in the original paper.

End-to-End Inference Throughput.

Setting1-step2-step3-step4-stepOurs (TF)Ours (H)
LongLive 1.3B (5s)26.4821.9818.7916.4221.7521.27
LongLive 1.3B (30s)25.4021.0217.9215.6320.4420.16
LingBot-World 2.019.6016.0613.6011.8014.2714.20
Krea Realtime 14B [4] (5s)7.345.644.573.856.266.12

All values are FPS, including the default VAE. Krea Realtime 14B is an additional larger-backbone evaluation conducted in response to the reviewers' concerns.

We report DiT-only throughput separately because VAE processing introduces a shared computational cost across sampling strategies, and its contribution can vary with the VAE implementation. This overhead can be independently reduced through lightweight video autoencoders, such as Tiny AutoEncoder [1] and LightVAE [2], or asynchronous streaming VAE decoding, as demonstrated in LongLive-2.0 [3].

Verification and KV-Cache Management Overhead.

KV-cache updates are required by both the uniform four-step baseline and our method. In our implementation, each denoising forward updates the K/V entries of active tokens, while committed tokens retain their cached representations without further recomputation.

To quantify the actual computational overhead, we additionally profiled the steady-state wall-clock latency on LongLive-30s and included the runtime breakdown in Appendix C.5 (Table G).

Component (ms/chunk)Uniform 4-stepOurs (Learned)Ours (Training-free)
Denoising forwards410.7235.8228.5
Clean-KV refresh76.076.076.0
Routing / verification—3.62.9
Total486.7315.5307.3

Routing and verification account for only 1.14% and 0.94% of the profiled latency for the learned and training-free verifiers, respectively.

The clean-KV refresh pass is also required by the uniform four-step baseline and incurs the same 76.0 ms/chunk cost. Token-wise commitment reduces denoising-forward latency from 410.7 ms/chunk to 235.8 ms/chunk for the learned verifier and 228.5 ms/chunk for the training-free verifier.

References

[1] Tiny AutoEncoder for Hunyuan Video (TAEHV). https://github.com/madebyollin/taehv.

[2] LightVAE: Towards Compact and Efficient Video Autoencoders. arXiv:2610.14726, 2026.

[3] LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation. arXiv:2605.18739, 2026.

[4] Millon, E. Krea Realtime 14B: Real-time Video Generation. GitHub repository, 2025. https://github.com/krea-ai/realtime-video.

Reviewer_WDVnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper observes that many tokens converge to their full-step output after the first one or two denoising steps in few-step autoregressive video generators, yet every token is still denoised to the end. It proposes Speculative Forcing, which treats intermediate clean estimates as drafts. A verifier accepts converged tokens early, accepted tokens stop being denoised but stay in the KV cache as context for the others. There are two verifiers, both used with a frozen generator: a training-free one based on similarity to history and stability across steps, and a learned lightweight head that predicts each token’s remaining gap from intermediate features. Experimental results on LongLive and on the Lingbot-World 2.0 world model show that the method roughly keeps four-step quality with about 1.7 and 2.5 mean steps.

Critical strengths
  1. The paper decides per token when to stop denoising and commits accepted tokens while keeping their KV cache, and it needs no change to the generator’s weights. This matters because the gains come on top of models that are already distilled to few steps and run in real time, where further speedups are hard to get.
  2. Both verifiers keep four-step quality across short video, long video and camera-controlled world generation. Ablations show that token-wise commitment beats chunk-wise acceptance and adapted baseline signals, and the learned verifier performs close to the oracle.
Critical weaknesses
  1. The gains over simple baselines are small and hard to judge. Uniform 2-step sampling is already faster than the method (40.9 vs. 37.8 fps on long video) with only a ~0.5-point drop in VBench Total Score. On Lingbot, uniform 3-step is only slightly worse than the proposed method. The paper reports no variance, significance tests or human study, so it is unclear whether the speed–quality trade-off is really better than just using fewer steps.

  2. There is no direct comparison with existing acceleration methods, and the tested scope is narrow. Related methods such as DSA, Surprise Forcing, MotionCache, RAS and HSA are only "adapted" as selection signals inside the authors' own framework, not compared end to end at matched speed. All experiments use two 1.3B, 4-step generators, so it is unclear how the method compares with the state of the art, or whether it generalizes to larger models or other step counts.

Questions for authors
  1. Can you report the variance across seeds or confidence intervals for the main VBench scores, so we can tell whether the ~0.3–0.5-point gains over uniform 2- or 3-step sampling are statistically meaningful?
  2. How does Speculative Forcing compare end to end, at matched throughput, with existing adaptive or caching methods such as DSA, Surprise Forcing, MotionCache or HSA, applied to the same generators?
  3. Does the reported throughput include all overheads (the verifier head, the nearest-neighbor search over the history bank, smoothing, and the extra clean pass that refreshes the KV cache), and what is the wall-clock breakdown?
  4. How sensitive are results to thresholds calibrated on one dataset or domain and applied to another, for example training-free thresholds calibrated on VidProM and applied to out-of-distribution prompts or scenes?
5 comments
AuthorsOpenReview ↗

Response to reviewer WDVn (1/3)

All changes are highlighted in blue in the revised manuscript. Table, section, and appendix references in our responses refer to the revised version.

W1 & Q1: Statistical Significance of Quality Improvements

We thank the reviewer for raising this important concern. Our goal is to achieve a more favorable speed–quality trade-off than uniform low-step sampling. While our method can be slower than uniform 2-step sampling on LongLive, the additional computation yields statistically supported quality improvements and reduced temporal drift, particularly in long-horizon generation. Furthermore, as detailed in our response to W2 (Generalization to Larger Models), our additional larger-model experiments demonstrate that our method can outperform uniform 2-step sampling in both throughput and generation quality, further supporting a favorable speed–quality trade-off across different model scales.

To establish the reliability of these improvements, we conducted additional prompt/scene-level paired bootstrap analyses (B = 2,000), recomputing the official VBench aggregation for each resample and reporting 95% confidence intervals for differences against the corresponding uniform-step baselines.

SettingComparisonΔ VBench (95% CI)Δ Drift (95% CI)
LongLive-5sOurs TF − 2-step+0.65 [+0.39, +0.92] (Total)—
LongLive-5sOurs H − 2-step+0.75 [+0.50, +1.01] (Total)—
LongLive-30sOurs TF − 2-step+0.44 [+0.08, +0.82] (Total)−0.31 [−0.38, −0.24]
LongLive-30sOurs H − 2-step+0.49 [+0.14, +0.88] (Total)−0.28 [−0.36, −0.22]
LingBot-World 2.0Ours TF − 3-step+0.21 [+0.09, +0.33] (Quality)−0.43 [−0.63, −0.25]
LingBot-World 2.0Ours H − 3-step+0.28 [+0.16, +0.39] (Quality)−0.52 [−0.72, −0.32]

All reported confidence intervals exclude zero, providing statistical support for the measured improvements over the corresponding uniform-step baselines.

We agree that aggregate metrics alone do not fully capture perceptual differences. We therefore also plan to conduct a human preference study to assess whether the measured improvements translate into perceptually meaningful gains, particularly in temporal consistency and visual quality.

Overall, our results demonstrate that adaptive token-wise computation achieves higher generation quality and reduced long-horizon drift compared with uniform few-step sampling at comparable computational budgets.

AuthorsOpenReview ↗

Response to reviewer WDVn (2/3)

W2 & Q2: Comparison with Existing Adaptive and Caching Methods

We thank the reviewer for emphasizing the importance of comparisons with existing acceleration methods. We agree that end-to-end comparisons are valuable. However, existing methods differ in their training requirements, sampling dynamics, and additional acceleration mechanisms, making controlled comparisons under identical generator settings challenging. To directly evaluate our core contribution of verification-based token-wise commitment, we compare denoising-step allocation strategies under the same generator and comparable denoising budgets, minimizing confounding effects from differences in training and caching mechanisms.

Compatibility with Existing Methods. We investigated several related approaches and identified the following differences:

  • HSA [1], MotionCache [2], and RAS [3] rely on reusing cached velocities, denoising residuals, or noise predictions across sampling steps. However, our DMD-distilled generators employ stochastic re-noising with newly sampled Gaussian noise between denoising steps [4, 5], unlike the sampling procedures assumed by these methods. Consequently, their cache-reuse mechanisms cannot be directly transferred without adapting them to the underlying sampling dynamics.
  • DSA [6] jointly trains its confidence estimator and generator, whereas our approach operates on a frozen generator. Without publicly available code and pretrained weights, a controlled comparison using identical generator checkpoints is not feasible. Nevertheless, DSA employs chunk-wise 1/4-step allocation with approximately 2.3 mean steps estimated from its reported NFE, whereas ours achieves 1.6–1.7 mean steps through finer-grained token-wise commitment. While not directly comparable across different settings, this suggests greater potential for computational savings by avoiding unnecessary refinement within each chunk.

Controlled Comparison with Surprise Forcing. Unlike the above methods, Surprise Forcing's step-allocation mechanism [7] is compatible with our generator. However, its reported end-to-end throughput is lower than that of the LongLive baseline (16.45 vs. 18.01 FPS for 5s), and its code is not publicly available, preventing direct reproduction under identical settings. To isolate the contribution of adaptive denoising from other system components, we therefore adapted its surprise-based step-allocation mechanism to the same LongLive generator, as detailed in Appendix C.3 (Tables C–F).

We further extend its original 2/4-step policy to 1/4-step allocation, enabling comparison at nearly identical mean denoising steps. Both methods retain four-step sampling for the initial chunk and are evaluated on LongLive-30s.

MethodMean Steps ↓FPS ↑Total ↑Quality ↑Drift ↓
Surprise-based allocation (adapted 1/4-step)1.7344.3882.4082.952.055
Ours (H)1.7237.7882.8183.321.707

At nearly identical denoising budgets, our method improves VBench Total by +0.41, Quality by +0.37, and reduces Drift by 16.9%. Our method delivers better generation quality and temporal stability at comparable denoising-step budgets. These results support the effectiveness of verification-based token-wise commitment relative to surprise-based chunk-wise allocation under matched computation budgets.

References

[1] HSA: Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation. arXiv:2605.06892, 2026.

[2] MotionCache: Motion-Aware Caching for Efficient Autoregressive Video Generation. ICML 2026; arXiv:2605.01725.

[3] RAS: Region-Adaptive Sampling for Diffusion Transformers. CVPR 2026; arXiv:2502.10389.

[4] DMD2: Improved Distribution Matching Distillation for Fast Image Synthesis. NeurIPS 2024; arXiv:2405.14867.

[5] Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. NeurIPS 2025; arXiv:2506.08009.

[6] DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation. CVPR Findings 2026; arXiv:2606.04432.

[7] Surprise Forcing: What to Remember, When to Skip in Long Video Generation. ECCV 2026; arXiv:2607.18436.

AuthorsOpenReview ↗

Response to reviewer WDVn (3/3)

W2: Generalization to Larger Models

We agree with the reviewer's concern regarding generalization to larger models. To further investigate this, we additionally evaluated Speculative Forcing on Krea Realtime 14B [1], a Self-Forcing-distilled autoregressive video generator based on Wan 2.1 14B. We have incorporated this experiment into Section 4.2 and Table 4 of the revised manuscript.

Interestingly, uniform 2–4-step sampling already achieves similar VBench scores on this larger backbone. Nevertheless, our method preserves near-four-step generation quality with only 1.62–1.63 mean denoising steps, while achieving higher throughput than uniform two-step sampling, even when including VAE processing.

Krea Realtime 14B 5s, VBench prompts

MethodMean Steps ↓FPS ↑Total ↑Quality ↑Semantic ↑
1-step1.0011.6881.6882.8377.10
2-step2.007.8884.5585.3281.50
3-step3.005.9584.7885.5681.65
4-step (dense)4.004.7884.8885.3782.89
Ours (TF)1.639.1584.8085.3682.56
Ours (H)1.628.8684.8585.4282.56

These results provide additional evidence that token-wise adaptive computation remains effective at a larger model scale, achieving better generation quality and higher throughput than uniform two-step sampling, while preserving near-four-step quality with fewer denoising steps.

References

[1] Millon, E. Krea Realtime 14B: Real-time Video Generation. GitHub repository, 2025. https://github.com/krea-ai/realtime-video.

Q3: Wall-Clock Breakdown and Runtime Overheads

We thank the reviewer for raising this question. Yes, our method achieves substantial wall-clock acceleration even after accounting for verification and KV-cache maintenance overheads. Specifically, our learned variant achieves a 1.55× throughput improvement over uniform four-step sampling (37.78 vs. 24.45 FPS in Table 2), with all inference-side overheads included. These include the learned verifier head (when applicable), nearest-neighbor search over the history bank, smoothing operations, and the additional clean-KV refresh pass.

To further quantify these overheads, we additionally profiled the steady-state inference latency on LongLive-30s and included the runtime breakdown in Appendix C.5 (Table G).

Component (ms/chunk)Uniform 4-stepOurs (Learned)Ours (Training-free)
Denoising forwards410.7235.8228.5
Clean-KV refresh76.076.076.0
Routing / verification—3.62.9
Total486.7315.5307.3

As shown above, routing and verification account for only 1.14% and 0.94% of the profiled latency for the learned and training-free variants, respectively. Denoising-forward latency decreases from 410.7 ms/chunk for uniform four-step sampling to 235.8 ms/chunk for the learned verifier and 228.5 ms/chunk for the training-free verifier. The clean-KV refresh cost of 76.0 ms/chunk is included for all methods.

Q4: Cross-Domain Robustness of Training-Free Thresholds

We thank the reviewer for raising this question. To evaluate cross-domain transfer, we additionally evaluated the LongLive generator on 200 held-out scenes from the LingBot-World 2.0 benchmark prompts, directly applying the LongLive-30s training-free thresholds calibrated on the held-out VidProM set without any recalibration or threshold tuning.

MethodMean Steps ↓FPS ↑Imaging Quality ↑Drift ↓
Uniform 1-step1.0061.0763.993.378
Uniform 2-step2.0040.6465.583.054
Uniform 3-step3.0030.4567.392.922
Uniform 4-step4.0024.3268.292.741
Ours TF (transferred)1.5338.2268.602.483

Despite the shift in evaluation domain, our method achieves higher Imaging Quality than uniform four-step sampling (68.60 vs. 68.29) and lower temporal Drift (2.483 vs. 2.741), while requiring only 1.53 mean denoising steps and achieving 1.57× higher throughput.

These results demonstrate that the original training-free thresholds retain a favorable quality-efficiency trade-off across different scene distributions without recalibration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.