acceptodds
Withdrawn from ICLR 2027

ForcingBooster: Adaptive Patch-wise Denoising for Fast Autoregressive Video Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Autoregressive video diffusion models have become a leading approach to real-time interactive video generation and video world modeling. However, existing methods typically allocate computation uniformly across space when generating the next video chunk, updating all patches with the same denoising schedule and number of function evaluations. This ignores the spatial heterogeneity of video content: static, simple, or recurring regions are often easier to denoise, while other regions require further refinement or additional context. A uniform budget thus leads to redundant computation on easy patches, whereas reducing the fixed step count can leave difficult patches insufficiently refined. We introduce ForcingBooster, a general framework that dynamically allocates denoising steps to individual patches: easy patches stop early to improve efficiency, while difficult patches receive more refinement steps. This design accelerates generation with negligible quality degradation. Our method is compatible with most existing few-step autoregressive diffusion models, requiring only a lightweight routing module with modest training. Experiments demonstrate that ForcingBooster reduces the total number of denoising steps applied across video patches by about 50% while maintaining visual quality comparable to the four-step baseline.

Primary area:
generative models
Keywords:
video generation, diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2zVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ForcingBooster trains a small router to stop denoising selected patches while keeping the video generator frozen. Stopped patches retain their clean-latent estimates and cached context; other patches continue. The training loss balances error against the full-step output with denoising cost. Tests cover Causal Forcing 1.3B, Wan2.1 14B, LingBot-World-V2, and LongLive. On Causal Forcing, patch-NFE drops by 50.19% and end-to-end speed improves by 1.30x, while VBench Total falls from 82.08 to 81.55. Random and chunk-wise allocation perform worse at similar runtimes. The practical benefit is plausible, but comparisons with closer adaptive methods are missing.

Critical strengths
  1. Only the small router needs training. Clean-latent exits and cached context let patches stop early without changing the generator’s weights, making the approach practical for existing few-step checkpoints.

  2. Tests cover four generators and include routing, caching, and decoding in runtime measurements. The random-allocation and chunk-wise controls show that spatial placement matters. Reporting both patch-NFE and end-to-end time also makes the actual speed benefit clear.

Critical weaknesses
  1. The closest alternatives are discussed but not tested. HSA already uses token-level budgets and preserves KV context; Patch Forcing learns patch-wise difficulty. Learned stopping on a frozen few-step generator differs from these methods, but the current controls do not show whether that difference is beneficial.

  2. The advantage over uniform two-step sampling is small or mixed on some models. On Wan2.1, two steps give 83.35 Total at 1.29x speedup, versus 83.32 at 1.31x for ForcingBooster. On LingBot, two steps are faster with a similar Total score. There is no uncertainty analysis or direct test of camera/action adherence and long-horizon drift. Section 3.3 also leaves unclear whether replay states come from the full-step or routed trajectory, which matters for the on-policy training claim.

Questions for authors
  1. How does the method compare with adapted HSA and token-caching or adaptive-step baselines on the same checkpoints at matched runtime?
  2. Please give training pseudocode showing the source of replay states, continuation-cost estimation, and the complete weighted loss.
  3. Are the quality differences reliable under paired sampling? How does the method perform on action/camera adherence and longer rollouts, especially given the near-tie with two-step sampling?
Reviewer_HAxQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ForcingBooster, a method that speeds up generation for few-step autoregressive video diffusion models by identifying "easy" patches and stopping their denoising early. Stopped patches keep their predicted clean latent and serve as clean context through the KV cache, while the model continues predicting only the remaining patches. The key component is a lightweight router, trained on top of the frozen generator with two losses: the error of the predicted clean latent (x^0\hat{x}_0) relative to the full-step output, and the expected denoising compute cost.

Critical strengths

1. Simple, lightweight, and general. The idea of stopping denoising early for easy regions is simple and intuitive. The method is lightweight, as it only needs to train a small router module (about 3.15M parameters, about 500 training steps) on top of a frozen few-step generator. It also works across many base models (Causal Forcing 1.3B, a causal Wan2.1 14B model, LingBot-World-V2, and LongLive), so it can be applied to existing few-step autoregressive video models.

2. Strong qualitative results. The qualitative results in the supplementary video show that videos generated with ForcingBooster are high quality, without noticeable artifacts. The predicted early-stop masks look accurate, adapting dynamically to the camera and object motion of each latent frame.

The closest prior works are cited and discussed in Section 2.

Critical weaknesses

1. No clear advantage over uniform 2-step sampling, and a small speedup. In Tables 1 and 2, the uniform 2-step baseline achieves very close, and sometimes better, VBench scores and speedups compared to ForcingBooster. On Causal Forcing (Table 1), ForcingBooster reaches a Total score of 81.55 at a 1.30× speedup, versus 81.29 at 1.29× for 2-step. In Table 2, 2-step has the higher score on Wan2.1 14B (83.35 vs. 83.32) and the higher speedup on LingBot-World-V2 (1.36× vs. 1.25×) and LongLive (1.21× vs. 1.15×). The ~1.3× overall speedup, which is the only benefit of this method, is not significant enough. The qualitative results (Fig. 3 and the supplementary video) suggest that ForcingBooster has higher quality than 2-step, but this is not clearly shown in the quantitative results. The authors may consider finding or designing proper metrics that measure these qualities, such as motion smear, to highlight this advantage.

2. Limited novelty and missing baselines. The idea of spending less computation on easy regions, including stopping early, is not new. It has been explored in prior works, including Patch Forcing (Schusterbauer et al., 2026), DSA (Le et al., 2026), MotionCache (Xu et al., 2026), HSA (Chu & Patel, 2026), RAS (Liu et al., 2026b), and DyDiT (Zhao et al., 2024) / DyDiT++ (Zhao et al., TPAMI 2026, arXiv:2504.06803). The experiments miss important baselines, such as the cited, published prior works Patch Forcing, DSA, and RAS, a training-free rule that stops a patch when its predicted x^0\hat{x}_0 stops changing, and TeaCache-style caching (Liu et al., 2025a). Apart from the random and chunk-wise allocation ablations in Table 3, the only baselines are uniform fewer-step sampling.

Minor (clarity). Some sentences in the paper are confusing, such as "Reconstruction and computational costs train the router through its stopping probabilities." (Fig. 2 caption, lines 188–189) and "patch-wise first-hit step distribution" (Fig. 7 caption, line 713).

Questions for authors

See weaknesses

Reviewer_KNFVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ForcingBooster, a lightweight learned router that adaptively allocates denoising steps to individual patches in autoregressive video diffusion models. It addresses the inefficiency of uniform computation, allowing easy regions to stop early while preserving further refinement for difficult regions. Experiments across video generators and world models show quality close to four-step baselines, with approximately 50% fewer patch-level denoising evaluations and a 1.30× end-to-end speedup on Causal Forcing.

Critical strengths
  1. Practical efficiency gains with limited trainable overhead. Only an approximately 3.15M-parameter router is trained for 500 steps in the main experiment, while the generator remains frozen. Timing includes routing, KV-cache construction, and VAE decoding. The approximately 1.30× end-to-end acceleration therefore demonstrates a practical benefit beyond the reduction in patch-NFE (Section 4.1, Table 1, Appendix A).

  2. Targeted ablations support spatially selective computation. At similar runtimes, patch-wise routing achieves a Total score of 81.54, compared with 79.03 for spatially randomized allocation and 80.01 for chunk-wise routing. Preserving the exit-step distribution while randomizing spatial assignments provides a useful control, supporting the importance of selecting specific locations for continued refinement (Table 3).

Critical weaknesses
  1. The incremental advantage over uniform two-step sampling is modest and inconsistent across backbones. Table 1 shows a 0.26-point Total improvement at similar runtime. In Table 2, the Wan2.1 14B results are nearly identical, while LingBot and LongLive gain only 0.11 and 0.12 points, respectively, with slower inference than two-step sampling. These results establish savings over four-step generation more clearly than a consistent improvement over a simple two-step baseline. Variability estimates and comparisons at matched runtime would help establish the reliability and practical significance of these small quality differences.
Questions for authors
  1. Which computational-cost weights, stopping thresholds, and combined training objective are used for the results in Tables 1–3?

  2. Could pseudocode clarify how full-step replay, current-router rollout, and the policy-generated histories described in Section 2.3 interact within a training update?

  3. Does constructing KV caches for stopped patches require an additional forward pass on their clean predictions? Which timestep representation is used for these cached features?

  4. Are the small quality gains over two-step sampling consistent across paired prompts and generation seeds? Under which runtime constraints does patch-wise routing provide the clearest advantage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.