acceptodds
Under review as a conference paper at ICLR 2027

BlockStep: Accelerating Autoregressive Video Diffusion via Online Multi-Model Routing Across Blocks and Steps

Abstract

Autoregressive video diffusion has become a compelling paradigm for streaming video synthesis, with step distillation as its standard path to fast inference. Yet step distillation reduces only the number of steps, not the size of the model run at each one: every step still invokes the full-size network – tens of billions of parameters – and replacing it with a smaller model can yield significant quality degradation. We introduce BlockStep, an online multi-model router: for each prompt, it routes every block and denoising step to a model of the appropriate size – the full-size base model or a smaller model distilled to the same few-step regime – invoking the base model only where it is needed to maintain quality while maximizing speedup. Unlike the leading autoregressive router, which scores each block with an external image-reward model – a costly extra network and an imperfect proxy for video quality – BlockStep does not rely on an extra reward model and is training-free. Specifically, BlockStep consists of three components: (1) a fixed block-0 anchor runs the base model on the first block, recovering most of the quality gap to the smaller model; (2) an online, block-level router upgrades selected blocks to the base model, based on the smaller model's attention entropy; and (3) within each upgraded block, a step-level schedule is selected based on block position. On Krea Realtime 14B with Wan2.1-1.3B Self-Forcing as the smaller model, BlockStep matches the base model's VBench total score at 2.37x wall-clock speedup.

Primary area:
generative models
Keywords:
video generation, autoregressive video generation, diffusion models, multi-model sampling, efficient video generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CXp5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BlockStep, a training-free multi-model routing framework for accelerating autoregressive video diffusion. The method dynamically allocates computation between a large target model and a smaller draft model across both autoregressive blocks and denoising steps. It has three components: a full-target anchor for the first block, an attention-entropy-based router to select subsequent blocks that need more computation, and position-dependent step schedules for upgraded blocks. Experiments using Krea Realtime 14B and Wan2.1-1.3B demonstrate a 2.37× end-to-end inference speedup while maintaining a VBench score comparable to the full target model.

Critical strengths
  1. The paper addresses an important computational bottleneck in autoregressive video generation by jointly allocating model capacity across blocks and denoising steps. The method is training-free and achieves a substantial 2.37× end-to-end speedup while maintaining the target model’s VBench score.

  2. The ablation results demonstrate that generating the initial block with the target model recovers most of the quality gap between the target and draft models. This provides a useful insight into how approximation errors propagate through autoregressive video generation.

Critical weaknesses
  1. The proposed block-level router assumes that higher attention entropy indicates greater uncertainty and therefore a stronger need for target-model computation. However, attention entropy is not a direct measure of draft-model error or the expected benefit of upgrading a block. Although the method outperforms random routing, the paper does not sufficiently establish the relationship between attention entropy and actual target-model improvement. This limits the justification for the central adaptive routing mechanism.

  2. The proposed position-dependent step scheduling provides only a marginal improvement over the simpler block-level routing strategy. In Table 1, BlockStep achieves a VBench score of 82.69 and 2.37× speedup, compared with 82.68 and 2.34× for BlockStep without step-level scheduling. Furthermore, the method is evaluated on only one target–draft model pair, making it difficult to assess whether the proposed scheduling principles generalise across different model architectures and generation settings. These limitations raise questions about whether the observed improvements justify the added scheduling complexity.

Questions for authors

Q1. Could the authors provide a direct analysis of the correlation between attention entropy and the actual benefit of upgrading a block to the target model? This would help validate the central routing assumption.

Q2. Given the marginal improvement over block-level routing alone (82.69 vs. 82.68 VBench), could the authors clarify the practical significance of the proposed position-dependent schedules? Are the improvements statistically significant across different prompts and seeds?

Q3. Does BlockStep achieve similar quality–efficiency trade-offs with other target–draft model pairs or different video lengths? Additional evidence would strengthen the generality of the proposed scheduling strategy.

Q4. Since the primary quality claim relies on matching the target model’s aggregate VBench score, could the authors provide more detailed evaluations of temporal consistency, motion quality, and prompt adherence? Human preference comparisons would also help establish whether the accelerated outputs are perceptually comparable to the target model.

2 comments

We thank the reviewer CXp5 for their constructive feedback. We are encouraged that the reviewer found BlockStep to have provided “useful insight into how approximation errors propagate through autoregressive video generation".

Below we address the (1) relationship between attention entropy and target-model improvement, (2) practical significance of the proposed position dependent schedules, (3) quality-efficiency trade-offs in different settings, and (4) more detailed evaluations of temporal consistency, motion quality, and prompt adherence.

Response to Concerns and Questions

Relationship Between Attention Entropy and Target-Model Improvement (W1, Q1)

Additional testing shows that while attention entropy is a weak predictor of per-block draft-target discrepancy and quality drop, it acts as a good clip-level allocation signal. Across 18 seeds, mean realized gains ranged from -0.000078 to +0.000083 for deciles 1-6 and +0.000152 to +0.000346 for deciles 7-10. Entropy’s overall rank correlation with the VBench score gain from a block upgrade is small, with a Spearman of +0.018 and per-decision AUC of 0.510. Analyzing the within-prompt discrepancy, however, attention entropy correlates with the decoded draft-target LPIPS of the VBench videos more strongly, with a Spearman of 0.185. In matched-budget offline tests, entropy routing beats random routing and matches or exceeds a LPIPS-based oracle on VBench score. We will describe entropy primarily as a clip-level allocation heuristic, presenting this evidence as motivation.

Practical Significance of Proposed Position Dependent Schedules (W2, Q2)

With a fixed router, implementing the position-dependent schedule instead of restarting every upgraded block only led to a -0.001 decrease in VBench score, with intervals of [-0.013, 0.012] across seeds and [-0.010, 0.010] across prompts. With this negligible decrease in score, the position-dependent schedule saved 0.50 target forward passes and 0.21 seconds per clip. The position-dependent schedules are thus an effective method of increasing efficiency even further.

Quality-Efficiency Trade-Offs in Different Settings (W2, Q2, Q3)

Replacing the draft model to create different model pairs had mixed results. Replacing the drafter with Rolling Forcing, BlockStep’s non-inferiority holds (+0.017 VBench score) and BlockStep beats the random router (+0.104 VBench score). In contrast, replacing the drafter with CausVid 3-step, BlockStep’s non-inferiority to the target model no longer holds (-0.617 VBench score), primarily due to the weaker draft model of CausVid 3-step. BlockStep with CausVid 3-step still beats the anchor-only baseline (+0.175 VBench score) and anchored random routing (+0.120 VBench score).

On 30-s, 40-block clips, BlockStep fails non-inferiority (-0.814 VBench score) and tends to upgrade later blocks more leniently because draft entropy rises with position. BlockStep’s entropy routing algorithm would need to be adjusted in order to address the rise in entropy with block position and make BlockStep more applicable to longer video clips.

Detailed Evaluations (Q4)

On the same videos used for the quality-speed comparison, we measure the average per-dimension VBench score across seeds and its standard deviation.

** Per-Dimension VBench Score Comparison**

DimensionTargetBlockStepBlockStep w/o step-level scheduleSDVGRandom routingDraft
Subject consistency95.99 ± 0.1395.64 ± 0.2395.63 ± 0.2389.92 ± 0.6695.25 ± 0.2094.84 ± 0.14
Background consistency95.33 ± 0.3095.09 ± 0.3395.10 ± 0.3393.08 ± 0.4794.59 ± 0.2394.47 ± 0.31
Temporal flickering96.87 ± 0.1997.85 ± 0.2997.84 ± 0.3098.20 ± 0.2598.00 ± 0.3798.17 ± 0.44
Motion smoothness98.38 ± 0.1498.30 ± 0.1598.29 ± 0.1698.43 ± 0.1398.32 ± 0.1498.26 ± 0.16
Dynamic degree72.78 ± 4.7771.94 ± 5.6871.94 ± 5.6872.22 ± 1.7073.89 ± 2.8573.61 ± 2.60
Aesthetic quality65.15 ± 0.1664.54 ± 0.1864.56 ± 0.1863.70 ± 0.5062.95 ± 0.4262.59 ± 0.53
Imaging quality70.06 ± 0.1369.66 ± 0.2969.66 ± 0.2870.12 ± 0.5870.07 ± 0.2369.98 ± 0.36
Object class91.98 ± 2.1291.79 ± 1.4591.68 ± 1.4192.25 ± 1.3983.20 ± 2.4981.82 ± 2.03
Multiple objects80.53 ± 1.7680.08 ± 1.9480.06 ± 1.9380.98 ± 2.9872.09 ± 1.8969.33 ± 2.62
Human action84.80 ± 1.4886.00 ± 1.2286.00 ± 1.2286.00 ± 2.2480.00 ± 2.5576.80 ± 2.39
Color86.60 ± 1.6987.44 ± 1.8587.36 ± 1.7988.30 ± 1.3888.22 ± 2.5988.23 ± 2.26
Spatial relationship86.13 ± 1.1585.88 ± 1.5385.90 ± 1.5884.15 ± 2.3580.07 ± 3.6177.95 ± 3.16
Scene39.84 ± 2.4138.82 ± 3.1638.82 ± 3.1637.95 ± 3.5428.81 ± 1.7328.66 ± 2.06
Appearance style19.91 ± 0.0619.85 ± 0.0619.85 ± 0.0619.35 ± 0.1119.32 ± 0.2019.23 ± 0.13
Temporal style24.74 ± 0.2424.57 ± 0.2924.57 ± 0.2923.98 ± 0.3323.71 ± 0.2423.51 ± 0.25
Overall consistency26.41 ± 0.2126.42 ± 0.2226.42 ± 0.2226.13 ± 0.2224.69 ± 0.2724.34 ± 0.13
Quality score84.22 ± 0.2484.25 ± 0.3484.25 ± 0.3482.98 ± 0.1284.12 ± 0.3483.97 ± 0.36
Semantic score76.52 ± 0.5676.44 ± 0.7576.41 ± 0.7275.91 ± 0.4771.02 ± 0.9669.74 ± 0.58
Total score82.68 ± 0.2682.69 ± 0.2982.68 ± 0.2981.57 ± 0.1481.50 ± 0.2281.12 ± 0.29

Overall, though there are significant gaps in score for dimensions such as temporal flickering (BlockStep is higher) and aesthetic quality (BlockStep is lower), when averaged out the total score of BlockStep matches that of the target model.

We are unable to conduct a human study at this time, and agree that the method could follow the target into a shared failure mode. Our method aims to accelerate the target model without quality loss, not necessarily to improve its quality.

Change List

  • Revise the entropy explanation.
  • Detail the efficiency benefits for adding step scheduling.
  • Report the results across model pairs and video lengths.
  • Report quality by dimension.
Reviewer_dBPyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose BlockStep, which allocates model computation across the two dimensions of video blocks and denoising steps. Through collaboration between a large and a small model, it accelerates a distilled autoregressive video diffusion model that has already undergone few-step distillation, and it uses attention entropy to control when the large model is invoked, achieving a significant speedup.

Critical strengths

1.The paper offers a useful application idea: when the number of denoising steps is very small, adjusting the model parameters used at each step, together with allocating computation according to the position of the video block, can speed up generation while reducing the impact on quality. 2.The routing signal comes from the small model's own attention and does not require running an additional reward model, which is a low-cost addition.

Critical weaknesses

1.The main quality gains come from the existing first-block anchoring strategy, and the necessity of the newly added routing and step-level scheduling has not yet been sufficiently supported. Since first-block anchoring is a design in SDVG, the experiments lack an evaluation of whether entropy routing and position-dependent step-level scheduling can stably improve quality and speed after fixing the same anchor frame.

2.The main contribution of this paper appears to be applying an existing design to few-step autoregressive generation and introducing position-dependent step-level switching. However, the current experiments have not yet clearly demonstrated how much independent gain this combination brings.

3.The attribution of contribution of the newly added components and the statement of the on-par quality conclusion are not sufficiently clear. In addition, the appendix also mentions the difference between the theoretical assumptions and the actual algorithm, explicitly acknowledging that the theory does not guarantee on-par VBench performance and does not prove that the chosen schedule is optimal.

Questions for authors

1.Does random routing fix the same large-model first block? The authors should compare entropy routing, random routing, and simple positional rules under the same anchor frame, context, and actual computation budget, and report the anchor-only results in full.

2.The paper should report the relationship between attention entropy and the actual realized upgrade gains, including the average gains after grouping by entropy.

3.According to the paper's description, the SDVG results in Table 1 are not reproduced results. The authors should then add how the data, step counts, and metrics of its implementation process are aligned with the method in this paper, because in my view this is a very important baseline.

4.Although the authors present a complete solution, they have not yet sufficiently established the independent contribution of the newly added method. The authors should complete the key comparisons above and demonstrate that the newly added components still bring stable gains after fixing the anchor frame.

1 comment

We thank the reviewer dBPy for their constructive feedback. We are encouraged that the reviewer found BlockStep to have provided “a useful application idea" that “does not require running an additional reward model”.

Below we address the (1) benefit of entropy routing and position-dependent step-level scheduling after fixing the first-block anchor, (2) relationship between attention entropy and realized upgrade gains, and (3) implementation process of SDVG.

Response to Concerns and Questions

Benefit of Entropy Routing and Position-Dependent Step-Level Scheduling (W1, W2, W3, Q1, Q4)

The original random baseline randomizes the first block, which we agree is where the main quality gain comes from. However, the first-block anchor alone is not sufficient to fully recover the target-drafter gap. With the newly added routing and step-level scheduling algorithms, we aim to recover this gap while still maximizing speedup. Below we evaluate whether entropy routing and position-dependent step-level scheduling stably improves quality and speed after fixing the anchor frame. We run different random and fixed schedule strategies on 946 VBench prompts and 15 seeds using the same block-0 anchor and compare them to BlockStep. BlockStep here scored 82.55 at 6.18 target forwards per clip.

Same-anchor comparatorTarget forwards per clipBlockStep gain in VBench Total, pp (95% CI)
Anchor-only5.00+0.118 [0.067, 0.168]
Random clips and blocks, same upgrade counts and step rule6.19+0.122 [0.070, 0.174]
Upgrade block 1 in randomly selected clips6.18+0.095 [0.001, 0.190]
Use the target for the last step of block 1 in every clip6.00+0.126 [0.060, 0.191]

BlockStep achieved higher VBench scores than the three schedule strategies at matched or near-matched target budget, with dynamic degree accounting for 84% of this gain, though static controls avoid 0.23-0.31s of routing overhead per clip.

With a fixed router, the position-dependent scheduling saved 0.50 target forwards at 0.21s per clip relative to restarting every upgraded block. The position cutoff was an empirical choice motivated by the conditional latent-error bounds of Appendix A, and we will present it as such.

Relationship Between Attention Entropy and Realized Upgrade Gains (Q2)

Additional testing shows that while attention entropy is a weak predictor of per-block draft-target discrepancy and quality drop, it acts as a good clip-level allocation signal. Across 18 seeds, mean realized gains ranged from -0.000078 to +0.000083 for deciles 1-6 and +0.000152 to +0.000346 for deciles 7-10. Entropy’s overall rank correlation with the VBench score gain from a block upgrade is small, with a Spearman of +0.018 and per-decision AUC of 0.510. Analyzing the within-prompt discrepancy, however, attention entropy correlates with the decoded draft-target LPIPS of the VBench videos more strongly, with a Spearman of 0.185. In matched-budget offline tests, entropy routing beats random routing and matches or exceeds a LPIPS-based oracle on VBench score. We will describe entropy primarily as a clip-level allocation heuristic, presenting this evidence as motivation.

Implementation Process of SDVG (Q3)

Because SDVG did not release their code, the original Table 1 row came from our SDVG reimplementation. While using the same model pair, prompts, seeds, and VBench scorer as BlockStep, we used SDVG’s four-step sampler and its own target-only reference. The reported 1.09x was the FPS ratio across pipelines.

For an aligned comparison, we used our five-step setup and emulated SDVG’s ImageReward decisions along the block-0 anchor trajectory. At matched target forwards, SDVG scores +0.012 points over the block-0 anchor, while BlockStep scored +0.137 points over SDVG.

Change List

  • Detail the efficiency benefits for adding step scheduling.
  • Revise the entropy explanation and add anchored random routing as a baseline.
  • Clarify implementation process of SDVG comparison.
Reviewer_wbXJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose BlockStep, a training-free online router that accelerates block-causal autoregressive video diffusion by coordinating multiple models. For every prompt, it assigns each block and each denoising step to either the full-size target or a smaller drafter from the same model family, through three components: a block-0 anchor always executed by the target, an entropy-driven block-level upgrade rule that inspects the drafter's self-attention entropy after three "peek" steps, and position-dependent step schedules that restart early upgraded blocks while continuing from the drafted prefix for later ones. On a single H100, the method matches the target's VBench total score (82.68 → 82.69) at 2.37× wall-clock speedup while calling the target for only 1.35 of 9 blocks per clip.

Critical strengths
  1. Relaxing uniform model capacity along both the block and the within-block step axes is, to my knowledge, novel for autoregressive video diffusion.
  2. The approach is training-free, requires no external reward model, and performs routing decisions in latent space.
  3. The reported 2.37× speedup at matched aggregate quality is of practical significance.
Critical weaknesses
  1. Even the parity claim is statistically unsupported: it relies on a single aggregate metric without cross-seed variance or significance testing, without a per-dimension VBench breakdown, and without a human study. Because LPIPS/PSNR/SSIM quantify similarity to the target output rather than absolute quality, the method could follow the target into a shared failure mode without this being detected.
  2. Attention entropy is not directly validated as a routing criterion. The paper itself describes entropy as "an observable of the drafter, not a direct measurement of its discrepancy from the target," yet supplies no such validation — for instance, a correlation between entropy and the actual quality drop, or a comparison of entropy-based routing with routing based on the true draft–target discrepancy — leaving entropy a plausible but unverified engineering proxy and the accompanying theory as post-hoc rationalization.
  3. Generalization remains untested, as the results stem from a single model pair, a single GPU, and a fixed 9-block clip.
Questions for authors

None

2 comments

We thank the reviewer wbXJ for their constructive feedback. We are encouraged that the reviewer found BlockStep’s method of relaxing uniform model capacity across the block- and step-wise axes “novel for autoregressive video diffusion”.

Below we address BlockStep’s (1) parity claim, (2) use of attention entropy as a routing criterion, (3) generalization.

Response to Concerns and Questions

Parity Claim (W1)

On the same videos used for the quality-speed comparison, we measure the average per-dimension VBench score across seeds and its standard deviation.

** Per-Dimension VBench Score Comparison**

DimensionTargetBlockStepBlockStep w/o step-level scheduleSDVGRandom routingDraft
Subject consistency95.99 ± 0.1395.64 ± 0.2395.63 ± 0.2389.92 ± 0.6695.25 ± 0.2094.84 ± 0.14
Background consistency95.33 ± 0.3095.09 ± 0.3395.10 ± 0.3393.08 ± 0.4794.59 ± 0.2394.47 ± 0.31
Temporal flickering96.87 ± 0.1997.85 ± 0.2997.84 ± 0.3098.20 ± 0.2598.00 ± 0.3798.17 ± 0.44
Motion smoothness98.38 ± 0.1498.30 ± 0.1598.29 ± 0.1698.43 ± 0.1398.32 ± 0.1498.26 ± 0.16
Dynamic degree72.78 ± 4.7771.94 ± 5.6871.94 ± 5.6872.22 ± 1.7073.89 ± 2.8573.61 ± 2.60
Aesthetic quality65.15 ± 0.1664.54 ± 0.1864.56 ± 0.1863.70 ± 0.5062.95 ± 0.4262.59 ± 0.53
Imaging quality70.06 ± 0.1369.66 ± 0.2969.66 ± 0.2870.12 ± 0.5870.07 ± 0.2369.98 ± 0.36
Object class91.98 ± 2.1291.79 ± 1.4591.68 ± 1.4192.25 ± 1.3983.20 ± 2.4981.82 ± 2.03
Multiple objects80.53 ± 1.7680.08 ± 1.9480.06 ± 1.9380.98 ± 2.9872.09 ± 1.8969.33 ± 2.62
Human action84.80 ± 1.4886.00 ± 1.2286.00 ± 1.2286.00 ± 2.2480.00 ± 2.5576.80 ± 2.39
Color86.60 ± 1.6987.44 ± 1.8587.36 ± 1.7988.30 ± 1.3888.22 ± 2.5988.23 ± 2.26
Spatial relationship86.13 ± 1.1585.88 ± 1.5385.90 ± 1.5884.15 ± 2.3580.07 ± 3.6177.95 ± 3.16
Scene39.84 ± 2.4138.82 ± 3.1638.82 ± 3.1637.95 ± 3.5428.81 ± 1.7328.66 ± 2.06
Appearance style19.91 ± 0.0619.85 ± 0.0619.85 ± 0.0619.35 ± 0.1119.32 ± 0.2019.23 ± 0.13
Temporal style24.74 ± 0.2424.57 ± 0.2924.57 ± 0.2923.98 ± 0.3323.71 ± 0.2423.51 ± 0.25
Overall consistency26.41 ± 0.2126.42 ± 0.2226.42 ± 0.2226.13 ± 0.2224.69 ± 0.2724.34 ± 0.13
Quality score84.22 ± 0.2484.25 ± 0.3484.25 ± 0.3482.98 ± 0.1284.12 ± 0.3483.97 ± 0.36
Semantic score76.52 ± 0.5676.44 ± 0.7576.41 ± 0.7275.91 ± 0.4771.02 ± 0.9669.74 ± 0.58
Total score82.68 ± 0.2682.69 ± 0.2982.68 ± 0.2981.57 ± 0.1481.50 ± 0.2281.12 ± 0.29

Overall, though there are significant gaps in score for dimensions such as temporal flickering (BlockStep is higher) and aesthetic quality (BlockStep is lower), when averaged out the total score of BlockStep matches that of the target model.

We are unable to conduct a human study at this time, and agree that the method could follow the target into a shared failure mode. Our method aims to accelerate the target model without quality loss, not necessarily to improve its quality.

Attention Entropy as a Routing Criterion (W2)

Additional testing shows that while attention entropy is a weak predictor of per-block draft-target discrepancy and quality drop, it acts as a good clip-level allocation signal. Across 18 seeds, entropy’s rank correlation with the VBench score gain from a block upgrade is small, with a Spearman of +0.018 and per-decision AUC of 0.510. Analyzing the within-prompt discrepancy, however, attention entropy correlates with the decoded draft-target LPIPS of the VBench videos more strongly, with a Spearman of 0.185. We will describe entropy primarily as a clip-level allocation heuristic, presenting this evidence as motivation.

Generalization (W3)

Replacing the draft model to create different model pairs had mixed results. Replacing the drafter with Rolling Forcing, BlockStep’s non-inferiority holds (+0.017 VBench score) and BlockStep beats the random router (+0.104 VBench score). In contrast, replacing the drafter with CausVid 3-step, BlockStep’s non-inferiority to the target model no longer holds (-0.617 VBench score). BlockStep with CausVid 3-step still beats the anchor-only baseline (+0.175 VBench score) and anchored random routing (+0.120 VBench score).

On a B300, BlockStep similarly outperforms the block-0 anchor and random routing, though its VBench score is slightly lower (-0.119) than that of the target model. At this point, BlockStep achieves 2.16x speedup over the target model.

On 30-s, 40-block clips, BlockStep fails non-inferiority (-0.814 VBench score) and tends to upgrade later blocks more leniently because draft entropy rises with position. BlockStep’s entropy routing algorithm would need to be adjusted in order to address the rise in entropy with block position and make BlockStep more applicable to longer video clips.

Change List

  • Add the 16 per-dimension VBench scores with means and standard deviations.
  • Revise the entropy explanation and add anchored random routing as a baseline.
  • Report the generalization results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.