acceptodds
Under review as a conference paper at ICLR 2027

Eliminating Context and Timestep Redundancy from Autoregressive Video Diffusion

Abstract

Few-step autoregressive video diffusion enables streaming and interactive generation, but commonly retains full temporal attention in every layer and a fixed denoising budget for every latent block. Our analysis reveals a consistent layer-wise pattern of temporal-context redundancy: similar subsets of Diffusion Transformer layers assign little attention to historical context across the evaluated training stages and datasets, as well as samples and timesteps within distribution matching distillation (DMD). In contrast, latent frames within a video vary in denoising difficulty and require different computational budgets. To address these two forms of redundancy, we introduce Static Pruning and Adaptive Denoising (SPAD), which reduces computation within each forward pass by removing historical cross-block attention from weak-temporal layers and adapts the number of denoising passes to content difficulty. To avoid online attention ranking, we identify weak-temporal layers from offline profiles and remove redundant historical-context computation using a fixed pruning pattern throughout training and inference. To allocate denoising steps on demand, a lightweight confidence head estimates proximity to the clean endpoint from the current trajectory state. We further introduce probabilistic timestep subsampling during DMD, exposing the generator to variable-span transitions and the states they induce to better accommodate inference-time skipping. Evaluations demonstrate that SPAD accelerates both training and inference while alleviating long-horizon degradation, supporting efficient autoregressive video generation without sacrificing visual quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion models, video generation, autoregressive generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VrsiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SPAD, a method for accelerating autoregressive video diffusion by addressing temporal-context redundancy and uneven denoising difficulty. It profiles layer-wise historical attention and statically prunes selected layers across training and inference, while a lightweight confidence head adaptively allocates denoising steps to each frame or chunk. Probabilistic timestep subsampling during DMD training further prepares the generator for inference-time skipping. Experiments on short- and long-video generation show reduced training and inference costs with competitive generation quality.

Critical strengths

The paper provides a reasonably thorough analysis of two distinct sources of computational redundancy. In particular, the consistency of layer-wise temporal-attention profiles across training stages and datasets motivates a practical pruning strategy rather than relying solely on an intuitive architectural modification.

The experiments show meaningful efficiency gains in both training and inference, including long-video generation. The component ablations also help explain the trade-off between adaptive skipping and generation quality, with timestep subsampling recovering much of the quality lost when adaptive skipping is introduced without corresponding training adaptation.

Critical weaknesses

The novelty of SPAD is somewhat incremental. Static attention pruning based on offline profiles and adaptive denoising for autoregressive video generation have both been explored in related work. The paper's main contribution therefore lies in its particular combination of cross-stage pruning, endpoint-based confidence supervision, and timestep-subsampled DMD. A clearer experimental demonstration of what this combination achieves beyond existing adaptive-generation and attention-acceleration methods would strengthen the contribution.

The ablations are cumulative rather than fully isolating the interaction between the main components. Although Table 3 shows that timestep subsampling recovers quality after confidence-guided skipping is introduced, it does not establish whether the benefit depends on this specific combination or how much the subsampling strategy contributes independently. This limits the conclusions that can be drawn about the necessity of the proposed training adaptation.

The evaluation demonstrates promising results, but the robustness of the reported gains remains unclear. Most short-video comparisons use a single backbone and training configuration, and the paper does not report variability across independent training runs. Moreover, the long-video results use a more aggressive skipping strength than the default short-video setting, making it important to distinguish the benefit of the method itself from the selected quality–efficiency operating point.

Questions for authors

How does SPAD compare with existing adaptive denoising methods, particularly DSA, under matched inference budgets and evaluation settings?

What is the effect of timestep subsampling when evaluated independently of confidence-guided skipping?

How sensitive are the main quality and efficiency gains to the pruning profile and training seed?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Vrsi (1/2)

Thank you for acknowledging our thorough analysis of two distinct sources of computational redundancy, the cross-stage and cross-dataset consistency of temporal-attention profiles that motivates a practical pruning strategy, the meaningful efficiency gains in both training and inference, and the component ablations showing that timestep subsampling recovers the quality lost by adaptive skipping. We address your concerns as follows:

W1: Novelty and What the Combination Achieves

SPAD differs from existing attention-acceleration and adaptive-generation methods in four aspects:

  1. An invariant found by profiling, rather than an online policy. Our profiles show that the layers which assign little attention to historical context are largely the same across the AR, CD, and DMD stages, across four training datasets, and across DMD samples and timesteps (Figure 2b). Existing attention-acceleration methods treat sparsity as an input-dependent quantity to be re-estimated at inference, so none of them can exploit such an invariant. Because the pattern is stable, it can be fixed once as a static mask and held throughout distillation and inference, which is what the training-side saving depends on.
  2. Acceleration of both training and inference. Existing attention-acceleration methods (e.g., Dummy Forcing, HeadCast, Forcing-KV) are training-free inference policies, and adaptive denoising methods (e.g., DSA) reduce the number of passes only at inference. SPAD's static mask is shared by every training stage and by inference, reducing per-update time by 12.2% to 18.1% and the total training time by 14.7%.
  3. Training adaptation to close the train-inference gap of skipping. Inference-time skipping produces variable-span transitions and intermediate states that a fixed-grid student never encounters during training. SPAD exposes the generator to these transitions through timestep-subsampled DMD, while existing adaptive methods leave the generator's training trajectories unchanged.
  4. Joint reduction of the cost and the number of forward passes. Static pruning reduces the cost of each pass, and adaptive denoising reduces the number of passes, so their savings compound. Neither requires online attention ranking.

These differences are reflected in the existing results:

  • Beyond attention acceleration. In both frame-wise and chunk-wise generation, SPAD is faster than all three attention-acceleration baselines and attains the highest VBench Total among them (Table 1). In addition, none of these baselines reduces training cost.
  • Complementarity of the two components. In frame-wise generation, profiled pruning alone reduces latency by 17.8% (12.06 s →\to 9.91 s), and adaptive denoising brings a further 21.3% reduction (9.91 s →\to 7.80 s), totaling 35.3%. In chunk-wise generation, the two components contribute 15.2% and 14.9%, totaling 27.8%. On 30-second videos, the total reduction reaches 48.9%.
  • Necessity of the training adaptation. Without timestep subsampling, confidence-guided skipping reduces frame-wise Semantic from 75.33 to 74.07 and Dynamic Degree from 74 to 61, and reduces chunk-wise VR from 10.20 to 9.37. Subsampling recovers them to 75.23, 77, and 10.23, respectively.
  • Beyond adaptive generation. We compare with DSA at its default setting (see Q1).

We have also summarized these differences in a property comparison table (new Table 13, §N).

W2 & Q2: Effect of Timestep Subsampling Independent of Confidence-Guided Skipping

The confidence loss updates only the confidence head, and training trajectories are sampled independently of the confidence prediction. The generator of the "+ Conf." setting in Table 3 is therefore trained on the fixed grid, while the SPAD generator is trained with timestep subsampling, and both use the same confidence-guided inference. Comparing these two rows thus measures the effect of timestep subsampling with the skipping rule held fixed. To separate subsampling from skipping entirely, we additionally run the SPAD generator on the full four-step grid (with confidence-guided skipping disabled). At the same NFE of 4.00, it reaches 83.07 Total against 82.97 for the fixed-grid-trained Profiled TP model of Table 3, so timestep subsampling preserves full-grid quality while improving the generator's robustness to skipped steps.

AuthorsOpenReview ↗

Response to Reviewer Vrsi (2/2)

W3 & Q3: Robustness of the Reported Gains

  1. Backbones and settings. SPAD is evaluated on four generation pipelines: Causal Forcing with chunk-wise and frame-wise generation, Rolling Forcing, and LongLive. They differ in temporal grouping, attention window, and video length (5 s and 30 s), and SPAD improves the quality-efficiency trade-off in all of them. All of these pipelines are built on Wan2.1-T2V-1.3B, which is the base model shared by current open-source AR distillation pipelines.
  2. Variability. We report results over 3 inference seeds with standard deviations in the new Table 7 (§I), where the std of VBench Total is 0.01 to 0.53. We also repeated the DMD stage with a different training seed from the same CD initialization (new Table 8, §I): the second run reaches 83.44 Total and 9.88 VR against 83.07 and 9.73 for the run reported in Table 1, with a slightly lower HPSv3 (9.74 against 9.96), so the result does not depend on one training run.
  3. Long-video operating point. α\alpha only scales the predicted confidence at inference and does not change the trained model. The confidence head trained in the long-video setting predicts lower confidence than in the 5-second setting, so we choose α=1.8\alpha=1.8 to align the average NFE (2.15) with the Fixed 2-step baseline (2.00) and compare them at a similar denoising budget. We also evaluated the 30-second setting at the default α=1.0\alpha=1.0: it keeps all four denoising steps (NFE 4.00) and takes 70.53 s, against 2.15 and 45.08 s at α=1.8\alpha=1.8, with comparable reward scores (VR 13.67, HPSv3 9.05). For 5-second generation, VBench Total varies only within 82.89 to 83.19 for α∈[0.6,1.4]\alpha \in [0.6, 1.4] (Figure 5b), so the gains are not tied to a single operating point.
  4. Sensitivity to the pruning profile. The profiles are highly consistent across three training stages, four training datasets, and DMD samples and timesteps (Figure 2b), so the selected layer set is insensitive to the profiling data and stage. Replacing the profiled set with uniformly spaced layers at the same budget lowers VBench Total by 0.55 (chunk-wise) and 0.51 (frame-wise) (Table 3), showing that the profile identifies the right layers. The pruning-budget sweep in Figure 5a and Table 6 further shows that quality degrades gradually up to 10 layers.

Q1: Comparison with DSA

DSA has not released its code. To compare under matched settings, we reimplemented it following its paper on the same Causal Forcing checkpoint and training data: a confidence head with learnable tokens cross-attending to DiT blocks 13, 21, and 29, trained jointly with DMD on the fixed four-step grid using the teacher-referenced confidence loss.

The table below compares both DSA variants with SPAD at its default setting.

MethodTotalQualitySemanticVRHPSTime (s)TANFE
DSA (our reimplementation)82.9185.0774.239.139.708.591.002.66
DSA + profiled TP82.8984.8874.928.149.427.420.672.76
SPAD (α=1.0\alpha=1.0, default)83.0785.0375.239.739.966.740.672.40

At its default setting, SPAD uses fewer forward passes and less wall-clock time than both DSA variants, and scores higher on VBench Total and Semantic and on both reward models. Methodologically, SPAD differs from DSA in its self-endpoint confidence target, its ability to skip intermediate steps instead of only exiting early, its training adaptation through timestep subsampling, and its additional reduction of per-pass cost, as summarized in Table 13.

If you have further questions, we are happy to provide further clarifications during the discussion phase. Thanks for your time and effort!

Thank you for the detailed responses and additional experiments. The new comparisons with DSA, the isolated ablations of adaptive denoising and timestep subsampling, and the additional robustness evaluations address most of my initial concerns. In particular, the results provide stronger evidence for the individual contributions of the proposed components and their effectiveness beyond the original cumulative ablations. While the novelty remains somewhat incremental and evaluation on additional backbones would further strengthen the paper, I find the current evidence sufficient to support acceptance. I will maintain my rating of 3.

Reviewer_JBMYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Static Pruning and Adaptive Denoising (SPAD) to accelerate autoregressive video diffusion. The method identifies layers with consistently low historical-attention mass and removes their cross-block attention throughout training and inference. A lightweight confidence head estimates endpoint proximity to allocate denoising steps, while timestep subsampling during distribution matching distillation prepares the generator for skipped transitions. Experiments cover frame-wise and chunk-wise generation, including 5-second and 30-second videos. The paper reports substantial denoising-latency reductions, a 14.7% reduction in estimated training time, and broadly preserved generation quality, together with reduced long-video reward drift.

Critical strengths
  • The paper addresses a practical efficiency problem with a relatively simple design. Reusing a fixed temporal-attention mask across distillation stages avoids online selection and provides benefits during both training and inference.

  • The temporal-profile analysis provides useful evidence across training stages, datasets, samples, and timesteps. Table 3 further supports the practical value of profiling through comparisons with uniformly spaced pruning at the same pruning budget and NFE.

  • The reported inference savings are substantial: 27.8% and 35.3% for the two short-video settings, and 48.9% for Rolling Forcing. The additional LongLive results broaden the empirical support beyond the main pipeline.

Critical weaknesses
  • The advantage of content-dependent step allocation is not sufficiently isolated. The biggest concern for me is whether the adaptive component improves the quality–efficiency trade-off over simpler schedules applied to the same trained generator. The Fixed 2-step baseline does not use temporal pruning, whereas SPAD combines pruning, confidence-guided scheduling, and timestep-subsampled training.

  • The confidence target needs validation as a predictor of safe skipping. Figures 1 and 2 measure distance to the generator’s final prediction on a full-grid rollout. However, Appendix G constructs training references from the sampled exit prediction, optionally followed by re-noising and one auxiliary forward pass at the smallest timestep.

  • The scope of the reported training savings is unclear. Equation (3) selects the pruning mask using profiles from AR, CD, and DMD checkpoints, while the same mask is then applied from the beginning of AR training. How are these checkpoints obtained before the accelerated pipeline starts? If existing checkpoints are reused, this is a reasonable deployment setting, but that assumption should be explicit. If separate unpruned runs are needed, their cost affects the claimed 14.7% total training reduction.

  • The long-horizon evaluation supports a narrower conclusion than general alleviation of degradation. Table 2 shows reduced drift in VR and HPSv3, which is encouraging. However, differences between early- and late-segment preference scores do not directly establish preservation of identity, action, or temporal coherence.

Questions for authors

See the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer JBMY (1/2)

Thank you for recognizing that SPAD addresses a practical efficiency problem with a simple design, that reusing a fixed temporal-attention mask across distillation stages avoids online selection and benefits both training and inference, and that our temporal-profile analysis provides useful evidence across training stages, datasets, samples, and timesteps. We address your concerns as follows:

W1: Isolating the Benefit of Content-Dependent Step Allocation

We isolate the adaptive component with two experiments.

  1. Same generator, different schedules. On the same SPAD generator (identical pruning mask and timestep-subsampled training), we replace confidence-guided scheduling with simpler schedules at a matched average NFE: fixed schedules with 4, 3, and 2 steps, random per-block skipping, and a position-based schedule that assigns more steps to earlier blocks. The step-allocation rule is the only difference among these settings. Results are reported in the new Table 10 (§L). At the matched average NFE of 2.91, confidence-guided allocation is better than both content-independent rules on every metric. It also matches the fixed 4-step schedule in VBench Total and Semantic while using 27% fewer forward passes.
  2. Adaptive denoising without temporal pruning. We repeat the cumulative ablation without temporal pruning, so that Fixed 2-step and the adaptive variants are compared on equal footing with respect to pruning. Results are reported in the new Table 11 (§L). Confidence-guided skipping alone reduces NFE from 4.00 to 3.00 but lowers VBench Total by 0.25, while adding timestep subsampling recovers it and exceeds the 4-step baseline, reaching 83.16 Total at an NFE of 2.69, with VR and HPSv3 also the highest of the four settings. The benefit of the adaptive components therefore does not depend on temporal pruning.

The existing Table 3 already provides partial evidence: with the same pruning mask, confidence-guided scheduling reduces chunk-wise NFE from 4.00 to 3.51 while keeping VBench Total unchanged (83.75 vs. 83.76), and timestep subsampling further reduces NFE to 3.29 while improving Total to 83.91.

W2: Validity of the Confidence Target as a Predictor of Safe Skipping

Unlike the full-grid rollout used in Figures 1 and 2, training builds the endpoint reference along the sampled trajectory, which saves up to three extra forward passes per block. This approximation is close by construction, for two reasons:

  1. The sampled grid size is 3 or 4 out of 4 points and always keeps the full-noise starting point, so at most one grid point is dropped.
  2. The reference always ends at the smallest grid point: the prediction is reused if this point is visited, and otherwise one auxiliary forward pass is run there. The final step is therefore never skipped.

To directly validate the target, we add two analyses in §M. First, we measure the discrepancy between the training reference and the full-grid reference on the same states (new Table 12). Skipping τ=750\tau=750 moves the endpoint by 0.050 in latent MSE and skipping τ=500\tau=500 by 0.055, both comparable to the 0.054 that we measure between two independent full-grid rollouts from the same state. The approximation therefore stays inside the sampling randomness of the full grid itself. At inference, confidence-guided skipping can also jump over both intermediate points, which never happens for a training reference. This path moves the endpoint by 0.049, also within the same noise floor. Second, at inference we compare the predicted confidence with the realized endpoint error with respect to the full-grid reference (new Figure 12). Over all pre-exit states of blocks 1 and later, the Spearman correlation between the predicted confidence and the realized error is −0.86, negative as expected, since higher confidence means smaller error. The predicted confidence deviates from its target by a mean absolute error of 0.13, and the correlation holds within each individual timestep (−0.76 to −0.83), so the head uses the content of the state and not only the timestep.

AuthorsOpenReview ↗

Response to Reviewer JBMY (2/2)

W3: Scope of the Reported Training Savings

The profiles in Eq. (3) are computed on the publicly released Causal Forcing checkpoints of the AR, CD, and DMD stages, so no separate unpruned runs were performed. The reported 14.7% reduction therefore contains no hidden profiling training, apart from the profiling forward passes themselves (about 65 minutes for 100 samples). As you noted, this is a reasonable deployment setting, and we have stated this assumption explicitly in §J.

The released checkpoints also allowed us to verify that the profiles are highly consistent across stages (Figure 2b). Consequently, for a new backbone without released checkpoints, a full unpruned pipeline is not required. A short unpruned AR run suffices, and those steps are then reused: the first 500 of the 5,000 AR steps run unpruned, the profile is taken from that checkpoint, and the same weights continue for the remaining 4,500 steps with the mask applied. The profile taken after those 500 steps already recovers all 10 of the layers used in the paper. Since the AR budget is unchanged, the extra cost is the per-step difference over 500 steps (0.36 h) plus one profiling pass, and the total reduction becomes 13.0% instead of 14.7%, which is still a substantial training speedup.

W4: Scope of the Long-Horizon Conclusion

Our drift metric follows the long-video protocol of Rolling Forcing, which measures the quality difference between the first and last segments of each video (following FramePack and CausVid), with its magnitude reflecting the severity of error accumulation. We apply it to VR and HPSv3 because these reward models jointly reflect visual quality and text alignment, and we keep the sign to distinguish degradation from improvement.

We agree that drift in preference scores alone does not directly establish the preservation of identity, action, or temporal coherence. We therefore add subject and background consistency over the full 30-second video, which measure how well the subject identity and the scene are preserved across the whole rollout, together with the original imaging-quality drift of Rolling Forcing, in new columns of Table 2 (§4.3). SPAD attains the smallest ΔDriftQuality\Delta\mathrm{Drift}_{\mathrm{Quality}} (0.21) and the highest subject (93.83) and background (94.84) consistency among all compared methods, so the reduced preference-score drift is accompanied by better preservation of the subject and the scene over 30 seconds.

The qualitative results in Figure 4 and §H also show that subject identity is preserved across the 30-second videos, e.g., the horse and its skates, and the hedgehog knight.

If you have further questions, we are happy to provide further clarifications during the discussion phase. Thanks for your time and effort!

AuthorsOpenReview ↗

[Reviewer JBMY] Invitation to Participate in Discussions for ICLR Submission 21875

We want to express our sincere gratitude for your constructive feedback, which acknowledges that SPAD addresses a practical efficiency problem with a simple design. According to your suggestions, we have provided further clarifications and revised the manuscript. As the discussion period is nearing its end in about only 1 day, we would like to invite you to join the discussion. If you require further clarification regarding your concerns, we are willing to provide more details during the discussion period. We believe that an active discussion will be of great benefit to the authors, the reviewers, and the whole ICLR community.

Thanks again for your time and effort!

Best,

ICLR Submission 21875 Authors

Thank you for the additional experiments. My main concern about whether content-dependent step allocation helps. Therefor I raise my score.

Reviewer_11XyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper accelerates few-step autoregressive video diffusion by addressing two sources of redundancy:

  • Context redundancy: SPAD identifies 10 of 30 DiT layers that consistently attend little to historical blocks and restricts them to within-block attention during training and inference.
  • Timestep redundancy: A lightweight confidence head enables adaptive skipping of denoising steps, supported by timestep subsampling during DMD training.

The method reduces inference latency by 28–35% for 5-second videos and 48.9% for 30-second videos, with comparable or slightly improved quality. Total distillation training time decreases by 14.7%.

Critical strengths
  1. Practical efficiency gains. SPAD reduces inference latency by 27.8% and 35.3% for chunk-wise and frame-wise 5-second generation, respectively, while preserving or improving VBench scores. The 48.9% speedup for 30-second videos is especially promising.
  2. Complementary efficiency techniques. Static temporal pruning reduces computation per forward pass, while adaptive denoising reduces the number of passes. The ablations in Table 3 demonstrate the contribution of each component.
  3. Solid experimental evaluation. The paper compares against recent efficiency methods, covers both frame-wise and chunk-wise generation, and extends its evaluation to 30-second videos using Rolling Forcing and LongLive.
Critical weaknesses
  1. Limited evidence for quality preservation. The reported VBench improvements (+0.38 and +0.09) are marginal, with no variance reported and reward metrics evaluated on only 100 prompts. Dynamic degree decreases in both settings, while the long-video experiments use a different α value (1.8 vs. 1.0). The proposed drift metric lacks validation, and the unusually high HPSv3 score for Fixed 2-step (13.67 vs. 8.80) remains unexplained. Moreover, SPAD is slower than Fixed 2-step, and the reported latency excludes VAE decoding and text encoding, potentially overstating end-to-end speedups.

  2. Limited novelty and insufficient justification of design choices.

    • Low historical attention does not necessarily imply negligible contribution, as value magnitudes and downstream effects are not considered. Direct per-layer ablations would better justify the pruning strategy.
    • Layer-level pruning is relatively coarse compared with head-level approaches such as HeadCast and Dummy Forcing. The improvement over uniform Grid Pruning is also small (~0.5 VBench).
    • Layer selection relies on profiling across AR, CD, and DMD stages, potentially requiring a full unpruned training pipeline. This overhead is not included in the reported 14.7% training-time reduction.
    • The confidence target measures similarity to the model's own final prediction rather than actual generation quality, raising concerns about its reliability.
    • Adaptive denoising for AR video has been explored in prior work (Le et al.; Liu et al., 2026a), making the contribution primarily an integration of existing techniques.
Questions for authors

Clarifications:

  1. Does layer selection require training the full unpruned AR→CD→DMD pipeline? If so, how does profiling overhead affect the reported training-time savings? Would profiling only the AR stage or a few hundred AR steps yield similar results?
  2. Were Dummy Forcing, HeadCast, and Forcing-KV applied training-free to the same Causal Forcing checkpoint, or retrained? Since SPAD is trained with its pruning mask, this may affect the fairness of the comparison.
  3. What motivates using α = 1.8 instead of the default 1.0 for 30-second generation? How do the results compare at α = 1.0, and what explains the unusually high HPSv3 score (13.67) for Fixed 2-step?
  4. What is the end-to-end inference latency, including text encoding, VAE decoding, confidence prediction, and the auxiliary reference forward pass?
  5. Is the layer selection based on frame-wise profiling also used for chunk-wise generation? If so, has its effectiveness been validated in that setting?

Experiments that would strengthen the paper:

  1. Report multi-seed VBench and reward scores with variance to establish whether the observed quality differences are statistically meaningful.
  2. Conduct per-layer ablations to verify that removing historical attention from the selected layers affects outputs less than removing it from other layers, supporting the proposed selection criterion.
  3. Evaluate head-level pruning baselines with the same adaptive denoising and DMD timestep subsampling to isolate the benefits of layer-level pruning.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 11Xy (1/4)

Thank you for recognizing the practical efficiency gains of SPAD, the complementarity between static temporal pruning and adaptive denoising, and the solid experimental evaluation across frame-wise, chunk-wise, and 30-second generation. We address your concerns as follows:

W1-1 & Q6: Evidence for Quality Preservation

As an acceleration method, SPAD aims to keep the quality of the full-computation model at lower cost. It reduces latency by 27.8% (chunk-wise) and 35.3% (frame-wise), and its VBench Total is the highest of all compared methods in both settings (83.91 vs. 83.53 and 83.07 vs. 82.98).

VBench scores are computed on all 946 augmented VBench prompts, and the long-video metrics on 200 MovieGen prompts following Rolling Forcing. For VisionReward and Dynamic Degree we follow the protocol of Causal Forcing, which uses the same 100 prompts with rich motion and complex actions. Self Forcing and Causal Forcing also report single-run point estimates without variance.

To better address your concern, we have repeated the evaluation with 3 inference seeds and report mean ±\pm std for VBench, VR, and HPSv3 in the new Table 7 (§I). The std of VBench Total is 0.27 to 0.53 in the chunk-wise setting and 0.01 to 0.28 in the frame-wise setting. In frame-wise generation SPAD has the highest mean on Total, VR, and HPSv3, and its per-seed margin over Causal Forcing, Grid Pruning, and Fixed 2-step is positive on all three seeds (+0.32, +0.71, +0.65 on Total). In chunk-wise generation, SPAD matches Causal Forcing and Grid Pruning in quality while being 28% and 15% faster, and exceeds Fixed 2-step on all three metrics.

W1-2: Dynamic Degree

Dynamic Degree is computed from optical flow, so abrupt frame-to-frame changes also count as motion. We therefore treat it as a measure of closeness to the unpruned model rather than as a score to maximize, and report it separately from quality.

By this measure, SPAD preserves motion while improving quality: its Dynamic Degree is 77 against 81 for the unpruned Causal Forcing baseline in frame-wise generation, the smallest gap among all compared acceleration methods, and 80 against 84 in chunk-wise generation. Over longer horizons the gap stays small (42 vs. 45 for 30-second Rolling Forcing, 89 vs. 86 on LongLive). In the same two settings SPAD also obtains the highest VBench Total, Quality, and Semantic scores and the highest VisionReward, so the retained motion does not come at the cost of frame quality.

Baselines with higher Dynamic Degree compromise quality: frame-wise Forcing-KV reaches 98, the highest value in Table 1, with VBench Quality 81.20 and VisionReward 8.31, against 85.03 and 9.73 for SPAD at 77. The HeadCast paper reports the same effect for Dummy Forcing. Deviation in the other direction is informative too: Grid Pruning drops to 49 in frame-wise generation, which is what pruning the wrong layers does to motion.

W1-3 & Q3: Choice of α\alpha for 30-Second Generation

α\alpha is an inference-time operating point that scales the predicted confidence and does not change the trained model. The confidence head trained in the long-video setting predicts systematically lower confidence than in the 5-second setting (mean 0.18 vs. 0.27), so α=1.0\alpha=1.0 corresponds to a considerably more conservative skipping strength there. We use α=1.8\alpha=1.8 to compensate for this scale difference, which aligns the average NFE (2.15) with the Fixed 2-step baseline (2.00) so that the two are compared at a similar denoising budget. As suggested, we also evaluated the 30-second setting at α=1.0\alpha=1.0. There the schedule keeps all four steps (NFE 4.00) at a latency of 70.53 s, against NFE 2.15 and 45.08 s at α=1.8\alpha=1.8, while the reward scores are comparable (VR 13.67 vs. 13.46, HPSv3 9.05 vs. 9.08) and the drift is smaller at α=1.8\alpha=1.8 (VR drift −0.74 vs. −0.93).

AuthorsOpenReview ↗

Response to Reviewer 11Xy (2/4)

W1-4: Validity of the Drift Metric

Our drift metric follows the long-video protocol of Rolling Forcing, which measures the difference in quality between the first and the last segments of each 30-second video (Δ\DeltaDrift, following FramePack and CausVid). Its magnitude reflects the severity of error accumulation. We apply the same protocol to VR and HPSv3 and keep the sign so that improvement and degradation can be distinguished. To check that our version measures the same thing, we additionally report the original Δ\DeltaDrift on VBench Imaging Quality, together with subject and background consistency over the full 30-second video, in new columns of Table 2 (§4.3). The original Δ\DeltaDrift ranks the four methods in exactly the same order as our VR drift (SPAD 0.21 < Rolling Forcing 0.49 < Grid Pruning 0.94 < Fixed 2-step 1.09), so replacing the quality score inside the established protocol with a reward score does not change which method is judged to drift least. SPAD with α=1.8\alpha=1.8 is the least drifting method under both metrics, at roughly half the latency of Rolling Forcing (45.08 s vs. 88.26 s).

W1-5 & Q3: The High HPSv3 Score of Fixed 2-Step

In the 30-second setting, Fixed 2-step collapses to nearly static, over-saturated videos: Dynamic Degree drops from 45 to 28, VR is the lowest among all methods (11.40), and VR drift is the worst (−3.72). With only two denoising steps over a long rollout, the mode-seeking DMD objective falls into a local shortcut that produces static, high-contrast, and saturated frames. HPSv3 is an image-level preference model applied to individual frames, so it is insensitive to motion and temporal coherence and tends to favor such saturated appearance, which inflates its score.

W1-6 & Q4: Speed of Fixed 2-Step and End-to-End Latency

Fixed 2-step is a speed-oriented reference point: in the 5-second setting it is faster than SPAD, but it reduces VBench Total by 1.29 and Semantic by 2.77 in chunk-wise generation, and Semantic by 1.42 in frame-wise generation. In the 30-second setting SPAD is 16.8% faster than Fixed 2-step (45.08 s vs. 54.20 s).

All methods are measured with the same protocol on the same NVIDIA H20 with batch size one, and we exclude text encoding and VAE decoding for every method. These two components are identical across methods (around 4.5 s for a 5-second video) and orthogonal to all compared acceleration techniques, so excluding them isolates the part that each method changes. There is no auxiliary reference forward pass at inference, as it is used only during training to construct the confidence target. The confidence head consists of two strided convolutions and a small MLP on pooled features, and has 0.128 M parameters (127,905), about 0.009% of the 1.3 B generator. We timed its forward pass on the same H20 at the shapes it sees during inference: 0.64 ms per call, which over the calls an actual run makes amounts to 36 ms per video in frame-wise generation (57.0 calls) and 12 ms in chunk-wise generation (19.0 calls), or 0.5% and 0.1% of the denoising latency in Table 1. Considering the shared 4.5 s of text encoding and VAE decoding together with the denoising latency of Table 1, end-to-end latency drops from about 16.1 s to 12.9 s (chunk-wise, −20.1%) and from about 16.6 s to 12.3 s (frame-wise, −25.7%) with SPAD acceleration.

W2-1 & Q7: Low Historical Attention vs. Negligible Contribution

We agree that attention mass is a proxy, and our pruning decision is therefore supported by interventional evidence rather than by the attention statistics alone. With an identical TA ratio and NFE, pruning the profiled layers outperforms pruning uniformly spaced layers (Grid Pruning) by 0.55 (chunk-wise) and 0.51 (frame-wise) in VBench Total, and retains much higher Dynamic Degree in frame-wise generation (74 vs. 49). The pruning-budget sweep (Figure 5a) further shows that quality decreases only gradually up to 10 layers and then drops sharply.

As suggested, we additionally conducted a training-free per-layer ablation on the unpruned frame-wise Causal Forcing checkpoint: for each of the 30 layers, we remove its historical attention alone, without retraining, and measure both the deviation of the output from the unpruned model and the quality (VisionReward and HPSv3) on the 100 CF prompts. In this training-free setting, the profiled set is no different from a random one in single-layer harm: 7 of the 30 layers lower VisionReward by more than two seed standard deviations when removed individually, 2 of which fall in the profiled set, against 2.3 expected for a random set of 10 layers. Removing attention from a model trained with full attention therefore cannot tell which layers are safe to prune, and the selection has to be compared under training with the mask. Training a separate model for every layer set is prohibitive, so Grid Pruning serves as the controlled comparison under training with the mask.

AuthorsOpenReview ↗

Response to Reviewer 11Xy (3/4)

W2-2 & Q8: Granularity of Pruning and the Margin over Grid Pruning

We chose layer-level granularity for two reasons.

  1. Effectiveness in training. SPAD applies one static mask to every training stage: teacher-forced AR training, CD initialization, and both the DMD generator and critic updates, yielding 12.2% to 18.1% per-update savings (Table 4). This requires a pattern that does not change with the input. Head-level patterns are less stable in this respect: Dummy Forcing reports about 25% discrepancy in dummy-head indices across text prompts and therefore re-assigns heads online, while HeadCast performs an online classification for each rollout after a full-context warm-up, which costs 5% to 8% of generation time on 5-second clips according to its paper. Such input-dependent patterns cannot be fixed as a training mask, which is why these methods are inference-only.
  2. Compatibility with modern attention kernels. A layer-wise restriction keeps all heads of a layer in a single dense FlashAttention call: a pruned layer simply attends within the current block and skips historical KV reads and writes. Head-wise policies split heads into groups with different context lengths, which requires restructuring the KV cache and custom attention paths.

Regarding the margin, a 0.5-point VBench Total improvement is substantial in this area. For reference, the headline improvement of Causal Forcing over Self Forcing is 0.30 in VBench Total (84.04 vs. 83.74), HeadCast regards staying within 0.15 points of full attention as quality-preserving, and DSA's adaptive sampling scores 83.95 vs. 83.94 for fixed sampling. Our profiled pruning improves over Grid Pruning by 0.55 (chunk-wise) and 0.51 (frame-wise) in VBench Total at identical cost.

To isolate the benefit of layer-level pruning (Q8), we apply HeadCast and Dummy Forcing head pruning to a model trained with the same adaptive denoising and timestep-subsampled DMD but without temporal pruning, and report the result in the new Table 9 (§K). At a comparable number of forward passes, head pruning reaches 82.74 and 82.80 VBench Total against 83.07 for SPAD in frame-wise generation, and 83.65 for both against 83.91 in chunk-wise generation, and neither reduces training cost.

W2-3 & Q1: Profiling Overhead

Layer selection does not require training an additional unpruned AR$\to$CD$\to$DMD pipeline. The profiles in the paper are computed on the publicly released Causal Forcing checkpoints of all three stages, so no extra unpruned training was performed, and the reported 14.7% training-time reduction contains no hidden profiling cost apart from the profiling forward passes themselves (about 65 minutes for 100 samples). This matches the common deployment setting in which pretrained checkpoints of the base pipeline are available. We have made this assumption explicit in §J.

For a new backbone without released checkpoints, the full pipeline is still unnecessary. The released checkpoints allowed us to verify that the profiles are highly consistent across stages (Figure 2b), so a short unpruned AR run is sufficient: profiling after only 500 unpruned AR steps already recovers all 10 of the layers used in the paper.

Those steps are not wasted: the first 500 of the 5,000 AR steps run unpruned, the profile is taken from that checkpoint, and the same weights then continue for the remaining 4,500 steps with the mask applied. The AR budget is unchanged, so the only extra training cost is the per-step difference over 500 steps (0.36 h), and one profiling pass adds about 65 minutes. The total training time becomes 74.19 h against 85.24 h unpruned, a 13.0% reduction, compared with 14.7% when released checkpoints are profiled. Profiling is a one-time cost per backbone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.