acceptodds
Under review as a conference paper at ICLR 2027

SEAE: Spectral Energy and Ablation Error for Structural Pruning of Few-step Video Diffusion Transformers

Abstract

The increasing size of Video Diffusion Models (VDMs) makes inference computationally expensive even after step distillation. Few-step distillation accelerates VDMs by reducing the denoising steps, but each step still executes the full width attention modules and the full depth Diffusion Transformer (DiT), leaving substantial memory usage and computation costs. Few-step VDMs Structural pruning is challenging because each solver step performs a larger portion of the noise-to-video transformation, giving pruning errors fewer subsequent steps in which to be corrected. We introduce SEAE, a structural pruning framework that reduces both attention width and DiT depth in few-step VDMs. Specifically, (i) our MHA-CSS method constructs Gram matrices from projected attention-head activations at all solver timesteps, uses their cumulative spectral energy to determine how many heads to retain in each self- and cross-attention module, and then selects the retained heads through CSS-inspired backward elimination with scalar refitting; (ii) we score each DiT blocks using Ablation Error, which temporarily removes each DiT block at a given timestep, completes the remaining sampling process, and measures its influence by comparing the final generated latent with the unablated reference. We form a low error candidate set at each solver timestep and retain their intersection as one static block pruning mask. Both masks are derived solely from forward passes over calibration set and define a fixed compact architecture shared across all timesteps, without runtime routing or cross-timestep feature caching. Recovery distillation finally trains the compact student using a weighted combination of teacher-student distillation and Flow Matching losses. Our model reduces the parameter count by 25.4% relative to the four-step Wan2.2-TI2V-5B teacher model, while nearly preserving its VBench-I2V total score for video generation.

Primary area:
generative models
Keywords:
Video diffusion models, Diffusion Transformers, Structured model pruning, Diffusion model compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oufhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SEAE, a structural pruning framework for few-step video diffusion transformers. It combines module-specific attention-head pruning with static block removal. Head budgets are derived from the spectra of projected head contributions, followed by CSS-inspired selection and scalar refitting. Blocks are scored by the final latent error caused by skipping them at individual solver timesteps, and low-error candidate sets are intersected across timesteps. The resulting compact model is recovered through distillation and evaluated on four-step Wan2.2-TI2V-5B Turbo.

Critical strengths
  1. The pruning criteria account for both module heterogeneity and timestep sensitivity. MHA-CSS distinguishes between determining head budgets and selecting individual heads, while evaluating candidate deletions by their worst-timestep reconstruction error. Table 3 provides useful evidence for the selection mechanism: with the same module-wise budgets and scalar refitting, CSS-based selection achieves 0.825 on VBench-I2V mini100, compared with 0.730 for random selection. Evaluating block importance through downstream endpoint error is also well motivated, since local feature similarity need not reflect the effect on the final generation.
  2. The method produces a genuinely smaller model with a fixed execution graph, rather than relying on runtime routing or feature caching. After recovery training, parameters decrease from 5B to 3.73B, reported DiT latency decreases from 1.20 s to 0.88 s, and peak VRAM decreases from 21.21 to 18.84 GiB, while the VBench-I2V total score changes from 0.8719 to 0.8702 (Table 1). The higher total score than the evaluated structural pruning baselines, together with the parameter and latency reductions, makes this a useful practical result.
Critical weaknesses
  1. The nearly unchanged aggregate score masks a substantial reduction in motion. In Table 4, dynamic degree decreases from 0.5276 for the teacher to 0.1537 for SEAE. The aggregate Quality score also decreases, while the I2V score improves. These results do not establish uniform preservation of generation quality and raise a concern that some appearance or consistency gains may accompany less dynamic outputs. This trade-off deserves explicit analysis, given the importance of motion in video generation.
  2. The block-pruning component is less thoroughly isolated than the head-pruning component. The direct comparison between endpoint Ablation Error and cosine similarity in Figure 9 is qualitative and covers two examples. There is no corresponding quantitative comparison at a matched block budget and recovery schedule. Consequently, the final combined result does not clearly establish how much the proposed block-selection criterion contributes beyond the attention-head pruning.
Questions for authors
  1. What accounts for the large dynamic-degree decrease in Table 4? Can intermediate results before and after recovery help distinguish whether this primarily arises from head pruning, block removal, or recovery training?
  2. At the same number of removed blocks and with identical recovery training, how does endpoint Ablation Error compare quantitatively with cosine-similarity-based selection? Please also specify the per-timestep candidate budgets used to obtain the final static block mask.
1 comment

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns.

The new results help clarify the source of motion degradation, showing that it already emerges during attention-head pruning and can be partially mitigated through recovery training. The additional quantitative comparison between Ablation Error and Cosine Similarity also provides stronger evidence for the effectiveness of the proposed block-selection criterion under matched settings.

I have also reviewed the authors' responses to the other reviewers and found no additional concerns that would change my overall assessment. The motion-quality trade-off remains an important limitation to acknowledge. I will maintain my original score of 3 (Weak Acceptance).

Reviewer_YiQvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors propose SEAE, a structural pruning method for few-step video diffusion transformers, with a focus on pruning of self/cross-attention and full blocks. For attention pruning, the method uses the spectral energy of projected head outputs to assign a separate head budget to each self-attention and cross-attention module. It then uses a CSS-based removal procedure with scalar refitting to select the retained heads. For depth pruning, the method temporarily removes each DiT block at each denoising step and measures the resulting error in the final latent. Blocks with consistently low error across all steps are removed. The resulting model has a fixed inference graph and doesn't require runtime routing or feature caching. The method is evaluated on the 4-step Wan2.2 5B model. After recovery training, the pruned model reduces the parameter count by ~25% and achieves a 1.36x DiT speedup while maintaining a similar overall VBench I2V score.

Critical strengths

The paper addresses a relevant practical problem with a clean structural solution. Few-step distillation reduces the number of denoising steps, but each remaining step still runs the full model. SEAE is complementary to step distillation and produces a smaller model with lower latency and memory use. The fixed inference graph is also attractive for deployment.

The pruning criteria are sensible and lead to a reasonable reduction. The method accounts for differences in redundancy across layers and between self-attention and cross-attention. Measuring block importance at the final latent is also better motivated than relying only on the local input-output similarity of a block. On the evaluated model, SEAE provides a good trade-off between parameter count, latency, and the aggregate VBench I2V score.

Critical weaknesses
  • The empirical scope is limited for a paper presented as a general method. All experiments use a single four-step backbone, Turbo Wan2.2 5B, in one TI2V setting. It is empirically unclear whether the observed head redundancy, the selected spectral threshold, and the block-pruning strategy transfer to another architecture, model size, task, or denoising schedule. The current evaluation supports a useful case study, but provides limited evidence for the broader method claims.

  • The final quality evidence is not yet fully convincing. The almost unchanged total VBench-I2V score hides a large reduction in dynamic degree, one of the most improtant video quality dimensions, from ~0.53 for the teacher to ~0.15 for SEAE. It is therefore unclear whether the student fully preserves perceptual video quality or reaches a similar total score through a different balance of appearance, conditioning, and motion. Further more, given the known limitations of the automatic vidoe quality benchamrks, adding a human preference study would be particularly useful here.

  • Certain helpful ablations for important design choices are missing. These include using one shared structure across all denoising steps despite the observed timestep-dependent specialization, the recovery objective, and the order of width and depth pruning. The current choices seem reasonable, but additional ablations would help clarify their impact on the final result. I detail these in the Questions for Authors.

Questions for authors
  1. Timestep-specific structures: Have the authors considered using a separate fixed structure for each of the four denoising steps? This would remain deterministic and router-free, while potentially making better use of the observed timestep-dependent specialization. What would be the resulting quality, latency, and storage trade-off?

  2. Generalization and calibration stability: Can the authors provide evidence on a second backbone or denoising-step setting? If this is not feasible, how stable are the selected heads and blocks across different calibration subsets and calibration-set sizes?

  3. Recovery objective: The use of the standard Flow Matching loss for turning a few-step model doesn't appear as the best option given few-step models are trained with different objectives. Could the authors report relacing FM loss with the orignally used dmd loss and/or with the KD-only? It would also be useful to clarify how the Flow Matching noise levels are sampled and why this target is well aligned with a teacher specialized for four-step inference.

  4. Perceptual quality: Could the authors provide a moderate-scale blinded human preference comparison against the teacher?

  5. Validation of block scoring: Figure 9 qualitatively compares endpoint Ablation Error with cosine similarity after removing two blocks. Could the authors provide an aggregate comparison using the final pruning budget and the same recovery setup? A random block baseline would also be useful.

  6. Clarification of comparisons and related work: The comparison with MagCache seems difficult to interpret directly, since the reported configuration is training-free, retains the full model, and uses two denoising steps, while the structural pruning methods use recovery training and four steps. Could the authors clarify whether MagCache is intended only as a broader system-level reference rather than a matched baseline? Also, could the authors clarify why MobileWan is categorized as a dynamic pruning method? If I understand correctly, its learned pruning decisions are materialized into a static model for inference.

1 comment

Thank you for the detailed response and additional experiments. The quantitative block-selection comparison, calibration stability analysis, and blinded user study address several of my main concerns. The clarifications regarding and are also helpful.

Some concerns are only partially resolved. The recovery-objective experiment does not exactly match the final SEAE setup, and the timestep-specific baseline mainly studies block skipping rather than timestep-specific head structures. More importantly, the evidence is still limited to the same Wan2.2-TI2V-5B architecture and task. Nevertheless, the rebuttal provides meaningful new evidence and improves my overall assessment. I will raise my score to 3.

Reviewer_muqmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SEAE, a structural pruning framework that reduces attention width and DiT depth in the four-step Wan2.2-TI2V-5B Turbo model. It first applies MHA-CSS: for each self- and cross-attention module, it builds Gram matrices of output-space head contributions over a calibration set, derives a head budget from cumulative spectral energy across solver timesteps, and selects retained heads through backward elimination with closed-form scalar refitting and a worst-timestep reconstruction criterion. The corresponding attention parameters are then physically sliced. Next, it scores each DiT block by temporarily replacing it with the identity at one timestep, completing the remaining trajectory, and measuring relative squared error at the final latent over the generated frames. Intersecting the per-timestep low-error candidate sets yields one static block-pruning mask. Both masks use forward-only calibration and define a fixed execution graph without runtime routing or cross-timestep feature caching. The compact model then undergoes 1,000 steps of recovery training with distillation and Flow Matching losses. The reported model reduces parameters from 5B to 3.73B and DiT latency from 1.20 s to 0.88 s, while achieving a VBench-I2V Total of 0.8702 versus the teacher's 0.8719.

Critical strengths
  1. Mask construction requires no backpropagation, and the resulting student is a fixed narrower, shallower network. The masks are derived from forward passes over 500 calibration pairs, and inference requires neither a runtime router nor a cross-timestep feature cache. The distinction from TinyFusion is the forward-only mask search; the static execution graph also offers a deployment benefit relative to routing and caching approaches.

  2. The Ablation Error criterion accounts for the downstream effects of block removal. Instead of relying only on local input–output cosine similarity, as described for SANA1.5/NeoDragon, it measures the perturbation at the final generated latent. The reported higher sensitivity at high-noise timesteps and shifts in block rankings across timesteps provide a clear motivation for intersecting the per-timestep low-error candidate sets.

  3. The component ablation provides useful evidence for CSS-based head selection. On VBench-I2V mini100 before recovery, CSS achieves 0.825 versus 0.730 for within-module random selection with the same module budgets and scalar refitting, isolating the effect of the selection rule at a fixed total of 889 retained heads. Table 3 also reports a separate refitting benefit under uniform budgets and random selection, improving the score from 0.671 to 0.721.

Critical weaknesses
  1. Near-parity in the aggregate score masks a substantial decline in dynamic degree. Table 4 reports a decrease from 0.5276 to 0.1537, approximately 71% in relative score terms, while eight of the ten individual dimensions improve. Quality falls from 0.7961 to 0.7805, but the higher I2V score largely offsets this decline in Total. The main text does not discuss the dynamic-degree drop, and the sampled-frame figures are insufficient to assess motion quality. Reduced motion could contribute to higher consistency or image-preservation scores, although the reported results do not establish this relationship. Motion-focused evaluation is therefore needed to support the broader quality-preservation claim, especially given TinyFusion's substantially higher dynamic-degree score of 0.3902.

  2. The comparison with other approaches to reducing per-step attention cost is incomplete. The main baselines cover structural pruning and caching, and the related work mentions token pruning, but there is no direct comparison or substantive discussion of attention sparsification. Structural pruning has a clear benefit in reducing the parameter set, which attention sparsification generally does not provide. Nevertheless, a comparison with a training-free sparse-attention method at matched quality or latency, also reporting memory use and offline calibration/recovery costs, would clarify the practical trade-offs. Testing their combination would further establish whether the approaches are complementary.

  3. The head-selection ablation establishes an advantage over random selection but does not compare CSS against a stronger importance-based criterion. Table 3 varies module budgets, random versus CSS selection, and scalar refitting; it does not evaluate a whole-head adaptation of Taylor importance, which the related work discusses for Diff-Pruning, or a simpler magnitude- or activation-based criterion. Such a comparison at matched head budgets would strengthen the justification for CSS. For gradient-based alternatives, reporting mask-search cost would help assess the trade-off against SEAE's forward-only calibration.

Questions for authors

Q1. Dynamic degree falls from 0.5276 to 0.1537 while eight of the ten individual VBench-I2V dimensions improve. Could you report motion measurements before and after recovery training, complemented by optical-flow statistics or an assessment of whether generated motion matches the prompts? Could you also investigate whether the higher I2V score (0.9598 versus 0.9477) is associated with reduced motion?

Q2. Could you provide end-to-end generation latency, including text encoding and VAE decoding, and specify the GPU model, batch size, precision, and timing protocol? For TinyFusion and NeoDragon, the displayed teacher/student latencies imply a speedup of 1.20/0.92 ≈ 1.30×, whereas Tables 1 and 4 report 1.20×. Could you reconcile these values and clarify whether the identical FLOPs, latency, and peak-memory entries for these two methods reflect matched configurations or rounding?

Q3. The paper motivates pruning in the few-step regime, but evaluates only one four-step backbone and acknowledges that a controlled multi-step/few-step comparison is missing. Could you compare pruning-induced quality loss on the four-step model and the corresponding multi-step base model, relative to their respective unpruned baselines and at matched pruning ratios and recovery budgets? Evidence from an appropriate eight- or sixteen-step configuration would also help assess generalization across sampling regimes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.