- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the cost of attention over spatiotemporal tokens in video diffusion transformers. It presents DraftAttention2, a training free framework that builds a low resolution draft attention map from average and max pooled queries and keys, and uses one shared ranking to jointly select attention blocks and assign their precision at 16, 8, or 4 bits or skip them, supported by an error decomposition analysis and a fused mixed precision kernel. Experiments on HunyuanVideo and MiniMax-H3, including few step variants, report up to 2.6 times DiT speedups with competitive VBench scores and better fidelity to dense outputs than sparse and low bit baselines.
Critical strengths
- The paper generalizes binary block selection into a joint allocation of sparsity and numerical precision, and the accompanying analysis gives this design principled support. The decomposition into sparsification and attention weighted quantization terms clarifies when low bit recovery is preferable to skipping, and justifies assigning higher precision to blocks with larger attention mass.
- The execution design translates fine grained precision assignments into measured speedups rather than theoretical savings. Fused operand preparation shares data movement between draft construction and quantization, and phased execution within a single kernel keeps precision transitions independent of the number of retained blocks.
- The empirical evaluation is broad and carefully controlled. Two model families with few step variants, seven baselines, matched speedup comparisons, and error versus latency ablations together make a convincing case that joint allocation improves the quality efficiency trade-off, particularly in the practical few step setting.
Critical weaknesses
- The quality evidence rests mainly on fidelity to dense outputs rather than standalone generation quality. On HunyuanVideo the strongest sparse baseline reaches a higher VBench average than all proposed configurations (76.81 versus at most 74.83), and in Table 3 adding max pooling slightly lowers the VBench average while improving similarity metrics, so the gains are clearer for dense output fidelity than for generation quality itself.
- The reported speedups are DiT level and hardware specific. Latency excludes text encoding and VAE decoding, dense attention is retained for the first 10 steps (first quarter in few step variants) without a sensitivity analysis, and all measurements use a single RTX 5090, so the 1.6 to 2.6 times figures do not translate directly to end to end application level acceleration.
- The theoretical results assume uniform error envelopes and exact attention masses, whereas the router uses draft scores as low cost proxies. The link between the error bounds and generation quality therefore remains empirical, and the per model precision quotas and pooling weight are tuned with limited guidance for transfer to new backbones.
Questions for authors
- The first 10 denoising steps (first quarter in few step variants) use dense attention. How sensitive are quality and speedup to the number of dense warm up steps, and can the framework operate with no dense steps at all?
- On HunyuanVideo all proposed configurations trail the strongest sparse baseline on VBench average despite better similarity metrics, and adding max pooling slightly lowers VBench in Table 3. How should this gap between standalone quality and dense output fidelity be interpreted, and is the VBench ranking stable across seeds?
- Theorem 3.6 assumes exact attention masses while routing relies on draft scores. Can any evidence quantify how closely the draft ranking tracks the exact mass ranking and how much the error bound degrades under this proxy?
2 comments
Response to Reviewer dxdz (2/2)
C3. Speedup including text encoding and VAE decoding (W2)
Latency excludes text encoding and VAE decoding
The acceleration remains substantial after including text encoding and VAE decoding. DraftAttention2 reaches E2E speedups of , , , and on HunyuanVideo, MiniMax-H3, and its 4/8-step variants, respectively, compared with , , , and for Sol-Attn. Sparsity denotes the fraction of skipped blocks; precision shares are measured among retained blocks.
HunyuanVideo
| Method | Sparsity (%) | 4/8/16-bit mix | DiT speedup | E2E speedup |
|---|---|---|---|---|
| Dense | 0.0 | 0%/0%/100% | ||
| Sol-Attn | 85.8 | 0%/0%/100% | ||
| DraftAttention2 | 80.7 | 75%/25%/0% |
MiniMax-H3
| Method | Sparsity (%) | 4/8/16-bit mix | DiT speedup | E2E speedup |
|---|---|---|---|---|
| Dense | 0.0 | 0%/0%/100% | ||
| Sol-Attn | 80.0 | 0%/0%/100% | ||
| DraftAttention2 | 74.3 | 75%/25%/0% |
4-step MiniMax-H3
| Method | Sparsity (%) | 4/8/16-bit mix | DiT speedup | E2E speedup |
|---|---|---|---|---|
| Dense | 0.0 | 0%/0%/100% | ||
| Sol-Attn | 65.0 | 0%/0%/100% | ||
| DraftAttention2 | 54.5 | 75%/25%/0% |
8-step MiniMax-H3
| Method | Sparsity (%) | 4/8/16-bit mix | DiT speedup | E2E speedup |
|---|---|---|---|---|
| Dense | 0.0 | 0%/0%/100% | ||
| Sol-Attn | 79.9 | 0%/0%/100% | ||
| DraftAttention2 | 74.0 | 50%/50%/0% |
C4. Draft approximation, the error bound, and configuration transfer (W3, Q3)
Can any evidence quantify how closely the draft ranking tracks the exact mass ranking and how much the error bound degrades under this proxy?
The draft retains 87.59–97.05% of the high-precision attention mass captured by the exact-mass oracle. We evaluate 4-step MiniMax-H3 at retained fractions of 35.05% and 64.95%, covering all query regions and all 56 heads. At each budget, the support and 75%/25% NVFP4/INT8 quotas are fixed across allocation rules. For comparison, uniform assignment with the same quota has an expected mass capture of 27.81–30.93%.
This measurement connects directly to Theorem 3.6. Let be the oracle-minus-draft difference in exact retained mass assigned to INT8. The difference in the theorem's assignment-dependent quantization term is
The measured head-averaged is 0.02546–0.09685, quantifying the effect of draft allocation in the same quantities as the bound. Ranking by exact pre-pruning masses captures 99.800–99.987% of the renormalized-mass oracle, showing that the normalization change has little effect on mass capture in this comparison.
the per model precision quotas and pooling weight are tuned with limited guidance for transfer to new backbones.
The pooling weight and retained precision ratios are shared across model settings. We use throughout and fixed 75%/25% or 50%/50% NVFP4/INT8 mixtures. Sparsity is adjusted to match the comparison latency, after which the configuration is fixed for quality evaluation. The percentages in Tables 1 and 2 of the submission refer to the whole map: for a fixed 75%/25% mixture, the 4/8-bit fractions are and at sparsity . Different reported percentages therefore reflect different density budgets, rather than independently selected precision ratios for each backbone.
Summary
The 4/8-bit results demonstrate higher PSNR at higher speedups than Sol-Attn, and higher PSNR and VBench scores at speedups matched to SpargeAttention, across all four model settings. The additional measurements establish low-cost draft allocation close to the mass oracle and substantial application-level speedup after including the surrounding generation stages.
Response to Reviewer dxdz (1/2)
We thank the reviewer for recognizing the joint allocation, theoretical analysis, and practical few-step gains. We address the quality assessment, warm-up protocol, application latency, and draft approximation below.
C1. Generation quality, dense fidelity, and seed stability (W1, Q2)
Our 4/8-bit configurations improve PSNR at higher DiT speedups than Sol-Attn across all four model settings. Tables 1 and 2 also show improvements in SSIM and LPIPS on HunyuanVideo and the two few-step models. On 4-step and 8-step MiniMax-H3, PSNR increases by 0.74 and 0.64 dB, while DiT speedups increase from 1.46×/1.80× to 1.84×/2.12×. The corresponding VBench averages are 74.00/79.89 versus 74.19/79.91.
At speedups matched to SpargeAttention, DraftAttention2 achieves higher PSNR and VBench averages in all four settings. On HunyuanVideo, for example, DraftAttention2 reaches 75.96 VBench at 2.51×, compared with 74.36 at 2.50× for SpargeAttention. The benefit of joint allocation is therefore supported by both fidelity and generation-quality measurements at comparable speed.
Evaluating fidelity alongside VBench follows established practice in video acceleration, including Sparse VideoGen [1, §5]. PSNR/SSIM/LPIPS measure preservation of the original model's output, while VBench [2] evaluates video-quality attributes. VMBench [3, §4.2] finds that several VBench motion metrics have limited alignment with human motion-quality judgments. A VBench average should therefore be considered alongside fidelity when assessing an attention approximation.
MaxPool helps the draft identify important attention contributions. In Table 3, adding MaxPool improves PSNR, SSIM, and LPIPS in all three precision settings, directly supporting this design choice.
All original quality evaluations use the same 326 prompts and a fixed seed of 0 for every method, without seed selection. Recent constraints on GPU availability have prevented us from completing the multi-seed evaluation within the rebuttal period.
C2. Sensitivity to dense warm-up steps (W2, Q1)
Dense warm-up follows the evaluation practice of prior sparse video-attention work. Sparse VideoGen [1, §5.1] retains dense computation for the first 25% of denoising steps, while AdaSpa [4] retains the first 10 steps for all methods. We adopt this convention for consistent comparison with existing approaches.
DraftAttention2 constructs its draft from the current queries and keys at each attention call. It does not require calibration statistics from earlier dense steps and can be enabled from the first denoising step. Recent constraints on GPU availability have prevented us from completing the quality evaluation without dense warm-up within the rebuttal period.
References
[1] Haocheng Xi et al. Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. ICML, 2025.
[2] Ziqi Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR, 2024.
[3] Xinran Ling et al. VMBench: A Benchmark for Perception-Aligned Video Motion Generation. ICCV, 2025.
[4] Yifei Xia et al. Training-free and Adaptive Sparse Attention for Efficient Long Video Generation. ICCV, 2025.