- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the fidelity loss of dynamic block sparse attention in video diffusion transformers, which it attributes to coarse block scoring, softmax truncation under top k selection, and tile sized selection units. It proposes AB-Attention, a training free method that scores small key/value blocks using query clusters formed within each query block, samples blocks in proportion to these scores with inverse inclusion probability reweighting so that one sparse pass estimates dense attention including the softmax normalizer, and gathers the sampled blocks into full tensor core tiles for efficient execution. On Wan2.1 and HunyuanVideo at 720p, the method skips 89 to 92 percent of attention computation, reports 1.84 to 2.23 times end to end speedups on an H20 GPU, and improves PSNR against dense references by 1.6 to 4.4 dB over the strongest baseline.
Critical strengths
- The paper decomposes the limitations of dynamic block sparse attention into three concrete design choices and addresses each with a matching component. The mechanism level analysis in Figure 2 supports the design with recall and error measurements against an oracle top k reference.
- Sampling with inclusion probability weighting is a statistically principled alternative to top k truncation and to correction branches. A single sparse pass estimates both the attention numerator and the softmax normalizer without bias, and the logit shift integrates this reweighting into the kernel at negligible cost.
- The evaluation is controlled and consistent across three models. All baselines share the same block sparse kernel and dense warm up protocol, and the proposed method is simultaneously the sparsest, the most faithful to dense attention, and the fastest in every comparison, with ablations isolating each component.
Critical weaknesses
- Each method is compared at a single operating point rather than along a sparsity or latency sweep. The baselines run at 67 to 81 percent sparsity while the proposed method runs at 89 to 92 percent, so quality versus speedup frontiers would be needed to exclude that part of the advantage reflects a more aggressive operating point rather than a better selection mechanism.
- The unbiasedness of SIP holds in expectation given current scores, but sampling plans are rebuilt every four steps while activations evolve. The effect of this staleness and the single sample variance of the estimator are not analyzed, and no seed or resampling variance is reported for the fidelity metrics.
- On the standalone VBench dimensions the method is close to dense attention but not consistently the closest, with RadialAttention and DFSAttn matching or exceeding it on aesthetic and imaging quality in Table 1. The central advantage therefore rests on same seed fidelity metrics, which measure closeness to the dense reference rather than perceptual quality itself, and the evaluation uses one prompt set, one resolution, and one GPU.
Questions for authors
- The baselines operate at 67 to 81 percent sparsity while the proposed method operates at 89 to 92 percent. How do the strongest baselines perform when their budgets are raised to matched sparsity, and can quality versus speedup frontiers be provided?
- Sampling plans are reused for four denoising steps while activations evolve, so the inclusion probabilities become stale. How do the fidelity metrics vary with the rebuild interval, and how large is the run to run variance of the sampling estimator across seeds?
- On the VBench aesthetic and imaging quality dimensions, RadialAttention and DFSAttn are as close or closer to dense attention than the proposed method in Table 1. How should this divergence between same seed fidelity metrics and standalone quality dimensions be interpreted?
4 comments
Response to Reviewer TDP9 (2/2)
Q3 & W3: On the VBench aesthetic and imaging quality dimensions, RadialAttention and DFSAttn are as close or closer to dense attention than the proposed method in Table 1. How should this divergence between same seed fidelity metrics and standalone quality dimensions be interpreted?
Thanks for raising this insightful observation. Fidelity metrics (PSNR, SSIM, and LPIPS) directly measure how accurately sparse attention reproduces the dense attention output, and are therefore the primary evaluation metrics across video acceleration works (e.g., SVG [1], SVG2 [2], RadialAttention [3], and DFSAttn [4]). By contrast, VBench's Aesthetic and Imaging Quality [5] are reference-free single-frame scores that never compare against the dense reference video and are much less sensitive to fine-grained approximation errors; across all VBench dimensions, AB-Attention remains on par with dense attention while improving PSNR by +3.1–6.7 dB over RadialAttention and DFSAttn. We have clarified this in Sections 5.1 and 5.2.
References:
[1] Xi et al. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025.
[2] Yang et al. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation, NeurIPS 2025.
[3] Li et al. Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation, NeurIPS 2025.
[4] Hu et al. DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation, ICML 2026.
[5] Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024.
Thanks for the detailed response. My concerns are mostly addressed, and I decide to improve my score.
Thank you for raising the score. We will further improve our manuscript.
Response to Reviewer TDP9 (1/2)
We sincerely appreciate this valuable suggestion. Raising SVG2 and DFSAttn to our ~89% sparsity lowers their PSNR by 1.83 and 1.55 dB, widening AB-Attention's lead to +3.64 and +4.61 dB, and AB-Attention consistently dominates the quality–speedup frontier across 1.49×–1.84× speedup.
Strongest baselines at matched ~89% sparsity: We evaluate SVG2 and DFSAttn at ~89% sparsity on 100 Wan2.1-T2V-14B prompts. Below, we present the experimental results:
Quality–speedup frontier: Sweeping AB-Attention's budget (P = 64 → 192, where P = 64 is the default setting in the manuscript) on the same 100 prompts on H20 traces a smooth quality–speedup frontier. Below, we present the experimental results:
Across 1.49×–1.84× speedup, AB-Attention spans 29.705 to 33.029 dB PSNR, exceeding matched-sparsity SVG2 at 1.83× by +3.64 dB, default DFSAttn at 1.65× by +5.09 dB, and default SVG2 at 1.50× by +5.13 dB. We have included the matched-sparsity comparison, budget sweep, and plotted frontiers in Appendix E.
We appreciate this thoughtful question on plan reuse and estimator stability. Fixed-activation unbiasedness holds strictly at plan-refresh steps (and at every step when C = 1), while plan reuse (C > 1) amortizes planning across nearby steps where activations evolve gradually. Empirically, fidelity decreases smoothly and modestly with the rebuild interval C (30.337 → 29.439 dB PSNR across C = 1 → 8 steps), and is insensitive to the sampling seed with minimal run-to-run variance (29.713 ± 0.009 dB across five 100-prompt runs).
Rebuild interval: We sweep the rebuild interval on 100 Wan2.1-T2V-14B prompts with fixed generation noise on H20 (C = 4 is the default setting in the manuscript). Below, we present the experimental results:
Fidelity across PSNR, SSIM, and LPIPS declines gradually as C increases from 1 to 8: rebuilding every step at C=1 adds 5.9% latency for +0.63 dB over C=4, making C=4 a favorable quality–speed operating point.
Sampling-seed variance: We evaluate five sampling seeds (40–44, with 42 as the default setting in the manuscript) on the same 100 prompts with fixed generation noise (500 videos). Below, we present the experimental results:
Dataset means span only 0.023 dB across seeds. In SIP, dominant blocks reach πij=1 and are retained deterministically with zero sampling variance, while systematic sampling with 1/πij weighting yields a low-variance estimate of the remaining tail; moreover, independent draws across query blocks, heads, and layers further reduce end-to-end sampling variance. We have added both sensitivity analyses to Appendix B.1.