acceptodds
Under review as a conference paper at ICLR 2027

AB-Attention: Accurate Block Attention for Accelerating Video Diffusion Transformers

19 views

Abstract

Video diffusion transformers generate high-quality videos, but the quadratic cost of 3D full attention dominates their inference. Dynamic block-sparse attention reduces this cost by selecting key/value blocks from the current activations and computing exact attention only on the selected ones. However, its fidelity degrades under aggressive sparsity: blocks are scored from coarse summaries or formed by costly global clustering, pruned blocks vanish from the softmax, and whole tiles are kept or dropped even when only a few of their keys matter. We present Accurate Block Attention (AB-Attention), a training-free method that addresses each limitation. Its Intra-Block Clustering Selector (IBCS) estimates block importance faithfully yet cheaply by clustering queries only within each query block, without permuting tokens. Sampling with Inclusion-Probability Weighting (SIP) then samples blocks in proportion to these estimates and reweights them by inverse inclusion probability, so that one pass estimates dense attention rather than truncating it. Finally, its kernel gathers key/value blocks much smaller than a tile into full tensor-core tiles, refining selection at no extra tensor-core cost. On Wan2.1 and HunyuanVideo, AB-Attention skips 89–92% of attention, whereas the baselines compute 1.8–3.6 as much, yet it still improves PSNR by up to 4.4 dB over the strongest baseline and achieves up to 2.23 end-to-end speedup.

Primary area:
generative models
Keywords:
sparse attention, diffusion transformer, video generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TDP9OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the fidelity loss of dynamic block sparse attention in video diffusion transformers, which it attributes to coarse block scoring, softmax truncation under top k selection, and tile sized selection units. It proposes AB-Attention, a training free method that scores small key/value blocks using query clusters formed within each query block, samples blocks in proportion to these scores with inverse inclusion probability reweighting so that one sparse pass estimates dense attention including the softmax normalizer, and gathers the sampled blocks into full tensor core tiles for efficient execution. On Wan2.1 and HunyuanVideo at 720p, the method skips 89 to 92 percent of attention computation, reports 1.84 to 2.23 times end to end speedups on an H20 GPU, and improves PSNR against dense references by 1.6 to 4.4 dB over the strongest baseline.

Critical strengths
  1. The paper decomposes the limitations of dynamic block sparse attention into three concrete design choices and addresses each with a matching component. The mechanism level analysis in Figure 2 supports the design with recall and error measurements against an oracle top k reference.
  2. Sampling with inclusion probability weighting is a statistically principled alternative to top k truncation and to correction branches. A single sparse pass estimates both the attention numerator and the softmax normalizer without bias, and the logit shift integrates this reweighting into the kernel at negligible cost.
  3. The evaluation is controlled and consistent across three models. All baselines share the same block sparse kernel and dense warm up protocol, and the proposed method is simultaneously the sparsest, the most faithful to dense attention, and the fastest in every comparison, with ablations isolating each component.
Critical weaknesses
  1. Each method is compared at a single operating point rather than along a sparsity or latency sweep. The baselines run at 67 to 81 percent sparsity while the proposed method runs at 89 to 92 percent, so quality versus speedup frontiers would be needed to exclude that part of the advantage reflects a more aggressive operating point rather than a better selection mechanism.
  2. The unbiasedness of SIP holds in expectation given current scores, but sampling plans are rebuilt every four steps while activations evolve. The effect of this staleness and the single sample variance of the estimator are not analyzed, and no seed or resampling variance is reported for the fidelity metrics.
  3. On the standalone VBench dimensions the method is close to dense attention but not consistently the closest, with RadialAttention and DFSAttn matching or exceeding it on aesthetic and imaging quality in Table 1. The central advantage therefore rests on same seed fidelity metrics, which measure closeness to the dense reference rather than perceptual quality itself, and the evaluation uses one prompt set, one resolution, and one GPU.
Questions for authors
  1. The baselines operate at 67 to 81 percent sparsity while the proposed method operates at 89 to 92 percent. How do the strongest baselines perform when their budgets are raised to matched sparsity, and can quality versus speedup frontiers be provided?
  2. Sampling plans are reused for four denoising steps while activations evolve, so the inclusion probabilities become stale. How do the fidelity metrics vary with the rebuild interval, and how large is the run to run variance of the sampling estimator across seeds?
  3. On the VBench aesthetic and imaging quality dimensions, RadialAttention and DFSAttn are as close or closer to dense attention than the proposed method in Table 1. How should this divergence between same seed fidelity metrics and standalone quality dimensions be interpreted?
4 comments
AuthorsOpenReview ↗

Response to Reviewer TDP9 (1/2)

Q1 & W1: The baselines operate at 67 to 81 percent sparsity while the proposed method operates at 89 to 92 percent. How do the strongest baselines perform when their budgets are raised to matched sparsity, and can quality versus speedup frontiers be provided?

We sincerely appreciate this valuable suggestion. Raising SVG2 and DFSAttn to our ~89% sparsity lowers their PSNR by 1.83 and 1.55 dB, widening AB-Attention's lead to +3.64 and +4.61 dB, and AB-Attention consistently dominates the quality–speedup frontier across 1.49×–1.84× speedup.

  • Strongest baselines at matched ~89% sparsity: We evaluate SVG2 and DFSAttn at ~89% sparsity on 100 Wan2.1-T2V-14B prompts. Below, we present the experimental results:

    MethodSparsity ↑PSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓Speedup ↑
    AB-Attention (P = 64)89.16%29.7050.91540.095625721.84×
    SVG2 (matched)88.88%26.0690.83770.218225871.83×
    DFSAttn (matched)89.14%25.0960.83130.210526561.79×
  • Quality–speedup frontier: Sweeping AB-Attention's budget (P = 64 → 192, where P = 64 is the default setting in the manuscript) on the same 100 prompts on H20 traces a smooth quality–speedup frontier. Below, we present the experimental results:

    AB-Attention budgetSparsity ↑PSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓Speedup ↑
    P = 64 (default)89.16%29.7050.91540.095625721.84×
    P = 9683.75%30.7710.92920.075927261.74×
    P = 12878.33%31.7320.94050.063728771.65×
    P = 16072.91%32.3630.94580.057030341.56×
    P = 19267.49%33.0290.95070.051431901.49×

    Across 1.49×–1.84× speedup, AB-Attention spans 29.705 to 33.029 dB PSNR, exceeding matched-sparsity SVG2 at 1.83× by +3.64 dB, default DFSAttn at 1.65× by +5.09 dB, and default SVG2 at 1.50× by +5.13 dB. We have included the matched-sparsity comparison, budget sweep, and plotted frontiers in Appendix E.

Q2 & W2: Sampling plans are reused for four denoising steps while activations evolve, so the inclusion probabilities become stale. How do the fidelity metrics vary with the rebuild interval, and how large is the run to run variance of the sampling estimator across seeds?

We appreciate this thoughtful question on plan reuse and estimator stability. Fixed-activation unbiasedness holds strictly at plan-refresh steps (and at every step when C = 1), while plan reuse (C > 1) amortizes planning across nearby steps where activations evolve gradually. Empirically, fidelity decreases smoothly and modestly with the rebuild interval C (30.337 → 29.439 dB PSNR across C = 1 → 8 steps), and is insensitive to the sampling seed with minimal run-to-run variance (29.713 ± 0.009 dB across five 100-prompt runs).

  • Rebuild interval: We sweep the rebuild interval on 100 Wan2.1-T2V-14B prompts with fixed generation noise on H20 (C = 4 is the default setting in the manuscript). Below, we present the experimental results:

    Reuse interval CPSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓
    C = 130.3370.92220.08782724
    C = 229.9910.91900.09172622
    C = 4 (default)29.7050.91540.09562572
    C = 829.4390.91160.10052546

    Fidelity across PSNR, SSIM, and LPIPS declines gradually as CC increases from 11 to 88: rebuilding every step at C=1C = 1 adds 5.9% latency for +0.63 dB over C=4C = 4, making C=4C = 4 a favorable quality–speed operating point.

  • Sampling-seed variance: We evaluate five sampling seeds (40–44, with 42 as the default setting in the manuscript) on the same 100 prompts with fixed generation noise (500 videos). Below, we present the experimental results:

    Sampling seedPSNR ↑SSIM ↑LPIPS ↓
    4029.7280.91550.0949
    4129.7100.91440.0957
    42 (default)29.7050.91540.0956
    4329.7070.91490.0954
    4429.7160.91410.0963
    Mean ± Std29.713 ± 0.0090.9149 ± 0.00060.0956 ± 0.0005

    Dataset means span only 0.023 dB across seeds. In SIP, dominant blocks reach πij=1\pi_{ij} = 1 and are retained deterministically with zero sampling variance, while systematic sampling with 1/πij1/\pi_{ij} weighting yields a low-variance estimate of the remaining tail; moreover, independent draws across query blocks, heads, and layers further reduce end-to-end sampling variance. We have added both sensitivity analyses to Appendix B.1.

AuthorsOpenReview ↗

Response to Reviewer TDP9 (2/2)

Q3 & W3: On the VBench aesthetic and imaging quality dimensions, RadialAttention and DFSAttn are as close or closer to dense attention than the proposed method in Table 1. How should this divergence between same seed fidelity metrics and standalone quality dimensions be interpreted?

Thanks for raising this insightful observation. Fidelity metrics (PSNR, SSIM, and LPIPS) directly measure how accurately sparse attention reproduces the dense attention output, and are therefore the primary evaluation metrics across video acceleration works (e.g., SVG [1], SVG2 [2], RadialAttention [3], and DFSAttn [4]). By contrast, VBench's Aesthetic and Imaging Quality [5] are reference-free single-frame scores that never compare against the dense reference video and are much less sensitive to fine-grained approximation errors; across all VBench dimensions, AB-Attention remains on par with dense attention while improving PSNR by +3.1–6.7 dB over RadialAttention and DFSAttn. We have clarified this in Sections 5.1 and 5.2.

References:
[1] Xi et al. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025.
[2] Yang et al. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation, NeurIPS 2025.
[3] Li et al. Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation, NeurIPS 2025.
[4] Hu et al. DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation, ICML 2026.
[5] Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024.

Thanks for the detailed response. My concerns are mostly addressed, and I decide to improve my score.

Thank you for raising the score. We will further improve our manuscript.

Reviewer_fGQJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AB-Attention is a training-free method for accelerating video diffusion through fine-grained block sampling. It uses query clustering within each block to estimate key/value-block importance and corrects sampled attention with inverse inclusion-probability weights. Its kernels pack small selected blocks into full execution tiles, with sampling plans reused across nearby denoising steps. Experiments on Wan2.1 and HunyuanVideo report up to 2.23× end-to-end speedup over dense attention.

Critical strengths
  1. The fixed-budget ablations provide useful evidence that each component contributes to the result.
Critical weaknesses
  1. The speedup over evaluated prior works is marginal, leaving the practical significance of the improvement unclear.

  2. The evaluation could be improved. For example, by selecting more informative quality metrics.

Please see the comments for more details.

Questions for authors

Thank you for your submission to ICLR. Here are my comments/questions.

  1. Table 1 shows only a 7.3–10.7% reduction in end-to-end latency relative to the fastest evaluated prior method for each model. My major concern is whether these results establish a substantial practical advantage over prior works. The improvement seems marginal.

  2. The VBench results are mixed and hard to read. It would be better to present only the Quality Score, Semantic Score, and Total Score, and move the results for each dimension to the appendix.

  3. The paper does not discuss a closely related prior work, Sol-Attn [1], which combines sparse attention with an approximation of unselected blocks. I understand Sol-Attn is an arXiv preprint, but it has been available since July. An experimental comparison, or at least a discussion of the key differences between the two methods, would help clarify the paper’s contribution.

  4. Appendix B.1 specifies a single sampling seed (42). It would be helpful to repeat the evaluation with several sampling seeds and report the mean and standard deviation to assess the consistency of the results.

  5. Table 3 reports quality metrics for different selection block sizes, but not their corresponding latencies. Smaller blocks may introduce additional selection and data-gathering overhead even when the number of retained tokens is fixed. Reporting latency for these variants would help clarify whether finer selection improves the quality–latency trade-off.

[1] Li, Haopeng, et al. "Sol-attn: Accelerating video generation inference via on-the-fly attention sparsification." arXiv preprint arXiv:2607.24027 (2026).

3 comments
AuthorsOpenReview ↗

Response to Reviewer fGQJ (1/2)

Q1 & W1: Table 1 shows only a 7.3–10.7% reduction in end-to-end latency relative to the fastest evaluated prior method for each model. My major concern is whether these results establish a substantial practical advantage over prior works. The improvement seems marginal.

Thanks. Because speedup is tunable via the sparsity budget, prior works typically operate at similar latency to earlier baselines and demonstrate their advantage through generation quality; AB-Attention achieves a 7.3–10.7% latency reduction while simultaneously delivering a +3.1–4.4 dB PSNR gain.

  • Standard practice in recent top-venue works: In their own main tables, prominent methods match or barely exceed the speed of their fastest baseline (0–3.1%) to showcase higher fidelity: SVG2 [1, Table 1] is 0–1.3% faster than SVG on Wan2.1-14B, PISA [2, Table 1] is 1.2–3.1% faster than its fastest baseline on HunyuanVideo and Wan2.1-14B, while DFSAttn [3, Table 2] and RadialAttention [4, Table 1] run 5.0–21.8% slower than the fastest baseline in their tables to recover quality.

  • Substantial quality gains alongside faster inference: Rather than merely matching baseline latency, AB-Attention cuts end-to-end latency by 7.3–10.7% while using 1.8–2.4× less attention and improving PSNR by +3.1–4.4 dB over the fastest baseline on every model. When instead tuned to matched latency on Wan2.1-14B, AB-Attention's PSNR lead widens to +5.09 dB over DFSAttn (at 1.65× speedup) and +5.13 dB over SVG2 (at 1.50× speedup) (Section 5.2 and Appendix E). Below, we summarize the comparison from Table 1:

    MetricWan2.1-1.3BWan2.1-14BHunyuanVideo
    Fastest baseline (s) ↓SVG2, 536.5DFSAttn, 2866.5DFSAttn, 2129.1
    AB-Attention (s) ↓479.12571.51974.6
    Latency reduction ↑10.7%10.3%7.3%
    PSNR ↑ (fastest → AB)30.02 → 34.4427.71 → 30.7930.25 → 34.35
    LPIPS ↓ (fastest → AB)0.1576 → 0.07340.1464 → 0.09640.1151 → 0.0700
    Sparsity ↑ (fastest → AB)80.12% → 89.16%81.03% → 89.16%81.29% → 92.35%

References:
[1] Yang et al. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation, NeurIPS 2025.
[2] Li et al. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers, NeurIPS 2026.
[3] Hu et al. DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation, ICML 2026.
[4] Li et al. Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation, NeurIPS 2025.

Q2 & W2: The VBench results are mixed and hard to read. It would be better to present only the Quality Score, Semantic Score, and Total Score, and move the results for each dimension to the appendix.

Thanks for the suggestion. Following prior sparse video attention works such as SVG [2] and DFSAttn [3], we evaluate on the Penguin Benchmark prompts [1], which do not carry VBench's prompt-level semantic labels. Consequently, VBench's official Quality, Semantic, and Total scores cannot be computed. We therefore followed the exact reporting format of prior baselines, which all list individual VBench dimensions directly in their main tables without composite scores: DFSAttn [3, Table 2] reports the exact same five columns (AQ, BC, IQ, MS, SC), and SVG [2, Table 1], RadialAttention [4, Table 2], and PISA [5, Table 1] likewise report individual dimensions (such as AQ, IQ, and SC) directly alongside PSNR, SSIM, and LPIPS. We have clarified this in Appendix C, and if preferred for readability, we are also happy to move all five VBench columns to the appendix so Table 1 focuses cleanly on PSNR, SSIM, LPIPS, and latency.

References:
[1] Kong et al. HunyuanVideo: A Systematic Framework for Large Video Generative Models, arXiv 2024.
[2] Xi et al. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025.
[3] Hu et al. DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation, ICML 2026.
[4] Li et al. Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation, NeurIPS 2025.
[5] Li et al. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers, NeurIPS 2026.

AuthorsOpenReview ↗

Response to Reviewer fGQJ (2/2)

Q3: The paper does not discuss a closely related prior work, Sol-Attn [1], which combines sparse attention with an approximation of unselected blocks. I understand Sol-Attn is an arXiv preprint, but it has been available since July. An experimental comparison, or at least a discussion of the key differences between the two methods, would help clarify the paper's contribution.

We appreciate the reviewer pointing out this closely related work. Section 2 already discusses Sol-Attn's predecessor PISA [2], and we have expanded it to cover Sol-Attn [1] directly alongside an experimental comparison:

  • Core difference (summary-based vs. sampling-based approximation of the tail): Sol-Attn computes the selected blocks exactly and approximates the remaining blocks with mean-pooled key/value summaries. AB-Attention instead samples from all blocks with score-proportional probabilities and reweights the sampled blocks by inverse inclusion probability, so a single sparse pass estimates the dense numerator and normalizer. The two methods thus account for the unselected tail differently, and we compare their accuracy empirically below.

  • Experimental comparison: We evaluated Sol-Attn via its official SM90 CuTe kernel with τ=1.0\tau = 1.0 on the same 100 Wan2.1-T2V-14B prompts on H20. Below, we present the experimental results:

    MethodSettingPSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓
    AB-AttentionP = 64, C = 4 (89.16% sparsity)29.7050.91540.09562572
    Sol-Attnτ = 1.0 (official)24.1940.82340.21112830

    AB-Attention outperforms Sol-Attn by +5.51 dB PSNR, +0.0920 SSIM, and −0.1155 LPIPS while also running faster.

We have added this discussion and comparison to Section 2 and Appendix E.

References:
[1] Li et al. Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification, arXiv 2026.
[2] Li et al. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers, NeurIPS 2026.

Q4: Appendix B.1 specifies a single sampling seed (42). It would be helpful to repeat the evaluation with several sampling seeds and report the mean and standard deviation to assess the consistency of the results.

Thanks for this helpful suggestion. Repeating Wan2.1-T2V-14B (P = 64, C = 4) across five sampling seeds (40–44, with 42 as the default setting in the manuscript) on 100 prompts with fixed generation noise (500 videos) confirms that results are insensitive to the sampling seed. Below, we present the experimental results:

Sampling seedPSNR ↑SSIM ↑LPIPS ↓
4029.7280.91550.0949
4129.7100.91440.0957
42 (default)29.7050.91540.0956
4329.7070.91490.0954
4429.7160.91410.0963
Mean ± Std29.713 ± 0.0090.9149 ± 0.00060.0956 ± 0.0005

Dataset means span only 0.023 dB across seeds. We have added this multi-seed evaluation to Appendix B.1.

Q5: Table 3 reports quality metrics for different selection block sizes, but not their corresponding latencies. Smaller blocks may introduce additional selection and data-gathering overhead even when the number of retained tokens is fixed. Reporting latency for these variants would help clarify whether finer selection improves the quality–latency trade-off.

We appreciate this insightful question on runtime trade-offs. Across block sizes 4 → 128 on Wan2.1-T2V-14B, full-video latency varies by only 2.2% (2572 s vs. 2516 s) while PSNR improves by +1.02 dB (28.686 → 29.705 dB), showing that finer selection improves the quality–latency trade-off. Below, we present the experimental results (block size 4 is the default setting in the manuscript):

Block sizePSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓Speedup ↑
b = 4 (default)29.7050.91540.095625721.84×
b = 829.6190.91500.095625391.87×
b = 1629.4550.91080.098525271.88×
b = 3229.2010.90920.101525241.88×
b = 6429.0690.90720.103225161.89×
b = 128 (tile-level)28.6860.90060.111325161.89×

Because planning runs once every C=4C = 4 sparse steps and gathered execution always operates on packed 128-token Tensor-Core tiles (staying flat at 94.6–100.9 ms across all block sizes), fine b=4b = 4 selection achieves higher fidelity at minimal latency cost. We have added these latency and speedup results and analysis to Section 5.3 and Appendix D (Table 5, formerly Table 3).

Dear Reviewer fGQJ,

Thank you for your constructive comments on practical efficiency, VBench reporting, Sol-Attn, and runtime stability. We wanted to briefly follow up on the five points you raised.

Practical advantage over prior baselines (Q1 & W1). While recent sparse attention works typically match baseline latency (0–3.1%) to improve fidelity, AB-Attention reduces end-to-end latency by 7.3–10.7% over the fastest baseline on each model while simultaneously improving PSNR by +3.1–4.4 dB; at matched latency on Wan2.1-14B, its PSNR margin widens to +5.09 dB over DFSAttn and +5.13 dB over SVG2 (Appendix E, Table 7, Figure 6).

VBench reporting (Q2 & W2). Appendix C clarifies that the Penguin Benchmark prompts lack VBench's category labels required for official Quality, Semantic, and Total aggregates, which is why prior works report standalone dimensions directly; we remain happy to move the five VBench columns to the appendix if preferred.

Discussion and comparison with Sol-Attn (Q3). Section 2 and Appendix E now contrast Sol-Attn's mean-pooled summary approximation of unselected blocks with AB-Attention's score-proportional block sampling and inverse-probability reweighting. On 100 Wan2.1-14B prompts on H20, AB-Attention outperforms official Sol-Attn (τ=1.0\tau = 1.0) by +5.51 dB PSNR (29.705 vs. 24.194 dB) with lower latency (2572 s vs. 2830 s) (Table 8).

Multi-seed consistency (Q4). Across five sampling seeds (40–44) on 100 prompts with fixed generation noise, dataset-mean PSNR is 29.713±0.00929.713 \pm 0.009 dB, spanning only 0.023 dB (Appendix B.1, Table 4).

Block-size latency trade-off (Q5). Shrinking the selection block from b=128b = 128 to b=4b = 4 increases end-to-end latency by only 2.2% (2516 s to 2572 s) while improving PSNR by +1.02 dB (Section 5.3; Appendix D, Table 5).

As the discussion draws to a close, we would welcome your feedback on whether these additions address your concerns. Please let us know if anything needs further clarification.

Best regards, Authors of Submission 38168

Reviewer_hqKXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AB-Attention, a training-free sparse attention method for accelerating video diffusion transformers. The method combines three components. First, the Intra-Block Clustering Selector constructs several query representatives within each query block and uses their weighted attention to pooled keys to estimate key/value block importance. Second, Sampling with Inclusion-Probability Weighting selects a fixed number of distinct blocks using systematic sampling and applies inverse-inclusion-probability weights through a negative log-probability shift of the attention logits. This produces unbiased estimates of the attention numerator and normalizer for fixed activations, although their ratio is generally biased. Third, a fused kernel gathers small key/value blocks into full execution tiles, separating selection granularity from tensor-core computation. Sampling plans are reused across nearby denoising steps.

Critical strengths
  1. The method connects its algorithmic choices to concrete execution constraints. Local query clustering provides multiple representatives without requiring a global token permutation, fixed-size sampling gives each query block a predictable computation budget, and gathering small key/value blocks into full tiles enables finer selection while retaining full-sized matrix operations. This combination is practically relevant for training-free acceleration of pretrained video diffusion models.

  2. Across three models and 600 prompts, AB-Attention achieves better PSNR, SSIM, and LPIPS than all evaluated sparse baselines while also reducing end-to-end latency. The PSNR improvements over the strongest evaluated baseline range from approximately 1.55 to 4.42 dB. The fixed-budget ablations are also informative: removing inclusion-probability weighting reduces PSNR by about 4 dB, while replacing query clustering or fine-grained selection causes smaller but consistent degradations. The timing protocol explicitly includes plan construction and sampling, which strengthens the practical relevance of the reported speedups.

Critical weaknesses
  1. Section 2 discusses Rectified SpaAttn, PISA, and SVG-EAR, all of which address the loss or distortion caused by discarding attention contributions. However, none is evaluated in Table 1. This omission is consequential because Table 2 identifies inclusion-probability weighting as the largest source of improvement. Comparisons with methods that directly discard the remainder demonstrate that the proposed approach is useful, but do not establish its advantage over existing correction-based alternatives. Moreover, Table 1 reports only one operating point per method. Although AB-Attention dominates the evaluated points, matched-quality or matched-latency comparisons and quality–latency curves are needed to substantiate the broader frontier claim. arxiv.org

  2. The estimator should be positioned against prior sampling-based attention, particularly MagicPIG, which already uses inverse sampling-probability correction implemented through a negative log-probability shift of attention logits. The potentially distinctive contribution here is therefore the fixed-budget block sampling design and its integration with local query clustering and GPU execution. However, the evidence for these aspects remains incomplete. Section 4.3 argues that selection block size drops out of kernel cost, but the block-size sweep reports only quality, not latency; equal tensor-core work does not establish equal gather and indexing costs. In addition, the evaluation specifies a single sampling seed and a fixed four-step reuse interval, without characterizing variability or reuse sensitivity. These omissions make it difficult to assess the robustness and practical cost of the proposed design beyond the reported configuration. arxiv.org

Questions for authors

Q1. Can you compare AB-Attention with representative correction-based methods discussed in Section 2, such as Rectified SpaAttn, PISA, or SVG-EAR, under matched model, precision, hardware, and timing settings? Quality–latency curves, or comparisons at matched quality and matched latency, would materially strengthen the assessment. If a comparison is infeasible, please explain the specific implementation or availability constraint.

Q2. How should SIP be distinguished from the self-normalized, inverse-probability-weighted attention estimator used in MagicPIG, particularly its Section 4 and Algorithm 1? Please clarify which contributions are specific to the fixed-size systematic block sampling design and its integration with video attention, as opposed to the weighting identity itself.

Q3. Does the claimed independence from selection block size hold for measured latency, or only for tensor-core arithmetic? Please provide a latency sweep over block sizes and a breakdown of clustering/scoring, inclusion-probability computation and sampling, and gathered sparse attention. Plan-storage or peak-memory measurements would also help assess the implementation trade-offs.

Q4. How sensitive are the results to the sampling seed and the reuse interval CC? Please report repeated sampling runs with fixed generation noise and a small reuse-interval sweep including C=1C=1. Since later activations may depend on the reused sampled set, please also clarify the scope of the fixed-activation unbiasedness argument under plan reuse. Separating reuse of proposal probabilities from reuse of the entire sampled set would be particularly informative. Finally, please specify how the attention-error results in Figure 2 are aggregated across layers, heads, denoising steps, and random draws.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hqKX (1/2)

Q1 & W1: Can you compare AB-Attention with representative correction-based methods discussed in Section 2, such as Rectified SpaAttn, PISA, or SVG-EAR, under matched model, precision, hardware, and timing settings? Quality–latency curves, or comparisons at matched quality and matched latency, would materially strengthen the assessment. If a comparison is infeasible, please explain the specific implementation or availability constraint.

Thanks for this valuable suggestion. We have added 100-prompt comparisons against SVG-EAR [1] and PISA [2] under matched Wan2.1-T2V-14B and H20 settings, together with AB-Attention's quality–latency curve; AB-Attention achieves higher fidelity and lower latency than both correction-based baselines.

  • Comparison with correction-based baselines: We evaluate both baselines with their official kernels on 100 Wan2.1-T2V-14B prompts:

    MethodSettingSparsity ↑PSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓
    AB-AttentionP = 64 (default)89.16%29.7050.91540.09562572
    SVG-EARtop-p = 0.40, Q100/K50087.54%28.4480.89180.12632667
    PISAdensity = 0.16, covariance routing84.00%23.8160.81450.22632709

    AB-Attention outperforms SVG-EAR by +1.26 dB PSNR and PISA by +5.89 dB while operating at higher sparsity and lower latency.

  • Quality–latency curve: Sweeping AB-Attention's budget (P = 64 → 192; P = 64 default) on the same 100 prompts traces a smooth quality–latency frontier:

    AB-Attention budgetSparsity ↑PSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓Speedup ↑
    P = 64 (default)89.16%29.7050.91540.095625721.84×
    P = 9683.75%30.7710.92920.075927261.74×
    P = 12878.33%31.7320.94050.063728771.65×
    P = 16072.91%32.3630.94580.057030341.56×
    P = 19267.49%33.0290.95070.051431901.49×

    Near the baselines' 2667–2709 s latency (P = 96, 2726 s), AB-Attention reaches 30.771 dB PSNR, widening its lead to +2.32 dB over SVG-EAR and +6.96 dB over PISA.

We have added these comparisons and quality–latency curves to Appendix E; due to rebuttal time and compute constraints, we will include Rectified SpaAttn [3] in the final version.

References:
[1] Zhou et al. SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing, ECCV 2026.
[2] Li et al. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers, NeurIPS 2026.
[3] Liu et al. Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation, arXiv 2025.

Q2 & W2: How should SIP be distinguished from the self-normalized, inverse-probability-weighted attention estimator used in MagicPIG, particularly its Section 4 and Algorithm 1? Please clarify which contributions are specific to the fixed-size systematic block sampling design and its integration with video attention, as opposed to the weighting identity itself.

Thanks. While both methods use the self-normalized inverse-probability weighting identity via the −log⁡π-\log \pi logit shift (now cited in Section 4.2 and contrasted with MagicPIG [1] in Appendix A.3), that identity only specifies how to reweight a given sample. MagicPIG and AB-Attention differ fundamentally in what is sampled, how the proposal and sample are constructed, and how the sample is executed:

DimensionMagicPIG [1] (Sec. 4, Alg. 1)AB-Attention (Sec. 4.1–4.3, Alg. 1)
Sampling unit & sharingPer-query independent sampling of individual tokens (scattered per-query token sets)Query-block shared sampling of fine-grained KV blocks (one sampled block set per query block)
Proposal distributionAngular SimHash collision probability (strictly below 1) from pre-built LSH tables, agnostic to vector norms and softmax temperatureOnline intra-block query-cluster softmax mixture over pooled keys without token permutation (IBCS)
Peak retention & budget controlIndependent Bernoulli collisions yielding a random, data-dependent sample size (requires a separate static sink/local window since collision probabilities stay below 1)Capped score-proportional inclusion probabilities (Eq. 4) via systematic sampling without replacement (SIP): unifies deterministic probability-1 peaks (zero variance) with a duplicate-free weighted tail at an exact block budget
Attention executionCoupled token-level selection and CPU gatherDecoupled fine-block selection and coarse-tile GPU execution: the fixed block budget packs non-contiguous blocks on-chip into full tiles and enables cross-step plan reuse

Reference:
[1] Chen et al. MagicPIG: LSH Sampling for Efficient LLM Generation, ICLR 2025.

AuthorsOpenReview ↗

Response to Reviewer hqKX (2/2)

Q3 & W2: Does the claimed independence from selection block size hold for measured latency, or only for tensor-core arithmetic? Please provide a latency sweep over block sizes and a breakdown of clustering/scoring, inclusion-probability computation and sampling, and gathered sparse attention. Plan-storage or peak-memory measurements would also help assess the implementation trade-offs.

Thanks. Independence holds for the gathered tensor-core pass, while measured end-to-end latency varies by only 2.2% across block sizes 4 → 128 (2572 s vs. 2516 s, gaining +1.02 dB PSNR), with no extra peak GPU memory over Dense (both 65.95 GiB). We have clarified this tensor-core scope in Section 4.3. Stage timings, latency, and memory on Wan2.1-T2V-14B (b = 4 default):

SettingCluster/score ↓Sample/pack ↓Gathered attn ↓Latency ↓Plan/layer ↓Peak mem ↓
Dense (baseline)———4744 s—65.95 GiB
b = 4 (default)58.8 ms42.4 ms96.9 ms2572 s369.4 MiB65.95 GiB
b = 851.5 ms21.9 ms100.9 ms2539 s184.7 MiB65.95 GiB
b = 1651.4 ms11.8 ms95.0 ms2527 s92.3 MiB65.95 GiB
b = 3251.3 ms6.5 ms95.4 ms2524 s46.2 MiB65.95 GiB
b = 6451.3 ms4.2 ms94.6 ms2516 s23.1 MiB65.95 GiB
b = 128 (tile-level)52.2 ms3.0 ms96.8 ms2516 s11.5 MiB65.95 GiB

Gathered attention remains flat (94.6–100.9 ms) across a 32× block-size range while only sampling scales (3.0 → 42.4 ms), running once every C=4C = 4 sparse steps. Inactive per-layer plans (11.5–369.4 MiB) are cached in host memory and transferred asynchronously one layer at a time, adding 0 GiB to Dense's 65.95 GiB peak GPU memory across all block sizes. We have added this breakdown to Appendix D.

Q4.1 & W2: How sensitive are the results to the sampling seed and the reuse interval CC? Please report repeated sampling runs with fixed generation noise and a small reuse-interval sweep including C=1C=1.

Thanks. Results are insensitive to both the sampling seed and moderate reuse intervals on 100 Wan2.1-T2V-14B prompts with fixed generation noise (added to Appendix B.1).

  • Sampling-seed sensitivity: Across five sampling seeds (40–44; 42 default) at C = 4, mean PSNR is 29.713 ± 0.009 dB (range 0.023 dB):

    Sampling seed (C = 4)PSNR ↑SSIM ↑LPIPS ↓
    4029.7280.91550.0949
    4129.7100.91440.0957
    42 (default)29.7050.91540.0956
    4329.7070.91490.0954
    4429.7160.91410.0963
    Mean ± Std29.713 ± 0.0090.9149 ± 0.00060.0956 ± 0.0005

    In SIP, dominant blocks reach πij=1\pi_{ij} = 1 deterministically with zero variance, while systematic sampling with 1/πij1/\pi_{ij} weighting gives a low-variance tail estimate; independent draws across query blocks, heads, and layers further reduce variance.

  • Reuse-interval sweep: Across C = 1 → 8 (C = 4 default), PSNR moves smoothly from 30.337 to 29.439 dB; rebuilding every step at C = 1 adds 5.9% latency for +0.63 dB over C = 4:

    Reuse interval CPSNR ↑SSIM ↑LPIPS ↓Latency (s) ↓
    C = 130.3370.92220.08782724
    C = 229.9910.91900.09172622
    C = 4 (default)29.7050.91540.09562572
    C = 829.4390.91160.10052546

Q4.2: Since later activations may depend on the reused sampled set, please also clarify the scope of the fixed-activation unbiasedness argument under plan reuse.

The fixed-activation unbiasedness guarantee (Appendix A.3) holds only at plan-refresh steps (and at every step when C = 1), where a fresh sample is drawn given current activations; it does not hold on reuse steps (C > 1) because later activations depend on the reused sampled set. We have clarified this in Section 4.3 and Appendix A.3.

Q4.3: Separating reuse of proposal probabilities from reuse of the entire sampled set would be particularly informative.

Plan reuse in AB-Attention reuses the entire sampled set (selected block indices and their log-weights), rather than reusing proposal probabilities to re-sample at each step. Directly reusing the sampled set avoids caching full proposal probabilities and per-step sampling, keeping storage and runtime overhead minimal; we have clarified this in Section 4.3.

Q4.4: Finally, please specify how the attention-error results in Figure 2 are aggregated across layers, heads, denoising steps, and random draws.

Figure 2 reports the relative output L2L_2 error on Wan2.1-T2V-14B (step 19, layer 20) across 10 prompts: for each prompt, errors are averaged over 4 attention heads (0, 10, 20, 30) and 4 random draws, and plotted as mean ± std across prompts. We have added these details to the Figure 2 caption.

Thank you for the additional experiments and clarifications. Please retain the precise scope of these conclusions in the revision. The unbiasedness statement applies to the numerator and normalizer with fresh sampling at fixed activations; it does not make their ratio unbiased or extend unchanged to reused plans. The block-size independence concerns tensor-core work, with measurable planning costs and host-memory storage. The added comparisons support an advantage at the tested operating points, without establishing every baseline's full quality-latency frontier. I will maintain my rating of 3, with stronger support after the rebuttal. These remaining points concern how the results are stated and do not change my positive recommendation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to ACs and All Reviewers

Dear ACs and Reviewers,

We sincerely appreciate the effort of the ACs and reviewers on our submission. We are encouraged that all three reviewers found the submission informative, and that Reviewers hqKX and fGQJ both rated its presentation "clear and easy to follow". Reviewer hqKX highlights that our design is "practically relevant for training-free acceleration of pretrained video diffusion models", Reviewer fGQJ recognizes the fixed-budget ablations as demonstrating that "each component contributes to the result", and Reviewer TDP9 finds that the proposed method is "simultaneously the sparsest, the most faithful to dense attention, and the fastest in every comparison". We thank the reviewers for this assessment and their constructive suggestions. We have carefully responded to each of the reviewers' questions and incorporated the corresponding clarifications and experimental results into the revised manuscript (highlighted in red).

Best regards,

Authors of submission-38168

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.