acceptodds
Withdrawn from ICLR 2027

DraftAttention2: Fast Video Diffusion with Low-Resolution-Guided Mixed-Precision Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video generation has broad applications in content creation and entertainment. Diffusion transformers have advanced the quality of generated videos, but attention over spatiotemporal tokens becomes increasingly expensive as video resolution and duration increase. We present DraftAttention2, a training-free framework that uses the low-resolution draft attention map to jointly select attention blocks and assign their numerical precision. Specifically, spatial 2D average- and max-pooled queries and keys capture complementary regional statistics to estimate block importance, and a shared ranking assigns higher precision to important blocks, lower precision to less important retained blocks, and skips the rest under configurable budgets. Our analysis separates sparsification error from attention-weighted quantization error, establishing when recovering skipped interactions with low-bit computation tightens the output-error bound. This analysis motivates retaining more interactions at low precision while reserving higher precision for blocks with larger attention mass. To translate these fine-grained assignments into practical speedups, we further develop fused operand preparation and a single attention kernel with precision-specific phases, sharing data movement, softmax statistics, and output accumulation across precisions. Experiments demonstrate that our method achieves a superior quality–efficiency trade-off over existing efficient video generation methods. Notably, its advantage is particularly pronounced for few-step video diffusion, where jointly combining sparsity with 4- and 8-bit mixed-precision computation substantially improves generation quality while retaining significant acceleration.

Primary area:
generative models
Keywords:
video diffusion, diffusion transformers, sparse attention, mixed-precision attention, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dxdzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the cost of attention over spatiotemporal tokens in video diffusion transformers. It presents DraftAttention2, a training free framework that builds a low resolution draft attention map from average and max pooled queries and keys, and uses one shared ranking to jointly select attention blocks and assign their precision at 16, 8, or 4 bits or skip them, supported by an error decomposition analysis and a fused mixed precision kernel. Experiments on HunyuanVideo and MiniMax-H3, including few step variants, report up to 2.6 times DiT speedups with competitive VBench scores and better fidelity to dense outputs than sparse and low bit baselines.

Critical strengths
  1. The paper generalizes binary block selection into a joint allocation of sparsity and numerical precision, and the accompanying analysis gives this design principled support. The decomposition into sparsification and attention weighted quantization terms clarifies when low bit recovery is preferable to skipping, and justifies assigning higher precision to blocks with larger attention mass.
  2. The execution design translates fine grained precision assignments into measured speedups rather than theoretical savings. Fused operand preparation shares data movement between draft construction and quantization, and phased execution within a single kernel keeps precision transitions independent of the number of retained blocks.
  3. The empirical evaluation is broad and carefully controlled. Two model families with few step variants, seven baselines, matched speedup comparisons, and error versus latency ablations together make a convincing case that joint allocation improves the quality efficiency trade-off, particularly in the practical few step setting.
Critical weaknesses
  1. The quality evidence rests mainly on fidelity to dense outputs rather than standalone generation quality. On HunyuanVideo the strongest sparse baseline reaches a higher VBench average than all proposed configurations (76.81 versus at most 74.83), and in Table 3 adding max pooling slightly lowers the VBench average while improving similarity metrics, so the gains are clearer for dense output fidelity than for generation quality itself.
  2. The reported speedups are DiT level and hardware specific. Latency excludes text encoding and VAE decoding, dense attention is retained for the first 10 steps (first quarter in few step variants) without a sensitivity analysis, and all measurements use a single RTX 5090, so the 1.6 to 2.6 times figures do not translate directly to end to end application level acceleration.
  3. The theoretical results assume uniform error envelopes and exact attention masses, whereas the router uses draft scores as low cost proxies. The link between the error bounds and generation quality therefore remains empirical, and the per model precision quotas and pooling weight are tuned with limited guidance for transfer to new backbones.
Questions for authors
  1. The first 10 denoising steps (first quarter in few step variants) use dense attention. How sensitive are quality and speedup to the number of dense warm up steps, and can the framework operate with no dense steps at all?
  2. On HunyuanVideo all proposed configurations trail the strongest sparse baseline on VBench average despite better similarity metrics, and adding max pooling slightly lowers VBench in Table 3. How should this gap between standalone quality and dense output fidelity be interpreted, and is the VBench ranking stable across seeds?
  3. Theorem 3.6 assumes exact attention masses while routing relies on draft scores. Can any evidence quantify how closely the draft ranking tracks the exact mass ranking and how much the error bound degrades under this proxy?
2 comments
AuthorsOpenReview ↗

Response to Reviewer dxdz (1/2)

We thank the reviewer for recognizing the joint allocation, theoretical analysis, and practical few-step gains. We address the quality assessment, warm-up protocol, application latency, and draft approximation below.

C1. Generation quality, dense fidelity, and seed stability (W1, Q2)

How should this gap between standalone quality and dense output fidelity be interpreted, and is the VBench ranking stable across seeds?

Our 4/8-bit configurations improve PSNR at higher DiT speedups than Sol-Attn across all four model settings. Tables 1 and 2 also show improvements in SSIM and LPIPS on HunyuanVideo and the two few-step models. On 4-step and 8-step MiniMax-H3, PSNR increases by 0.74 and 0.64 dB, while DiT speedups increase from 1.46×1.46\times/1.80×1.80\times to 1.84×1.84\times/2.12×2.12\times. The corresponding VBench averages are 74.00/79.89 versus 74.19/79.91.

At speedups matched to SpargeAttention, DraftAttention2 achieves higher PSNR and VBench averages in all four settings. On HunyuanVideo, for example, DraftAttention2 reaches 75.96 VBench at 2.51×2.51\times, compared with 74.36 at 2.50×2.50\times for SpargeAttention. The benefit of joint allocation is therefore supported by both fidelity and generation-quality measurements at comparable speed.

Evaluating fidelity alongside VBench follows established practice in video acceleration, including Sparse VideoGen [1, §5]. PSNR/SSIM/LPIPS measure preservation of the original model's output, while VBench [2] evaluates video-quality attributes. VMBench [3, §4.2] finds that several VBench motion metrics have limited alignment with human motion-quality judgments. A VBench average should therefore be considered alongside fidelity when assessing an attention approximation.

MaxPool helps the draft identify important attention contributions. In Table 3, adding MaxPool improves PSNR, SSIM, and LPIPS in all three precision settings, directly supporting this design choice.

All original quality evaluations use the same 326 prompts and a fixed seed of 0 for every method, without seed selection. Recent constraints on GPU availability have prevented us from completing the multi-seed evaluation within the rebuttal period.

C2. Sensitivity to dense warm-up steps (W2, Q1)

How sensitive are quality and speedup to the number of dense warm up steps, and can the framework operate with no dense steps at all?

Dense warm-up follows the evaluation practice of prior sparse video-attention work. Sparse VideoGen [1, §5.1] retains dense computation for the first 25% of denoising steps, while AdaSpa [4] retains the first 10 steps for all methods. We adopt this convention for consistent comparison with existing approaches.

DraftAttention2 constructs its draft from the current queries and keys at each attention call. It does not require calibration statistics from earlier dense steps and can be enabled from the first denoising step. Recent constraints on GPU availability have prevented us from completing the quality evaluation without dense warm-up within the rebuttal period.

References

[1] Haocheng Xi et al. Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. ICML, 2025.

[2] Ziqi Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR, 2024.

[3] Xinran Ling et al. VMBench: A Benchmark for Perception-Aligned Video Motion Generation. ICCV, 2025.

[4] Yifei Xia et al. Training-free and Adaptive Sparse Attention for Efficient Long Video Generation. ICCV, 2025.

AuthorsOpenReview ↗

Response to Reviewer dxdz (2/2)

C3. Speedup including text encoding and VAE decoding (W2)

Latency excludes text encoding and VAE decoding

The acceleration remains substantial after including text encoding and VAE decoding. DraftAttention2 reaches E2E speedups of 2.45×2.45\times, 2.56×2.56\times, 1.70×1.70\times, and 2.01×2.01\times on HunyuanVideo, MiniMax-H3, and its 4/8-step variants, respectively, compared with 2.13×2.13\times, 2.02×2.02\times, 1.39×1.39\times, and 1.73×1.73\times for Sol-Attn. Sparsity denotes the fraction of skipped blocks; precision shares are measured among retained blocks.

HunyuanVideo

MethodSparsity (%)4/8/16-bit mixDiT speedup ↑\uparrowE2E speedup ↑\uparrow
Dense0.00%/0%/100%1.00×1.00\times1.00×1.00\times
Sol-Attn85.80%/0%/100%2.16×2.16\times2.13×2.13\times
DraftAttention280.775%/25%/0%2.51×2.51\times2.45×2.45\times

MiniMax-H3

MethodSparsity (%)4/8/16-bit mixDiT speedup ↑\uparrowE2E speedup ↑\uparrow
Dense0.00%/0%/100%1.00×1.00\times1.00×1.00\times
Sol-Attn80.00%/0%/100%2.04×2.04\times2.02×2.02\times
DraftAttention274.375%/25%/0%2.60×2.60\times2.56×2.56\times

4-step MiniMax-H3

MethodSparsity (%)4/8/16-bit mixDiT speedup ↑\uparrowE2E speedup ↑\uparrow
Dense0.00%/0%/100%1.00×1.00\times1.00×1.00\times
Sol-Attn65.00%/0%/100%1.46×1.46\times1.39×1.39\times
DraftAttention254.575%/25%/0%1.84×1.84\times1.70×1.70\times

8-step MiniMax-H3

MethodSparsity (%)4/8/16-bit mixDiT speedup ↑\uparrowE2E speedup ↑\uparrow
Dense0.00%/0%/100%1.00×1.00\times1.00×1.00\times
Sol-Attn79.90%/0%/100%1.80×1.80\times1.73×1.73\times
DraftAttention274.050%/50%/0%2.12×2.12\times2.01×2.01\times

C4. Draft approximation, the error bound, and configuration transfer (W3, Q3)

Can any evidence quantify how closely the draft ranking tracks the exact mass ranking and how much the error bound degrades under this proxy?

The draft retains 87.59–97.05% of the high-precision attention mass captured by the exact-mass oracle. We evaluate 4-step MiniMax-H3 at retained fractions of 35.05% and 64.95%, covering all query regions and all 56 heads. At each budget, the support and 75%/25% NVFP4/INT8 quotas are fixed across allocation rules. For comparison, uniform assignment with the same quota has an expected mass capture of 27.81–30.93%.

This measurement connects directly to Theorem 3.6. Let G8G_8 be the oracle-minus-draft difference in exact retained mass assigned to INT8. The difference in the theorem's assignment-dependent quantization term is ΔJ=(ε4−ε8)G8.\Delta J=(\varepsilon_4-\varepsilon_8)G_8.

The measured head-averaged G8G_8 is 0.02546–0.09685, quantifying the effect of draft allocation in the same quantities as the bound. Ranking by exact pre-pruning masses captures 99.800–99.987% of the renormalized-mass oracle, showing that the normalization change has little effect on mass capture in this comparison.

the per model precision quotas and pooling weight are tuned with limited guidance for transfer to new backbones.

The pooling weight and retained precision ratios are shared across model settings. We use λ=0.2\lambda=0.2 throughout and fixed 75%/25% or 50%/50% NVFP4/INT8 mixtures. Sparsity is adjusted to match the comparison latency, after which the configuration is fixed for quality evaluation. The percentages in Tables 1 and 2 of the submission refer to the whole map: for a fixed 75%/25% mixture, the 4/8-bit fractions are 0.75(1−s)0.75(1-s) and 0.25(1−s)0.25(1-s) at sparsity ss. Different reported percentages therefore reflect different density budgets, rather than independently selected precision ratios for each backbone.

Summary

The 4/8-bit results demonstrate higher PSNR at higher speedups than Sol-Attn, and higher PSNR and VBench scores at speedups matched to SpargeAttention, across all four model settings. The additional measurements establish low-cost draft allocation close to the mass oracle and substantial application-level speedup after including the surrounding generation stages.

Reviewer_wyASOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DraftAttention2 is a training-free method for accelerating video diffusion through block sparsity and mixed-precision attention. It combines drafts from average- and max-pooled queries and keys to rank regional interactions and assign them to 16-bit, INT8, NVFP4, or skipped paths under configurable quotas. The paper analyzes sparsification and quantization errors and implements the allocation through fused operand preparation and a single attention kernel with precision-specific phases. Experiments on HunyuanVideo and MiniMax-H3, including few-step variants, show favorable speed–fidelity trade-offs at several operating points.

Critical strengths
  1. The engineering contribution is valuable. The fused implementation addresses the overhead of mixing precisions, helping translate cheaper attention computation into practical inference speedups.
Critical weaknesses
  1. The practical advantage over prior sparse attention works, such as Sol-Attn, remains insufficiently demonstrated.

  2. The contribution of the specific allocation rule is insufficiently demonstrated. The experiments do not isolate whether sharing budgets across query regions improves on simpler per-query allocation.

Please refer to the comments for more details.

Questions for authors

Thanks for the submission to ICLR. Below are my comments/quesitons.

  1. Theorem 3.6 uses attention masses renormalized over the retained support, whereas Equation (2) ranks a draft computed before pruning. These orderings need not coincide when retained mass differs across query regions. Remark 3.2 also establishes mean-logit preservation rather than attention-mass preservation. An empirical comparison between the draft ranking and the importance measure used in the theorem, or an oracle precision assignment at fixed support and quotas, would help clarify how well the proposed ranking supports both block selection and precision allocation.

  2. Table 3 examines MaxPool, and Figure 5 examines precision mixtures, but neither directly compares head-wide allocation with assigning a fixed budget to each query region. Since sharing the budget across query regions is an important feature of the proposed method, it would be helpful to isolate its contribution. A comparison using the same pooled scores, total precision quotas, and execution backend would clarify whether this flexibility improves the quality–latency trade-off over the simpler per-query alternative.

  3. Tables 1–2 report several sparsity and precision allocations. It would be helpful to briefly describe how these configurations were selected empirically and whether the same selection procedure was used across the evaluated models. This would make the reported operating points easier to reproduce.

  4. My major concern is that, across Tables 1–2, the proposed design does not show significant or meaningful improvement over prior work Sol-Attn. On HunyuanVideo, for example, the proposed design just achieves a slightly higher speedup (2.38× versus 2.16×) with a lower VBench score (74.70 versus 76.81). This demonstrates a trade-off, while leaving the practical advantage over Sol-Attn unresolved.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wyAS (1/3)

We thank the reviewer for recognizing the engineering contribution. We first highlight the practical gains of 4/8-bit attention, then address the allocation and configuration questions.

C1. Practical advantage over Sol-Attn (W1, Q4)

The practical advantage over prior sparse attention works, such as Sol-Attn, remains insufficiently demonstrated.

Our 4/8-bit configurations achieve higher PSNR at higher DiT speedups than Sol-Attn across all four model settings. These results illustrate the central finding of our work: spending the compute budget on greater interaction coverage at lower precision improves output fidelity. The HunyuanVideo point highlighted in the review uses 8/16-bit attention; the tables below show the 4/8-bit results from Tables 1 and 2, alongside Sol-Attn and SpargeAttention. Sparsity denotes the fraction of skipped blocks; the 4/8/16-bit mix is measured among retained blocks.

HunyuanVideo

MethodSparsity (%)4/8/16-bit mixVBench Avg ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDiT speedup ↑\uparrow
Sol-Attn85.80%/0%/100%76.8125.210.8050.1782.16×2.16\times
SpargeAttn85.00%/100%/0%74.3626.020.8240.1592.50×2.50\times
DraftAttention280.775%/25%/0%75.9626.370.8140.1572.51×2.51\times

MiniMax-H3

MethodSparsity (%)4/8/16-bit mixVBench Avg ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDiT speedup ↑\uparrow
Sol-Attn80.00%/0%/100%78.2520.950.6910.1982.04×2.04\times
SpargeAttn80.00%/100%/0%77.4720.230.6620.2322.59×2.59\times
DraftAttention274.375%/25%/0%77.7421.020.6900.2052.60×2.60\times

4-step MiniMax-H3

MethodSparsity (%)4/8/16-bit mixVBench Avg ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDiT speedup ↑\uparrow
Sol-Attn65.00%/0%/100%74.1919.540.6640.1911.46×1.46\times
SpargeAttn65.00%/100%/0%73.7419.260.6410.2061.86×1.86\times
DraftAttention254.575%/25%/0%74.0020.280.6870.1791.84×1.84\times

8-step MiniMax-H3

MethodSparsity (%)4/8/16-bit mixVBench Avg ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDiT speedup ↑\uparrow
Sol-Attn79.90%/0%/100%79.9118.190.5750.2621.80×1.80\times
SpargeAttn80.00%/100%/0%79.7018.150.5660.2772.12×2.12\times
DraftAttention274.050%/50%/0%79.8918.830.5990.2472.12×2.12\times

Compared with Sol-Attn, DraftAttention2 gains 0.74/0.64 dB PSNR at higher speedups on 4-step/8-step MiniMax-H3, with comparable VBench averages. At speedups comparable to SpargeAttention, DraftAttention2 improves PSNR by 0.35–1.02 dB across all four settings.

Evaluating both fidelity and VBench follows established practice in video acceleration; for example, Sparse VideoGen [1, §5] evaluates PSNR/SSIM/LPIPS alongside VBench. Preserving the original model's output is a central objective when replacing its attention computation.

References

[1] Haocheng Xi et al. Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. ICML, 2025.

AuthorsOpenReview ↗

Response to Reviewer wyAS (2/3)

C2. Draft ranking and the attention masses in the theorem (Q1)

An empirical comparison between the draft ranking and the importance measure used in the theorem, or an oracle precision assignment at fixed support and quotas, would help clarify how well the proposed ranking supports both block selection and precision allocation.

The draft captures 87.59–97.05% of the oracle's high-precision attention mass. We construct the requested oracle using exact retained, renormalized attention masses WabW_{ab}, with support and integer precision quotas fixed. We evaluate 4-step MiniMax-H3 at retained fractions of 35.05% and 64.95%, covering all query regions and all 56 heads. INT8 receives 25% of retained blocks and NVFP4 receives the remainder.

Precision allocationINT8 attention mass relative to the oracle ↑\uparrow
Uniform assignment (expected)27.81–30.93%
Draft ranking87.59–97.05%
Exact pre-pruning mass ranking99.800–99.987%

The last row directly tests the normalization concern: ranking by exact pre-pruning mass recovers nearly all the mass assigned by the renormalized-mass oracle. The draft, computed at low resolution, retains most of this allocation benefit at low cost.

The effect on the theorem's quantization term can also be measured directly. For fixed support and 4/8-bit quotas, let G8G_8 denote the oracle-minus-draft difference in exact mass assigned to INT8. Then

=(\varepsilon_4-\varepsilon_8)G_8, \qquad J(p)=\sum_{(a,b)\\ \mathrm{retained}}W_{ab}\varepsilon_{p_{ab}}.$$ This relates the measured mass capture to the assignment-dependent bound in Theorem 3.6. ### C3. Head-wide versus per-query allocation (W2, Q2) > A comparison using the same pooled scores, total precision quotas, and execution backend would clarify whether this flexibility improves the quality–latency trade-off over the simpler per-query alternative. **Sharing the budget across query regions increases retained attention mass.** We compare head-wide and per-query allocation in attention-operator experiments on HunyuanVideo and MiniMax-H3, using identical pooled scores, total retained-block counts, precision quotas, and execution backend. Both use a 75%/25% NVFP4/INT8 mixture among retained blocks. Across five sparsity levels, head-wide allocation consistently captures more original dense attention mass, with gains of 0.029–0.589 percentage points. The table reports the mean increase in retained attention mass over per-query allocation for each model, in percentage points. | Sparsity (%) | HunyuanVideo $\uparrow$ | MiniMax-H3 $\uparrow$ | | --- | --- | --- | | 50 | +0.213 | +0.145 | | 65 | +0.234 | +0.175 | | 80 | +0.227 | +0.188 | | 90 | +0.189 | +0.312 | | 95 | +0.216 | +0.311 | Budget sharing allows a query region to retain additional high-score blocks beyond a fixed per-query quota. The consistent increase in captured mass demonstrates this allocation benefit. **The routing cost is small.** At 13.4%/13.1% sparsity for HunyuanVideo/MiniMax-H3, respectively, with the same 75%/25% precision mixture, selection and route packing take 1.44–1.49 ms and 2.68–2.73 ms, accounting for 0.59–0.66% of the complete attention operator.
AuthorsOpenReview ↗

Response to Reviewer wyAS (3/3)

C4. Configuration selection and the role of 16-bit computation (Q3)

It would be helpful to briefly describe how these configurations were selected empirically and whether the same selection procedure was used across the evaluated models.

We use a common configuration-selection procedure across all four model settings. The 4/8-bit mixtures use fixed 75%/25% or 50%/50% ratios among retained blocks, and all four settings share λ=0.2\lambda=0.2. For each mixture, sparsity is adjusted to match the comparison latency; the resulting configuration is fixed for quality evaluation.

Tables 1 and 2 of the submission report percentages of the entire attention map. For example, a 75%/25% NVFP4/INT8 mixture at sparsity ss has 0/4/8/16-bit fractions ss, 0.75(1−s)0.75(1-s), 0.25(1−s)0.25(1-s), and zero. The differing table percentages therefore arise from latency-matched sparsity levels, while the retained precision ratio stays the same.

The operator experiments provide a budget-allocation principle: use lower precision to recover more interactions, and increase precision as density grows. Compared with FP16 attention at 80.3% sparsity, the fixed 75%/25% mixture reduces relative L2 output error by 44.1–64.2% at comparable complete-attention latency in operator experiments on HunyuanVideo and MiniMax-H3. As sparsification error decreases, higher precision becomes more valuable: at fixed support, promoting 5% of interactions from INT8 to 16-bit reduces error by 53.88–79.01% at full support, compared with 0.38–2.69% at 65% sparsity. This explains both the advantage of 4/8-bit mixtures and the role of 16-bit computation in further recovering accuracy.

Summary

The 4/8-bit results establish a clear gain in fidelity at higher speedups. The oracle experiment supports the draft's precision ranking, and the operator experiments explain how density and precision should share the compute budget.

Reviewer_gkgnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DraftAttention2, a training-free attention acceleration method for video diffusion. It uses low-resolution draft attention maps constructed from average- and max-pooled queries and keys to jointly rank attention blocks and assign them to skipped, 4-bit, 8-bit, or 16-bit computation. The core idea is that at similar latency, retaining more moderately important interactions at low precision is preferable to skipping them entirely. The authors support this idea through output-error bound analysis, fused kernels, and video generation experiments.

Critical strengths
  1. The paper analyzes sparsification error and quantization error within a unified framework.

  2. DraftAttention2 fuses reordering, pooling, and quantization, while sharing softmax statistics and output accumulation across precisions

  3. The method demonstrates clear improvements in fidelity to dense-attention outputs.

Critical weaknesses
  1. Combining sparsity with quantization and using pooled guidance for mixed-precision attention both have precedents.
  2. The main contributions are unified allocation and efficient execution, but lack of direct comparisons with the closest methods, such as BMAttn and ThriftAttention.
Questions for authors
  1. Could the authors compare with BMAttn and ThriftAttention on the same hardware at matched latency?
  2. Could the authors provide a latency breakdown of the kernel optimizations and report routing and operand-preparation computational overhead?
2 comments
AuthorsOpenReview ↗

Response to Reviewer gkgn (1/2)

We thank the reviewer for recognizing our unified error analysis, fused implementation, and improvements in fidelity. We address the distinction from prior work and provide the requested comparisons below.

C1. Novelty over existing approaches (W1)

Combining sparsity with quantization and using pooled guidance for mixed-precision attention both have precedents.

Our experiments establish a practical principle for sparse attention: lowering precision to retain more interactions improves fidelity at comparable latency. This changes how the compute budget is spent: recovering discarded interactions can be more valuable than computing a smaller set at high precision.

Tables 1 and 2 of the original submission show this benefit across HunyuanVideo, MiniMax-H3, and its 4-step and 8-step variants. Compared with 16-bit Sol-Attn, our 4/8-bit configurations retain more interactions and achieve higher PSNR at higher DiT speedups in all four settings. Sparsity denotes the fraction of skipped blocks; the 4/8/16-bit mix is measured among retained blocks.

HunyuanVideo

MethodSparsity (%)4/8/16-bit mixVBench Avg ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDiT speedup ↑\uparrow
Sol-Attn85.80%/0%/100%76.8125.210.8050.1782.16×2.16\times
DraftAttention280.775%/25%/0%75.9626.370.8140.1572.51×2.51\times

MiniMax-H3

MethodSparsity (%)4/8/16-bit mixVBench Avg ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDiT speedup ↑\uparrow
Sol-Attn80.00%/0%/100%78.2520.950.6910.1982.04×2.04\times
DraftAttention274.375%/25%/0%77.7421.020.6900.2052.60×2.60\times

4-step MiniMax-H3

MethodSparsity (%)4/8/16-bit mixVBench Avg ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDiT speedup ↑\uparrow
Sol-Attn65.00%/0%/100%74.1919.540.6640.1911.46×1.46\times
DraftAttention254.575%/25%/0%74.0020.280.6870.1791.84×1.84\times

8-step MiniMax-H3

MethodSparsity (%)4/8/16-bit mixVBench Avg ↑\uparrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowDiT speedup ↑\uparrow
Sol-Attn79.90%/0%/100%79.9118.190.5750.2621.80×1.80\times
DraftAttention274.050%/50%/0%79.8918.830.5990.2472.12×2.12\times

In attention-operator experiments on HunyuanVideo and MiniMax-H3, a fixed 75%/25% NVFP4/INT8 mixture reduces relative L2 output error by 44.1–64.2% compared with FP16 attention at 80.3% sparsity.

We adjust sparsity for each precision scheme to match the latency budget. The following tables report mean complete-attention latency and mean relative L2 output error. The 75%/25% ratio is fixed among retained blocks.

HunyuanVideo

PrecisionSparsity (%)Latency (ms) ↓\downarrowRelative L2 error (%) ↓\downarrow
FP1680.3142.1613.30
INT860.5142.657.26
4/8-bit (75%/25%)50.0144.165.50

MiniMax-H3

PrecisionSparsity (%)Latency (ms) ↓\downarrowRelative L2 error (%) ↓\downarrow
FP1680.3295.0111.17
INT860.5296.267.06
4/8-bit (75%/25%)50.0300.736.13

Lower arithmetic precision thus produces a more accurate attention output by enabling greater coverage. Our error decomposition explains when recovering skipped interactions outweighs the added quantization error. DraftAttention2 turns this insight into a joint allocation over 0/4/8/16 bits, using a single online draft to decide which interactions to retain and at what precision.

This differs from BMAttn's offline-calibrated affine boundaries and ThriftAttention's allocation between FP4 and FP16 over unmasked blocks. DraftAttention2 makes spatial support and numerical precision part of the same decision. Its execution is designed around this allocation: fused reordering, pooling, and quantization share operand preparation, while precision-specific phases share softmax statistics and output accumulation within one kernel. The comparisons below establish the benefits of both the allocation and its execution.

AuthorsOpenReview ↗

Response to Reviewer gkgn (2/2)

C2. Comparison with BMAttn and ThriftAttention (W2, Q1)

Could the authors compare with BMAttn and ThriftAttention on the same hardware at matched latency?

DraftAttention2 reduces relative L2 output error by 62.7% compared with ThriftAttention at matched latency. We evaluate both native implementations on the same video-attention input and RTX 5090. Both retain all video interactions; ThriftAttention uses its default 5% high-precision allocation, and DraftAttention2 uses equal NVFP4/INT8 quotas. Latency includes operand preparation, selection, attention, and output processing.

MethodLatency (ms) ↓\downarrowRelative L2 output error (%) ↓\downarrow
ThriftAttention579.274.938
DraftAttention2574.741.840

The latencies differ by less than 1%. With sparsity removed from this comparison, the result directly demonstrates the advantage of our mixed-precision allocation and execution.

Recent constraints on GPU availability have prevented us from completing the video-quality evaluation with ThriftAttention within the rebuttal period.

BMAttn's public implementation [1] provides quantization/dequantization followed by floating-point attention; the native INT4/INT8 backend described in the paper is not released. We therefore provide the direct runtime comparison with ThriftAttention's released native implementation.

C3. Runtime breakdown and preparation overhead (Q2)

Could the authors provide a latency breakdown of the kernel optimizations and report routing and operand-preparation computational overhead?

Draft construction and routing take approximately 3.4 ms per attention call. The table reports mean latencies over 144 attention calls in MiniMax-H3 with a 75%/25% NVFP4/INT8 mixture. Each percentage is relative to the complete attention operator at the corresponding sparsity.

Stage13.12% sparsity (ms) ↓\downarrowShare (%)54.47% sparsity (ms) ↓\downarrowShare (%)
Operand preparation37.898.0137.9613.67
Draft construction and routing3.420.723.371.21
Attention computation429.5490.79234.0184.29
Output assembly2.220.472.220.80
Other and synchronization0.050.010.050.02
Complete attention operator473.12100.00277.61100.00

Draft construction and routing account for only 0.72–1.21% of attention runtime. Attention computation dominates at both sparsity levels.

Fusing reordering, pooling, and quantization reduces redundant memory accesses and kernel launches during operand preparation.

Summary

The empirical finding, error analysis, and efficient implementation together establish how exchanging precision for greater interaction coverage improves sparse attention. The ThriftAttention comparison further demonstrates the benefits of our precision allocation and its execution.

References

[1] BMAttn: Public implementation. GitHub repository.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.