- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a novel mixed-precision method for sparse attention. Motivated by the imbalanced importance among selected query–key interactions, the authors compute important interactions with higher precision and less important ones with lower precision, and use the saved compute to include additional interactions in the attention coverage. The method is supported by theoretical analysis and extensive experiments ranging from video diffusion to LLMs. The kernel implementation would also be insightful for future research.
Critical strengths
-
Strongly supported motivation and insight for future research. The idea is supported by experiments and theoretical analysis, not only by its effectiveness in diffusion and LLM applications. This work also guides future research on sparse attention, considering not just where to compute but also how precisely to compute.
-
Comprehensive experiments with broad applicability. The evaluation covers both video generation and long-context language understanding, multiple models, and various quality metrics, including quality–compute frontiers, context-length analysis, attention-output error, and kernel efficiency. The method is also applied to a wide range of existing sparse attention methods (Radial, SVG2, FlexPrefill, XAttention, SeerAttention). Because it reuses each method’s own importance signal without a new selection mechanism, it can be easily plugged into existing approaches.
Critical weaknesses
-
Additional controlled experiments would strengthen the motivation. The paper claims that, under the same compute budget, converting the excess precision of less important interactions into broader coverage improves quality. However, it is somewhat unclear whether this gain comes from importance-aware precision allocation or mainly from the expanded coverage. It would be helpful to see results with a reversed allocation (high precision for low-attention interactions), with lower precision applied only to less important interactions without expanding coverage, and with only the few most important interactions kept. Such experiments would clarify how much attention importance actually contributes to the performance gain.
-
Insufficient method details and comparisons. The paper describes the underlying sparse attention mechanisms only briefly through their control parameters, and a more detailed explanation would be helpful. The precision allocation is also somewhat heuristic, without ablation on its design choices. In addition, Table 2 lacks a cost comparison to BF16 like the budget matching in Table 1, which makes it hard to compare efficiency.
Questions for authors
-
For XAttention, Figure 4(d) and Figure 7(a) show the smallest gain, yet Table 2 shows the largest improvement. Could this gap come from a weak sparse baseline at the chosen operating point?
-
Is the precision conversion done inside the kernel or before the inputs are fed into it? If the latter, is this additional cost included in the reported kernel time?
4 comments
Response to Reviewer Qu7X (Part 2/2)
Ablations on importance and coverage (Reviewer Qu7X Weakness 1)
We ran the three requested ablations in Weakness 1.
- Reversed precision: the same tiles and the same cost as mixed precision, but with the precision order flipped, so the least important tiles get BF16. This tests whether it matters which tiles get the high precision.
- No added coverage: only the tiles that the original sparse method selects. The less important ones are computed at lower precision, as in mixed precision, but the saved compute is not spent on new tiles. This tests what the added coverage contributes.
- Top-few only: only the most important tiles, the ones that mixed precision computes in BF16. This tests whether the remaining interactions are needed at all.
Table R5: Gain of mixed precision over each ablation (task-score points).
| Model | Selector | Setting | Reversed precision | No added coverage | Top-few only |
|---|---|---|---|---|---|
| Llama-3.1-8B | FlexPrefill | γ = .85 | 3.45 | 3.02 | 8.85 |
| XAttention | τ = .9 | 3.73 | 2.03 | 77.25 | |
| SeerAttention | density .24 | 0.28 | 1.96 | 20.18 | |
| Phi-4-14B | FlexPrefill | γ = .85 | 2.58 | 0.29 | 9.09 |
| XAttention | τ = .9 | 4.10 | 5.00 | 75.05 |
Each number is the score of mixed precision minus the score of the ablation. A positive number means mixed precision is better, and a negative number means the ablation is better.
Through these ablations we find that both expanded coverage and importance-aware precision contribute to the gain:
- Expanded coverage matters. The "No added coverage" and "Top-few only" ablations shows that, without the added coverage the score drops, most sharply when only the most important tiles are kept. The interactions that sparse attention skips are worth recovering, even at low precision.
- Choosing precision by importance matters. The "Reversed precision" ablation gives the high precision to the least important tiles at the same cost. It leads to worse performance, showing that allocating precision by importance does matter.
Table 2 versus Figures 4(d) and 7(a) (Reviewer Qu7X Question 1)
Table 2 reports benchmark scores. Figures 4(d) and 7(a) report the error of the attention output against dense attention. Attention error is a good proxy for the benchmark score, but the relation is not strict: the same change in attention error can change many answers on one task and few on another. So the selector with the smallest error reduction does not necessarily show the smallest benchmark gain.
Allocation design and compute budget of Table 2 (Reviewer Qu7X Weakness 2)
In the paper we keep the allocator deliberately simple. It follows each sparse method's own layout, ranking and selection strategy, and only assigns precisions along that ranking, which makes it easy to apply to any sparse method. A better-designed allocator can improve the result further. As an example, we tried a new knapsack allocator for Radial on Wan2.1-14B.
Table R6: Gain of the knapsack allocator over the paper's allocator at four compute budgets (Radial, Wan2.1-14B, PSNR in dB).
| β = .75 | β = 1.0 | β = 1.5 | β = 2.0 | |
|---|---|---|---|---|
| Knapsack − paper's allocator | 1.10 | 0.16 | 0.29 | 0.35 |
Each number is the PSNR of the knapsack allocator minus the PSNR of the paper's allocator, on the same prompts. A positive number means the knapsack is better.
- What the knapsack does. The knapsack allocator assigns eligible Radial blocks to BF16, FP8, FP4 or skip to minimize estimated error under a fixed budget. Error estimates use attention mass from one calibration prompt. It first covers many blocks at FP4, then upgrades important blocks. The map is solved offline and reused across layers and denoising steps, adding no inference selection time.
- What it changes. The knapsack is ahead of the paper's allocator on average at all four budgets, most clearly at the lowest one.
We add a cost column to Table 2, as in Table 1. The costs are:
Table R7: Attention cost of the Table 2 settings, as a percentage of dense BF16 attention.
| Model | Selector | Setting | Sparse | Mixed |
|---|---|---|---|---|
| Llama-3.1-8B | FlexPrefill | γ = .85 | 26.4% | 26.4% |
| XAttention | τ = .5 | 14.3% | 14.4% | |
| SeerAttention | density .16 | 17.7% | 17.7% | |
| Llama-3.3-70B | FlexPrefill | γ = .85 | 25.3% | 25.2% |
| XAttention | τ = .5 | 16.7% | 16.7% | |
| Phi-4-14B | FlexPrefill | γ = .85 | 28.8% | 28.7% |
| XAttention | τ = .5 | 14.3% | 14.3% |
Full results are in the new Appendices C–E of the revised paper.
Dear reviewer, thank you for your helpful suggestions! As we approach the end of the discussion period, are there any other questions that we can help answer?
Thank you for the detailed rebuttal. I especially like the experiments regarding W1. They show the effectiveness of mixed precision and the need for increasing coverage, which supports the motivation of the paper.
Regarding Q1, the answer in the rebuttal still leaves some concerns. The rebuttal explains that attention error and the benchmark score are not strictly related, but this is only explained in words. I think a breakdown analysis would give better observations. For example, the cases with large attention error could be separated into questions that are already answered wrong even in full precision.
I think it is still good work compared to other attention precision works, so I want to keep my positive score. Good luck!
Response to Reviewer Qu7X (Part 1/2)
Total runtime: selection and quantization overhead (Reviewer Qu7X Question 2)
Our equal-compute evaluation already includes the quantization cost: we match the native sparse attention kernel time with the mixed-precision kernel time plus quantization time. Quantization is inexpensive, only 0.2–0.5% of the native layer's forward time (Table R1).
The precision conversion happens both before and inside the kernel, and both are counted in the equal compute budgent matching.
Other layer computations, including projections and the FFN, are unchanged between mixed and native sparse attention methods. Therefore, the selection time (scoring, ranking attention interactions, choosing which to compute and at what precision, and building the kernel’s index lists) is the only remaining difference between the methods.
Table R1: Selection and quantization time per layer, native sparse attention vs mixed precision.
Δ is the difference in selection time (mixed − native), so negative values mean faster selection with mixed precision. Δ / kernel and Δ / layer express this difference as percentages of the native attention kernel and full-layer times.
We do not inlcude selection time in our compute budget matching because it depends on implementation. For each native method, we use its original codebase. Both the original sparse attnetion implemenation and our mixed precision implementation use the same scoring and ranking code, so choosing precisions is the only thing that differs. Our selection logic is even slightly faster, because it is more optimized
We keep these implementation savings outside the matched budget, so they do not give mixed precision additional attention compute.