acceptodds
Under review as a conference paper at ICLR 2027

Compute Less, Capture More: Improving Sparse Attention with Mixed Precision

Abstract

Sparse attention reduces computation by selecting which query–key interactions to evaluate, but typically computes every selected interaction at the same numerical precision. We find that this is inefficient: importance remains highly uneven within the selected set, while useful interactions are still omitted. We introduce mixed-precision sparse attention, which converts this excess precision into broader coverage. It preserves high precision for important interactions, lowers the precision of less important retained interactions, and uses the saved compute to recover interactions that the original sparse method would otherwise skip. By reusing the geometry or importance ranking of existing sparse methods, it expands coverage without a new selection mechanism or additional compute. Our analysis shows that this exchange can reduce attention-output error because broader coverage better preserves dense attention distributions and its benefit can outweigh the error introduced by lower precision. We further develop a fused kernel that executes BF16, FP8, and FP4 interactions within a single attention operation with little overhead. Across video generation and long-context language understanding, mixed-precision sparse attention consistently improves the quality–compute frontier of existing sparse attention methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, mixed precision, quantization, efficient inference, long-context language models, video generation, GPU kernel optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Qu7XOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel mixed-precision method for sparse attention. Motivated by the imbalanced importance among selected query–key interactions, the authors compute important interactions with higher precision and less important ones with lower precision, and use the saved compute to include additional interactions in the attention coverage. The method is supported by theoretical analysis and extensive experiments ranging from video diffusion to LLMs. The kernel implementation would also be insightful for future research.

Critical strengths
  1. Strongly supported motivation and insight for future research. The idea is supported by experiments and theoretical analysis, not only by its effectiveness in diffusion and LLM applications. This work also guides future research on sparse attention, considering not just where to compute but also how precisely to compute.

  2. Comprehensive experiments with broad applicability. The evaluation covers both video generation and long-context language understanding, multiple models, and various quality metrics, including quality–compute frontiers, context-length analysis, attention-output error, and kernel efficiency. The method is also applied to a wide range of existing sparse attention methods (Radial, SVG2, FlexPrefill, XAttention, SeerAttention). Because it reuses each method’s own importance signal without a new selection mechanism, it can be easily plugged into existing approaches.

Critical weaknesses
  1. Additional controlled experiments would strengthen the motivation. The paper claims that, under the same compute budget, converting the excess precision of less important interactions into broader coverage improves quality. However, it is somewhat unclear whether this gain comes from importance-aware precision allocation or mainly from the expanded coverage. It would be helpful to see results with a reversed allocation (high precision for low-attention interactions), with lower precision applied only to less important interactions without expanding coverage, and with only the few most important interactions kept. Such experiments would clarify how much attention importance actually contributes to the performance gain.

  2. Insufficient method details and comparisons. The paper describes the underlying sparse attention mechanisms only briefly through their control parameters, and a more detailed explanation would be helpful. The precision allocation is also somewhat heuristic, without ablation on its design choices. In addition, Table 2 lacks a cost comparison to BF16 like the budget matching in Table 1, which makes it hard to compare efficiency.

Questions for authors
  1. For XAttention, Figure 4(d) and Figure 7(a) show the smallest gain, yet Table 2 shows the largest improvement. Could this gap come from a weak sparse baseline at the chosen operating point?

  2. Is the precision conversion done inside the kernel or before the inputs are fed into it? If the latter, is this additional cost included in the reported kernel time?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Qu7X (Part 1/2)

Total runtime: selection and quantization overhead (Reviewer Qu7X Question 2)

Our equal-compute evaluation already includes the quantization cost: we match the native sparse attention kernel time with the mixed-precision kernel time plus quantization time. Quantization is inexpensive, only 0.2–0.5% of the native layer's forward time (Table R1).

The precision conversion happens both before and inside the kernel, and both are counted in the equal compute budgent matching.

  • Before the kernel: Q, K and V are converted in one pass per attention call. This is the "Quantization" column of Table R1, and it is included in the matched budget.
  • Inside the kernel: the attention probabilities are converted on the fly, so this is already also part of the reported kernel time.

Other layer computations, including projections and the FFN, are unchanged between mixed and native sparse attention methods. Therefore, the selection time (scoring, ranking attention interactions, choosing which to compute and at what precision, and building the kernel’s index lists) is the only remaining difference between the methods.

Table R1: Selection and quantization time per layer, native sparse attention vs mixed precision.

SettingLayer (ms)Attention kernel (ms)Selection, native (ms)Selection, mixed (ms)Δ (ms)Δ / kernelΔ / layerQuantization, mixed: ms (% of layer)
Wan2.1 · Radial4101200.33≈0−0.33−0.3%−0.1%2.12 (0.5%)
Wan2.1 · SVG2, warm k-means47913831.430.5−0.95−0.7%−0.2%1.58 (0.3%)
Wan2.1 · SVG2, cold k-means1024143571.0569.7−1.3−0.9%−0.1%1.58 (0.2%)
Llama-3.1 · XAttention, 32K95.813.96.235.65−0.57−4.1%−0.6%0.36 (0.4%)
Llama-3.1 · XAttention, 64K21037.926.222.4−3.8−9.9%−1.8%0.70 (0.3%)
Llama-3.1 · FlexPrefill, 32K85.35.554.764.49−0.26−4.8%−0.3%0.36 (0.4%)
Llama-3.1 · FlexPrefill, 64K17118.08.368.01−0.35−1.9%−0.2%0.70 (0.4%)

Δ is the difference in selection time (mixed − native), so negative values mean faster selection with mixed precision. Δ / kernel and Δ / layer express this difference as percentages of the native attention kernel and full-layer times.

We do not inlcude selection time in our compute budget matching because it depends on implementation. For each native method, we use its original codebase. Both the original sparse attnetion implemenation and our mixed precision implementation use the same scoring and ranking code, so choosing precisions is the only thing that differs. Our selection logic is even slightly faster, because it is more optimized

  • Radial: We precompute its static schedule instead of rebuilding it at every attention call.
  • SVG2: We fuse precision assignment into top-p selection and replace FlashInfer’s planner with simpler index construction.
  • XAttention: We replace the original chunked selection with fused selection and index construction, reducing planning time.

We keep these implementation savings outside the matched budget, so they do not give mixed precision additional attention compute.

AuthorsOpenReview ↗

Response to Reviewer Qu7X (Part 2/2)

Ablations on importance and coverage (Reviewer Qu7X Weakness 1)

We ran the three requested ablations in Weakness 1.

  • Reversed precision: the same tiles and the same cost as mixed precision, but with the precision order flipped, so the least important tiles get BF16. This tests whether it matters which tiles get the high precision.
  • No added coverage: only the tiles that the original sparse method selects. The less important ones are computed at lower precision, as in mixed precision, but the saved compute is not spent on new tiles. This tests what the added coverage contributes.
  • Top-few only: only the most important tiles, the ones that mixed precision computes in BF16. This tests whether the remaining interactions are needed at all.

Table R5: Gain of mixed precision over each ablation (task-score points).

ModelSelectorSettingReversed precisionNo added coverageTop-few only
Llama-3.1-8BFlexPrefillγ = .853.453.028.85
XAttentionτ = .93.732.0377.25
SeerAttentiondensity .240.281.9620.18
Phi-4-14BFlexPrefillγ = .852.580.299.09
XAttentionτ = .94.105.0075.05

Each number is the score of mixed precision minus the score of the ablation. A positive number means mixed precision is better, and a negative number means the ablation is better.

Through these ablations we find that both expanded coverage and importance-aware precision contribute to the gain:

  1. Expanded coverage matters. The "No added coverage" and "Top-few only" ablations shows that, without the added coverage the score drops, most sharply when only the most important tiles are kept. The interactions that sparse attention skips are worth recovering, even at low precision.
  2. Choosing precision by importance matters. The "Reversed precision" ablation gives the high precision to the least important tiles at the same cost. It leads to worse performance, showing that allocating precision by importance does matter.

Table 2 versus Figures 4(d) and 7(a) (Reviewer Qu7X Question 1)

Table 2 reports benchmark scores. Figures 4(d) and 7(a) report the error of the attention output against dense attention. Attention error is a good proxy for the benchmark score, but the relation is not strict: the same change in attention error can change many answers on one task and few on another. So the selector with the smallest error reduction does not necessarily show the smallest benchmark gain.


Allocation design and compute budget of Table 2 (Reviewer Qu7X Weakness 2)

In the paper we keep the allocator deliberately simple. It follows each sparse method's own layout, ranking and selection strategy, and only assigns precisions along that ranking, which makes it easy to apply to any sparse method. A better-designed allocator can improve the result further. As an example, we tried a new knapsack allocator for Radial on Wan2.1-14B.

Table R6: Gain of the knapsack allocator over the paper's allocator at four compute budgets (Radial, Wan2.1-14B, PSNR in dB).

β = .75β = 1.0β = 1.5β = 2.0
Knapsack − paper's allocator1.100.160.290.35

Each number is the PSNR of the knapsack allocator minus the PSNR of the paper's allocator, on the same prompts. A positive number means the knapsack is better.

  • What the knapsack does. The knapsack allocator assigns eligible Radial blocks to BF16, FP8, FP4 or skip to minimize estimated error under a fixed budget. Error estimates use attention mass from one calibration prompt. It first covers many blocks at FP4, then upgrades important blocks. The map is solved offline and reused across layers and denoising steps, adding no inference selection time.
  • What it changes. The knapsack is ahead of the paper's allocator on average at all four budgets, most clearly at the lowest one.

We add a cost column to Table 2, as in Table 1. The costs are:

Table R7: Attention cost of the Table 2 settings, as a percentage of dense BF16 attention.

ModelSelectorSettingSparseMixed
Llama-3.1-8BFlexPrefillγ = .8526.4%26.4%
XAttentionτ = .514.3%14.4%
SeerAttentiondensity .1617.7%17.7%
Llama-3.3-70BFlexPrefillγ = .8525.3%25.2%
XAttentionτ = .516.7%16.7%
Phi-4-14BFlexPrefillγ = .8528.8%28.7%
XAttentionτ = .514.3%14.3%

Full results are in the new Appendices C–E of the revised paper.

Dear reviewer, thank you for your helpful suggestions! As we approach the end of the discussion period, are there any other questions that we can help answer?

Thank you for the detailed rebuttal. I especially like the experiments regarding W1. They show the effectiveness of mixed precision and the need for increasing coverage, which supports the motivation of the paper.

Regarding Q1, the answer in the rebuttal still leaves some concerns. The rebuttal explains that attention error and the benchmark score are not strictly related, but this is only explained in words. I think a breakdown analysis would give better observations. For example, the cases with large attention error could be separated into questions that are already answered wrong even in full precision.

I think it is still good work compared to other attention precision works, so I want to keep my positive score. Good luck!

Reviewer_pnqWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper jointly allocates attention coverage and precision. Existing importance rankings assign tiles to BF16, FP8, FP4, or skipping, using lower precision to recover otherwise omitted interactions. A fused kernel shares online softmax across precision groups. Language-model and video-generation experiments report better quality at matched kernel cost, or lower cost at matched quality.

Critical strengths

1.Jointly allocating precision and coverage is a useful extension of binary sparse selection and can reuse several existing selectors.

2.Evaluations span language and video models, with both matched-cost quality and matched-quality cost comparisons supporting the approach's broader applicability.

Critical weaknesses

1.Please compare against uniform FP8/FP4 with expanded coverage at matched measured cost. The existing ThriftAttention comparison does not isolate whether mixed allocation improves over simply computing more interactions at lower precision.

2.Kernel timings exclude parts of selection and preparation. Please report complete prefill and video-generation times, including precision allocation and quantization, to establish end-to-end benefits.

3.The allocation policy needs clearer cost coefficients, quantization settings, and calibration procedures. Sensitivity to the BF16 budget and calibration choices is needed to assess reproducibility and robustness.

Questions for authors

See weakness above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer pnqW (Part 1/3)

Total runtime: selection and quantization overhead (Reviewer pnqW Weakness 2)

Our equal-compute evaluation already includes the quantization cost: we match the native sparse attention kernel time with the mixed-precision kernel time plus quantization time. Quantization is inexpensive, only 0.2–0.5% of the native layer's forward time (Table R1).

Other layer computations, including projections and the FFN, are unchanged between mixed and native sparse attention methods. Therefore, the selection time (scoring, ranking attention interactions, choosing which to compute and at what precision, and building the kernel’s index lists) is the only remaining difference between the methods.

Table R1: Selection and quantization time per layer, native sparse attention vs mixed precision.

SettingLayer (ms)Attention kernel (ms)Selection, native (ms)Selection, mixed (ms)Δ (ms)Δ / kernelΔ / layerQuantization, mixed: ms (% of layer)
Wan2.1 · Radial4101200.33≈0−0.33−0.3%−0.1%2.12 (0.5%)
Wan2.1 · SVG2, warm k-means47913831.430.5−0.95−0.7%−0.2%1.58 (0.3%)
Wan2.1 · SVG2, cold k-means1024143571.0569.7−1.3−0.9%−0.1%1.58 (0.2%)
Llama-3.1 · XAttention, 32K95.813.96.235.65−0.57−4.1%−0.6%0.36 (0.4%)
Llama-3.1 · XAttention, 64K21037.926.222.4−3.8−9.9%−1.8%0.70 (0.3%)
Llama-3.1 · FlexPrefill, 32K85.35.554.764.49−0.26−4.8%−0.3%0.36 (0.4%)
Llama-3.1 · FlexPrefill, 64K17118.08.368.01−0.35−1.9%−0.2%0.70 (0.4%)

Δ is the difference in selection time (mixed − native), so negative values mean faster selection with mixed precision. Δ / kernel and Δ / layer express this difference as percentages of the native attention kernel and full-layer times.

We do not inlcude selection time in our compute budget matching because it depends on implementation. For each native method, we use its original codebase. Both the original sparse attnetion implemenation and our mixed precision implementation use the same scoring and ranking code, so choosing precisions is the only thing that differs. Our selection logic is even slightly faster, because it is more optimized

  • Radial: We precompute its static schedule instead of rebuilding it at every attention call.
  • SVG2: We fuse precision assignment into top-p selection and replace FlashInfer’s planner with simpler index construction.
  • XAttention: We replace the original chunked selection with fused selection and index construction, reducing planning time.

We keep these implementation savings outside the matched budget, so they do not give mixed precision additional attention compute.

AuthorsOpenReview ↗

Response to Reviewer pnqW (Part 2/3)

Uniform FP8 / FP4 with expanded coverage (Reviewer pnqW Weakness 1)

We compare mixed precision with two new baselines, uniform FP8 and uniform FP4, to test whether choosing precision by importance helps beyond using one cheaper precision and computing more interactions.

Each uniform baseline uses the same selector, ranking and inputs as mixed precision. It computes every tile in one precision (FP8 or FP4) and spends the savings on more tiles, in ranking order, up to the same attention-kernel cost as mixed precision.

Table R2: Gain of mixed precision over uniform FP4 / FP8 at the same attention-kernel cost (mean task score for language models, PSNR in dB for video models).

ModelSelectorSettingGain vs uniform FP4Gain vs uniform FP8
Llama-3.1-8BFlexPrefillγ = .853.891.11
XAttentionτ = .94.531.60
SeerAttentiondensity .245.031.44
Phi-4-14BFlexPrefillγ = .853.151.08
XAttentionτ = .94.731.91
Wan2.1-14BRadialβ = 1.01.530.65
SVG2top-p = .851.541.21
HunyuanVideo-13BRadialβ = 1.01.192.39
SVG2top-p = .852.461.40

Each number is the score of mixed precision minus the score of the uniform baseline. A positive number means mixed precision is better, and a negative number means the uniform baseline is better.

Through these experiments we find that:

  1. Mixed precision is ahead of both uniform baselines in all tested settings.
  2. The gain does not only come from more coverage. Uniform FP4 covers the most tiles but computes the important ones too coarsely. Uniform FP8 computes them more accurately but covers fewer. Choosing precision by importance gets both.

We also repeated the comparison at four budgets per method, to check whether the gain is consistent across budgets. Budgets increase from left to right.

Table R3: Gain of mixed precision over uniform FP4 / FP8 at four compute budgets (mean task score for language models, PSNR in dB for video models).

ModelSelector (budgets)vsBudget 1Budget 2Budget 3Budget 4
Llama-3.1-8BFlexPrefill (γ .50 / .70 / .85 / .95)FP43.261.083.891.10
FP81.420.591.110.12
SeerAttention (density .08 / .16 / .24 / .32)FP43.324.505.036.05
FP81.890.881.440.78
XAttention (τ .5 / .7 / .8 / .9)FP44.621.983.034.53
FP811.011.961.811.60
Phi-4-14BFlexPrefill (γ .50 / .70 / .85 / .95)FP41.153.253.151.87
FP82.383.411.08−0.77
XAttention (τ .5 / .7 / .8 / .9)FP43.565.774.074.73
FP811.854.140.341.91
Wan2.1-14BRadial (β .75 / 1.0 / 1.5 / 2.0)FP4−0.191.532.943.06
FP8−0.680.650.730.44
SVG2 (top-p .70 / .85 / .90 / .95)FP40.161.541.862.58
FP80.981.210.780.35

The numbers are read the same way as in Table R2.

Across budgets we find that:

  1. Against uniform FP4, the gain grows with budget on SeerAttention and both video methods. An offline analysis shows that this is due to, at very low compute budget, the coverage matters more than smart precision choice.
  2. Against uniform FP8, the gain on language tasks is largest at low budgets, reaching 11–12 points on XAttention. It narrows at high budgets, where uniform FP8 is already close to dense attention.
AuthorsOpenReview ↗

Response to Reviewer pnqW (Part 3/3)

In the paper we keep the allocator deliberately simple. It follows each sparse method's own layout, ranking and selection strategy, and only assigns precisions along that ranking, which makes it easy to apply to any sparse method. A better-designed allocator can widen the gain over the uniform baselines. As an example, we tried a new knapsack allocator for Radial on Wan2.1-14B.

Table R4: Gain over uniform FP4 / FP8 with the paper's allocator and with the knapsack allocator (Radial, Wan2.1-14B, PSNR in dB).

vsAllocatorβ = .75β = 1.0β = 1.5β = 2.0
FP4Paper's (from Table R3)−0.191.532.943.06
Knapsack0.921.683.233.42
FP8Paper's (from Table R3)−0.680.650.730.44
Knapsack0.420.801.000.96

The two allocators were evaluated on different prompts.

  • What the knapsack does. The knapsack allocator assigns eligible Radial blocks to BF16, FP8, FP4 or skip to minimize estimated error under a fixed budget. Error estimates use attention mass from one calibration prompt. It first covers many blocks at FP4, then upgrades important blocks. The map is solved offline and reused across layers and denoising steps, adding no inference selection time.
  • It adds no selection time at inference. The map is solved once, offline, and reused for every layer and denoising step.
  • What it changes. With the knapsack, mixed precision is ahead of both uniform baselines at every budget. Its gain is larger than the paper's allocator's in seven of the eight cases.

Quantization settings and calibration (Reviewer pnqW Weakness 3)

  • Formats. BF16 tiles keep the model's BF16 operands. FP8 tiles use the E4M3 format (4 exponent bits, 3 mantissa bits). FP4 tiles use NVFP4: 4-bit E2M1 values (2 exponent bits, 1 mantissa bit) in groups of 16, where each group shares one E4M3 scale, set to the group's largest magnitude divided by 6, the largest E2M1 value.
  • What is quantized. Q, K and V are converted to each tile's precision on the fly, in one pass per attention call. In the low-precision tiles, the attention probabilities P are also converted inside the kernel, before they multiply V. Model weights are not changed.
  • What stays in high precision. Both products, QKᵀ and PV, accumulate in FP32. The online-softmax state (running maximum, normalization sum and output accumulator) is also FP32 and is shared by all three precisions.
  • No calibration. All scales are computed from the current tensors at run time. Quantization uses no calibration data, no learned parameters and no fine-tuning.

Full results are in the new Appendices C–E of the revised paper.

Reviewer_BidEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to compute more attention pairs by spending fewer bits on the less important ones. The paper reuses importance scores from prior sparse attention methods to choose a precision for each tile, then processes the tiles together in one kernel. At matched attention-kernel time, tests on two video models and three language models improve video fidelity and most reported language-task scores.

Critical strengths

The method improves several existing sparse attention methods, so its usefulness is not tied to one selector. Video PSNR gains are 1.90-6.39 dB at matched kernel time, and most language-task categories improve too.

The output-error measurements back up the quality results: mean error falls by 15.9-68.0% across 11 combinations. Mixing precisions in one kernel adds little overhead compared with the times predicted from running each precision separately.

Critical weaknesses

The reported timing covers the attention kernel. It should also include choosing each tile’s precision and building the index lists, so we can see how much time the method actually saves.

A useful baseline is missing: use the same low precision for every selected tile, and spend the saved time on more attention pairs. That would show whether choosing precision by importance helps beyond simply using cheaper arithmetic.

Questions for authors

What is the total runtime, including choosing precisions, quantizing values, and building index lists?

At the same total runtime, how does using FP8 for all selected tiles and computing more pairs compare?

4 comments
AuthorsOpenReview ↗

Response to Reviewer BidE (Part 1/3)

Total runtime: selection and quantization overhead (Reviewer BidE Question 1)

Our equal-compute evaluation already includes the quantization cost: we match the native sparse attention kernel time with the mixed-precision kernel time plus quantization time. Quantization is inexpensive, only 0.2–0.5% of the native layer's forward time (Table R1).

Other layer computations, including projections and the FFN, are unchanged between mixed and native sparse attention methods. Therefore, the selection time (scoring, ranking attention interactions, choosing which to compute and at what precision, and building the kernel’s index lists) is the only remaining difference between the methods.

Table R1: Selection and quantization time per layer, native sparse attention vs mixed precision.

SettingLayer (ms)Attention kernel (ms)Selection, native (ms)Selection, mixed (ms)Δ (ms)Δ / kernelΔ / layerQuantization, mixed: ms (% of layer)
Wan2.1 · Radial4101200.33≈0−0.33−0.3%−0.1%2.12 (0.5%)
Wan2.1 · SVG2, warm k-means47913831.430.5−0.95−0.7%−0.2%1.58 (0.3%)
Wan2.1 · SVG2, cold k-means1024143571.0569.7−1.3−0.9%−0.1%1.58 (0.2%)
Llama-3.1 · XAttention, 32K95.813.96.235.65−0.57−4.1%−0.6%0.36 (0.4%)
Llama-3.1 · XAttention, 64K21037.926.222.4−3.8−9.9%−1.8%0.70 (0.3%)
Llama-3.1 · FlexPrefill, 32K85.35.554.764.49−0.26−4.8%−0.3%0.36 (0.4%)
Llama-3.1 · FlexPrefill, 64K17118.08.368.01−0.35−1.9%−0.2%0.70 (0.4%)

Δ is the difference in selection time (mixed − native), so negative values mean faster selection with mixed precision. Δ / kernel and Δ / layer express this difference as percentages of the native attention kernel and full-layer times.

We do not inlcude selection time in our compute budget matching because it depends on implementation. For each native method, we use its original codebase. Both the original sparse attnetion implemenation and our mixed precision implementation use the same scoring and ranking code, so choosing precisions is the only thing that differs. Our selection logic is even slightly faster, because it is more optimized

  • Radial: We precompute its static schedule instead of rebuilding it at every attention call.
  • SVG2: We fuse precision assignment into top-p selection and replace FlashInfer’s planner with simpler index construction.
  • XAttention: We replace the original chunked selection with fused selection and index construction, reducing planning time.

We keep these implementation savings outside the matched budget, so they do not give mixed precision additional attention compute.

AuthorsOpenReview ↗

Response to Reviewer BidE (Part 2/3)

Uniform FP8 / FP4 with expanded coverage (Reviewer BidE Question 2)

We compare mixed precision with two new baselines, uniform FP8 and uniform FP4, to test whether choosing precision by importance helps beyond using one cheaper precision and computing more interactions.

Each uniform baseline uses the same selector, ranking and inputs as mixed precision. It computes every tile in one precision (FP8 or FP4) and spends the savings on more tiles, in ranking order, up to the same attention-kernel cost as mixed precision.

Table R2: Gain of mixed precision over uniform FP4 / FP8 at the same attention-kernel cost (mean task score for language models, PSNR in dB for video models).

ModelSelectorSettingGain vs uniform FP4Gain vs uniform FP8
Llama-3.1-8BFlexPrefillγ = .853.891.11
XAttentionτ = .94.531.60
SeerAttentiondensity .245.031.44
Phi-4-14BFlexPrefillγ = .853.151.08
XAttentionτ = .94.731.91
Wan2.1-14BRadialβ = 1.01.530.65
SVG2top-p = .851.541.21
HunyuanVideo-13BRadialβ = 1.01.192.39
SVG2top-p = .852.461.40

Each number is the score of mixed precision minus the score of the uniform baseline. A positive number means mixed precision is better, and a negative number means the uniform baseline is better.

Through these experiments we find that:

  1. Mixed precision is ahead of both uniform baselines in all tested settings.
  2. The gain does not only come from more coverage. Uniform FP4 covers the most tiles but computes the important ones too coarsely. Uniform FP8 computes them more accurately but covers fewer. Choosing precision by importance gets both.

We also repeated the comparison at four budgets per method, to check whether the gain is consistent across budgets. Budgets increase from left to right.

Table R3: Gain of mixed precision over uniform FP4 / FP8 at four compute budgets (mean task score for language models, PSNR in dB for video models).

ModelSelector (budgets)vsBudget 1Budget 2Budget 3Budget 4
Llama-3.1-8BFlexPrefill (γ .50 / .70 / .85 / .95)FP43.261.083.891.10
FP81.420.591.110.12
SeerAttention (density .08 / .16 / .24 / .32)FP43.324.505.036.05
FP81.890.881.440.78
XAttention (τ .5 / .7 / .8 / .9)FP44.621.983.034.53
FP811.011.961.811.60
Phi-4-14BFlexPrefill (γ .50 / .70 / .85 / .95)FP41.153.253.151.87
FP82.383.411.08−0.77
XAttention (τ .5 / .7 / .8 / .9)FP43.565.774.074.73
FP811.854.140.341.91
Wan2.1-14BRadial (β .75 / 1.0 / 1.5 / 2.0)FP4−0.191.532.943.06
FP8−0.680.650.730.44
SVG2 (top-p .70 / .85 / .90 / .95)FP40.161.541.862.58
FP80.981.210.780.35

The numbers are read the same way as in Table R2.

Across budgets we find that:

  1. Against uniform FP4, the gain grows with budget on SeerAttention and both video methods. An offline analysis shows that this is due to, at very low compute budget, the coverage matters more than smart precision choice.
  2. Against uniform FP8, the gain on language tasks is largest at low budgets, reaching 11–12 points on XAttention. It narrows at high budgets, where uniform FP8 is already close to dense attention.
AuthorsOpenReview ↗

Response to Reviewer BidE (Part 3/3)

In the paper we keep the allocator deliberately simple. It follows each sparse method's own layout, ranking and selection strategy, and only assigns precisions along that ranking, which makes it easy to apply to any sparse method. A better-designed allocator can widen the gain over the uniform baselines. As an example, we tried a new knapsack allocator for Radial on Wan2.1-14B.

Table R4: Gain over uniform FP4 / FP8 with the paper's allocator and with the knapsack allocator (Radial, Wan2.1-14B, PSNR in dB).

vsAllocatorβ = .75β = 1.0β = 1.5β = 2.0
FP4Paper's (from Table R3)−0.191.532.943.06
Knapsack0.921.683.233.42
FP8Paper's (from Table R3)−0.680.650.730.44
Knapsack0.420.801.000.96

The two allocators were evaluated on different prompts.

  • What the knapsack does. The knapsack allocator assigns eligible Radial blocks to BF16, FP8, FP4 or skip to minimize estimated error under a fixed budget. Error estimates use attention mass from one calibration prompt. It first covers many blocks at FP4, then upgrades important blocks. The map is solved offline and reused across layers and denoising steps, adding no inference selection time.
  • It adds no selection time at inference. The map is solved once, offline, and reused for every layer and denoising step.
  • What it changes. With the knapsack, mixed precision is ahead of both uniform baselines at every budget. Its gain is larger than the paper's allocator's in seven of the eight cases.

Full results are in the new Appendices C–E of the revised paper.

Dear reviewer, thank you for your helpful suggestions! As we approach the end of the discussion period, are there any other questions that we can help answer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.