acceptodds
Under review as a conference paper at ICLR 2027

ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention

Abstract

Efficient attention algorithms are critical to mitigate the quadratic cost of attention in long-context workloads. Such methods often rely on low-bit quantisation and/or hard sparsity to accelerate inference. FP4 attention on Blackwell GPUs offers large speedups but degrades sharply relative to FP16 as context length grows while sparsity methods discard query-key interactions outright which can degrade model quality. We address both failure modes through ThriftAttention, a mixed-precision attention mechanism that uses a heuristic to promote only the most important interactions to FP16 while computing the remainder in FP4. ThriftAttention relies on our finding that the output impact of quantisation error is highly non-uniform and increases with the importance of each query-key interaction, concentrating functionally relevant error in a small number of attention blocks that contain the most important tokens. Our experiments show across long-context benchmarks and model families that by computing only 5% of the attention blocks in FP16, ThriftAttention recovers on average 92.3% of the FP4FP16 performance gap. We demonstrate ThriftAttention's advantage as context length grows, the regime where FP4 attention performs worst.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Training-free, FP4, attention, efficient, sparsity, mixed-precision, inference, llm

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vUaTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ThriftAttention addresses the quality degradation caused by FP4 attention in long-context settings. It estimates block importance using dot products between block-averaged query and key vectors, restores a small fraction of the highest-scoring blocks to FP16 computation, and retains FP4 computation for the remaining blocks. The outputs from the two precision paths are then combined. The paper also develops fused kernels and optimizes exponentiation, normalization, and data movement in the FP4 path. The experiments show that the method recovers most of the quality loss introduced by FP4 while achieving some end-to-end speedup.

Critical strengths

1.Balances model quality and practical runtime performance.

The method does not discard less important blocks; instead, it retains low-precision computation for these blocks and increases precision only for a small fraction of important blocks. Table 4 shows that, across four models and three long-context benchmarks, promoting just 5% of the blocks to FP16 recovers, on average, 92.3% of the quality gap between FP4 and FP16. This indicates that selectively increasing precision can effectively mitigate the quality degradation caused by FP4.

In addition to reporting kernel-level speedups, the paper evaluates end-to-end gains. According to Tables 2 and 3, under the tested configuration for Ministral-3-8B, the time to first token and per-token decoding latency correspond to speedups of approximately 1.15× and 1.21×, respectively, over FP16. Meanwhile, LongBench v2 accuracy increases from 31.89% with FP4 to 34.89%, approaching the FP16 result of 35.55%. These results suggest a degree of practical value.

Critical weaknesses

1.The methodological novelty remains relatively limited.

The core approach is to select a small fraction of blocks based on importance and switch them from FP4 back to FP16, improving quality through additional high-precision computation. Block selection itself relies on dot products between block means and Top-k selection, and Section 4.5 explicitly describes this heuristic as SpargeAttn-style Top-k. Thus, in terms of its methodological design, the work is closer to applying an existing block-importance estimation approach to precision allocation than to introducing a clearly distinct selection mechanism.

I acknowledge the authors’ analysis of the non-uniformity of quantization errors, as well as their implementation work on approximate exponentiation and fused kernels. However, these aspects primarily support the rationale and computational efficiency of the proposed approach. The paper still needs to clarify what substantive methodological contribution it offers beyond “switching important blocks back to FP16” and the associated engineering optimizations. The current results demonstrate that this combination is effective, but they do not yet convince me that its methodological novelty is sufficiently substantial.

2.A comprehensive comparison with optimized non-sparse FP8 attention is missing.

The main comparisons in the paper are against FP16, FP4, and sparse attention methods, yet Section 2 already discusses SageAttention’s INT8/FP8 approach. Therefore, FP8 attention that does not rely on block dropping should be an important baseline. Merely showing better quality than uniform FP4 and higher speed than uniform FP16 is insufficient to determine whether this selective mixed-precision design is more competitive than an intermediate-precision alternative.

Questions for authors

1.Could you provide a comparison with optimized non-sparse FP8 attention under the same hardware, model, and context-length settings, reporting downstream quality, end-to-end latency, and memory usage? This comparison would help establish whether selective FP16/FP4 computation offers a better quality–efficiency trade-off than an intermediate-precision alternative.

Reviewer_K9umOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ThriftAttention is a training-free mixed-precision attention method that promotes selected query–key block interactions to FP16 while computing the remaining interactions in FP4. Selection uses a top-k heuristic based on block-mean queries and keys. The two paths are combined through online-softmax accumulation, and the FP4 path uses an approximate exponential construction implemented with arithmetic and bit operations. Evaluation includes long-context benchmarks across four models, NLL analysis, and kernel-level and end-to-end measurements on an RTX PRO 6000.

Critical strengths
  1. The quality improvements over uniform FP4 are substantial and demonstrated across several model families and benchmarks. A 5% FP16 budget recovers an average of 92.3% of the FP4-to-FP16 benchmark gap. The NLL and error-capture analyses provide additional evidence that a small subset of interactions accounts for a disproportionate share of consequential quantization error, although recovery remains incomplete in some settings.

  2. The paper connects the algorithm to a working implementation and useful controlled comparisons. Kernel timings include quantization and block selection, and the reported 5% configuration achieves an average 1.97x speedup over FP16 attention. The comparison using the same selection heuristics for mixed precision and hard sparsity is particularly informative: it supports retaining low-precision contributions from unselected blocks rather than discarding them at approximately comparable computational budgets.

Critical weaknesses
  1. The evaluation does not establish the method’s advantage over strong intermediate-precision attention alternatives. The main quality–efficiency comparisons emphasize FP16, uniform FP4, and sparse methods, without corresponding FP8/INT8 attention baselines under the same hardware and evaluation settings. These alternatives are directly relevant to the proposed tradeoff: they may recover much of the FP4 quality loss without query-dependent precision routing and dual KV-cache storage. Consequently, the results demonstrate a convincing improvement over uniform FP4, but do not yet establish a compelling advantage over the broader set of efficient quantized-attention options.

  2. The practical benefit under a fixed memory budget remains insufficiently characterized. In Table 3, decoding improves from 15.74 to 13.02 ms/token, while resident memory increases from 20.21 to 25.78 GiB because both FP16 and FP4 KV caches are maintained. This approximately 21% throughput improvement comes with approximately 28% more resident memory at the reported operating point. The additional memory could limit context length or concurrency, yet the paper does not provide corresponding end-to-end tradeoff curves across sequence lengths and batch sizes under a common memory limit. The principal systems evidence also comes from one consumer/workstation Blackwell platform; the B200 appendix evaluates a different low-precision kernel configuration rather than establishing the complete method’s deployment tradeoff there.

Overall assessment: The mixed-precision strategy is promising and the recovery over uniform FP4 is well supported. However, the missing intermediate-precision comparisons and incomplete characterization of the memory–throughput tradeoff leave the practical advantage insufficiently established.

Questions for authors
  1. How does ThriftAttention compare with strong FP8/INT8 attention implementations on the same GPU, reporting downstream quality, kernel latency, end-to-end latency, and memory? This comparison would materially affect whether selective FP16 promotion provides a worthwhile advantage over simpler precision choices.

  2. Under a fixed GPU-memory budget, how does dual-cache storage affect the maximum context length, supported batch size, and aggregate decoding throughput? Measurements across several context lengths and batch sizes would help determine whether the reported latency advantage survives the additional memory cost.

Reviewer_aEHVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NVFP4 attention on Blackwell GPUs is fast but can result in quality loss on long-context LLM tasks. The paper proposes ThriftAttention, to score each pair of query and key blocks cheaply, using the dot product of their mean vectors. It computes the top-scoring blocks in FP16 and the rest in FP4, then merges the two. It also speeds up the FP4 path by rounding attention weights to powers of two, which runs on faster hardware units than a true exponential. With 5% of blocks in FP16, it recovers 92.3% of FP4’s quality loss across four sub-10B scale models on RULER, LongBench and HELMET, runs 1.8-2.1× faster than FlashAttention-2 on an RTX PRO 6000.

Critical strengths

The work tackles a real deployment problem and provides working fused kernel on actual FP4 hardware, evaluated end to end against both quantized and sparse baselines. The paper also documents that FP4 attention, so far validated mainly on image and video generation, keeps only 31–58% of FP16 quality on long-context LLM tasks, which is a useful warning for practitioners. Also the paper shows, at equal compute, keeping unselected blocks in FP4 clearly beats discarding them (Table 7).

Critical weaknesses

Missing baselines. (a) The paper does show clear wins over sparse methods: at similar prefill speed, ThriftAttention beats SpargeAttention on LongBench v2 (34.89 vs 29.57, Table 2). But its quality is measured mainly against FP4, a weak reference point. (b) 8-bit attention (SageAttention2++) is near-lossless on Llama-3.1-8B and about 3× faster than FlashAttention-2 on an RTX 4090. ThriftAttention at 5% reaches 1.8–2.1× and still loses quality (Llama-3.1 RULER drops from 0.859 to 0.770, Table 4). SpargeAttention, the paper’s only 8-bit point, also skips blocks. Running dense 8-bit attention (e.g., SageAttention2++) on the same RTX PRO 6000 and context lengths, as a point in Figure 1 and a row in Table 4, would show whether ThriftAttention beats this simplest option. Modest end-to-end gains, and serving baselines. The decode comparison omits KV-cache quantization, such as the NVFP4 KV cache now standard in TensorRT-LLM and SGLang, which cuts memory roughly fourfold at about a point of long-context accuracy. The memory overhead may cancel the speed gain at larger batch sizes, and no batched throughput is reported.

Questions for authors
  1. How does ThriftAttention compare with dense 8-bit attention, for example SageAttention2++, run on the same RTX PRO 6000 and at the same context lengths, as a point in Figure 1 and a row in Table 4?

  2. For decoding, how does ThriftAttention compare with an NVFP4 KV cache? How does throughput compare at batch sizes where memory is the limit?

  3. SageAttention3 is measured here at 2.2-2.3x over FlashAttention-2, whereas in its own paper reports a peak speedup of 5x on an RTX 5090. Since the speedup barely changes from 16K to 131K tokens (Figure 3), context length seems unlikely to explain it. Could the authors provide some more insights on this?

  4. How was SpargeAttention’s 25% budget in Table 2 chosen relative to ThriftAttention’s 5%? Were the two matched on speed? What about other settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.