- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ThriftAttention addresses the quality degradation caused by FP4 attention in long-context settings. It estimates block importance using dot products between block-averaged query and key vectors, restores a small fraction of the highest-scoring blocks to FP16 computation, and retains FP4 computation for the remaining blocks. The outputs from the two precision paths are then combined. The paper also develops fused kernels and optimizes exponentiation, normalization, and data movement in the FP4 path. The experiments show that the method recovers most of the quality loss introduced by FP4 while achieving some end-to-end speedup.
Critical strengths
1.Balances model quality and practical runtime performance.
The method does not discard less important blocks; instead, it retains low-precision computation for these blocks and increases precision only for a small fraction of important blocks. Table 4 shows that, across four models and three long-context benchmarks, promoting just 5% of the blocks to FP16 recovers, on average, 92.3% of the quality gap between FP4 and FP16. This indicates that selectively increasing precision can effectively mitigate the quality degradation caused by FP4.
In addition to reporting kernel-level speedups, the paper evaluates end-to-end gains. According to Tables 2 and 3, under the tested configuration for Ministral-3-8B, the time to first token and per-token decoding latency correspond to speedups of approximately 1.15× and 1.21×, respectively, over FP16. Meanwhile, LongBench v2 accuracy increases from 31.89% with FP4 to 34.89%, approaching the FP16 result of 35.55%. These results suggest a degree of practical value.
Critical weaknesses
1.The methodological novelty remains relatively limited.
The core approach is to select a small fraction of blocks based on importance and switch them from FP4 back to FP16, improving quality through additional high-precision computation. Block selection itself relies on dot products between block means and Top-k selection, and Section 4.5 explicitly describes this heuristic as SpargeAttn-style Top-k. Thus, in terms of its methodological design, the work is closer to applying an existing block-importance estimation approach to precision allocation than to introducing a clearly distinct selection mechanism.
I acknowledge the authors’ analysis of the non-uniformity of quantization errors, as well as their implementation work on approximate exponentiation and fused kernels. However, these aspects primarily support the rationale and computational efficiency of the proposed approach. The paper still needs to clarify what substantive methodological contribution it offers beyond “switching important blocks back to FP16” and the associated engineering optimizations. The current results demonstrate that this combination is effective, but they do not yet convince me that its methodological novelty is sufficiently substantial.
2.A comprehensive comparison with optimized non-sparse FP8 attention is missing.
The main comparisons in the paper are against FP16, FP4, and sparse attention methods, yet Section 2 already discusses SageAttention’s INT8/FP8 approach. Therefore, FP8 attention that does not rely on block dropping should be an important baseline. Merely showing better quality than uniform FP4 and higher speed than uniform FP16 is insufficient to determine whether this selective mixed-precision design is more competitive than an intermediate-precision alternative.
Questions for authors
1.Could you provide a comparison with optimized non-sparse FP8 attention under the same hardware, model, and context-length settings, reporting downstream quality, end-to-end latency, and memory usage? This comparison would help establish whether selective FP16/FP4 computation offers a better quality–efficiency trade-off than an intermediate-precision alternative.