- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces ARGUS, a low-bit FlashAttention approach that employs the NVFP4 format. It specifies the precision choices for prefill computation, persistent KV cache storage, and decode computation separately. Both prefill and KV cache use NVFP4, and the decode dequantizes historical cache to BF16 and compute attention in BF16. The method also uses offline calibration for QK centers to reduce quantization error. The experiments are mainly conducted on 4-8B Large language models. This includes attention, reconstruction error, ablation, reasoning, and long-context language tasks. They also report operator latency compared with other quantization attention kernel designs and achieve impressive speedup performance. The results show practical benefits, while model quality varies across models and tasks.
Critical strengths
-
The paper is generally understandable. The different profile, decoder configurations, and cache designs are reasonably clear.
-
It reports actual practical speed-up using a customized low-bit attention kernel integrated with SGLang. They assess the kernel with different text lengths and cache capacities. The speedup result is impressive compared with other SOTA low-bit quantization attention kernels.
Critical weaknesses
- Llama3.1-8B-Instruct BF16 results in Table 2 are substantially below published results. For example, this paper reports 10.58 on TriviaQA, 4.52 on SAMSum, and 21.67 on TREC, compared with 91.65, 43.50, and 72.50 in the QServe BF16.
- Llama3.1-8B-Instruct on LongBench v1 falls from 75 to 39, and 64K ruler scores fall from 82.83 to 77.61. The accuracy drop is a little bit unacceptable. The degradation on downstream tasks doesn't have further explanation. So, it is suggested to provide a performance explanation of the benchmark degradation and an analysis of which kind of task is suitable for the method.
- Please distinct the new design proposed in ARGUS relative to the closest prior methods. For example, QK centering and score correction have been proposed in SageAttention, and 4over6 is Also, an existing technique. Different caching methods for K and V, current token or previous history, are also alread used in current implementations. It is strongly suggested that the authors discuss the contribution compared with the proposed method and the already existing techniques.
Questions for authors
- The settings for the calibration data sample count, input length, and how the centers are calibrated for each layer of each model under different tasks.
- From Table 5, the accuracy performance mainly comes from 4over6 technique, but other components contribute little. This would reduce the Actual practical value of the proposed method. By the way, the Accuracy performance evaluation study should be evaluated on actual downstream tasks instead of just measuring their quantization error using distance.
- Are SageAttention3 and Thrift both using the same HF/SGLang framework for speed-up evaluation, with the same workload and cache settings?
- The experiments mainly cover 4-8B Language models. It's recommended to include a smaller or larger model if feasible.
Thank you for the detailed rebuttal and additional experiments. The authors have adequately addressed my major concerns, particularly regarding the KV-only comparison, accuracy evaluation, and calibration analysis. I am satisfied with the clarifications and will maintain my positive assessment.Good luck.