acceptodds
Under review as a conference paper at ICLR 2027

ARGUS: Seeing Beyond the Kernel in Stage-Aware NVFP4 Attention

Abstract

NVFP4 attention leverages high-throughput FP4 arithmetic on NVIDIA Blackwell GPUs, but existing training-free methods largely optimize the main fused kernel and apply similar low-precision execution across inference stages. This kernel-centric focus overlooks two important bottlenecks: preprocessing and cache construction can constitute a substantial fraction of prefill operator latency, whereas autoregressive decoding is primarily constrained by streaming the growing KV cache. We introduce **ARGUS**, a stage-aware NVFP4 attention design that looks beyond kernel-level acceleration and places low precision where each inference stage benefits from it most. This stage-aware design is realized through two complementary components. **Cache-Ready Prefill** replaces input-dependent statistics with **Offline Sparse Centering**, reducing runtime preprocessing while directly materializing a persistent NVFP4 KV cache. **Cache-Native Decoding** directly consumes this compressed cache with BF16 attention, decoupling cache precision from compute precision to reduce memory traffic without unnecessary low-precision transformations. Across reasoning and long-context evaluations on GQA models, ARGUS maintains competitive model quality and achieves up to **4.09×** prefill and **2.26×** decode operator-level speedups, yielding up to **2.40×** end-to-end acceleration for the evaluated 128K-input workload on SGLang.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Quantization, NVFP4 attention, GQA

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_16hDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces ARGUS, a low-bit FlashAttention approach that employs the NVFP4 format. It specifies the precision choices for prefill computation, persistent KV cache storage, and decode computation separately. Both prefill and KV cache use NVFP4, and the decode dequantizes historical cache to BF16 and compute attention in BF16. The method also uses offline calibration for QK centers to reduce quantization error. The experiments are mainly conducted on 4-8B Large language models. This includes attention, reconstruction error, ablation, reasoning, and long-context language tasks. They also report operator latency compared with other quantization attention kernel designs and achieve impressive speedup performance. The results show practical benefits, while model quality varies across models and tasks.

Critical strengths
  1. The paper is generally understandable. The different profile, decoder configurations, and cache designs are reasonably clear.

  2. It reports actual practical speed-up using a customized low-bit attention kernel integrated with SGLang. They assess the kernel with different text lengths and cache capacities. The speedup result is impressive compared with other SOTA low-bit quantization attention kernels.

Critical weaknesses
  1. Llama3.1-8B-Instruct BF16 results in Table 2 are substantially below published results. For example, this paper reports 10.58 on TriviaQA, 4.52 on SAMSum, and 21.67 on TREC, compared with 91.65, 43.50, and 72.50 in the QServe BF16.
  2. Llama3.1-8B-Instruct on LongBench v1 falls from 75 to 39, and 64K ruler scores fall from 82.83 to 77.61. The accuracy drop is a little bit unacceptable. The degradation on downstream tasks doesn't have further explanation. So, it is suggested to provide a performance explanation of the benchmark degradation and an analysis of which kind of task is suitable for the method.
  3. Please distinct the new design proposed in ARGUS relative to the closest prior methods. For example, QK centering and score correction have been proposed in SageAttention, and 4over6 is Also, an existing technique. Different caching methods for K and V, current token or previous history, are also alread used in current implementations. It is strongly suggested that the authors discuss the contribution compared with the proposed method and the already existing techniques.
Questions for authors
  1. The settings for the calibration data sample count, input length, and how the centers are calibrated for each layer of each model under different tasks.
  2. From Table 5, the accuracy performance mainly comes from 4over6 technique, but other components contribute little. This would reduce the Actual practical value of the proposed method. By the way, the Accuracy performance evaluation study should be evaluated on actual downstream tasks instead of just measuring their quantization error using distance.
  3. Are SageAttention3 and Thrift both using the same HF/SGLang framework for speed-up evaluation, with the same workload and cache settings?
  4. The experiments mainly cover 4-8B Language models. It's recommended to include a smaller or larger model if feasible.
Reviewer_i6NDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ARGUS, a stage-aware NVFP4 attention method that uses different compute and cache precisions for prefill and decoding. It uses Offline Sparse Centering to reduce online preprocessing during prefill, then reads the NVFP4 KV cache and computes attention in BF16 during decoding. The authors evaluate the method on an RTX 5090 through accuracy tests, operator-level and end-to-end performance measurements, and ablations of precision placement.

Critical strengths
  1. The paper accounts for preprocessing and KV-cache costs rather than focusing solely on the main attention kernel. The precision-placement ablation shows that storing the cache in NVFP4 while retaining BF16 decode computation is preferable to using NVFP4 computation in both stages, providing direct support for the stage-aware design.
  2. Alongside operator-level measurements, the paper includes end-to-end experiments in SGLang and multiple accuracy benchmarks. These help establish whether operator-level gains translate into serving improvements and clarify the associated accuracy trade-offs, making the results more convincing.
Critical weaknesses
  1. The accuracy results are less consistent than the paper’s discussion suggests. ARGUS shows substantial degradation on some long-context tasks, including a drop from 75.00 to 39.00 on the LongBench passage-retrieval task for Llama-3.1-8B. These losses deserve closer analysis, since average scores can obscure failures in capabilities the method aims to support.
  2. The end-to-end evidence is concentrated on Qwen3-4B on a single RTX 5090, with 128 generated tokens per request. This leaves the practical benefit under longer generation workloads and other model configurations unclear. Since the design depends on the balance between compute, memory traffic, and conversion overhead, broader serving measurements would materially strengthen the claimed advantage.
Questions for authors
  1. What causes the drop from 75.00 to 39.00 on Llama-3.1-8B’s LongBench passage-retrieval task? Can the existing ablations distinguish the effects of NVFP4 prefill, KV-cache quantization, and offline centering?
  2. Do you have end-to-end results with more than 128 generated tokens per request or with models other than Qwen3-4B? If so, how does the benefit change as decoding accounts for more of the request latency?
  3. How sensitive is Offline Sparse Centering to the calibration data? Do the reported accuracy results use the same fixed centers across all benchmarks, and is there evidence that they remain reliable when the input distribution changes?
Reviewer_ej2HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ARGUS combines NVFP4 prefill attention with a persistent NVFP4 KV cache and BF16 decode computation. Offline sparse Q/K centering replaces input-dependent statistics during prefill. The prefill operator also writes an append-friendly cache that decoding consumes through load-path dequantization. The design targets different bottlenecks in the two stages: prefill computation and preprocessing, and decode cache traffic.

On an RTX 5090, the paper reports up to 4.09× prefill-operator speedup, 2.26× decode-operator speedup, and 2.40× end-to-end speedup for Qwen3-4B with 128K input tokens and 128 generated tokens. Evaluation includes reasoning, long-context tasks, precision-placement ablations, and serving measurements. Quality is close to BF16 in several settings, but some model–task combinations show substantial losses. The main open issues concern task-dependent quality losses and the gains from the proposed centering and prefill path beyond compressed-cache decoding.

Critical strengths
  1. The precision-placement study supports treating prefill, cache storage, and decode arithmetic separately. NVFP4 cache storage with BF16 decode achieves a 2.04× decode speedup, whereas extending NVFP4 to decode arithmetic slows it to 0.62×. Combining the cache with NVFP4 prefill gives a 3.23× prefill speedup. This is useful evidence that lower arithmetic precision can lose its benefit once conversion and layout costs are included.

  2. The evaluation measures complete operators and serving behavior, including cache construction, different output lengths, absolute latency on a second model, and capacity. Qwen3-8B end-to-end latency falls from 12,582.44 to 7,764.84 ms at 64K with matched YaRN settings. The reported 3.0–3.5× capacity increase also shows a practical benefit of cache compression beyond kernel speed.

  3. The cache layout connects prefill and incremental decoding without requantizing old cache entries. The centering compensation preserves pre-quantization softmax semantics for fixed centers. Quality results also show useful operating points: Qwen3-8B retains a RULER average of 81.52 versus 81.64 for BF16, and GSM8K accuracy of 88.93 versus 89.08.

Critical weaknesses
  1. ARGUS shows substantial accuracy losses on several evaluated tasks despite the claim in Section 4.2 that it consistently stays close to BF16. In Table 2, Llama-3.1-8B-Instruct falls from 17.51 to 15.07 on average, with passage retrieval dropping from 75.00 to 39.00. Table 7 reports a GSM8K drop from 32.60 to 26.31 for Llama-3.1-8B, and Table 3 reports a LongBench-v2 drop from 34.19 to 31.21 for Qwen3-8B.

  2. The offline sparse centering study lacks a matched comparison of latency and downstream quality. In Table 5, centering reduces MSE from 0.012553 to 0.003569, while sparsifying the centers after 4over6 changes it only from 0.003051 to 0.003025. Table 11 compares whole methods, and Table 15 reports reconstruction metrics for additional calibration variants. These results do not report operator latency or downstream differences between online dense, offline dense, and offline sparse centering, or sensitivity to the fixed mask cj=μj1(∣μj∣≥Mg/24)c_j=\mu_j\mathbf{1}(|\mu_j|\geq M_g/24) in Equation 8.

  3. The KV-only baseline is absent from the downstream and end-to-end serving comparisons. In Table 4, A2 and ARGUS both achieve 2.04× decode speedup, while A2 has lower output error: MSE 0.0872 versus 0.2846. ARGUS adds a 3.23× prefill speedup, but the corresponding task-quality and total-latency differences between A2 and ARGUS are not reported.

Questions for authors
  1. Provide the calibration data, sample count, lengths, held-out split, and cost for Equation 2. Were any evaluation examples used to select the centers or the threshold in Equation 8? How does the fixed calibration transfer to longer contexts and different domains?

  2. Compare online dense, offline dense, and offline sparse centering with the cache layout, quantizer, and kernel schedule fixed. Report operator latency, retained-channel fractions, and downstream quality, including sensitivity to the threshold in Equation 8. Evaluate A2 from Table 4 on the same serving and affected quality workloads as ARGUS, reporting TTFT, TPOT, total latency, and task scores under matched execution settings. For Figure 5, specify backend versions, chunked-prefill, CUDA-graph and scheduling settings, timing repetitions, and variability. State whether it uses the repetition protocol reported for Table 14 in Appendix F.

  3. Reconcile the 67.75% Qwen3-8B BF16 AIME25 result in Table 1 with the protocol in Table 6. Averaging binary correctness over 30×8=24030\times8=240 responses gives increments of 100/240≈0.4167100/240\approx0.4167 percentage points, so 67.75% cannot result from that protocol after rounding to two decimals. Is this a typo or a different aggregation? Provide question-level paired uncertainty for the small AIME25 differences, accounting for repeated samples of each problem.

1 comment

Thank you for the detailed rebuttal and additional experiments. The authors have adequately addressed my major concerns, particularly regarding the KV-only comparison, accuracy evaluation, and calibration analysis. I am satisfied with the clarifications and will maintain my positive assessment.Good luck.

Reviewer_oUDuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ARGUS uses four-bit floating point (NVFP4) differently in the two stages of language model inference. During prefill, when the model processes the prompt, it uses NVFP4 attention and subtracts precomputed offsets from selected query and key channels. This avoids computing centering statistics for each request. The quantized keys can go straight into the persistent key-value (KV) cache, while values are packed separately in a layout that supports appending one token at a time. During decoding, ARGUS reads the compressed cache and converts blocks to BF16 as it loads them for attention. On an RTX 5090, the authors report up to 2.40× end-to-end speedup over their BF16 SGLang baseline for Qwen3-4B with a 128K-token prompt and 128 generated tokens. Accuracy changes vary by model and task.

Critical strengths

The paper measures costs that a kernel benchmark can miss. It includes preprocessing and cache construction in prefill latency and explains why the format that works well for prefill arithmetic differs from the format needed for incremental cache updates. In particular, prefill packs values across tokens, while the persistent cache packs them within each token so that appending a token does not require repacking earlier ones.

The precision ablation supports the decode design. At a 4K cache length, Table 4 reports the same 2.04× decode speedup for the KV-only configuration and full ARGUS. Performing decode attention itself in NVFP4 slows the tested implementation to 0.62× the BF16 baseline. This comparison helps explain why the authors compress cache storage and keep the decode arithmetic in BF16.

Critical weaknesses

The main serving result does not isolate ARGUS’s additional value over KV-cache compression. Table 4 includes a useful KV-only control with BF16 prefill, an NVFP4 cache, and BF16 decode. Figure 5 reports the headline serving speedup against a BF16 baseline without showing that control in the same serving experiment. Since the KV-only configuration matches ARGUS’s decode speedup in Table 4, an end-to-end comparison would show how much time the proposed prefill path actually saves. Appendix F’s absolute latency breakdown for Qwen3-8B is helpful.

A comparable breakdown for the headline Qwen3-4B workload would make the 2.40× result easier to interpret. The paper establishes the value of centering more clearly than the value of making it sparse. Table 5 shows a substantial reconstruction improvement from Q/K centering, while dense and sparse centering with 4over6 produce nearly identical results. The introduction would be clearer if it separated the runtime benefit of moving statistics offline from the claimed robustness benefit of dropping small offsets, and identified which measurements support each.

Questions for authors

Can you add the Table 4 KV-only configuration to the SGLang serving comparison, using the same decode kernel and server settings as ARGUS? For the 128K Qwen3-4B result, please report absolute time to first token, time per output token, and total latency, and specify the BF16 attention backend and chunked-prefill settings.

Can you report the Table 4 precision configurations on Llama’s passage-retrieval and 64K RULER tasks? Which stage accounts for most of the quality loss?

Where does sparse offline centering improve on dense offline centering in measured latency or downstream quality? How sensitive is that difference to the calibration data?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarify Regarding Reviewer 16hD's Follow-up

We appreciate the reviewer's acknowledgment of our resolved concerns. However, several statements in the follow-up overlook existing evidence or mischaracterize prior techniques, which lowers the confidence of the reviewer's judgement and implies that the reviewer does not understand the whole field (not professional enough). We respectfully clarify these points for the Area Chair.

(1) Complete re-evaluation of Table 2. Contrary to the reviewer's suggestion that Table 2 still needs to be rerun, we have already rerun all four methods (BF16, SageAttention3, ThriftAttention, and ARGUS) on both Llama-3.1-8B-Instruct and Qwen3-8B under the corrected LongBench-E evaluation, which are already in the revised script. We also include the explanation in the response to W1.

MethodQasperMultiFieldQAHotpotQA2WikiMQAGovReportMultiNewsTRECTriviaQASAMSumPassageCPassageRLCCRepoBench-PAvg
BF1643.2455.6160.5246.7334.1825.7571.3391.8542.7016.8999.6766.8252.4954.44
Sage343.0355.8957.8746.1533.4425.2770.0091.6741.6811.8199.3368.1054.9853.79
Thrift42.9550.6850.4341.5732.9524.4062.0088.5341.358.5196.5054.5947.1949.36
ARGUS43.5255.5458.6442.4633.5325.0869.3392.0641.9314.2098.3369.3856.7553.90

Table: Llama-3.1-8B-Instruct (RoPE)

MethodQasperMultiFieldQAHotpotQA2WikiMQAGovReportMultiNewsTRECTriviaQASAMSumPassageCPassageRLCCRepoBench-PAvg
BF1641.1351.3159.9646.5933.5023.4270.3389.1342.5216.6499.1768.0360.1653.99
Sage340.7948.6657.1443.4933.3623.4970.3387.8441.3311.2893.5067.9457.9252.08
Thrift39.8248.2553.7943.2533.1822.9768.6786.9539.916.7994.0063.4152.6850.28
ARGUS40.0649.1456.0243.2833.6023.3668.3387.1142.1412.0094.3369.8659.5252.21

Table: Qwen3-8B(YaRN)

(2) Mischaracterization of SmoothQuant. The reviewer describes Q/K centering as "SmoothQuant-style per-channel mean subtraction." However, SmoothQuant employs per-channel multiplicative scaling with corresponding weight compensation and not mean subtraction. The reviewer do not understand one of the foundation works of quantization and misuse it.

(3) Misinterpretation of baseline comparisons. All four methods were evaluated under matched HF settings, with speedups of 1.00×/0.52×/0.74×/1.13× on Llama (already in the original script). SGLang comparisons independently use identical serving settings for ARGUS and BF16. No claimed acceleration arises from cross-framework comparisons.

(4) Decoding acceleration and matched ablations. ARGUS achieves 1.31× TPOT acceleration on Qwen3-8B at 64K context. Matched KV-only and full ARGUS achieve separately 1.04× and 1.62× E2E speedups.

(5) Accuracy–efficiency trade-off and novelty. We admit degradation on precision-sensitive RULER 64K tasks: 5% although some long context retrieval subsets may degrade more, but ARGUS (77.61%) still remains close to the best of sage3 (78.21%) and thrift (28.70%), but Llama LongBench-E retains 53.90 vs. 54.44 BF16 accuracy with 1.13× HF acceleration. Offline centering also improves Qwen3-8B accuracy from 49.50 to 52.21, demonstrating benefits beyond naive quantization.

We respectfully ask the Area Chair to consider these corrected results, controlled comparisons, and technical clarifications when assessing the remaining concerns regarding soundness, practical efficiency, and novelty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.