- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes NibbleMLA, an FP4 quantization and computation method for Multi-head Latent Attention (MLA). The method consists of several steps: (1) applying a Hadamard co-rotation to the shared KV latent, including both NoPE and RoPE dimensions, and the corresponding queries; (2) quantizing the latent into FP4 with a single E4M3 scale per token; (3) computing QK directly using FP4 Tensor Cores; (4) performing softmax; (5) multiplying the softmax probabilities by the corresponding per-token E4M3 scales before quantizing them into FP4; and (6) computing PV in FP4 and applying the inverse Hadamard transformation to the output.
The resulting cache requires only 0.44× the storage of the original FP8 cache, while maintaining comparable accuracy across multiple benchmarks. The proposed implementation also achieves 1.10–1.27× speedup in decode-attention kernel latency compared with the FP8 baseline.
Critical strengths
-
An interesting and practical approach to enabling FP4 attention. The paper proposes an effective way to address the scale-axis mismatch in MLA by folding the per-token E4M3 scales of the KV cache into the softmax probabilities before FP4 quantization. This enables both QK and PV to operate directly on the FP4 cache without dequantization to BF16.
-
A relevant problem motivated by real-world LLM serving. The paper focuses on MLA, which is already used in production LLMs, and addresses the practical challenges of KV cache compression and efficient attention computation. The reported memory savings, competitive accuracy, and decode-attention kernel speedups demonstrate the practical value of the proposed approach.
Critical weaknesses
-
Incomplete experimental evaluation and insufficiently polished presentation. The manuscript feels somewhat unfinished in places, particularly the discussion in Section 5.3. More importantly, several evaluations are missing, including prefill kernel performance and a latency breakdown of the Hadamard transformations and other additional operations. The experiments also do not clearly specify the exact Blackwell GPU, which affects reproducibility. Given the available space in the main text (9 pages in ICLR), some important implementation and experimental details from the appendix could be incorporated to improve clarity and completeness. These limitations make it difficult to fully assess the practical benefits and overheads of the proposed method.
-
Limited performance improvement over the existing FP4 baseline. Although the paper reports a 1.06× average speedup over the 288-byte FP4 baseline, this improvement is largely driven by an unusually slow baseline result at batch size 8 and top-k 512. Excluding this configuration, the average speedup drops to approximately 1.02×, indicating only a marginal improvement. This raises concerns about whether directly computing PV in FP4 provides a meaningful performance advantage over the existing FP4 cache implementation that dequantizes to BF16, particularly considering the additional transformations and quantization operations introduced by NibbleMLA.
Questions for authors
-
Accuracy comparison with group-wise FP4 quantization. While the paper shows that Hadamard rotation improves per-token FP4 quantization, could the authors provide a more controlled comparison against per-16-channel FP4 quantization, isolating the effects of Hadamard rotation and scale search, to clarify whether the proposed approach actually achieves better quantization accuracy?
-
Kernel optimization breakdown and the benefit of FP4 PV. Given the limited speedup over the existing 288-byte FP4 baseline, could the authors provide a kernel-level latency breakdown or ablation to clarify the actual benefit of computing PV directly with FP4 MMA rather than dequantizing to BF16, and whether dequantization-based compression approaches such as TurboQuant might offer a more favorable accuracy–performance trade-off?
3 comments
Response to Reviewer aZwx (2/2): controlled per-16 comparison, benefit of FP4 PV, TurboQuant
Q1 Controlled comparison with per-16-channel FP4. We added this comparison on the serving path (new Table 2). Every arm serves DeepSeek-V4-Flash with the cache format under test, and we score 24 LongBench-v2 contexts of 32K tokens with teacher forcing (786K predicted tokens per arm). Reported: increase in next-token NLL over the FP8 cache (×10⁻³, 95% CI over documents). Two FP8 runs are bitwise identical, so all differences come from the format.
| Cache format | B/token | FP4 PV possible | ΔNLL vs FP8 |
|---|---|---|---|
| per-16 (DeepSeek-V4.1 layout) | 288 | no | +18.4 [13.9, 23.1] |
| per-16 + per-block scale search | 288 | no | +8.3 [5.4, 11.7] |
| per-16 + H512 | 288 | no | +8.8 [6.2, 11.8] |
| per-16 + H512 + per-block search | 288 | no | +1.9 [0.1, 3.8] |
| per-token, no rotation | 257 | yes | +30.5 [23.1, 38.6] |
| per-token + H512 | 257 | yes | +4.9 [1.9, 8.3] |
| per-token + H512 + search (NibbleMLA), FP4 attention | 257 | yes | +2.3 [0.1, 4.7] |
NibbleMLA is far more accurate than the per-16 cache as deployed (better on 24 of 24 documents). Rotation and search help both layouts, and with both applied the two reach the same accuracy (paired difference −0.4, CI [−2.1, +1.2]). The per-16 layout then still needs 31 more bytes per token and still cannot feed PV in FP4. The per-token layout, the only one both GEMMs can consume, therefore costs no accuracy once the latent is co-rotated.
Q2 What computing PV in FP4 buys. We isolated PV at the kernel level. In our kernel, we replaced only the FP4 PV GEMM with a BF16 one on the same 257 B cache: V is converted from E2M1 and transposed in shared memory, and P stays in BF16 without any quantization. Replacing only the FP4 PV makes the kernel about 5× slower (0.16–0.22× the speed of the FP8 kernel on the nine shapes), whereas NibbleMLA with FP4 PV is 1.10–1.27× faster than FP8 (revised App. H). The reason is the volume of work:
- a 64-token block of V^T holds 512 × 64 = 32,768 values that a BF16 PV must convert;
- the P tile it multiplies holds 64 × 64 = 4,096 values, which is all that FP4 PV quantizes.
The official kernel avoids that penalty only by dequantizing each tile to BF16 once and using it for both GEMMs. NibbleMLA performs no conversion of the cache at all and is faster on 12 of 13 shapes while reading 11% fewer bytes. Stage timers (App. H) show that the MMA warp issues the FP4 QK and PV MMAs for only 20–24% of its time. The remaining time is the softmax path that every attention kernel pays. Blackwell Ultra doubles the throughput of the exponential unit and raises FP4 throughput to 3× that of FP8, which shifts the balance further toward FP4 attention. End to end, the same design yields 1.21–1.38× serving throughput when the KV cache is the bottleneck (part 1/2). In accuracy, FP4 attention on the NibbleMLA cache matches BF16 attention on the same cache (+0.2×10⁻³ NLL, CI [−1.2, +1.8]).
TurboQuant. We implemented TurboQuant's MSE-optimal codebook (16-level Lloyd–Max) in our serving harness: same H512 rotation, same per-token E4M3 scale, same 257 B (Table 2).
- Offline, its codebook reconstructs the latent slightly better than E2M1.
- End to end, this does not translate into better accuracy: ΔNLL +2.8 [1.1, 4.7] vs +2.3 for NibbleMLA (paired difference +0.5, CI [−1.5, +2.2]).
- Serving cost: its 4-bit codes are table indices, not tensor-core operands. Every attention read must first decode them, so at best it runs like the dequantizing 288 B kernel that NibbleMLA already outperforms.
- No serving data: the TurboQuant paper reports no decode kernel or serving throughput.
At equal accuracy, NibbleMLA is the format that runs both GEMMs natively on FP4 tensor cores. TurboQuant is now discussed in the related work.
Follow-up: presentation, breakdown, and per-16 comparison
Dear Reviewer aZwx, thank you again for your detailed review. As the discussion period closes, a brief recap of what we added for your points:
- Presentation: the 10th page now holds the controlled ablation, stage breakdown, end-to-end serving, the exact GPU, and a summary for §5.3.
- Speedup is stable, not an outlier: 1.075 / 1.075 / 1.074× over the official FP4 kernel in three further rounds, faster on 12 of 13 shapes; replacing only FP4 PV by BF16 PV makes the kernel ~5× slower, which shows what FP4 PV buys.
- Prefill and overheads: per-operation costs and prefill are reported (App. I); they are hidden end to end, while long-form generation throughput rises 1.38× on GB200 and 2.19× on B200.
- Controlled per-16 comparison (Q1): per-16 as deployed is far behind (24/24 documents); with the same rotation and search it ties NibbleMLA but needs 11% more bytes and cannot run PV in FP4. TurboQuant is statistically equal in accuracy but cannot feed tensor cores.
We would be grateful if you could let us know whether these address your concerns, and if so, consider updating your score.
Response to Reviewer aZwx (1/2): presentation, GPU, prefill, overhead breakdown, and why the speedup is real
Thank you for the detailed review and for recognizing that folding the per-token scale into P is what lets both GEMMs read the FP4 cache directly. We answer in two parts. All new material is marked in blue in the revised PDF.
W1 Presentation and missing evaluations. We used the 10th page for the evaluations you list:
GPU: NVIDIA GB200 (Blackwell, compute capability 10.0, 152 SMs, 185 GiB HBM3e, 2062 MHz max SM clock, CUDA 13.0).
Prefill: the stored cache is read through a dequantization kernel, after which prefill runs the stock FlashMLA sparse prefill, so prefill compute is unchanged. Reading the cache costs 2.0 µs per layer for 128 tokens and 11.3 µs for 16K tokens. In serving, time to first token equals the official configuration at equal load and is far lower when the KV cache is the bottleneck (below).
Hadamard and other added operations, per layer and decode step (profiles of DeepSeek-V4.1-Flash serving, App. I; last row from Table 3):
These costs do not show up end to end: at equal batch the forward pass takes the same time (batch 256: 17.03 vs 17.05 ms; 65K context, batch 200: 15.70 vs 15.63 ms).
W2 The speedup over the official FP4 kernel is stable, not driven by an outlier.
index_topk = 512), so this case is part of normal decoding, not a corner case to set aside. Across all top-k 512 shapes from batch 8 to 512, NibbleMLA is faster at every batch size, with 1.11× lower total latency.This holds while NibbleMLA reads 11% fewer bytes and runs both GEMMs in FP4. Part 2/2 shows with a kernel-level ablation that FP4 PV is what makes this possible.
The decisive benefit is at the system level. We designed NibbleMLA for long-input, long-output serving such as long-form writing and document continuation, where the KV cache limits how many requests run at once. In SGLang serving of DeepSeek-V4.1-Flash on four GB200 GPUs, NibbleMLA replaces the official cache (C8A16 sliding window + C4A16 compressed) with C4A4 for both (§5.6):