acceptodds
Under review as a conference paper at ICLR 2027

NibbleMLA: Dual-Axis FP4 Attention over a Single Latent Cache

Abstract

Multi-head latent attention (MLA) reduces the key-value cache to a single latent per token that serves as both key and value. What remains is numeric: how many bits store that latent, and in what precision the two attention GEMMs read it. FP4 latent caches already exist, but they dequantize to BF16 before attention. We show that this is forced by a structural mismatch rather than by accuracy: block-scaled FP4 tensor cores accept scale factors only along the contraction dimension, whereas the shared latent is contracted along channels by QK and along tokens by PV. We prove that a scale written once per token can be consumed by both GEMMs only if it is a per-token scalar times a static channel vector—precisely the layout that suits raw latents worst, because their channel magnitudes are highly uneven. NibbleMLA resolves this tension by changing the basis instead of the scale: a fixed -dimensional Hadamard co-rotation, which costs no storage, equalizes the channels so that one E4M3 scalar per token suffices. The resulting -byte-per-token cache ( FP8) feeds both attention GEMMs on FP4 tensor cores without dequantization. On DeepSeek-V4-Flash, FP4 attention matches BF16 attention computed on the same cache (output cosine in all layers); end-task accuracy is statistically indistinguishable from the FP8 cache on GSM8K, AIME, and LongBench-v2 and within points on -task RULER at K; and the decode-attention kernel is – faster than the FP8 kernel and, on average, faster than the official FP4 kernel, which dequantizes.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Multi-head Latent Attention, FP4 attention, KV cache compression, paged attention, long-context inference, low-precision inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aZwxOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes NibbleMLA, an FP4 quantization and computation method for Multi-head Latent Attention (MLA). The method consists of several steps: (1) applying a Hadamard co-rotation to the shared KV latent, including both NoPE and RoPE dimensions, and the corresponding queries; (2) quantizing the latent into FP4 with a single E4M3 scale per token; (3) computing QK directly using FP4 Tensor Cores; (4) performing softmax; (5) multiplying the softmax probabilities by the corresponding per-token E4M3 scales before quantizing them into FP4; and (6) computing PV in FP4 and applying the inverse Hadamard transformation to the output.

The resulting cache requires only 0.44× the storage of the original FP8 cache, while maintaining comparable accuracy across multiple benchmarks. The proposed implementation also achieves 1.10–1.27× speedup in decode-attention kernel latency compared with the FP8 baseline.

Critical strengths
  1. An interesting and practical approach to enabling FP4 attention. The paper proposes an effective way to address the scale-axis mismatch in MLA by folding the per-token E4M3 scales of the KV cache into the softmax probabilities before FP4 quantization. This enables both QK and PV to operate directly on the FP4 cache without dequantization to BF16.

  2. A relevant problem motivated by real-world LLM serving. The paper focuses on MLA, which is already used in production LLMs, and addresses the practical challenges of KV cache compression and efficient attention computation. The reported memory savings, competitive accuracy, and decode-attention kernel speedups demonstrate the practical value of the proposed approach.

Critical weaknesses
  1. Incomplete experimental evaluation and insufficiently polished presentation. The manuscript feels somewhat unfinished in places, particularly the discussion in Section 5.3. More importantly, several evaluations are missing, including prefill kernel performance and a latency breakdown of the Hadamard transformations and other additional operations. The experiments also do not clearly specify the exact Blackwell GPU, which affects reproducibility. Given the available space in the main text (9 pages in ICLR), some important implementation and experimental details from the appendix could be incorporated to improve clarity and completeness. These limitations make it difficult to fully assess the practical benefits and overheads of the proposed method.

  2. Limited performance improvement over the existing FP4 baseline. Although the paper reports a 1.06× average speedup over the 288-byte FP4 baseline, this improvement is largely driven by an unusually slow baseline result at batch size 8 and top-k 512. Excluding this configuration, the average speedup drops to approximately 1.02×, indicating only a marginal improvement. This raises concerns about whether directly computing PV in FP4 provides a meaningful performance advantage over the existing FP4 cache implementation that dequantizes to BF16, particularly considering the additional transformations and quantization operations introduced by NibbleMLA.

Questions for authors
  1. Accuracy comparison with group-wise FP4 quantization. While the paper shows that Hadamard rotation improves per-token FP4 quantization, could the authors provide a more controlled comparison against per-16-channel FP4 quantization, isolating the effects of Hadamard rotation and scale search, to clarify whether the proposed approach actually achieves better quantization accuracy?

  2. Kernel optimization breakdown and the benefit of FP4 PV. Given the limited speedup over the existing 288-byte FP4 baseline, could the authors provide a kernel-level latency breakdown or ablation to clarify the actual benefit of computing PV directly with FP4 MMA rather than dequantizing to BF16, and whether dequantization-based compression approaches such as TurboQuant might offer a more favorable accuracy–performance trade-off?

3 comments
AuthorsOpenReview ↗

Response to Reviewer aZwx (1/2): presentation, GPU, prefill, overhead breakdown, and why the speedup is real

Thank you for the detailed review and for recognizing that folding the per-token scale into P is what lets both GEMMs read the FP4 cache directly. We answer in two parts. All new material is marked in blue in the revised PDF.

W1 Presentation and missing evaluations. We used the 10th page for the evaluations you list:

  • a controlled ablation of rotation, search and layout (new Table 2);
  • kernel stability and an in-kernel stage breakdown (§5.5, App. H);
  • a new end-to-end serving section (§5.6, App. I);
  • the exact GPU in §5.1, and a closing summary for §5.3.

GPU: NVIDIA GB200 (Blackwell, compute capability 10.0, 152 SMs, 185 GiB HBM3e, 2062 MHz max SM clock, CUDA 13.0).

Prefill: the stored cache is read through a dequantization kernel, after which prefill runs the stock FlashMLA sparse prefill, so prefill compute is unchanged. Reading the cache costs 2.0 µs per layer for 128 tokens and 11.3 µs for 16K tokens. In serving, time to first token equals the official configuration at equal load and is far lower when the KV cache is the bottleneck (below).

Hadamard and other added operations, per layer and decode step (profiles of DeepSeek-V4.1-Flash serving, App. I; last row from Table 3):

OperationDeepSeek-V4.1 official (C8A16 + C4A16)NibbleMLA (C4A4)
query RoPE (+ H512 + NVFP4 quantization)2.0 µs3.7 µs
cache write (+ rotation, quantization, 3-way search)2.1 µs≈ 3.5 µs
inverse rotation of the output–folded into the split-combine kernel
decode-attention kernel, (128, 2048)55.8 µs55.0 µs

These costs do not show up end to end: at equal batch the forward pass takes the same time (batch 256: 17.03 vs 17.05 ms; 65K context, batch 200: 15.70 vs 15.63 ms).

W2 The speedup over the official FP4 kernel is stable, not driven by an outlier.

  • Not noise. Three additional interleaved rounds on the same GPU give a mean speedup over the official DeepSeek-V4.1 FP4 kernel (C4A16, 288 B) of 1.075, 1.075 and 1.074 on the nine shapes. The official kernel takes 18.36–18.37 µs at (8, 512) in every round, against 11.0 µs for NibbleMLA. This is a reproducible property of the official kernel at that shape.
  • A shape the model actually uses. Top-k 512 is DeepSeek-V4-Flash's native indexer setting (index_topk = 512), so this case is part of normal decoding, not a corner case to set aside. Across all top-k 512 shapes from batch 8 to 512, NibbleMLA is faster at every batch size, with 1.11× lower total latency.
  • Consistent elsewhere. Over 13 measured shapes, NibbleMLA is faster on 12, and within 1% on the remaining one. At batch 256–512, not covered in the paper, it is 1.04–1.10× faster than the official kernel and 1.05–1.18× faster than FP8 (App. H).

This holds while NibbleMLA reads 11% fewer bytes and runs both GEMMs in FP4. Part 2/2 shows with a kernel-level ablation that FP4 PV is what makes this possible.

The decisive benefit is at the system level. We designed NibbleMLA for long-input, long-output serving such as long-form writing and document continuation, where the KV cache limits how many requests run at once. In SGLang serving of DeepSeek-V4.1-Flash on four GB200 GPUs, NibbleMLA replaces the official cache (C8A16 sliding window + C4A16 compressed) with C4A4 for both (§5.6):

  • KV capacity: 1.27× (throughput-first recipe, no speculative decoding) and 1.90× (low-latency recipe, DSpark speculative decoding).
  • Output throughput: 1.38× at 24K in / 16K out with 256 concurrent requests, and 1.21× at 16K/16K.
  • Median time to first token: 238 s → 15 s and 55 s → 12 s.
AuthorsOpenReview ↗

Response to Reviewer aZwx (2/2): controlled per-16 comparison, benefit of FP4 PV, TurboQuant

Q1 Controlled comparison with per-16-channel FP4. We added this comparison on the serving path (new Table 2). Every arm serves DeepSeek-V4-Flash with the cache format under test, and we score 24 LongBench-v2 contexts of 32K tokens with teacher forcing (786K predicted tokens per arm). Reported: increase in next-token NLL over the FP8 cache (×10⁻³, 95% CI over documents). Two FP8 runs are bitwise identical, so all differences come from the format.

Cache formatB/tokenFP4 PV possibleΔNLL vs FP8
per-16 (DeepSeek-V4.1 layout)288no+18.4 [13.9, 23.1]
per-16 + per-block scale search288no+8.3 [5.4, 11.7]
per-16 + H512288no+8.8 [6.2, 11.8]
per-16 + H512 + per-block search288no+1.9 [0.1, 3.8]
per-token, no rotation257yes+30.5 [23.1, 38.6]
per-token + H512257yes+4.9 [1.9, 8.3]
per-token + H512 + search (NibbleMLA), FP4 attention257yes+2.3 [0.1, 4.7]

NibbleMLA is far more accurate than the per-16 cache as deployed (better on 24 of 24 documents). Rotation and search help both layouts, and with both applied the two reach the same accuracy (paired difference −0.4, CI [−2.1, +1.2]). The per-16 layout then still needs 31 more bytes per token and still cannot feed PV in FP4. The per-token layout, the only one both GEMMs can consume, therefore costs no accuracy once the latent is co-rotated.

Q2 What computing PV in FP4 buys. We isolated PV at the kernel level. In our kernel, we replaced only the FP4 PV GEMM with a BF16 one on the same 257 B cache: V is converted from E2M1 and transposed in shared memory, and P stays in BF16 without any quantization. Replacing only the FP4 PV makes the kernel about 5× slower (0.16–0.22× the speed of the FP8 kernel on the nine shapes), whereas NibbleMLA with FP4 PV is 1.10–1.27× faster than FP8 (revised App. H). The reason is the volume of work:

  • a 64-token block of V^T holds 512 × 64 = 32,768 values that a BF16 PV must convert;
  • the P tile it multiplies holds 64 × 64 = 4,096 values, which is all that FP4 PV quantizes.

The official kernel avoids that penalty only by dequantizing each tile to BF16 once and using it for both GEMMs. NibbleMLA performs no conversion of the cache at all and is faster on 12 of 13 shapes while reading 11% fewer bytes. Stage timers (App. H) show that the MMA warp issues the FP4 QK and PV MMAs for only 20–24% of its time. The remaining time is the softmax path that every attention kernel pays. Blackwell Ultra doubles the throughput of the exponential unit and raises FP4 throughput to 3× that of FP8, which shifts the balance further toward FP4 attention. End to end, the same design yields 1.21–1.38× serving throughput when the KV cache is the bottleneck (part 1/2). In accuracy, FP4 attention on the NibbleMLA cache matches BF16 attention on the same cache (+0.2×10⁻³ NLL, CI [−1.2, +1.8]).

TurboQuant. We implemented TurboQuant's MSE-optimal codebook (16-level Lloyd–Max) in our serving harness: same H512 rotation, same per-token E4M3 scale, same 257 B (Table 2).

  • Offline, its codebook reconstructs the latent slightly better than E2M1.
  • End to end, this does not translate into better accuracy: ΔNLL +2.8 [1.1, 4.7] vs +2.3 for NibbleMLA (paired difference +0.5, CI [−1.5, +2.2]).
  • Serving cost: its 4-bit codes are table indices, not tensor-core operands. Every attention read must first decode them, so at best it runs like the dequantizing 288 B kernel that NibbleMLA already outperforms.
  • No serving data: the TurboQuant paper reports no decode kernel or serving throughput.

At equal accuracy, NibbleMLA is the format that runs both GEMMs natively on FP4 tensor cores. TurboQuant is now discussed in the related work.

AuthorsOpenReview ↗

Follow-up: presentation, breakdown, and per-16 comparison

Dear Reviewer aZwx, thank you again for your detailed review. As the discussion period closes, a brief recap of what we added for your points:

  • Presentation: the 10th page now holds the controlled ablation, stage breakdown, end-to-end serving, the exact GPU, and a summary for §5.3.
  • Speedup is stable, not an outlier: 1.075 / 1.075 / 1.074× over the official FP4 kernel in three further rounds, faster on 12 of 13 shapes; replacing only FP4 PV by BF16 PV makes the kernel ~5× slower, which shows what FP4 PV buys.
  • Prefill and overheads: per-operation costs and prefill are reported (App. I); they are hidden end to end, while long-form generation throughput rises 1.38× on GB200 and 2.19× on B200.
  • Controlled per-16 comparison (Q1): per-16 as deployed is far behind (24/24 documents); with the same rotation and search it ties NibbleMLA but needs 11% more bytes and cannot run PV in FP4. TurboQuant is statistically equal in accuracy but cannot feed tensor cores.

We would be grateful if you could let us know whether these address your concerns, and if so, consider updating your score.

Reviewer_zL7FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes NibbleMLA, which combines FP4 storage and FP4 computation for the shared MLA latent cache (K = V). The central challenge is that NVFP4 block scales must align with the contraction dimension, while QK contracts over channels and PV contracts over tokens. Consequently, a conventional per-16-channel scale layout cannot be consumed directly by both GEMMs. Under the write-once constraint, the paper shows that the compatible scale layouts must factor into a per-token scalar and a static channel vector. The proposed solution applies a fixed 512-dimensional Hadamard co-rotation to the latent and query. This adds no cache storage and spreads energy across channels, making a single E4M3 scale per token sufficiently accurate. The method achieves modest attention-kernel speedups while largely preserving downstream accuracy.

Critical strengths
  1. I find the central idea interesting and well motivated. The paper connects a hardware constraint to a concrete quantization design, and the Hadamard co-rotation provides a natural solution. Although the current speedups are modest, the approach may become more beneficial on future hardware.

  2. The accuracy degradation is generally small across the evaluated benchmarks, despite using FP4 for both cache storage and attention computation.

  3. The evaluation uses actual quantization and inference within a real inference framework. I also appreciate the candid reporting of the speedups and their limitations.

Critical weaknesses
  1. The speedups remain modest, particularly relative to the existing FP4-cache baseline. This limits the demonstrated performance benefit of native FP4 computation.

  2. A comparison against FP4 cache storage with FP8 attention computation would help clarify the tradeoff between storage precision and compute precision, including whether FP4-to-FP8 conversion could offer competitive performance.

Questions for authors

Could the authors discuss the differences from SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving, particularly regarding cache layout and scale handling?

2 comments
AuthorsOpenReview ↗

Response to Reviewer zL7F: speedup and end-to-end gains, FP4 storage with FP8 compute, SAW-INT4

Thank you for the positive assessment. New results are marked in blue in the revised PDF.

W1 Modest speedups. At the operator level, NibbleMLA's decode attention is 1.10–1.27× faster than the DeepSeek-V4-Flash FP8 kernel (C8A16, 584 B) on every shape, 1.19× on average. It is also faster on average than the official DeepSeek-V4.1 FP4 kernel (C4A16, 288 B), even though it runs both GEMMs in FP4 and stores 11% fewer bytes (Table 3). Above all, the kernel is not where the method pays most: the smaller cache entry raises the number of requests a GPU can serve. We designed NibbleMLA for long-input, long-output serving such as long-form writing and document continuation, where the KV cache limits concurrency; video-grounded writing, whose visual tokens lengthen the context further, is our next target. In SGLang serving of DeepSeek-V4.1-Flash on four GB200 GPUs (new §5.6, App. I), replacing the official cache (C8A16 sliding window + C4A16 compressed) with our C4A4 cache gives:

  • 1.27–1.90× more KV capacity, depending on the serving recipe;
  • 1.38× output throughput at 24K in / 16K out with 256 concurrent requests, and 1.21× at 16K/16K;
  • median time to first token 238 s → 15 s and 55 s → 12 s;
  • unchanged forward time at equal batch, i.e. the rotation and quantization costs are hidden.

W2 FP4 storage with FP8 attention. This is a useful axis to separate, and we added the comparison (new App. J). Proposition 1 constrains 4-bit operands only. An E4M3 operand can absorb a per-16-channel scale into its values: each 64-token block is converted in shared memory to E4M3(code·s₁₆·2⁻ᵉ), with one power of two per block folded into the logits and into P. E4M3 MMAs accept MN-major operands, so PV reads the same tile. We built such a kernel (C4A8) and compared on three axes.

  • Speed. At decode, 8-bit arithmetic is competitive. On DeepSeek-V4.1-Flash decode shapes, our C4A8 kernel over the 288 B cache is 1.18–1.26× faster than the official BF16-arithmetic kernel on layers with 512 compressed entries, because E4M3 needs no two-term quantization of P. At decode, that quantization is part of the softmax path, which dominates our kernel's time (App. H).
  • KV capacity. Capacity is set by the storage format alone, not by the arithmetic. The 257 B format is the smallest of the options and serves any arithmetic: the per-token α folds into the logit scale and P for E4M3 just as it does for FP4.
  • Accuracy. We measured attention-output error (1 − cos, ×10⁻³, mean over all 43 layers) of cache and arithmetic together:
CacheBF16 attn.E4M3 attn.FP4 attn.
per-16, 288 B3.143.49not consumable
NibbleMLA, 257 B2.872.992.90

The 257 B cache is the more accurate store under every arithmetic, and on it NibbleMLA's two-term FP4 arithmetic is closer to BF16 than single-term E4M3 is. End to end, FP4 attention on our cache matches BF16 attention on the same cache (teacher-forced NLL +0.2×10⁻³, CI [−1.2, +1.8]; Table 2).

Why NibbleMLA computes in FP4. Its goal is a single compact cache that is a native operand of the lowest-precision tensor cores for both attention GEMMs, so that memory and compute scale together. FP4 delivers this at no accuracy cost on our cache. FP4 tensor throughput is 2× that of FP8 on Blackwell and 3× on Blackwell Ultra, which also doubles the exponential-unit throughput that bounds today's softmax path. FP4 compute therefore gains as attention does more work per cache byte, for example with larger batches or speculative verification of several tokens per request. The 257 B format also leaves 8-bit arithmetic open as a deployment option. NibbleMLA thus fixes the storage choice that is best under every arithmetic, and pairs it with the arithmetic that has the most headroom.

Q Differences from SAW-INT4.

  • Cache layout: SAW-INT4 stores keys and values as separate INT4 tensors with one scale and one zero point per token and head. MLA has a single latent shared by K and V, so one set of scales must serve both GEMMs, which is exactly what Proposition 1 characterizes.
  • Scale handling: SAW-INT4's asymmetric zero point is not a tensor-core block scale, so its cache must be dequantized inside the attention kernel. NibbleMLA's single E4M3 scale per token is consumed directly, as the key block scale in QK and folded into P in PV.
  • Rotation: SAW-INT4 rotates keys only, with block-diagonal Hadamards (default size 128). With a shared latent the rotation also enters PV and must be undone on the output, which NibbleMLA does with one fixed full-width H512.
  • Accuracy on the MLA latent (revised App. G, offline 1 − cos ×10⁻³): SAW-INT4's quantizer reaches 9.30 without rotation and 3.71 with BDR-128 at 260 B; NibbleMLA reaches 2.87 at 257 B.
  • Scope: SAW-INT4 targets GQA models (Qwen3, GLM-4.7); NibbleMLA targets shared-latent MLA.

SAW-INT4 is now discussed in the related work.

AuthorsOpenReview ↗

Follow-up on FP4 storage vs. FP8 compute and SAW-INT4

Dear Reviewer zL7F, thank you again for your supportive review. In response to your questions we added (i) the FP4-storage / FP8-compute comparison on speed, KV capacity and accuracy (App. J): the 257 B cache is the more accurate store under every arithmetic, and FP4 has the most tensor-core headroom on current and next-generation GPUs; (ii) a point-by-point comparison with SAW-INT4 (layout, scale handling, rotation) and its accuracy on the MLA latent; and (iii) end-to-end serving results: 1.27–1.90× KV capacity and up to 1.38× throughput on GB200 (§5.6), 3.07× and 2.19× on B200. We hope these strengthen your assessment, and we would be happy to clarify anything further.

Reviewer_G8JxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

NibbleMLA studies how a shared MLA latent cache can serve both attention matrix multiplications directly in FP4. The same latent is reduced over channels in QK and over tokens in PV, creating incompatible requirements for conventional block scales. Under independent token writes and direct consumption of unchanged cache codes, the paper characterizes compatible scales as a token scalar times a static channel vector. It then uses a fixed Hadamard co-rotation and a single scale per token, storing 257 bytes per main-latent entry.

Experiments on DeepSeek-V4-Flash compare FP4 and BF16 arithmetic on identical caches, evaluate downstream reasoning and long-context tasks, and measure Blackwell decode-attention kernels. The proposed cache retains similar aggregate quality to FP8 at 0.44 times its main-latent footprint, with measured kernel speedups of 1.10–1.27 times over FP8. The contribution connects a hardware constraint to cache representation and low-precision execution.

Critical strengths
  1. The paper identifies and formalizes the scale-axis mismatch between QK and PV in shared-latent attention, providing a clear motivation for the proposed design.
  2. NibbleMLA enables direct FP4 computation on both attention GEMMs without dequantizing the persistent cache, achieving a compact 257 B/token representation.
  3. The evaluation includes downstream accuracy, numerical error analysis, and kernel benchmarks. The same-cache FP4/BF16 comparison is particularly useful for isolating arithmetic precision effects.
Critical weaknesses
  1. Limited methodological novelty. The solution largely combines established techniques, including Hadamard rotation, per-token quantization, and low-precision attention. Once the hardware constraint is identified, the design appears relatively straightforward. Moreover, the paper does not compare against simpler alternatives, such as static per-channel scaling, which is also permitted by Proposition 1.
  2. Insufficient ablation and methodological details. The contributions of Hadamard rotation and scale search are not isolated. Furthermore, the offline calibration procedure and the reconstruction-based and attention-aware scale-selection objectives are insufficiently specified, making it difficult to assess their necessity and reproduce the method.
  3. Limited performance gains over existing FP4 implementations. The average speedup over the official FP4 kernel is only 1.06×, decreasing to approximately 1.02× after excluding its documented slow shape. The lack of end-to-end evaluation also makes the practical benefits unclear.
Questions for authors
  1. Could the authors compare Hadamard rotation against simpler alternatives, particularly calibrated static per-channel scaling, under the same cache size and computation precision?
  2. Could the authors provide an ablation separating Hadamard rotation and scale search, and clarify the offline calibration procedure, including the calibrated parameter and the attention-aware scale-selection objective?
  3. How do the reported kernel-level improvements translate to end-to-end inference performance, including the overhead of rotation and cache preparation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer G8Jx (1/2): what is new, static per-channel scaling, rotation vs. search

Thank you for the careful review and for recognizing the scale-axis constraint and the same-cache FP4/BF16 comparison. We answer in two parts: this part covers novelty and the ablations (Q1, Q2); part 2/2 covers speed, capacity and end-to-end serving (W3, Q3). New results are marked in blue in the revised PDF.

W1 Novelty. Hadamard rotations, per-token scales and low-bit attention exist individually. What did not exist is a way to make the persistent MLA cache a 4-bit operand of both attention GEMMs. Every FP4 latent cache before ours, including DeepSeek-V4.1's, dequantizes to BF16, and the reason was not known. Proposition 1 identifies it: because one tensor is contracted over channels in QK and over tokens in PV, the only layouts that both block-scaled FP4 GEMMs can consume are a per-token scalar times a static channel vector. This turns an open systems question into a single, precisely posed accuracy problem. The design that follows is not "straightforward" in the sense of following common practice. It contradicts the prevailing evidence that global rotations bring little or even hurt FP4 accuracy when scales cover 16–32 elements: under the one-scale-per-token regime, the full-width co-rotation removes 84% of the accuracy loss (table below). That is the paper's contribution: the constraint, plus the regime in which it becomes cheap to satisfy.

Q1 Static per-channel scaling, same bytes and same arithmetic. We added the controlled ablation you asked for (revised Table 2). Every arm serves DeepSeek-V4-Flash with the cache format under test on the real store path, and we score 24 LongBench-v2 contexts of 32K tokens with teacher forcing (786K predicted tokens per arm). Reported: increase in next-token NLL over the FP8 cache (×10⁻³, 95% bootstrap CI over documents). Two FP8 runs give bitwise-identical log-probabilities, so every difference is caused by the format.

Cache format (257 B unless noted)AttentionΔNLL vs FP8
per-token scaleBF16+30.5 [23.1, 38.6]
+ calibrated static per-16 vector bBF16+18.5 [14.2, 23.2]
+ calibrated static per-16 vector bFP4+13.9 [9.0, 19.2]
+ H512 co-rotationFP4+4.9 [1.9, 8.3]
+ H512 + write-time search (NibbleMLA)FP4+2.3 [0.1, 4.7]
NibbleMLA cacheBF16+2.1 [−0.1, 4.7]

A calibrated static vector, which Proposition 1 permits, recovers only 39% of the loss; the co-rotation recovers 84%, and NibbleMLA beats the static-vector cache on all 24 documents. The reason is structural. A static vector can rescale channels only on average, but the channels that set each token's maximum change from token to token (for example, RoPE pairs rotate with position). The rotation spreads whichever channel is loud over all 512 coordinates, so no calibration is needed. Over all 43 layers, a per-channel vector b_c = RMS_c^γ with γ fitted per layer is no better than the per-16 one.

Q2 Rotation vs. search, calibration, objective. The two components are separated in the same table. Search alone barely moves the per-token cache (+30.5 → +29.7): without rotation, the SSE-optimal scale clips on 98% of tokens and saturates the few dominant channels. After rotation, the per-token kurtosis drops from 9.5 to 2.95 (Gaussian: 3), and the same search lowers the loss by a further 2.6 (paired CI [0.9, 4.5]). The rotation is the main lever; the search is a free refinement on top of it. The rule is now stated exactly (revised Eq. 4), with x~=H512Ct\tilde x = H_{512} C_t:

αt=arg⁡min⁡α∈{⌈rmax⁡c∣x~c∣/6⌉E4M3: r∈{1, 0.85, 0.7}} ∑c(α E2M1(x~c/α)−x~c)2\alpha_t=\arg\min_{\alpha\in\{\lceil r\max_c|\tilde x_c|/6\rceil_{\mathrm{E4M3}}:\ r\in\{1,\,0.85,\,0.7\}\}}\ \sum_c\big(\alpha\,\mathrm{E2M1}(\tilde x_c/\alpha)-\tilde x_c\big)^2

It is computed at write time for every entry and never in the decode path. On calibration: in the raw basis, a label-free calibrated per-16 vector spreads by 2.4–4.8× across channels; in the H512 basis it is flat to within 1.0–1.15×. NibbleMLA therefore needs no calibrated parameter at all (b ≡ 1), and an attention-weighted objective gives no further gain over the reconstruction objective above. §4.2 now says this explicitly.

AuthorsOpenReview ↗

Response to Reviewer G8Jx (2/2): speed is stable, and the end-to-end win is KV capacity

W3 Speedup over the official FP4 kernel. We agree that the decode-attention margin over the official DeepSeek-V4.1 FP4 kernel (C4A16, 288 B) is moderate. However, it is robust, it comes with a cache that is 11% smaller and with both GEMMs in FP4, and kernel latency is not where NibbleMLA pays most: its main payoff is KV capacity (Q3 below). Three additional interleaved rounds on the same GB200 give a mean speedup over the official FP4 kernel of 1.075, 1.075 and 1.074 on the paper's nine shapes, and 1.19× over the DeepSeek-V4-Flash FP8 kernel (C8A16, 584 B) in every round. The official kernel takes 18.36–18.37 µs at (8, 512) in all three rounds, so that point reflects how the official kernel handles the shape, not noise. At larger batches, which the paper did not include, NibbleMLA is 1.04–1.10× faster than the official FP4 kernel and 1.05–1.18× faster than FP8 (revised App. H). In-kernel stage timers explain why the margin is bounded at decode: the tensor-core GEMMs are no longer the bottleneck. At batch ≥ 64, 57–71% of the critical path is softmax and P quantization on CUDA cores, while the FP4 QK and PV MMAs occupy the MMA warp only 20–24% of the time.

Q3 End-to-end serving, including rotation and cache preparation. The main benefit appears at the system level, where the smaller cache entry increases the number of requests a GPU can hold. We built NibbleMLA for long-input, long-output workloads such as long-form writing and document continuation, where the KV cache, not compute, caps concurrency. Video-grounded writing, whose visual tokens lengthen the context further, is the next target. To measure this, we integrated the 257 B cache and the kernel into SGLang for DeepSeek-V4.1-Flash.

Configurations compared (CxAy = x-bit cache, y-bit attention arithmetic):

  • DeepSeek-V4-Flash production: C8A16 in every layer (584 B). This is the FP8 baseline of the paper's accuracy and kernel experiments.
  • DeepSeek-V4.1-Flash official: C8A16 sliding-window cache (528 B) + C4A16 compressed cache (288 B), BF16 attention.
  • NibbleMLA: C4A4 (257 B) for both caches.

Setup. One machine with four GB200 GPUs, TP4/EP4, arms interleaved, two cycles, 95% CI on the log ratio. Two recipes:

  • throughput-first: up to 256 running requests, no speculative decoding;
  • low-latency: DSpark speculative decoding (block 5), CUDA graphs for decode, static memory fraction 0.8.

Results vs. the V4.1 official configuration (new §5.6, App. I):

  • KV capacity: 1.27× (throughput-first) and 1.90× (low-latency) more tokens. Both are conservative, because the planner still sizes the compressed pool of both arms at 584 B.
  • Long-form generation, 24K in / 16K out, 256 concurrent: output throughput 1.38× [1.36, 1.40]; median time to first token 238 s → 15 s.
  • 16K in / 16K out, 256 concurrent: 1.21× [1.19, 1.24]; median TTFT 55 s → 12 s.
  • 64K in / 16K out (throughput-first): 1.04–1.05× throughput, TTFT up to 1.24× lower.
  • Mechanism: in the 64K workload the official cache runs a median of 196 requests with its pool 99% full and 31 queued; NibbleMLA runs all 256.

Overheads. Per layer and decode step, query RoPE with H512 and NVFP4 quantization takes 3.7 µs (official RoPE 2.0 µs). The cache write with rotation, quantization and the 3-way search takes about 3.5 µs (official 2.1 µs). The inverse rotation is folded into the split-combine kernel. These costs are hidden end to end: at equal batch the forward pass takes the same time (batch 256: 17.03 vs 17.05 ms; 65K context, batch 200: 15.70 vs 15.63 ms). In that profile, the compressed-layer attention kernel takes 37–46 µs against 46.6–47.5 µs for the official one. DSpark acceptance lengths of the two arms agree within 1.5%.

In short, NibbleMLA gives a moderate and stable kernel speedup and, more importantly, a larger KV capacity at FP8-level accuracy, which in long-generation serving translates into substantially higher throughput and much lower time to first token.

AuthorsOpenReview ↗

Follow-up: have the new ablations addressed your concerns?

Dear Reviewer G8Jx, thank you again for your review. As the discussion period closes, we briefly recap the additions made for your questions:

  • Static per-channel scaling (Q1) at the same 257 B and arithmetic recovers only 39% of the per-token loss, versus 84% for the co-rotation; NibbleMLA is better on 24/24 documents (Table 2).
  • Rotation vs. search (Q2) are now separated end to end, the exact search rule is given (Eq. 4), and no calibrated parameter is needed (§4.2).
  • End-to-end (Q3): 1.90× KV capacity and 1.38× long-form generation throughput on GB200, and 3.07× and 2.19× on B200, where less memory is left for the cache, with rotation overheads hidden (§5.6, App. I); the kernel margin over the official FP4 kernel is reproducible across rounds.

We would be grateful if you could let us know whether these address your concerns, and if so, consider updating your assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the author response for the AC and reviewers

We thank the reviewers and the AC. Below is a summary of the new evidence; all of it is in the revised PDF (changes in blue).

1. Controlled ablation on the serving path (new Table 2; G8Jx Q1–Q2, aZwx Q1). Teacher-forced NLL on 24 LongBench-v2 contexts of 32K tokens (786K predicted tokens per arm); scoring is deterministic (two FP8 runs are bitwise identical). Increase in NLL over FP8 (×10⁻³):

  • per-token scale 30.5; + calibrated static channel vector (allowed by Prop. 1) 18.5; + H512 4.9; NibbleMLA 2.3.
  • The co-rotation removes 84% of the loss, a static vector only 39%; NibbleMLA is better on 24/24 documents.
  • The per-16 (288 B) cache with the same rotation and search reaches the same accuracy (paired −0.4, CI [−2.1, +1.2]) but needs 31 more bytes per token and cannot feed PV in FP4.
  • FP4 attention vs. BF16 attention on the same NibbleMLA cache: +0.2, CI [−1.2, +1.8].
  • TurboQuant's codebook on the same rotation and 257 B: +2.8, statistically equal to NibbleMLA, while its codes are not tensor-core operands.

2. Kernel (G8Jx W3, aZwx W2, zL7F W1). 1.19× faster than FP8 in every round; three additional interleaved rounds give 1.075 / 1.075 / 1.074× over the official FP4 kernel; faster on 12 of 13 shapes; 1.04–1.10× at batch 256–512. Replacing only FP4 PV by BF16 PV makes the kernel ~5× slower, which is why the cache must be an FP4 operand of PV.

3. End-to-end serving (G8Jx Q3, aZwx W1). SGLang, DeepSeek-V4.1-Flash, 4× GB200, NibbleMLA C4A4 vs. the official C8A16 + C4A16 cache: 1.27–1.90× KV capacity, 1.38× output throughput for 24K-in/16K-out long-form generation at 256 concurrent requests (median TTFT 238 s → 15 s), 1.21× at 16K/16K; added rotation and quantization work is hidden (equal forward time at equal batch). The gain grows as memory for the KV cache shrinks: with the same recipe on B200, capacity rises 3.07× and 16K/16K throughput 2.19× (median TTFT 619.5 s → 18.1 s); on B300 (268 GB), NibbleMLA serves 1,024 concurrent requests for 1.28× throughput.

4. FP4 storage with FP8 compute, SAW-INT4 (zL7F). The 257 B cache is the more accurate store under BF16, E4M3 and FP4 arithmetic alike, and FP4 tensor throughput is 2× (Blackwell) to 3× (Blackwell Ultra) that of FP8. SAW-INT4's separate K/V INT4 cache with zero points cannot be consumed by tensor cores; on the MLA latent it is less accurate (3.71 vs 2.87 ×10⁻³ offline).

5. Presentation and details (aZwx, G8Jx). Exact GPU and configurations (§5.1), the exact scale-search rule and why no calibration is needed (§4.2, Eq. 4), a closing summary for §5.3, prefill and per-operation overheads (App. I).

AuthorsOpenReview ↗

Results on two more GPUs (B200, B300): the gain grows as memory for the KV cache shrinks

We repeated the end-to-end comparison of §5.6 on two more NVIDIA GPUs with the same model (DeepSeek-V4.1-Flash), serving stack (SGLang) and arms: NibbleMLA (C4A4, 257 B for both caches) against the official configuration (C8A16 sliding-window cache + C4A16 compressed cache).

B200: less memory left for the KV cache than GB200. Same low-latency recipe as §5.6 (DSpark speculative decoding, CUDA graphs, static memory fraction 0.8), 256 concurrent requests, two cycles, 95% intervals.

  • KV capacity: 3.07× (7.15M vs 2.33M tokens), compared with 1.90× on GB200. B200 leaves 18.3 GB for the KV cache. The sliding-window cache for 256 running requests is reserved first (14.6 GB official, 7.1 GB ours), so the compressed history gets 3.7 GB in the official configuration and 11.1 GB in ours.
  • Long-form generation, 16K in / 16K out: 2.19× output throughput ([2.16, 2.22]); median time to first token 619.5 s → 18.1 s.
  • 24K in / 16K out: 1.84× ([1.68, 2.02]); median time to first token 914.5 s → 70.9 s.
  • Mechanism, from the server logs at 24K: the official cache is full (98% used) with a median of 72 running requests and 157 waiting; NibbleMLA runs 201 requests with 1 waiting.
  • Already at 8K in / 8K out, throughput rises by 1.39×.

B300 (Blackwell Ultra, 268 GB per GPU). Here both caches hold 256 long requests, so the gain comes from running more requests at once. At 16K in / 16K out, NibbleMLA serves 1,024 concurrent requests at 22,461 output tok/s, 1.28× the best configuration of the official stack (17,539 tok/s at 256 concurrent requests; above 256 it does not start in this setup, because its scheduling-metadata kernel exceeds the shared-memory limit). Median time to first token falls from 202 s to 138 s. With long inputs and short outputs (64K in / 128 out, 384 concurrent requests), throughput is 1.13× higher (6,050 vs 5,336 tok/s).

Across the three GPUs, the gain follows the memory left for the KV cache, as the capacity mechanism predicts, and it is largest where that memory is tightest. Longer contexts, video tokens and larger models all push serving into this regime, which is the setting NibbleMLA targets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.