- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies low-bit quantization of MLA caches. It introduces value-aware and score-aware objectives for the shared compressed latent and the decoupled RoPE key, respectively, and learns invertible transformations through alternating optimization without backpropagation through the model. Using INT2 latents and INT4 RoPE keys, the method achieves approximately 2.5 effective bits per element (BPE), with average reasoning and coding scores close to BF16 across four MLA models. The evaluation also includes long-context benchmarks, component ablations, and model-level decoding latency measurements.
Critical strengths
-
A useful compression–accuracy trade-off. The method quantizes both the latent and the RoPE key, achieving approximately 2.5 BPE while maintaining average reasoning and coding scores close to BF16 across four models. It also achieves long-context retrieval performance comparable to OSCAR at a lower BPE.
-
Supporting analysis and implementation evidence. The paper includes component ablations, a controlled comparison between role-aware and ordinary reconstruction objectives, layer-wise error evaluations, and model-level TPOT measurements. These provide evidence for the method's effectiveness and practical feasibility.
Critical weaknesses
-
Insufficient evidence for the generality of the role analysis. Neglecting the latent's content-key path is a central approximation, but the main supporting analysis focuses on GLM-4.7-Flash. Logit attribution alone does not establish output sensitivity. Although the paper provides additional validation, the INT4 path-wise observations leave uncertainty about the approximation under the deployed INT2 latent configuration and across other models.
-
Limited methodological novelty. Function-aware quantization objectives and invertible transformations are established ideas. The main additions are MLA-specific role metrics and an alternating optimization procedure that avoids backpropagation through the model. I view these as incremental adaptations of existing quantization approaches rather than a substantial methodological advance.
-
Limited coverage of system efficiency. The reported model-level TPOT speedup is approximately 1.07–1.10×, but these measurements use batch size 1, a 512-token prompt, and total context lengths up to approximately 8.7K. The separate layer-level overhead tests do not establish serving benefits under long contexts, high concurrency, or cache-capacity pressure. The connection between theoretical cache-capacity gains and practical throughput improvements therefore remains insufficiently demonstrated.
-
Noticeable long-context accuracy degradation. In Table 2, average RULER-NIAH accuracy decreases from 96.95 with BF16 to 93.66, and the 128K score decreases from 87.22 to 79.13. These results already use higher precision in the first two and last two layers, yielding 2.66 BPE overall. While the method matches OSCAR at a lower BPE, the remaining gap to BF16 warrants further analysis given the paper's emphasis on long-context applications.
Questions for authors
Please address the four concerns outlined under Critical Weaknesses.
5 comments
W3 — Limited coverage of system efficiency.
Following the three reviews, we extended the serving measurements: measured cache capacity and peak memory, TTFT, TPOT across context lengths and batch sizes, and throughput under different memory limits. All numbers are on two H20 GPUs with GLM-4.7-Flash (TP2).
Measured memory. We integrated the RAQ-MLA cache (INT2 latent, INT4 RoPE key) into vLLM as a KV-cache dtype and measured it with about 45 GiB of KV-cache memory per GPU for every format:
| KV cache | Capacity allocated by vLLM | vs. BF16 |
|---|---|---|
| BF16 | 0.92M tokens | 1× |
| FP8 (FlashMLA) | 1.83M tokens | 2.0× |
| RAQ-MLA (ours) | 6.09M tokens | 6.65× |
Peak GPU memory is the same for all formats (about 83 GB per GPU) because vLLM pre-allocates the budget, so the saving appears as capacity: at a 128K context the budget holds about 46 concurrent requests with RAQ-MLA, versus 7 with BF16 and 14 with FP8.
Decode speed. We also rewrote the decode kernel during the rebuttal: tokens rather than heads now occupy the M dimension of the tensor-core instruction, and the split-KV schedule follows each request's actual length. Each configuration below is served alone on the same two GPUs, and every number is the mean of three runs:
| TPOT (ms) | RAQ-MLA (ours) | BF16 FlashMLA | BF16 FlashAttention | FP8 FlashMLA |
|---|---|---|---|---|
| 32K context, batch 1 | 6.37 | 8.12 | 9.36 | 6.24 |
| 32K context, batch 4 | 14.56 | 18.46 | 18.55 | 15.21 |
| 32K context, batch 8 | 25.86 | 32.85 | 33.39 | 26.97 |
| 64K context, batch 1 | 6.66 | 9.59 | 13.06 | 7.11 |
| 64K context, batch 4 | 27.97 | 35.92 | 36.35 | 29.97 |
| 64K context, batch 8 | 56.84 | 71.61 | 73.13 | 60.79 |
| 128K context, batch 1 | 7.75 | 12.63 | 20.69 | 8.83 |
| 128K context, batch 4 | 72.95 | 88.48 | 89.80 | 77.27 |
| 128K context, batch 8 | 159.62 | 157.64 | 159.18 | 168.67 |
At batch 1 RAQ-MLA is 1.27×, 1.44× and 1.63× faster than BF16 FlashMLA at 32K, 64K and 128K context, and 1.5–2.7× faster than vLLM's BF16 FlashAttention backend; it matches FP8 FlashMLA at 32K and is 6% and 12% faster at 64K and 128K. The benefit is not limited to batch 1: at batch 4 and 8 over 32K–128K context it is 18–22% faster than BF16 FlashMLA and 4–7% faster than FP8 FlashMLA.
Prefill cost. Time to first token, same runs:
| TTFT (s) | RAQ-MLA (ours) | BF16 FlashMLA | BF16 FlashAttention | FP8 FlashMLA |
|---|---|---|---|---|
| 32K context, batch 1 | 3.54 | 3.52 | 3.52 | 3.52 |
| 32K context, batch 4 | 9.44 | 9.32 | 9.10 | 9.30 |
| 32K context, batch 8 | 17.13 | 16.94 | 16.34 | 16.88 |
| 64K context, batch 1 | 11.26 | 11.21 | 11.22 | 11.22 |
| 64K context, batch 4 | 29.02 | 28.95 | 28.52 | 28.92 |
| 64K context, batch 8 | 52.66 | 52.34 | 51.62 | 52.52 |
| 128K context, batch 1 | 39.42 | 39.35 | 39.26 | 39.27 |
| 128K context, batch 4 | 99.89 | 100.10 | 99.32 | 99.91 |
| 128K context, batch 8 | 179.48 | 190.83 | 190.25 | 179.09 |
Prefill is unaffected: RAQ-MLA's TTFT is within 1% of the BF16 and FP8 caches, because prefill dominates at these lengths and quantizing the cache writes adds almost nothing. The exception is 128K at batch 8, where RAQ-MLA is faster (179.5 s against 190.8 s): eight such requests exceed the BF16 cache capacity, so vLLM has to queue them, while RAQ-MLA runs all eight at once.
The larger advantage of a low-bit KV cache is that, when GPU memory is the limit, it lets more requests run concurrently and so raises throughput. To show this, we swept the KV-cache memory budget under a fixed decode-heavy load (8K prompt, 4K output, 64 concurrent requests) and measured aggregate output throughput:
| GPU memory | BF16 (GiB) | tok/s | FP8 (GiB) | tok/s | RAQ-MLA (GiB) | tok/s |
|---|---|---|---|---|---|---|
| 0.35 | 3.16 | 436 | 3.13 | 746 | 1.86 | 1051 |
| 0.45 | 12.66 | 842 | 12.63 | 1230 | 11.36 | 1692 |
| 0.55 | 22.16 | 956 | 22.13 | 1597 | 20.86 | 1727 |
| 0.65 | 31.66 | 1036 | 31.63 | 1599 | 30.36 | 1735 |
| 0.80 | 45.91 | 1177 | 45.88 | 1601 | 44.61 | 1733 |
At equal memory RAQ-MLA delivers 2.0× the throughput of BF16 and 1.4× that of FP8; with a quarter of the memory it still exceeds BF16 at the full budget by 44%, and at the full budget it decodes 1.53× faster than BF16.
W4 — Noticeable long-context accuracy degradation.
You are right that a gap to BF16 remains at long context, and we report it as a limitation rather than smooth it over. Two-bit KV-cache quantization is still an open problem at long range: on RULER-NIAH at 128K, OSCAR reports a drop from to on Qwen3-8B and from to on Qwen3-4B-Thinking, while rotation-only INT2 collapses to zero.
We also find the problem eases on larger models. On GLM-5.2, the largest model of Table 1, RULER-NIAH stays above at every length including 128K and LongBench-v1 is within points of BF16; OSCAR reports the same pattern on its own largest model.
| GLM-5.2, RULER-NIAH | BPE | 4K | 8K | 16K | 32K | 64K | 128K | Avg. |
|---|---|---|---|---|---|---|---|---|
| BF16 | 16 | 100.00 | 100.00 | 100.00 | 99.88 | 99.88 | 99.78 | 99.92 |
| OSCAR | 3.78 | 99.72 | 100.00 | 100.00 | 99.94 | 99.47 | 99.75 | 99.81 |
| RAQ-MLA | 2.50 | 99.81 | 99.97 | 100.00 | 99.88 | 99.66 | 99.62 | 99.82 |
| GLM-5.2, LongBench-v1 | BPE | Single-QA | Multi-QA | Summ. | Few-shot | Synth. | Code | Avg. |
|---|---|---|---|---|---|---|---|---|
| BF16 | 16 | 49.90 | 53.90 | 23.72 | 68.43 | 75.18 | 76.26 | 55.32 |
| OSCAR | 3.78 | 49.05 | 53.55 | 23.78 | 67.80 | 74.02 | 75.63 | 54.76 |
| RAQ-MLA | 2.50 | 49.74 | 53.63 | 23.68 | 68.11 | 74.32 | 75.98 | 55.03 |
The revised paper adds a Limitations section stating this. We hope the community will continue to push on this regime, and we will keep working on it.
We thank the reviewer for the detailed reading. We address the four concerns below; all changes are marked in blue in the revised PDF.
W1 — Insufficient evidence for the generality of the role analysis.
We repeated the role analysis with the same code on three MLA models (GLM-4.7-Flash, GLM-5.2 and Moonlight), both at INT4 as in the paper and at the deployed INT2 setting.
Output errors are relative L2 errors of the attention output on five representative layers.
The revised paper adds an appendix with the INT2 counterpart of Figure 3(b) for the three models. The conclusion of Section 3 also holds on the other models and at the deployed INT2 setting: within the latent, "the value readout carries most of its error through to the output, while the attention score is dominated by kR". Neglecting the latent's content-key path therefore remains a good approximation.
W2 — Limited methodological novelty.
We agree that function-aware objectives and invertible transforms are established in general. This paper is one of the first studies of low-precision KV-cache quantization for MLA-based models (see also Reviewer 73GY's Strengths), and what it adds is an analysis of how quantization error propagates inside the MLA cache, together with a method built on it. Specifically: