acceptodds
Under review as a conference paper at ICLR 2027

RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches

Abstract

Multi-Head Latent Attention (MLA) has been adopted by recent frontier large language models. Instead of per-head keys and values, MLA caches a compressed KV latent that is up-projected into keys and values at decode time, together with a narrow positional key that carries position information into the attention score. Existing KV-cache quantization methods are designed around the keys and values of standard attention and do not account for what MLA actually caches, which costs accuracy at low bit widths. We analyze the MLA cache both in activation distribution and in how quantization error propagates to the attention output, and show that the two cached tensors play different roles: the latent mainly supplies the values, whereas the positional key dominates the attention score. We therefore propose RAQ-MLA (Role-Aware Quantization for Multi-Head Latent Attention Caches), which fits an invertible transform to each cached tensor against the error that tensor delivers to the attention output: attention-weighted value error for the latent and score perturbation for the positional key. Without training, RAQ-MLA quantizes the full MLA cache to about bits per element, a increase in theoretical KV-cache capacity, while remaining nearly lossless on reasoning and coding tasks across four MLA models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLA, KV cache, Quanitzation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9HdTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies low-bit quantization of MLA caches. It introduces value-aware and score-aware objectives for the shared compressed latent and the decoupled RoPE key, respectively, and learns invertible transformations through alternating optimization without backpropagation through the model. Using INT2 latents and INT4 RoPE keys, the method achieves approximately 2.5 effective bits per element (BPE), with average reasoning and coding scores close to BF16 across four MLA models. The evaluation also includes long-context benchmarks, component ablations, and model-level decoding latency measurements.

Critical strengths
  1. A useful compression–accuracy trade-off. The method quantizes both the latent and the RoPE key, achieving approximately 2.5 BPE while maintaining average reasoning and coding scores close to BF16 across four models. It also achieves long-context retrieval performance comparable to OSCAR at a lower BPE.

  2. Supporting analysis and implementation evidence. The paper includes component ablations, a controlled comparison between role-aware and ordinary reconstruction objectives, layer-wise error evaluations, and model-level TPOT measurements. These provide evidence for the method's effectiveness and practical feasibility.

Critical weaknesses
  1. Insufficient evidence for the generality of the role analysis. Neglecting the latent's content-key path is a central approximation, but the main supporting analysis focuses on GLM-4.7-Flash. Logit attribution alone does not establish output sensitivity. Although the paper provides additional validation, the INT4 path-wise observations leave uncertainty about the approximation under the deployed INT2 latent configuration and across other models.

  2. Limited methodological novelty. Function-aware quantization objectives and invertible transformations are established ideas. The main additions are MLA-specific role metrics and an alternating optimization procedure that avoids backpropagation through the model. I view these as incremental adaptations of existing quantization approaches rather than a substantial methodological advance.

  3. Limited coverage of system efficiency. The reported model-level TPOT speedup is approximately 1.07–1.10×, but these measurements use batch size 1, a 512-token prompt, and total context lengths up to approximately 8.7K. The separate layer-level overhead tests do not establish serving benefits under long contexts, high concurrency, or cache-capacity pressure. The connection between theoretical cache-capacity gains and practical throughput improvements therefore remains insufficiently demonstrated.

  4. Noticeable long-context accuracy degradation. In Table 2, average RULER-NIAH accuracy decreases from 96.95 with BF16 to 93.66, and the 128K score decreases from 87.22 to 79.13. These results already use higher precision in the first two and last two layers, yielding 2.66 BPE overall. While the method matches OSCAR at a lower BPE, the remaining gap to BF16 warrants further analysis given the paper's emphasis on long-context applications.

Questions for authors

Please address the four concerns outlined under Critical Weaknesses.

5 comments

We thank the reviewer for the detailed reading. We address the four concerns below; all changes are marked in blue in the revised PDF.


W1 — Insufficient evidence for the generality of the role analysis.

We repeated the role analysis with the same code on three MLA models (GLM-4.7-Flash, GLM-5.2 and Moonlight), both at INT4 as in the paper and at the deployed INT2 setting.

ModelRoPE-key share of the logitINT4 latent: output error via content key / via valueINT2 latent: via content key / via valueRole metric vs exact output error (Pearson / Spearman)
GLM-4.7-Flash92.5%1.7% / 8.3%5.8% / 27.4%0.997 / 0.925
GLM-5.294.5%1.9% / 10.4%7.0% / 27.4%0.967 / 0.848
Moonlight94.2%2.0% / 13.5%6.7% / 44.4%0.983 / 0.937

Output errors are relative L2L_2 errors of the attention output on five representative layers.

The revised paper adds an appendix with the INT2 counterpart of Figure 3(b) for the three models. The conclusion of Section 3 also holds on the other models and at the deployed INT2 setting: within the latent, "the value readout carries most of its error through to the output, while the attention score is dominated by kR\mathbf{k}^{\mathrm{R}}". Neglecting the latent's content-key path therefore remains a good approximation.


W2 — Limited methodological novelty.

We agree that function-aware objectives and invertible transforms are established in general. This paper is one of the first studies of low-precision KV-cache quantization for MLA-based models (see also Reviewer 73GY's Strengths), and what it adds is an analysis of how quantization error propagates inside the MLA cache, together with a method built on it. Specifically:

  • The latent behaves like a Value cache and the RoPE key like a Key cache (Section 3). The RoPE key carries 92.5% of the attention-score attribution, while the latent's error reaches the output mainly through the value (8.3%) rather than the content key (1.7%). Our response to W1 confirms this on more models and at the deployed INT2 setting.
  • Each design choice follows from this analysis (Section 4; see also the Strengths of Reviewers cR2v and 73GY). The latent gets per-token quantization and a value-aware transform, and the RoPE key gets per-channel quantization and a score-aware transform, matching their distributions and roles. The role metric plays the part of the Hessian in second-order post-training quantization, derived in closed form from the error each tensor delivers to the attention output, and both transforms are fitted against it by alternating minimization with the quantizer in the loop. This is itself part of our methodological contribution: the fit needs no training and no gradient-based optimization such as SGD.

W3 — Limited coverage of system efficiency.

Following the three reviews, we extended the serving measurements: measured cache capacity and peak memory, TTFT, TPOT across context lengths and batch sizes, and throughput under different memory limits. All numbers are on two H20 GPUs with GLM-4.7-Flash (TP2).

Measured memory. We integrated the RAQ-MLA cache (INT2 latent, INT4 RoPE key) into vLLM as a KV-cache dtype and measured it with about 45 GiB of KV-cache memory per GPU for every format:

KV cacheCapacity allocated by vLLMvs. BF16
BF160.92M tokens1×
FP8 (FlashMLA)1.83M tokens2.0×
RAQ-MLA (ours)6.09M tokens6.65×

Peak GPU memory is the same for all formats (about 83 GB per GPU) because vLLM pre-allocates the budget, so the saving appears as capacity: at a 128K context the budget holds about 46 concurrent requests with RAQ-MLA, versus 7 with BF16 and 14 with FP8.

Decode speed. We also rewrote the decode kernel during the rebuttal: tokens rather than heads now occupy the M dimension of the tensor-core instruction, and the split-KV schedule follows each request's actual length. Each configuration below is served alone on the same two GPUs, and every number is the mean of three runs:

TPOT (ms)RAQ-MLA (ours)BF16 FlashMLABF16 FlashAttentionFP8 FlashMLA
32K context, batch 16.378.129.366.24
32K context, batch 414.5618.4618.5515.21
32K context, batch 825.8632.8533.3926.97
64K context, batch 16.669.5913.067.11
64K context, batch 427.9735.9236.3529.97
64K context, batch 856.8471.6173.1360.79
128K context, batch 17.7512.6320.698.83
128K context, batch 472.9588.4889.8077.27
128K context, batch 8159.62157.64159.18168.67

At batch 1 RAQ-MLA is 1.27×, 1.44× and 1.63× faster than BF16 FlashMLA at 32K, 64K and 128K context, and 1.5–2.7× faster than vLLM's BF16 FlashAttention backend; it matches FP8 FlashMLA at 32K and is 6% and 12% faster at 64K and 128K. The benefit is not limited to batch 1: at batch 4 and 8 over 32K–128K context it is 18–22% faster than BF16 FlashMLA and 4–7% faster than FP8 FlashMLA.

Prefill cost. Time to first token, same runs:

TTFT (s)RAQ-MLA (ours)BF16 FlashMLABF16 FlashAttentionFP8 FlashMLA
32K context, batch 13.543.523.523.52
32K context, batch 49.449.329.109.30
32K context, batch 817.1316.9416.3416.88
64K context, batch 111.2611.2111.2211.22
64K context, batch 429.0228.9528.5228.92
64K context, batch 852.6652.3451.6252.52
128K context, batch 139.4239.3539.2639.27
128K context, batch 499.89100.1099.3299.91
128K context, batch 8179.48190.83190.25179.09

Prefill is unaffected: RAQ-MLA's TTFT is within 1% of the BF16 and FP8 caches, because prefill dominates at these lengths and quantizing the cache writes adds almost nothing. The exception is 128K at batch 8, where RAQ-MLA is faster (179.5 s against 190.8 s): eight such requests exceed the BF16 cache capacity, so vLLM has to queue them, while RAQ-MLA runs all eight at once.

The larger advantage of a low-bit KV cache is that, when GPU memory is the limit, it lets more requests run concurrently and so raises throughput. To show this, we swept the KV-cache memory budget under a fixed decode-heavy load (8K prompt, 4K output, 64 concurrent requests) and measured aggregate output throughput:

GPU memoryBF16 (GiB)tok/sFP8 (GiB)tok/sRAQ-MLA (GiB)tok/s
0.353.164363.137461.861051
0.4512.6684212.63123011.361692
0.5522.1695622.13159720.861727
0.6531.66103631.63159930.361735
0.8045.91117745.88160144.611733

At equal memory RAQ-MLA delivers 2.0× the throughput of BF16 and 1.4× that of FP8; with a quarter of the memory it still exceeds BF16 at the full budget by 44%, and at the full budget it decodes 1.53× faster than BF16.

W4 — Noticeable long-context accuracy degradation.

You are right that a gap to BF16 remains at long context, and we report it as a limitation rather than smooth it over. Two-bit KV-cache quantization is still an open problem at long range: on RULER-NIAH at 128K, OSCAR reports a drop from 78.278.2 to 45.045.0 on Qwen3-8B and from 81.081.0 to 39.539.5 on Qwen3-4B-Thinking, while rotation-only INT2 collapses to zero.

We also find the problem eases on larger models. On GLM-5.2, the largest model of Table 1, RULER-NIAH stays above 99.499.4 at every length including 128K and LongBench-v1 is within 0.30.3 points of BF16; OSCAR reports the same pattern on its own largest model.

GLM-5.2, RULER-NIAHBPE4K8K16K32K64K128KAvg.
BF1616100.00100.00100.0099.8899.8899.7899.92
OSCAR3.7899.72100.00100.0099.9499.4799.7599.81
RAQ-MLA2.5099.8199.97100.0099.8899.6699.6299.82
GLM-5.2, LongBench-v1BPESingle-QAMulti-QASumm.Few-shotSynth.CodeAvg.
BF161649.9053.9023.7268.4375.1876.2655.32
OSCAR3.7849.0553.5523.7867.8074.0275.6354.76
RAQ-MLA2.5049.7453.6323.6868.1174.3275.9855.03

The revised paper adds a Limitations section stating this. We hope the community will continue to push on this regime, and we will keep working on it.

Reviewer_73GYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RAQ-MLA, a post training quantization method for large language models with multi head latent attention (MLA). MLA caches latents ctKVc^{KV}_t (which is used to generate keys and values) and decoupled position keys ktRk^R_t (which is obtained to RoPE) while regular attention caches keys and values.

RAQ-MLA proposes four main things :

  1. Asymmetric quantization axis which uses per token quantization for the latent ctKVc^{KV}_t and per channel quantization for the position keys ktRk^R_t.
  2. Asymmetric precision for latent (INT2) and position keys (INT4).
  3. Transform based quantization where transform is value aware for latents and pre softmax attention score aware for position keys.
  4. Alternating minimization approach to fit transforms using calibration data and without gradient descent.

The method achieves 2.5 effective bits per cached element and shows strong performance on downstream evaluations surpassing QuaRot and OSCAR baselines.

Critical strengths
  1. The paper is one of the first studies in low precision kv cache quantization for MLA based models.
  2. The paper provides a coherent explanation for different quantization behavior of latents and position keys (Figure 3). In particular, value aware transform for latents and pre softmax attention score aware transform for position keys is a new finding.
  3. Alternating minimization approach for learning the transforms is a new contribution and it prevents gradient based learning used by SpinQuant (https://arxiv.org/abs/2405.16406).
  4. The local objective diagnosis is meaningful because it shows strong correlation to attention output error (Figure 5).
  5. The additional transforms used to aid quantization add minimal overhead in terms of memory and inference latency.
  6. Downstream evaluations for reasoning and coding tasks are strong and shows the benefits of RAQ-MLA (Table 1).
Critical weaknesses
  1. The central contribution needs further downstream isolation. Per channel/ per token quantization analysis has already been demonstrated in KIVI (https://arxiv.org/abs/2402.02750) and KVlinC (https://arxiv.org/abs/2510.05373). Higher precision for MLA position keys is used by FlashMLA/SnapMLA (https://arxiv.org/abs/2602.10718). And, OSCAR already derives attention aware key/value transforms. The distinctive contribution is MLA specific role metrics and fitting of invertible transforms. Table 3 provides some insights on learned transforms at INT2/BF16 but the final INT2/INT4 method is not compared.

  2. The paper compares with only 2 baselines : Quarot and OSCAR. Even these baselines are adapted by the authors to work on MLA models. The authors choose to leave position keys in full precision which increases bits per element for the baselines. I believe this is unfair and the baselines should be evaluated in the same settings as RAQ-MLA. Additionally, KVarN baseline should be included for all models.

  3. Long context evaluations are not thorough. Paper only shows RULER NIAH on a single model (Table 2) where performance of RAQ-MLA is almost same as OSCAR (RAQ-MLA : 93.66 v/s OSCAR : 93.60). Additionally, LongBench-v1 eval in Appendix Table 7 shows that RAQ-MLA is slightly worse than OSCAR. Since kv cache is only a bottleneck at long sequence length, it is essential to include more long context evaluations on additional datasets and models.

  4. The paper lacks thorough ablation studies on calibration dataset choice. In particular how does calibration dataset, its sequence length and number of samples impact downstream evaluation performance.

  5. Practical memory and serving benefits remain insufficiently measured. Figure 6 only shows theoretical memory used while the paper does not report measured peak memory usage.

  6. Appendix E states that some sampled-task results are averaged over three seeds, without identifying those entries or reporting uncertainty.

Questions for authors
  1. In implementation details (line 422) authors mention that recent tokens are kept in full precision and only quantized when the cache page is full. Do the baselines Quarot and OSCAR also use the same setting?
  2. Why are all downstream evaluations for KVarN not included in the paper? Can the authors include KVarN for all tasks?
  3. At the final INT2-latent/INT4-position setting, can you compare clipping-only, fixed Hadamard/OSCAR, learned identity-metric (Mt=IM_t = I) , and learned role-aware transforms under identical cache policies? Replacing one branch at a time would help identify the contribution of each learned transform.
  4. In Appendix E.2 line 990, authors mention that some of the reported results are averaged across three seeds. Can the authors specify which results are these? This is essential to understand if all the comparisons are fair.
  5. Can you report measured cache capacity or peak memory and end-to-end serving performance at long context and more than one batch/concurrency level, including prefill or TTFT cost?
  6. Can the authors also provide the end to end calibration time for obtaining optimized transforms in RAQ-MLA?
3 comments

e thank the reviewer for the detailed and constructive review. We address each point below; all changes are marked in blue in the revised PDF.


W1 and W2 (first half), and Q3 — downstream isolation at the final setting, and baselines under the same setting.

Existing methods for the MLA cache — FlashMLA, SnapMLA and KVarN — quantize only the latent and keep the positional key in BF16, and we implemented the baselines, including those designed for standard attention, under that same convention. We think that analysing what the positional key costs, especially once the latent is quantized to low precision, is one of the contributions of this work.

To isolate the transforms from that budget difference, Table 3 now ablates each branch one at a time at the final INT2 / INT4 setting, with the offline attention-output error of each branch next to the downstream scores. GLM-4.7-Flash, on the five benchmarks of Table 1:

StepLatentkR\mathbf{k}^{\mathrm{R}}BPELatent err.kR\mathbf{k}^{\mathrm{R}} err.AIME25GPQAMATH-500GSM8KHumanEvalAvg.
AnaiveBF163.78–083.3371.72––––
Bclip + tailBF163.780.266087.7876.4397.7396.0394.7190.54
CHadamardBF163.780.256088.8973.9197.7395.6893.0989.86
DOSCARBF163.780.253087.7872.0697.4095.7594.3189.46
Mlearned with Mt=IM_t=IBF163.780.233092.2274.7597.3395.7394.1090.83
Evalue-awareBF163.780.211092.2276.0997.2095.4094.9291.17
Fvalue-awaredirect INT42.470.2110.046287.7874.2497.3395.5094.9289.95
F″value-awareOSCAR2.470.2110.048388.8975.7697.7395.6394.7190.54
Gvalue-awarescore-aware2.470.2110.042192.2275.2597.6796.0894.9291.23

On the latent, our value-aware metric gives the lowest error (0.2110.211, against 0.2330.233 for the identity metric, 0.2530.253 for OSCAR and 0.2560.256 for Hadamard), and is lowest on all 4747 layers; downstream, the two learned transforms are clearly ahead of the two fixed ones (91.1791.17 and 90.8390.83 against 89.8689.86 and 89.4689.46). On the positional key, the score-aware transform is the only one that improves on no transform (0.04210.0421 against 0.04620.0462; OSCAR 0.04830.0483, Hadamard 0.05010.0501), and it raises the five-benchmark average from 89.9589.95 to 91.2391.23.


W2 (second half) and Q2 — KVarN should be included for all models.

Thank you for the suggestion. We have added KVarN on all five tasks of the other models of Table 1, with the same setting as its GLM-4.7-Flash row (INT4 storage, effective INT3, 4.67 BPE) and the same evaluation protocol as the other rows of each model. Both rows are three-seed means on the sampled tasks.

GLM-4.7-FlashBPEAIME25GPQAMATH-500GSM8KHumanEvalMean
KVarN INT34.6785.5672.0597.9396.0895.1289.35
RAQ-MLA2.4792.2275.2597.6796.0894.9291.23
MethodBPEARC-CPIQAMATH-500GSM8KHumanEvalMean
MoonlightKVarN INT34.6751.9652.6756.8079.9864.2261.13
RAQ-MLA2.4751.7952.6761.8083.8567.2763.48
DS-V2-LiteKVarN INT34.6770.2275.0360.3383.7280.4973.96
RAQ-MLA2.4768.6074.5460.6083.5278.2573.10

At about half the average bits, RAQ-MLA is ahead of KVarN on GLM-4.7-Flash and Moonlight and within one point of it on DS-V2-Lite.

W3 — Long-context evaluations are not thorough.

We added MRCR v2 [Vodrahalli et al., 2024; OpenAI, 2025], which asks the model to reproduce a specified one among several near-identical earlier answers in conversations of up to 128K tokens, on GLM-4.7-Flash. We also evaluated the largest model of Table 1, GLM-5.2, on MRCR v2, LongBench-v1 and RULER-NIAH.

MRCR v2MethodBPE2 needles4 needles8 needlesOverall
GLM-4.7-FlashBF161646.8529.1019.2232.58
OSCAR3.7841.0623.5916.0827.71
RAQ-MLA2.4741.9223.7016.7128.27
GLM-5.2BF161688.0263.5343.4666.28
OSCAR3.7878.8255.1134.8357.50
RAQ-MLA2.5082.2253.4936.5358.80
RULER-NIAHMethodBPE4K8K16K32K64K128KAvg.
GLM-4.7-FlashBF1616100.0099.5999.2298.4196.7587.0096.83
OSCAR3.7899.5698.6498.3496.0190.3978.6593.60
RAQ-MLA2.4798.8898.9798.0396.1992.5079.1693.95
GLM-5.2BF1616100.00100.00100.0099.8899.8899.7899.92
OSCAR3.7899.72100.00100.0099.9499.4799.7599.81
RAQ-MLA2.5099.8199.97100.0099.8899.6699.6299.82
GLM-5.2 LongBench-v1BPESingle-QAMulti-QASumm.Few-shotSynth.CodeAvg.
BF161649.9053.9023.7268.4375.1876.2655.32
OSCAR3.7849.0553.5523.7867.8074.0275.6354.76
RAQ-MLA2.5049.7453.6323.6868.1174.3275.9855.03

RAQ-MLA also outperforms OSCAR on GLM-4.7-Flash, with 28.2728.27 against 27.7127.71 on MRCR v2 and 93.9593.95 against 93.6093.60 on RULER-NIAH, and on GLM-5.2, with 58.8058.80 against 57.5057.50 on MRCR v2 and 55.0355.03 against 54.7654.76 on LongBench-v1.


W4 — No thorough ablation of the calibration data.

We ablated the calibration data along the three axes you name, starting from the Table 1 configuration of GLM-4.7-Flash. Each variant is refitted with the same fitter and hyperparameters; the Table 1 rows are quoted from the paper.

Calibration dataAIME25GPQAHumanEvalMean
RedPajama92.2275.2594.9287.46
Self-generated dataset91.1175.5994.7187.14
RULER prompts86.6774.7595.1285.51
GPQA prompts91.1174.5893.9086.53
Calibration data sizeAIME25GPQAHumanEvalMean
1691.1175.2594.5186.96
3287.7877.1194.1086.33
6492.2275.2594.9287.46
Sequence lengthAIME25GPQAHumanEvalMean
8K91.1172.9096.1486.71
16K90.0074.0794.7186.26
16K+32K92.2275.2594.9287.46

The calibration data does affect the result: with other sources, the three-task mean ranges from 85.5 to 87.1, against 87.5 with RedPajama. RAQ-MLA, however, converges quickly in the amount of calibration data: 16 sequences, or sequences cut to 8K tokens, already come within 1 point of Table 1. These results are now in Appendix G of the revised paper.

W5 and Q5 — measured memory, and end-to-end serving at long context and higher concurrency.

Following the three reviews, we extended the serving measurements: measured cache capacity and peak memory, TTFT, TPOT across context lengths and batch sizes, and throughput under different memory limits. All numbers are on two H20 GPUs with GLM-4.7-Flash (TP2).

Measured memory. We integrated the RAQ-MLA cache (INT2 latent, INT4 RoPE key) into vLLM as a KV-cache dtype and measured it with about 45 GiB of KV-cache memory per GPU for every format:

KV cacheCapacity allocated by vLLMvs. BF16
BF160.92M tokens1×
FP8 (FlashMLA)1.83M tokens2.0×
RAQ-MLA (ours)6.09M tokens6.65×

Peak GPU memory is the same for all formats (about 83 GB per GPU) because vLLM pre-allocates the budget, so the saving appears as capacity: at a 128K context the budget holds about 46 concurrent requests with RAQ-MLA, versus 7 with BF16 and 14 with FP8.

Decode speed. We also rewrote the decode kernel during the rebuttal: tokens rather than heads now occupy the M dimension of the tensor-core instruction, and the split-KV schedule follows each request's actual length. Each configuration below is served alone on the same two GPUs, and every number is the mean of three runs:

TPOT (ms)RAQ-MLA (ours)BF16 FlashMLABF16 FlashAttentionFP8 FlashMLA
32K context, batch 16.378.129.366.24
32K context, batch 414.5618.4618.5515.21
32K context, batch 825.8632.8533.3926.97
64K context, batch 16.669.5913.067.11
64K context, batch 427.9735.9236.3529.97
64K context, batch 856.8471.6173.1360.79
128K context, batch 17.7512.6320.698.83
128K context, batch 472.9588.4889.8077.27
128K context, batch 8159.62157.64159.18168.67

At batch 1 RAQ-MLA is 1.27×, 1.44× and 1.63× faster than BF16 FlashMLA at 32K, 64K and 128K context, and 1.5–2.7× faster than vLLM's BF16 FlashAttention backend; it matches FP8 FlashMLA at 32K and is 6% and 12% faster at 64K and 128K. The benefit is not limited to batch 1: at batch 4 and 8 over 32K–128K context it is 18–22% faster than BF16 FlashMLA and 4–7% faster than FP8 FlashMLA.

Prefill cost. Time to first token, same runs:

TTFT (s)RAQ-MLA (ours)BF16 FlashMLABF16 FlashAttentionFP8 FlashMLA
32K context, batch 13.543.523.523.52
32K context, batch 49.449.329.109.30
32K context, batch 817.1316.9416.3416.88
64K context, batch 111.2611.2111.2211.22
64K context, batch 429.0228.9528.5228.92
64K context, batch 852.6652.3451.6252.52
128K context, batch 139.4239.3539.2639.27
128K context, batch 499.89100.1099.3299.91
128K context, batch 8179.48190.83190.25179.09

Prefill is unaffected: RAQ-MLA's TTFT is within 1% of the BF16 and FP8 caches, because prefill dominates at these lengths and quantizing the cache writes adds almost nothing. The exception is 128K at batch 8, where RAQ-MLA is faster (179.5 s against 190.8 s): eight such requests exceed the BF16 cache capacity, so vLLM has to queue them, while RAQ-MLA runs all eight at once.

The larger advantage of a low-bit KV cache is that, when GPU memory is the limit, it lets more requests run concurrently and so raises throughput. To show this, we swept the KV-cache memory budget under a fixed decode-heavy load (8K prompt, 4K output, 64 concurrent requests) and measured aggregate output throughput:

GPU memoryBF16 (GiB)tok/sFP8 (GiB)tok/sRAQ-MLA (GiB)tok/s
0.353.164363.137461.861051
0.4512.6684212.63123011.361692
0.5522.1695622.13159720.861727
0.6531.66103631.63159930.361735
0.8045.91117745.88160144.611733

At equal memory RAQ-MLA delivers 2.0× the throughput of BF16 and 1.4× that of FP8; with a quarter of the memory it still exceeds BF16 at the full budget by 44%, and at the full budget it decodes 1.53× faster than BF16.

Reviewer_cR2vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new KV cache quantization method called RAQ-MLA for multi-head latent attention layers. The main idea is that errors in different tensors have different levels of impacts to attention scores, which is the actual "downstream" results that matter. Based on this, RAQ-MLA uses value-aware transform for compressed latent tensors and score-aware transform for positional key tensors. RAQ-MLA also uses alternating minimization to fit invertible transforms, which avoids any fine-tuning of the model. Experiments on reasoning/coding shows promising results across 4 MLA models with up to 2.5 BPE achieved.

Critical strengths

The intuition -> analysis -> techniques chain is sound, strong and easy to understand. The empirical results are promising.

Critical weaknesses
  • "nearly lossless" might be a bit over-claiming as there are certain evaluation shows noticeable performance drop. I think the work is already strong and such claims should be tuned to be more technical.
  • Benefits seems to be limited to batch size 1 and the method doesn't seem to work well for long-context setups, which is one of the application that motivates the work.
Questions for authors
  • Which results in Table 1 are averaged across multiple seeds? Could you share per-seed results or variance for the main comparisons? I'd like to understand how much of them are "on average better" or "consistently better"; showing consistently better might help you make the "nearly lossless" like claim, I think.
  • Why did you use a higher-precision setup (2.66 BPE) for the long-context evaluation? How does the default setup at around 2.5 BPE perform on RULER?
  • Re. the score-aware transform beats direct INT4 quantization of the positional key in Table 3, how would an OSCAR transform perform on the positional key using the same quantizer and BPE? This would make it clearer how much the score-aware objective helps compared with an existing transform.
3 comments

We thank the reviewer for the careful reading and for the constructive suggestions. We address each point below; all changes are marked in blue in the revised PDF.


W1 — "nearly lossless" is over-claiming.

We agree that "nearly lossless" is not a precise claim, and we have removed it and similar wording from the paper in favour of quantitative statements. In the abstract it now reads: "Extensive experiments on GLM-4.7-Flash, GLM-5.2, Moonlight, and DeepSeek-Coder-V2-Lite validate the effectiveness of RAQ-MLA in preserving reasoning and coding accuracy."


W2 — Benefits seem limited to batch size 1, and the method does not seem to work well for long context.

Following the three reviews, we extended the serving measurements: measured cache capacity and peak memory, TTFT, TPOT across context lengths and batch sizes, and throughput under different memory limits. All numbers are on two H20 GPUs with GLM-4.7-Flash (TP2).

Decode speed. We also rewrote the decode kernel during the rebuttal: tokens rather than heads now occupy the M dimension of the tensor-core instruction, and the split-KV schedule follows each request's actual length. Each configuration below is served alone on the same two GPUs, and every number is the mean of three runs:

TPOT (ms)RAQ-MLA (ours)BF16 FlashMLABF16 FlashAttentionFP8 FlashMLA
32K context, batch 16.378.129.366.24
32K context, batch 414.5618.4618.5515.21
32K context, batch 825.8632.8533.3926.97
64K context, batch 16.669.5913.067.11
64K context, batch 427.9735.9236.3529.97
64K context, batch 856.8471.6173.1360.79
128K context, batch 17.7512.6320.698.83
128K context, batch 472.9588.4889.8077.27
128K context, batch 8159.62157.64159.18168.67

At batch 1 RAQ-MLA is 1.27×, 1.44× and 1.63× faster than BF16 FlashMLA at 32K, 64K and 128K context, and 1.5–2.7× faster than vLLM's BF16 FlashAttention backend; it matches FP8 FlashMLA at 32K and is 6% and 12% faster at 64K and 128K. The benefit is not limited to batch 1: at batch 4 and 8 over 32K–128K context it is 18–22% faster than BF16 FlashMLA and 4–7% faster than FP8 FlashMLA.

The larger advantage of a low-bit KV cache is that, when GPU memory is the limit, it lets more requests run concurrently and so raises throughput. To show this, we swept the KV-cache memory budget under a fixed decode-heavy load (8K prompt, 4K output, 64 concurrent requests) and measured aggregate output throughput:

GPU memoryBF16 (GiB)tok/sFP8 (GiB)tok/sRAQ-MLA (GiB)tok/s
0.353.164363.137461.861051
0.4512.6684212.63123011.361692
0.5522.1695622.13159720.861727
0.6531.66103631.63159930.361735
0.8045.91117745.88160144.611733

At equal memory RAQ-MLA delivers 2.0× the throughput of BF16 and 1.4× that of FP8; with a quarter of the memory it still exceeds BF16 at the full budget by 44%, and at the full budget it decodes 1.53× faster than BF16.

Q1 — Which results in Table 1 are averaged across seeds? Can you share per-seed results or variance?

In Table 1, we ran three seeds (0, 1, 2) for OSCAR and RAQ-MLA and report their mean on the sampled tasks. We prioritized these two because OSCAR is the strongest baseline and its gap to RAQ-MLA is the smallest, so this is the comparison where seed variance matters most.

Per-seed five-task means on GLM-4.7-Flash:

seed 0seed 1seed 2Mean
BF1691.3090.4391.5491.09
KVarN (4.67 BPE)89.8290.1988.0489.35
OSCAR (3.78 BPE)89.2389.8689.3089.46
RAQ-MLA (2.47 BPE)91.2691.2791.1591.23

RAQ-MLA is higher than OSCAR on every seed (+1.77+1.77) and lies within the seed range of BF16 (+0.14+0.14). Given the limited time, we will add multi-seed results for all models and datasets as far as possible.


Q2 — Why a higher-precision setup (2.66 BPE) for long context? How does the default ~2.5 BPE perform on RULER?

We added MRCR v2 [Vodrahalli et al., 2024; OpenAI, 2025], which asks the model to reproduce a specified one among several near-identical earlier answers in conversations of up to 128K tokens, on GLM-4.7-Flash with all layers at 2.47 BPE. We also evaluated the largest model of Table 1, GLM-5.2, on MRCR v2, LongBench-v1 and RULER-NIAH.

MRCR v2MethodBPE2 needles4 needles8 needlesOverall
GLM-4.7-FlashBF161646.8529.1019.2232.58
OSCAR3.7841.0623.5916.0827.71
RAQ-MLA2.4741.9223.7016.7128.27
GLM-5.2BF161688.0263.5343.4666.28
OSCAR3.7878.8255.1134.8357.50
RAQ-MLA2.5082.2253.4936.5358.80
RULER-NIAHMethodBPE4K8K16K32K64K128KAvg.
GLM-4.7-FlashBF1616100.0099.5999.2298.4196.7587.0096.83
OSCAR3.7899.5698.6498.3496.0190.3978.6593.60
RAQ-MLA2.4798.8898.9798.0396.1992.5079.1693.95
GLM-5.2BF1616100.00100.00100.0099.8899.8899.7899.92
OSCAR3.7899.72100.00100.0099.9499.4799.7599.81
RAQ-MLA2.5099.8199.97100.0099.8899.6699.6299.82
GLM-5.2 LongBench-v1BPESingle-QAMulti-QASumm.Few-shotSynth.CodeAvg.
BF161649.9053.9023.7268.4375.1876.2655.32
OSCAR3.7849.0553.5523.7867.8074.0275.6354.76
RAQ-MLA2.5049.7453.6323.6868.1174.3275.9855.03

The 2.66 BPE setting was a conservative choice for long context. At the default 2.472.47 BPE, RAQ-MLA reaches 93.9593.95 on RULER-NIAH (79.1679.16 at 128K), higher than the 2.662.66 BPE configuration of Table 2 (93.6693.66 and 79.1379.13); we have added this run to Appendix H. At 2.472.47–2.502.50 BPE, RAQ-MLA also outperforms OSCAR at 3.783.78 BPE on MRCR for both models.

Q3 — How would an OSCAR transform perform on the positional key at the same quantizer and BPE?

We added OSCAR's rotation on the RoPE key, with the same latent, quantizer and BPE as our score-aware transform. The rows of Table 3 that differ only in the kR\mathbf{k}^{\mathrm{R}} transform are:

kR\mathbf{k}^{\mathrm{R}} transformBPEkR\mathbf{k}^{\mathrm{R}} err.AIME25GPQAMATH-500GSM8KHumanEvalAvg.
BF163.78092.2276.0997.2095.4094.9291.17
direct INT42.470.046287.7874.2497.3395.5094.9289.95
OSCAR2.470.048388.8975.7697.7395.6394.7190.54
score-aware2.470.042192.2275.2597.6796.0894.9291.23

All four share the same value-aware latent, so the only variable is the transform on the RoPE key. OSCAR's rotation averages 90.5490.54 over the five benchmarks against 91.2391.23 for the score-aware transform, and offline it is slightly worse than using no transform at all on this branch (0.04830.0483 against 0.04620.0462, which the score-aware transform lowers to 0.04210.0421).

Comments on the paper

3 comments

W6 and Q4 — which entries are averaged over three seeds, and the uncertainty.

In Table 1, we ran three seeds (0, 1, 2) for OSCAR and RAQ-MLA and report their mean on the sampled tasks. We prioritized these two because OSCAR is the strongest baseline and its gap to RAQ-MLA is the smallest, so this is the comparison where seed variance matters most.

Per-seed five-task means on GLM-4.7-Flash:

seed 0seed 1seed 2Mean
BF1691.3090.4391.5491.09
KVarN (4.67 BPE)89.8290.1988.0489.35
OSCAR (3.78 BPE)89.2389.8689.3089.46
RAQ-MLA (2.47 BPE)91.2691.2791.1591.23

RAQ-MLA is higher than OSCAR on every seed (+1.77+1.77) and lies within the seed range of BF16 (+0.14+0.14). Given the limited time, we will add multi-seed results for all models and datasets as far as possible.


Q1 — Do QuaRot and OSCAR use the same page setting?

Our strongest baseline, OSCAR, is evaluated under exactly our cache setting (clipping ratio 0.940.94 and a 128128-token BF16 open page), so that comparison is like for like. QuaRot follows its own original recipe, which uses neither. We have made this explicit in §5.1 and Appendix E.1.


Q6 — End-to-end calibration time.

Calibration is one-time and offline, with no backpropagation through the model. Each layer is fitted independently, so layers run in parallel across GPUs.

ModelLayersCapture (whole model, once)Value-aware transform fit (one layer, one GPU)Score-aware transform fit (one layer, one GPU)
GLM-4.7-Flash4722 min, 2 × H20136 s20 s
GLM-5.27847 min, 16 × H207.5 min13 min

End to end on 8 GPUs, calibration takes about 40 min for GLM-4.7-Flash and about 4 h for GLM-5.2, against 29–48 h and 10–16 h for one evaluation pass over the five benchmarks. The revised paper adds this as Appendix E.2.

Reviewer_73GYOpenReview ↗

Response to authors

I thank the authors for their rebuttal. It has addressed my concerns and I will maintain my positive rating. Good luck!

Thank you very much for carefully reviewing our responses, for your support, and for the kind wishes. Your comments have helped us improve the paper, and we are glad that our responses have addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.