acceptodds
Withdrawn from ICLR 2027

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memory still scales linearly with context length and batch size. Existing methods primarily quantize the content cache to FP8 while retaining the RoPE key cache at high precision. Low-bit RoPE quantization remains poorly understood, leaving joint low-bit compression of the content and RoPE caches largely unexplored. In this work, we establish a systematic model of MLA's dual-path quantization errors, characterizing their distinct effects on attention-output distortion and explaining the pronounced amplification of RoPE-path errors. Guided by this analysis, we introduce QuantMLA, a function-aligned framework for low-bit dual-path quantization. We derive path-specific transformation spaces that preserve full-precision computation while remaining fully fusible into model parameters offline, eliminating online transformation overhead. Within these spaces, QuantMLA learns path-specific transformations with function-aligned objectives: attention-output reconstruction captures the content path's coupled matching and aggregation errors, while positional QK reconstruction preserves the RoPE-induced component of the attention logits and admits a theoretical bound on output distortion. Across four MLA model families, QuantMLA enables, to our knowledge, the first reported joint INT4 caching of the content and RoPE caches with minimal accuracy degradation. Further compressing the content cache to INT2 while retaining the RoPE key cache at INT4 maintains competitive performance on challenging reasoning and code benchmarks. We develop a native low-bit MLA attention kernel that integrates unpacking and dequantization directly into attention computation. The physical cache layout provides 3.59× compression at 128K context, while a cache-pressure serving workload achieves 5.168× higher whole-job output throughput than BF16. Our code is provided in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Head Latent Attention, KV Cache Quantization, Low-Bit Quantization, Model Compression, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6Z1TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies low-bit KV-cache quantization for Multi-head Latent Attention (MLA), whose cache contains a shared content latent and a decoupled RoPE key. By injecting errors of the same relative size into each cache path, the authors find that RoPE-key errors distort the attention output more than content-latent errors. QuantMLA therefore learns path-specific, offline-fusible transformations: a rotation and scaling on the normalized content latent trained to reconstruct the attention output, and RoPE-commuting pair rotations with tied scales trained to reconstruct positional QK scores. QuantMLA keeps the first 4 sink tokens and the most recent 128 tokens in BF16 and quantizes the remaining cache. Across four MLA models, it stays close to BF16 accuracy at C4R4 and outperforms RTN, SmoothQuant, and QuaRot, while achieving 3.59x cache compression at 128K and 5.168x throughput in a memory-constrained serving setting.

Critical strengths
  1. The paper identifies an asymmetry between MLA's two cache paths: although the RoPE key is much smaller than the content latent, the matched-error study on multiple models shows its errors can distort the output more.
  2. The paper proposes learned rotation-and-scaling transformations tailored to each cache path, which require only lightweight calibration and can be fused into existing weights offline with no inference-time computation.
Critical weaknesses

QuantMLA reduces cache quantization error by learning rotation and scaling transformations, an approach closely related to existing learned-transformation methods such as SpinQuant and OSTQuant. However, the comparison only includes RTN, SmoothQuant, and a fixed Hadamard rotation (QuaRot), so it is unclear how much of the improvement comes from the MLA-specific design rather than from learning the transformations.

Questions for authors
  1. Were the baselines in Tables 1 and 2 evaluated with the same BF16 sink protection as QuantMLA? If not, please report the baselines with the same protection, as this affects how much of the gain comes from the proposed method.
  2. Section 2.2 scales both paths to the same relative error, but in practice the error depends on each path's value distribution, so the same bit-width can give quite different errors. Using the same metric as Figure 2, does the RoPE path still hurt more when both are simply quantized to INT4 for each model?
  3. Prior work such as SpinQuant and OSTQuant also learns rotation and scaling transformations for quantization. Comparing against one of them on the MLA cache, with the same calibration data, would show how much the MLA-specific design adds on top of learning the transformations.
  4. It would be helpful to extend the evaluation to show the effect of each path's transformation separately, and to re-measure the Figure 2 distortion after the transformations.
Reviewer_dsQwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes QuantMLA, a post training quantization method for large language models with multi head latent attention (MLA). MLA caches content latent (which is used to generate keys and values) and decoupled rope keys. QuantMLA is one of the first works to explore low bit quantization for kv cache in MLA based models (I do see several contemporary submissions to ICLR 2027 also exploring low bit MLA KV cache quantization but they are too recent to compare).

QuantMLA proposes :

  1. Analysis on how quantization error in content path and RoPE path impact attention output error. Key observation is that at matched cache error, RoPE path leads to substantially higher attention output error.
  2. Learned and offline fusible transformations to aid quantization of content and RoPE path. The content transform minimizes attention output error while RoPE transform minimizes positional-QK reconstruction error.
  3. Efficient hardware implementation via custom attention decode kernel. The decode kernel includes a low bit MLA backend with unpacking and dequantization fused into the attention operation. QuantMLA achieves 3.59X compression at 128K context and 5.168X higher output throughput compared to BF16.
Critical strengths

QuantMLA has following strengths :

  1. QuantMLA is one of the first works to explore low bit quantization for kv cache in MLA based models
  2. Analysis on how quantization error in content path and RoPE path impact attention output error is highly valuable. Key observation is that at matched cache error, RoPE path leads to substantially higher attention output error.
  3. Evaluation tasks are pretty thorough for short context evaluations involving reasoning, coding and information extraction. The technique shows sufficient generalizability across different model backbones.
  4. The system efficiency results are thorough. The authors have done an impressive job on creating the attention decode kernel and integrating it to vLLM.
Critical weaknesses
  1. The paper is not written well and it is difficult to understand. The main text does not provide sufficient details on how the transforms are optimized. These details are eventually provided in Appendix C.7 Table 7 where the authors mention that they use gradient based learning and Cayley parameterization to learn transforms. These details should be in the main paper. The attention kernel descripting (section 3.3) includes details about mixed precision execution stating terms like “protected BF16” without providing details on what they mean. Only after fully reading the paper twice and using AI (chatgpt Sol 6.1) I was able to understand the proposed approach clearly.
  2. The baseline comparison is limited to RTN and adaptations of two relatively old methods: QuaRot, using fixed Hadamard rotations, and SmoothQuant, using calibrated channel scaling. Although existing methods may not directly target joint low-bit quantization of MLA’s content and RoPE caches, the authors should adapt representative recent KV-cache quantization methods developed for conventional attention, particularly those using learned or attention-aware transformations.
  3. long context evaluation is highly insufficient. The authors only benchmark on RULER-NIAH tasks and that too at 32K context length. Since kv cache is only a bottleneck at long contexts, the authors need to expand their long context evaluations.
  4. The baselines are not evaluated fairly. QuantMLA keeps recent tokens and sink tokens in BF16 while no such mixed precision execution exists for RTN, Smoothquant or Quarot baseline.
Questions for authors

Please see weaknesses.

Reviewer_7ovbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QuantMLA presents an analysis and methodology for low-bit KV cache quantization for Multi-head Latent Attention (MLA), specifically targeting both the content cache and RoPE decoupled cache. QuantMLA learns two offline transformations, targeting the content path aggregation and matching objectives and the decoupled RoPE path that respects the pairwise rotational structure.

Critical strengths
  • First low-bit quantization method that actually targets the RoPE cache in MLA. The paper characterizes the sensitivity of this cache across multiple models across 6 models up to the 1T scale.
Critical weaknesses
  • The main results presented in Table 1 through 4 look good, especially at the C2R4 configuration. But is unclear whether the baselines compared with are also protected under the same conditions as QuantMLA. Additionally, I am unsure if the QuantMLA uses the sink4 or local128 protection configuration.
  • Table 3 ablates the impact of each learned transformation being present. However, no true analysis of function-aligned objectives is provided. This brings up the question if whether a learned sclae-only transformation is sufficient.
Questions for authors
  • Do Table 1 and Table 2 report protected QuantMLA accuracy, but unprotected numbers for other baselines? Furthermore, are accuracy values in Table 1 and 2 with sink4 or local128 protection?
  • In Fig. 7b, the local gain ratio across layer index seems to be unit at the majority of layer indices. This makes me wonder how much of the matched NMSE is simply a normalization effect?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.