- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies low-bit KV-cache quantization for Multi-head Latent Attention (MLA), whose cache contains a shared content latent and a decoupled RoPE key. By injecting errors of the same relative size into each cache path, the authors find that RoPE-key errors distort the attention output more than content-latent errors. QuantMLA therefore learns path-specific, offline-fusible transformations: a rotation and scaling on the normalized content latent trained to reconstruct the attention output, and RoPE-commuting pair rotations with tied scales trained to reconstruct positional QK scores. QuantMLA keeps the first 4 sink tokens and the most recent 128 tokens in BF16 and quantizes the remaining cache. Across four MLA models, it stays close to BF16 accuracy at C4R4 and outperforms RTN, SmoothQuant, and QuaRot, while achieving 3.59x cache compression at 128K and 5.168x throughput in a memory-constrained serving setting.
Critical strengths
- The paper identifies an asymmetry between MLA's two cache paths: although the RoPE key is much smaller than the content latent, the matched-error study on multiple models shows its errors can distort the output more.
- The paper proposes learned rotation-and-scaling transformations tailored to each cache path, which require only lightweight calibration and can be fused into existing weights offline with no inference-time computation.
Critical weaknesses
QuantMLA reduces cache quantization error by learning rotation and scaling transformations, an approach closely related to existing learned-transformation methods such as SpinQuant and OSTQuant. However, the comparison only includes RTN, SmoothQuant, and a fixed Hadamard rotation (QuaRot), so it is unclear how much of the improvement comes from the MLA-specific design rather than from learning the transformations.
Questions for authors
- Were the baselines in Tables 1 and 2 evaluated with the same BF16 sink protection as QuantMLA? If not, please report the baselines with the same protection, as this affects how much of the gain comes from the proposed method.
- Section 2.2 scales both paths to the same relative error, but in practice the error depends on each path's value distribution, so the same bit-width can give quite different errors. Using the same metric as Figure 2, does the RoPE path still hurt more when both are simply quantized to INT4 for each model?
- Prior work such as SpinQuant and OSTQuant also learns rotation and scaling transformations for quantization. Comparing against one of them on the MLA cache, with the same calibration data, would show how much the MLA-specific design adds on top of learning the transformations.
- It would be helpful to extend the evaluation to show the effect of each path's transformation separately, and to re-measure the Figure 2 distortion after the transformations.