acceptodds
Under review as a conference paper at ICLR 2027

RateQuant: Attention-Aware Rate–Distortion Allocation for KV Cache Quantization

Abstract

Effective KV cache compression combines accurate quantization with precision allocation to preserve model quality within a memory budget. Allocation is coupled through existing attention error and inputs from earlier quantized layers. We introduce RateQuant, a rate–distortion framework allocating KV precision across layers and heads through measured attention responses. These responses capture how adjacent bit-width changes reinforce or cancel the existing attention error. Dynamic programming combines them into budget-preserving joint updates, and remeasurement refreshes the responses for the next round. Full-model cross-entropy on separate calibration data then selects the final allocation. Our analysis connects normalized attention distortion to task loss under bounded suffix sensitivity and establishes exact optimization of the local additive objective. Across three models and two quantizers, RateQuant achieves the lowest perplexity in all 18 settings spanning 2–3-bit budgets, with reductions of 12.6–94.5% at 2 bits and 3.2–10.3% at 2.5 bits versus each setting's strongest tested baseline. Frozen allocations achieve the highest three-model mean scores in all 30 LongBench evaluation settings (five tasks, two quantizers, and three budgets), without task-specific recalibration.

Primary area:
optimization
Keywords:
KV cache quantization, mixed-precision quantization, rate-distortion theory, large language model inference, post-training quantization, model compression, memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_M68uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies mixed-precision KV-cache allocation under a fixed budget. It measures how bit-width changes across layers and KV heads affect projected attention-output error under the current configuration, uses dynamic programming to construct budget-preserving joint updates, and remeasures after each update. The final allocation is selected using model loss on a separate calibration set and frozen for perplexity and LongBench evaluation.

Critical strengths
  1. A reasonable allocation criterion and clear methodology. The method considers interactions between bit-width changes and existing quantization error, with remeasurement adapting the criterion to configuration changes. The analysis also clearly distinguishes exact optimization of the local objective from full-model outcomes.

  2. Consistent reported improvements and a reasonable transfer protocol. The method achieves the lowest perplexity across all 18 tested configurations. Distortion measurement and allocation selection use separate calibration sets, and allocations are frozen before LongBench evaluation, providing evidence of cross-task transfer.

Critical weaknesses
  1. The methodological increment and contributions of the core components are insufficiently established. Attention-aware mixed precision and rate–distortion allocation have been studied previously, but the paper lacks controlled comparisons that remove the residual cross term, disable remeasurement, or use simpler allocation strategies at the same granularity. Since additional precision is heavily concentrated in the first layer in some configurations, a budget-matched first-layer protection baseline is also needed.

  2. Limited comparison with advanced allocation methods. Experiments mainly compare Uniform, KV-sep, and an adapted KVTuner, without empirical comparisons against related allocation methods such as RDKV and AATC. KVTuner also uses a different allocation granularity and leaves some budget unused in several configurations, making it difficult to isolate the effects of the scoring criterion, search space, and realized budget.

  3. Practical system benefits remain unverified. Evaluation uses eager quantization and dequantization without a packed mixed-precision cache or corresponding kernels, and nominal budgets exclude quantizer metadata. Actual memory usage, decoding latency, and calibration time are not measured, leaving deployment benefits unclear.

  4. Limited quality references and generalization evidence. Experiments cover only 4B and 8B models and five LongBench tasks, with generation capped at 64 or 128 tokens. Long-chain reasoning and execution-based code generation are not evaluated. The main results also lack an unquantized FP16 reference, making the remaining quality gap difficult to assess.

Questions for authors

Please address the four concerns outlined under Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Revised PDF uploaded

Dear Reviewer M68u,

Thank you again for your detailed comments on the ablations, the baselines, deployment, and the evaluation. We have uploaded the revised PDF, with all changes in blue. The new results are in a new Section 4.6 and a new Appendix F with Tables 9–21. Your points are addressed as follows:

  • W1 (cross-term ablation and component controls): Section 4.6 and Appendix F.1, Tables 9 and 10.
  • W2 (RDKV, KVTuner, and effective storage): Appendix F.2, Tables 11–13, and Appendix F.3, Table 14.
  • W3 (packed cache, compression, decoding throughput, calibration time): Appendix F.4, Tables 15–17. Section 6 and Appendix D now describe the packed implementation.
  • W4 (Qwen3-32B on AIME, HumanEval+, and MBPP+ with an FP16 reference): Appendix F.5, Tables 18–20.

Regards from the authors.

Thank you for the detailed responses and additional experiments. The new ablations, comparisons, and evaluations have addressed most of my main concerns. I am raising my rating from 2 to 3.

AuthorsOpenReview ↗

Thank You for Your Feedback and Updated Rating

Dear Reviewer M68u,

Thank you for reading our responses and for raising your rating. Your suggestions on the ablations, the baseline comparisons, the deployment measurements, and the broader evaluation made the paper more complete, and we will keep all of these additions in the final version.

Regards from the authors.

Reviewer_bg21OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RateQuant, an offline precision allocation method for mixed-precision KV cache quantization. Its core idea is to measure how increasing or decreasing the precision of a single component by one bit affects the post-projection attention reconstruction error, using inputs produced by the current quantized network. This preserves the interaction between precision changes and the existing attention error residual. Dynamic programming then constructs budget-preserving joint updates within a bounded neighborhood of adjacent bit-width changes, and the responses are remeasured at the new allocation.

The paper provides a loss–distortion upper bound under an assumption on downstream network sensitivity, together with an optimality proof for the dynamic programming solution to the local additive allocation problem. Experiments cover three models, two base quantizers, and three precision budgets, reporting the best results in all 18 WikiText-2 perplexity configurations and the highest scores averaged across the three models in all 30 LongBench task–quantizer–budget combinations.

Critical strengths

The allocation objective accounts for the actual attention error in the current quantized computation, and both the method and its theoretical analysis are clearly presented. The finite-change formulation in Eq. (8) preserves the interaction between an update and the existing residual, allowing the method to accommodate signed, nonmonotonic precision responses. The paper also clearly distinguishes exact optimization of the local surrogate objective from the full-model objective, with Eq. (9) separating joint-update interactions from changes in downstream inputs.

Critical weaknesses
  1. Key ablation studies are missing. The paper combines conditional distortion measurements, head-wise allocation, dynamic programming, and other design choices without isolating their individual contributions. Comparisons under the same budget should include a simple strategy that assigns higher precision to early layers, fixed curves without remeasurement, greedy balanced transfers, and layer-wise versus head-wise allocation to establish the necessity of the complete algorithm.

  2. Baseline coverage and comparison controls are insufficient. RateQuant can allocate K/V precision separately for each KV head in each layer, whereas the adapted KVTuner baseline searches over grouped-layer configurations, resulting in different search spaces. Related methods such as RDKV and AATC have already investigated bit-width allocation based on attention distortion, but the paper provides no experimental comparisons with them. Consequently, its advantages over closely related methods remain insufficiently substantiated. Additional results should match the actual bit budgets and compare different scoring criteria and search methods at the same allocation granularity. For methods using different representations, the relationship between actual storage consumption and quality should be reported.

Questions for authors
  1. How much does each mechanism contribute to the overall improvement? Under the same total bit budget, could you provide ablations using fixed curves without iterative remeasurement and replacing dynamic programming with greedy balanced transfers?

  2. Can simpler precision allocation strategies explain the observed gains? Under the same total budget, how large is the gap between RateQuant and a simple strategy that assigns higher precision to the first few layers while correspondingly reducing precision elsewhere? In addition, when the scoring criterion and search method are held fixed, what results are obtained with layer-wise versus head-wise allocation?

  3. What advantages does RateQuant offer over closely related attention-aware allocation methods? Could you provide representative experimental comparisons with RDKV or AATC? Could you also report the relationship between quality and actual storage consumption, including necessary overheads such as quantization metadata, and clarify any differences in applicable settings or implementations?

1 comment
AuthorsOpenReview ↗

Revised PDF uploaded

Dear Reviewer bg21,

Thank you again for your suggestions on the ablations and the comparisons with related methods. We have uploaded the revised PDF, with all changes in blue. The new results are in a new Section 4.6 and a new Appendix F with Tables 9–21. Your points are addressed as follows:

  • W1, Q1, Q2 (fixed curves, greedy transfers, early-layer protection, layer-wise versus head-wise allocation): Section 4.6 and Appendix F.1, Table 10.
  • W2-1 (KVTuner at the same granularity, and the settings where it uses its full budget): Appendix F.2, Tables 12 and 13.
  • W2-2, Q3 (RDKV in all 18 settings): Appendix F.2, Table 11.
  • W2-3, Q3 (storage including quantizer parameters): Appendix F.3, Table 14.

Regards from the authors.

Reviewer_tupmOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose an algorithm for allocating a fixed precision budget to different layers/attention heads/KV cache components of an LLM to minimize an approximation of cross-entropy loss. They compare their method to uniform precision and baselines called KV-sep and KVTuner, on Qwen3-4B, Qwen3-8B, and Llama-3.1-8B. They get better perplexity on WikiText-2 and better performance on the MFQA, 2Wiki, SamSum, Retrieval, and LCC long-context tasks.

Critical strengths

They seem to propose a principled method which significantly outperforms prior work, and show that this holds across various models and datasets. I see no major concerns, though I am not very familiar with the related work.

Critical weaknesses

They compare to baselines at fixed average KV bits, but it would be nice to see a comparison of VRAM/wall-clock time on actual hardware in order to assess whether their method is efficient with modern hardware/deep learning libraries.

Questions for authors
  • Why did you choose to compare models with 2, 2.5, 3 bits? What happens at lower/higher budgets?
  • How does your method allocate its precision over heads/layers/KV components? Did you observe clear patterns/trends?
  • Does the optimal precision allocation depend on workload, e.g. whether the model is doing math vs. coding vs. writing, or whether the KV cache is for a chain of thought vs. final answer?
1 comment
AuthorsOpenReview ↗

Revised PDF uploaded

Dear Reviewer tupm,

Thank you again for your questions on hardware efficiency, the choice of budgets, and the allocation patterns. We have uploaded the revised PDF, with all changes in blue. The new results are in a new Section 4.6 and a new Appendix F with Tables 9–21. Your points are addressed as follows:

  • W1 (memory and decoding throughput on hardware): Section 4.6 and Appendix F.4, Tables 15–17.
  • Q1 (the 2–3-bit range and other budgets): Section 4.1 and Appendix F.2, Table 12. The comparison with the 4-bit NVFP4 format is in Appendix F.5, Tables 19 and 20.
  • Q2 (allocation across K, V, and layers): Section 4.4, Appendix F.6, Table 21, and Appendix F.1, Table 10.
  • Q3 (reasoning and code generation without task-specific recalibration): Section 4.6 and Appendix F.5, Tables 18 and 20.

Regards from the authors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their time and effort in providing careful and constructive feedback on our submission.


We are encouraged to see that our reviewers recognize the strengths of RateQuant in its allocation criterion, its analysis, and its empirical results.

  • Reviewer bg21 recognizes that our allocation objective "accounts for the actual attention error in the current quantized computation", and Reviewer M68u notes a "reasonable allocation criterion and clear methodology".
  • Reviewer bg21 finds that the paper "clearly distinguishes exact optimization of the local surrogate objective from the full-model objective", and Reviewer M68u likewise notes that the analysis "clearly distinguishes exact optimization of the local objective from full-model outcomes".
  • Reviewer tupm notes that the improvement over prior work "holds across various models and datasets", and Reviewer M68u notes consistent "reported improvements and a reasonable transfer protocol".

We highlight our contributions as follows:

  • RateQuant allocates KV cache precision across layers and heads through Measured Attention Responses: under the current allocation, it measures how the attention error changes when a K or V component gains or loses one bit, retaining the cross term with the existing error.
  • By combining these responses with budget-preserving Dynamic-Programming Joint Updates and Iterative Remeasurement, RateQuant achieves the lowest perplexity in all 18 settings of Table 1 and the highest three-model mean score in all 30 LongBench settings of Table 2, without task-specific recalibration.

As suggested by the reviewers, we revised our manuscript and added Section 4.6 and Appendix Section F. The new experiments in our responses to each reviewer now appear in Appendix Section F, with Table 9 through Table 21, and are summarized in Section 4.6. New and revised text is marked in blue.

Additional Results & Experiments

  • Added a paired single-round ablation of the residual cross term in Table 9 and Appendix Section F.1, as suggested by Reviewer M68u.
  • Added matched-budget comparisons with fixed curves without remeasurement, greedy transfers in place of dynamic programming, layer-wise allocation, and first-layer and first-two-layer protection in Table 10 and Appendix Section F.1, as suggested by Reviewers bg21 and M68u.
  • Added the comparison with RDKV in all 18 model–quantizer–budget settings in Table 11 and Appendix Section F.2, as suggested by Reviewers bg21 and M68u.
  • Added the comparison between KVTuner and RateQuant restricted to layer-wise allocation in Table 12, and the eight settings where KVTuner uses its full budget in Table 13, both in Appendix Section F.2, as suggested by Reviewers bg21 and M68u.
  • Added effective storage that includes quantizer parameters and layout information, together with task scores, in Table 14 and Appendix Section F.3, as suggested by Reviewers bg21 and M68u.
  • Added the evaluation of a packed mixed-precision KV cache with matching attention kernels in vLLM, with cache compression in Table 15, decoding throughput under the same GPU memory in Table 16, and offline calibration time in Table 17, all in Appendix Section F.4, in response to Reviewers M68u and tupm.
  • Added Qwen3-32B evaluations on long-chain reasoning (AIME 2024 and 2025) and execution-based code generation (HumanEval+ and MBPP+), with an FP16 reference for both and FP8 and NVFP4 references for code generation, in Table 18 through Table 20 and Appendix Section F.5, as suggested by Reviewer M68u and in response to Reviewer tupm.
  • Added the mean K, V, and first-layer bit widths of the selected allocations in all six model–quantizer configurations in Table 21 and Appendix Section F.6, in response to Reviewer tupm.

Other Changes & Modifications

  • We have added Section 4.6 to summarize the results above in the main text, and updated Section 7 accordingly.
  • We have stated in Section 4.1 why we study 2–3-bit budgets and that other average budgets only change the budget constraint, in response to Reviewer tupm.
  • We have reported the allocation statistics of all six configurations in Section 4.4, in response to Reviewer tupm.
  • We have revised Section 6 and Appendix Section D to describe the packed implementation in vLLM, in response to Reviewers M68u and tupm.
  • We have linked Appendix Sections C.2 and E.2 to the measured calibration time in Table 17 and to the full-budget KVTuner settings in Table 13.
AuthorsOpenReview ↗

Summary of the Discussion

Dear Area Chair and Reviewers,

We thank the Area Chair and all reviewers for their time and constructive comments, which have made this paper more complete. Reviewer tupm recommended clear acceptance (4) in the initial review. During the discussion, Reviewer M68u confirmed that the new ablations, comparisons, and evaluations "have addressed most of my main concerns" and raised the rating from 2 to 3 (Weak acceptance), and Reviewer bg21 also raised the rating from 2 to 3 (Weak acceptance). All three reviewers currently recommend acceptance.

We responded to the comments of all three reviewers with new experiments. Reviewers tupm and M68u asked about memory usage and speed on actual hardware. Our measurements (Tables 15 and 16) show that, with the same GPU memory, RateQuant reaches 2.0–3.0 times the decoding throughput of FP16. The concerns shared by Reviewers bg21 and M68u are the four points below. Each score is the mean over five LongBench tasks, three models, and two quantizers, and each range covers the 2-, 2.5-, and 3-bit budgets.

  • Contribution of each component. With the budget and granularity unchanged, removing remeasurement lowers the score by 1.76–4.99 points, and replacing dynamic programming with greedy transfers lowers it by 1.13–3.13 points. In a paired single-round comparison, removing the residual cross term lowers it by 2.16–7.27 points (Tables 9 and 10). Each component therefore contributes to the full method.
  • Simpler allocation strategies. At the same budget, RateQuant exceeds first-layer and first-two-layer protection by 1.88–8.93 points, and with the scoring and search unchanged, head-wise allocation exceeds layer-wise allocation by 1.06–3.01 points (Table 10). Protecting early layers therefore recovers only part of the gain.
  • Matched comparison with KVTuner and RDKV. Restricted to layer-wise allocation, RateQuant still exceeds KVTuner by 2.34–9.78 points, and it scores higher in all eight settings where KVTuner uses its full budget (Tables 12 and 13). With the same quantizer and budget, RateQuant outperforms RDKV in all 18 settings, by 9.77–14.01 points on average (Table 11). The advantage therefore does not come from a finer search space or a larger realized budget.
  • Quality at equal actual storage. With quantizer parameters and layout information included, RateQuant, Uniform, and KV-sep all use 2.818 bits per element at the 2.5-bit budget, and RateQuant scores 20.15 and 8.55 points higher (Table 14).

In summary, these results show that each component of RateQuant contributes to its gains, and that its advantage over related methods holds under matched granularity, budget, and storage. We thank the Area Chair and all reviewers again for their consideration.

Regards from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.