acceptodds
Under review as a conference paper at ICLR 2027

Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation

Abstract

Low-bit key–value (KV) caches reduce the memory required for autoregressive decoding, but the resulting quality loss depends on the model and quantizer. We keep the quantizer fixed and distill the floating-cache model's behavior into low-rank Q/K/V projection updates while the student executes a physically packed incremental cache. Across three seeds, 4-bit affine-cache adapters recover of the held-out perplexity gap on TinyLlama-1.1B and on Gemma-4-12B. On the same frozen NF4 Llama-3.1-8B base, one validation-selected run per quantizer recovers under KIVI K2V2 and under KVarN K4V2, while preserving 180-case associative retrieval. Gemma's score on an official 4K/8K RULER subset rises from 42.80 with the unadapted 4-bit cache to 48.33 after adaptation (46.15 floating), with substantial task heterogeneity. Finally, a 2-bit rank–token sweep reduces TinyLlama's 2-bit PPL from 576.10 to across three seeds, versus 10.3988 floating, but restores only 11–12 of 180 retrieval cases. These results show that low-rank projection adaptation can recover held-out quality across fixed cache formats, while perplexity recovery need not restore long-context retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache quantization, LoRA, error correction, knowledge distillation, cache compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aqteOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether the quality degradation caused by low-bit KV-cache quantization can be recovered without modifying the quantizer itself. The proposed approach keeps both the base model and cache format fixed, and trains low-rank adapters on the attention Q/K/V projections through self-distillation from the floating-cache model.

Critical strengths
  • The method itself seems to be working well. The experiment results look not bad on several benchmarks.
Critical weaknesses
  • The writing is very hard to understand; in the introduction, a lot of things are not defined. In the methodology section, the author uses techniques from KIVI and KVarN without giving any preliminaries.

  • The paper did not report metrics like latency and throughput. Even I understand this paper is focusing more on the effectiveness of the adapter itself, but I'm still curious about the inference overhead it introduces and also how long it takes to train this adapter. Is it really worth it?

  • Lack of motivation: it's like simply combining quantization and adapter together. Why should this adapter be designed specifically for this quantization scenario?

Questions for authors
  • This setting seems to be working on the online updating scenarios. Which means if you see a new data sample, you can use this sample to update the LoRA adapter. This experiment is not required at the rebuttal, but I hope the author can give some discussion and it would be great to see some preliminary testing.
2 comments
AuthorsOpenReview ↗

Response to Reviewer aqte

Thank you for the review. The presentation concerns led to a restructured paper.

W1. Writing and preliminaries. We rewrote the paper. Section 2 now introduces the KV cache, the quantization scheme, KIVI and KVarN before they are used; each term is defined at first use, and the method has a schematic. Does the revised version read better?

W2. Overhead and training time; is it worth it?

  • Inference. The adapter adds no cache bytes. Kept separate over 4-bit weights, it adds 72–84 MiB and at most 11.4% to decode latency (ms/token at batch size 1). Merged into full-precision weights, it is within ±2% of the adapter-free model.
  • Training. Calibration takes 4.5–4.9 minutes on the 1.1B model at 4 bits and 11–20 hours for the 8B and 12B models, on one 20 GB GPU.
  • Is it worth it? It closes 54% of the quality gap with 0.1% of the parameters that full-weight QAT trains (Section 6.2).

W3. Motivation. In practice, a deployment rarely chooses its cache format for quality alone. The format has to match an attention kernel that is hardware-accelerated and maintained by the serving framework's community. vLLM, for example, ships a fixed FP8 cache with native kernels motivated by hardware support and reports small quality losses that persist for some models (https://vllm.ai/blog/2026-04-22-fp8-kvcache). Going below 8 bits saves more memory but costs more quality, so cheap recovery under a fixed format is one ingredient for moving this practice to low-bit caches. Adapting the model provides that recovery without adding cache memory. The query, key and value projections are the weights that write and read the cache, so the adapters go there and are trained through the quantized cache.

Q. Online updating. The setting allows it since training is label-free. In a preliminary test (60 label-free updates on three MMLU subjects, 581 held-out examples), no accuracy improvement was observed: 50.6% against 52.3% for the static adapter (−1.72 points, paired 95% interval [−4.48, +1.04]). A tuned study of online schedules is beyond the scope of this paper.

Thank you for the effort on rewriting the paper, and also the additional overhead report. I encourage the author to keep refine the paper. I'll keep my original score.

Reviewer_MwXBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper keeps a KV-cache quantizer fixed and trains LoRA adapters on Q/K/V by distilling the floating-cache model into a student that runs the packed low-bit cache, with an identity STE through the quantizer. At 4-bit affine caches it recovers 54% (TinyLlama-1.1B, QKV) and 76% (Gemma-4-12B on NF4 weights, QK) of the held-out perplexity gap over three seeds; single runs on Llama-3.1-8B NF4 recover 60% under KIVI K2V2 and 38% under KVarN K4V2. At 2-bit, TinyLlama perplexity falls from 576 to 11.4 (float 10.4) but NIAH retrieval recovers only 11–12 of 180 cases. A four-task RULER subset on Gemma rises from 42.8 to 48.3 (float 46.2) with strong per-task redistribution.

Critical strengths
  • Careful, honest reporting (seed-level statistics, paired fix/break counts, explicit one-seed caveats). The 2-bit result is a useful negative finding: near-floating perplexity does not restore long-context retrieval, which argues against perplexity-only evaluation of KV-cache quantizers.
  • The recipe is quantizer-agnostic and was validated against two external pipelines (KIVI, KVarN) with exact tensor matching.
Critical weaknesses
  • Incremental contribution with no baselines. LoRA self-distillation through a quantized forward pass is a standard QAT/QLoRA recipe, yet no prior correction method (GEAR, KVLinC, STAR-KV) and no cheap alternative (tuning norms/biases, full QAT, a larger KIVI residual window) is compared at matched calibration budget, so the recovery percentages cannot be placed.
  • The method matters least where it works and most where it fails. At 4-bit the absolute gaps are 0.3 PPL (TinyLlama) and 1.1 PPL (Gemma) and adaptation closes about half; at 2-bit retrieval stays at 6% vs 94% floating. The RULER "recovery" above floating (single-needle drops 84→56 at 4K while CWE rises 30→69) looks like behavioral drift from WikiText distillation, and no general benchmark checks for regressions. Gemma's ~2% skipped non-finite updates are unexplained.
Questions for authors
  1. Could you add one prior correction baseline and one trivial baseline (same token budget; norms/biases only, or full QAT) under identical caches?
  2. Does adapted Gemma regress on a general benchmark (MMLU, LongBench)? Why does single-needle accuracy drop sharply while the aggregate rises?
  3. What causes the non-finite updates at learning rate 1e-6 on Gemma (STE, NF4 dequantization, scale underflow)?
  4. Could you report measured decode latency for the NF4 runs that keep explicit adapters?
  5. Is the 2-bit retrieval failure capacity-limited (rank/tokens still rising) or structural? One run at rank 96 or 40M tokens would settle this.
1 comment
AuthorsOpenReview ↗

Response to Reviewer MwXB

Thank you for your review and for the specific questions, each of which led to a new experiment.

W1 / Q1. Baselines. We added the baselines that fit a fixed cache format, on the same model and evaluation (Section 6.2; the table is in the general comment).

  • Norms-only tuning closes 29% of the gap, KVLoRA 54%, and full-weight QAT about 100%.
  • GEAR, in one memory-matched configuration of our implementation, scores below the unadapted cache.
  • Enlarging KIVI's window from 128 to 384 tokens reaches 7.029 PPL with 31% more cache memory; the adapter reaches 6.860 at the original window (separate runs).
  • KVLinC and STAR-KV define their own cache formats, so Section 3 positions them by what they change.

We agree that the training components are established. The contribution is the fixed-format, cache-only setting and its measured place among these alternatives on the cost-quality tradeoff.

W2. Where the method matters. The gain is larger where the gap is larger: under KIVI's 2-bit cache on Llama (one run), the adapter closes 60% of a 0.49 PPL gap and regains 1.4 of the 4.8 MMLU points lost. Deployed formats also tend to be simple ones. The vLLM project blog recommends FP8 cache as the default starting point because it is simple and often good enough. Going below 8 bits saves more memory but costs more quality. Our aim is cheap quality recovery on established low-bit cache formats, not more storage methods.

Q2. General benchmarks; single-needle drop. The evidence shows no regression on Gemma: the adapted model is at or above the unadapted 4-bit cache on both benchmarks.

Full precisionQuantizedAdapted
Gemma MMLU71.5870.3570.49
Gemma LongBench17.6018.3120.29
Llama MMLU64.3659.5660.97
Llama LongBench19.8118.2017.97

The single-needle drop came from our prompts lacking RULER's answer prefix. With the official prompt, single-needle retrieval is 100% in every condition. The rise in common-word extraction is reported as a general adapter effect and not as quantization recovery, because the 4-bit cache already scores above full precision on that task (Section 6.4).

Q3. Non-finite updates. In a failing update captured in replay, the first non-finite value is an FP16 overflow in the gradient through the quantizer's key scale (1.46 × 10^5, above the FP16 maximum of 65,504). Of your three candidates, this evidence points to the STE backward pass. The scales were nonzero, which argues against underflow; NF4's contribution was not tested. Holding the scale in FP32 removes the overflow (Appendix H); the reported runs use the original implementation.

Q4. Decode latency with unmerged adapters over NF4. Measured at batch size 1 with framework attention on the reconstructed cache, the adapter adds at most 11.4% to decode latency across six context lengths. It adds 72–84 MiB of memory and no cache bytes. The two endpoints are below, and the full table is in Appendix C.

ms/token512 tokens32,768 tokens
Gemma85.2 → 93.8 (+10.2%)107.3 → 110.0 (+2.5%)
Llama with KIVI134.5 → 141.2 (+5.0%)913.1 → 912.9 (0%)

Q5. Two-bit: capacity or structure? The evidence suggests structural. Across ranks 6–96 and up to 40M tokens, retrieval never exceeds 12/180, against 170/180 with the full-precision cache. The rank-96 run you suggested retrieves 9/180 at 40M tokens (it has its own schedule and suite, so it is not a paired rank comparison). On the unadapted model, 2-bit keys with 16-bit values retrieve 0/180 and 16-bit keys with 2-bit values 158/180, which points to key precision.

Reviewer_ZLUeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies quality recovery after the KV-cache quantization format has been fixed. It freezes the base model and quantizer and trains low-rank updates to attention projections by distilling predictions from the same base model with a floating-point cache. The student uses a physically packed incremental cache and straight-through gradients through cache quantization.

Experiments cover TinyLlama, Gemma, and Llama-3.1, including affine/symmetric caches and implementations of KIVI and KVarN. The main experiments demonstrate reproducible reductions in held-out perplexity, while approximately parameter-matched comparisons favor QKV adaptation. A rank–training-budget study and retrieval evaluations also demonstrate an important limitation: substantial perplexity recovery, particularly at 2 bits, need not restore long-context retrieval.

Critical strengths

(1) Tables 1–2 show consistent held-out perplexity improvements, with three seeds for the principal affine-cache results and approximately parameter-matched adapter-placement comparisons.

(2) The paper reports absolute perplexity, task-level heterogeneity, and statistical limitations rather than relying solely on favorable aggregate recovery percentages. This improves the interpretability of the findings.

Critical weaknesses

(1) The incremental contribution over relevant training-based approaches is insufficiently established. LLM-QAT already studies distillation-based quantization-aware training, including KV-cache quantization; LR-QAT develops low-rank QAT and evaluates configurations including KV quantization; and KV-Distill uses parameter-efficient distillation for compressed contexts, although its compression mechanism differs. The proposed fixed-format, attention-only setting is a meaningful distinction, but the paper does not sufficiently explain or demonstrate what advantage it provides over these related approaches.

(2) The practical quality–cost benefit remains insufficiently characterized. The main perplexity evaluations use short reset windows, while the RULER evaluation covers only four tasks, two lengths. No end-to-end latency, peak-memory, or training-time measurements are reported. Consequently, the evidence establishes perplexity recovery more clearly than a favorable deployment trade-off.

Questions for authors

(1) What is the principal technical or empirical distinction from LLM-QAT and LR-QAT beyond restricting adaptation to attention projections and fixing the cache format?

(2) Do existing logs contain calibration wall-clock time and peak training memory for the principal settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZLUe

Thank you for your detailed review.

W1 / Q1. Distinction from LLM-QAT, LR-QAT and KV-Distill. We freeze the deployed weights and the cache format, and train only adapters on the attention projections. LLM-QAT and LR-QAT quantize the cache together with weights and activations, so the deployed weights change; KV-Distill changes what the cache stores. The advantage is that the adapter can be added to an existing deployment (shown with KIVI and KVarN) while training under 1% of the parameters. The cost is measured in Section 6.2: KVLoRA closes 54% of the gap while our full-weight QAT control closes about 100% with 957× the parameters.

W2. Quality–cost benefit. Added: decode latency and memory (Section 6.6), and MMLU plus six LongBench tasks with inputs up to 32,768 tokens (Section 6.4). RULER is re-run on two tasks with the official prompt. On Llama with KIVI K2V2, KVLoRA lowers PPL from 7.159 to 6.860 at 2,048-token resets and regains 1.4 out of 4.8 points on MMLU. Left unmerged with NF4 weights, the adapter costs 72 MiB of memory and a 0-5% increase in decode latency (BS=1).

Q2. Calibration time and peak memory. Wall time was logged (Table 1): 4.5–4.9 minutes for TinyLlama at 4 bits, 13 hours for Gemma, 11–20 hours for Llama. Peak memory was not, so we measured it over the first 50 updates of the TinyLlama and NF4 Gemma recipes (batch size 1, eight sequences per update): 3.7 GB and 14.0 GB. All runs fit one 20 GB GPU.

Thank you for your response. It would be great to incorporate the content into a future endeavor.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes

We thank the reviewers. The main revisions are as follow:

1. Controlled comparison (Section 6.2). TinyLlama, 4-bit cache; trained rows share data, objective and token budget.

MethodTrainable paramsTest PPLGap closedRetrieval /180
Full-precision cache–10.399–171
4-bit cache, no adaptation010.7230%163
GEAR, one memory-matched configuration (our implementation)011.160<0158
Norms-only tuning92K10.62829%165
KVLoRA (ours, 3 seeds)1.15M10.547 ± 0.00854 ± 2.5%167–169
LoRA on all linear layers1.15M10.48773%168
Full-weight QAT (upper reference)1,100M10.393≈100%173

2. Cost (Section 6.6). No added cache bytes. Kept separate over 4-bit weights, the adapter adds 72–84 MiB and at most 11.4% decode latency. Calibration takes 4.5–4.9 minutes on TinyLlama at 4 bits and 11–20 hours for the 8B and 12B models, on one 20 GB GPU.

3. Transfer (Section 6.4). MMLU and LongBench added. Our earlier RULER prompts lacked the benchmark's answer prefix; two tasks are re-run with the official prompt and replace the earlier four-task results.

4. Two-bit limit (Section 6.5). A rank-96 run with up to 40M tokens did not restore retrieval; the evidence points to key precision.

5. Presentation. New background section, reorganised related work, method schematic.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.