acceptodds
Under review as a conference paper at ICLR 2027

When Per-Head KV Precision Helps: Interactions, Allocation, and Scale Timing

Abstract

A common approach to mixed-precision KV caches allocates bits per attention head using isolated sensitivity scores: quantize one head, measure the loss increase, and assign more bits to the heads whose quantization increases loss most. We test three assumptions behind this recipe with paired, teacher-forced measurements on Qwen3-0.6B, Qwen3-1.7B, and Mistral-7B. First, joint low-bit damage is superadditive: when every head is quantized to two bits under a symmetric integer code, joint damage is about 3.3 times the sum of single-head damages on Qwen3-1.7B and 3.0 times on Mistral-7B, whereas four-bit damage on Qwen3-1.7B is nearly additive. Second, under a min-max code, whether keys or values are more sensitive depends on when quantization scales are computed. When scales use complete token groups, two-bit values cause much greater loss than two-bit keys; when scales use only entries already written, this value-side penalty disappears on all three models and the ordering reverses on both Qwen models. Third, the best allocator depends on the operating point: at about 3.3 average bits, measured selection on Qwen3-1.7B lowers loss by 0.044 nats relative to an equal-cost static first-layer rule, whereas at 2.75 bits a static first-16-layer rule gives the lowest held-out loss on Mistral-7B. In a serially filling cache, two-bit keys with four-bit values improve on uniform three bits by 0.16 nats on Mistral-7B at equal modeled cost. Per-head scores thus support useful allocation when combined with joint evaluation of the assembled cache, an explicit scale rule, and an equal-cost static baseline.

Primary area:
generative models
Keywords:
KV cache quantization, mixed precision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N37bOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Paper Summary

In one sentence: a common way to compress a language model's memory is to give extra bits to the parts that hurt most when compressed one at a time; this paper tests whether that works and finds it only holds in some settings.

Problem:

A language model keeps a memory of past tokens (the KV cache), and for long texts this memory is large. Storing it with fewer bits saves space but adds error. Some parts of the memory are more sensitive than others, so a common approach gives those parts more bits. To find them, it compresses one part at a time and checks how much worse the model gets.

Goal:

Check whether testing one part at a time still picks the right parts when the whole memory is compressed together.

Method:

The paper does not propose a new method; it runs tests on three models (Qwen3-0.6B, Qwen3-1.7B, Mistral-7B) using long books, always at the same total memory. It asks three questions: Is the damage from compressing everything at once equal to the sum of the damages from compressing each part alone? Does it matter more to give extra bits to keys or to values, and does the answer change with how the compression range is set? Does testing each part beat a simple rule like "give extra bits to the first layers"?

Results:

  • With 4 bits, the damages roughly add up (shown on Qwen3-1.7B). With 2 bits, compressing everything at once does about 3 times more damage than the sum of the parts.
  • Whether keys or values need extra bits is not fixed; the answer changes with how the compression range is set.
  • Testing each part sometimes beats the simple rule and sometimes does not, depending on the model and the memory budget.

Difference between this paper and the closest baseline:

The closest baseline is KVTuner, from "KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference" (Li et al., 2025, arXiv:2502.04420). KVTuner already gives extra bits to the parts that need them most. This paper adds no new method; it adds measurements of when testing parts one at a time can be trusted.

Critical strengths

Critical Strengths

S1: The paper shows clearly that testing parts one at a time can be misleading at low bits. With 2 bits, compressing everything at once does about 3 times more damage than the sum of the parts, on both Qwen3-1.7B and Mistral-7B.

Why it matters: It tells practitioners to test the whole compressed memory, not just add up per-part scores.

Critical weaknesses

Critical Weaknesses

W1: The "scale timing" claim in the title is not supported. The paper says that whether keys or values need more bits depends on when the compression range is set: from a whole block of tokens (including later ones), or only from tokens already written. But the paper's own setup closest to real use, which also uses only written tokens, does not show this flip. The paper itself admits that "future reading alone does not explain it" and that it has not separated timing from other changes in how the range is set.

Why it matters: The title names scale timing as a main finding, but the evidence does not show that timing is the cause.

W2: KVTuner already does this, and goes further. KVTuner already gives keys and values in each layer their own bits based on measured sensitivity, picks the setting by testing the whole compressed model, and notes that the compression setting changes whether keys or values matter more. The paper says this itself in its related work. KVTuner also tests on math reasoning tasks and reports real speedups (up to 21% higher throughput than 8-bit KIVI), while this paper only simulates compression and never runs KVTuner itself; it compares only to its own "KVTuner-inspired" search.

Why it matters: The paper's only addition is measurements, and without a direct comparison it is unclear whether they would change what KVTuner picks.

W3: No standard KV-cache benchmarks. All tests use loss on PG-19 books and a retrieval test built from the same books. The paper does not report results on benchmarks commonly used for KV-cache compression, such as LongBench, RULER, or Needle-in-a-Haystack, nor on math reasoning tasks like those KVTuner uses.

Why it matters: Without these, it is unknown whether the loss differences matter for real tasks.

Questions for authors

Questions For Authors

See weaknesses.

Reviewer_2kRcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when isolated per-head KV quantization scores support mixed-precision allocation. Experiments show low-bit superadditivity, quantizer-dependent K/V sensitivity, and budget-dependent advantages of measured versus static allocation. A serial-cache experiment also finds K2/V4 superior to uniform 3-bit quantization.

Critical strengths
  1. The research questions are interesting and well motivated.

  2. The experiments are extensive, covering multiple models and quantization settings.

Critical weaknesses
  1. The headline allocation gains lack sufficiently clean generalization evidence. Figure 3(a) and Table 3 select and evaluate heads on the same documents. Appendix A.1b appears to repeat selection on new pools, which establishes reproducibility rather than frozen-allocation generalization. The transfer and held-out Mistral experiments help, but do not directly validate the headline Qwen comparison.

  2. The connection to causal cache operation remains incomplete. Most superadditivity and allocation results use future-inclusive FUT scales. PRE changes scale contents and granularity alongside timing, while the serial experiment does not directly compare measured versus static head allocation or reproduce the joint-to-isolated ratios. Thus, the central findings’ relevance to online allocation remains uncertain.

Questions for authors
  1. Does measured allocation still outperform the static baseline on Qwen3-1.7B at 3.286 bits when calibration and evaluation use disjoint documents?

  2. Do the observed superadditivity and measured-versus-static allocation rankings persist under causal serial-cache quantization?

Reviewer_gPUMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an empirical study of per-head mixed-precision KV-cache quantization. It examines whether isolated head sensitivities predict joint quantization damage, how quantization scale rules affect the relative sensitivity of keys and values, and when measured head allocation outperforms static layer allocation. Experiments primarily use paired, teacher-forced NLL measurements on PG-19-derived samples with Qwen3 and Mistral models, supplemented by context-length, quantizer, and serial-cache controls. Principal findings include substantial low-bit superadditivity, configuration-dependent allocation benefits, and a serial Mistral result where K2/V4 improves over uniform three-bit quantization by approximately 0.160.16 nats at equal modeled payload-plus-residual cost.

Critical strengths
  1. Useful empirical evidence about the limitations of isolated sensitivity measurements. At two bits, joint damage is approximately 3.3×3.3\times the summed isolated damage on Qwen3-1.7B with symmetric quantization and 3.0×3.0\times on Mistral-7B (Table 1). This quantitatively demonstrates why an allocation should be evaluated jointly rather than assessed by summing individual head scores. The distinction between a useful ranking signal and an accurate loss predictor is practically consequential.

  2. Informative comparisons against simple, equal-budget baselines. The paper tests whether calibration actually improves allocation over static layer rules. On held-out Mistral documents at 2.75 payload bits, upgrading the first 16 layers achieves lower loss than measured head selection and the tested layer-width search. Reporting this alongside settings where measured allocation wins provides useful guidance and avoids presenting sensitivity-based allocation as universally beneficial. Paired document-level comparisons and negative results strengthen this contribution.

Critical weaknesses
  1. The conceptual novelty, mechanistic explanation, and predictive guidance remain limited. The paper provides useful empirical findings, but sensitivity-based mixed precision and evaluation of assembled configurations are established approaches; Section 8 acknowledges these connections to KVTuner. The additional contribution is principally a collection of quantified observations and controls. The study does not establish a validated mechanism explaining the observed superadditivity or a criterion predicting when measured allocation will outperform a static rule. Appendix A.9 explicitly leaves the interaction mechanism unresolved. Consequently, the practical guidance largely remains to evaluate each model–quantizer–budget combination. For an empirical contribution, I would find either a more explanatory account or stronger evidence of a transferable design principle necessary to make the significance compelling.

  2. The generalization evidence for the measured-allocation advantage is uneven. Figure 3(a) and Table 3 select and evaluate heads on the same documents, so these comparisons do not independently establish that the selected allocations outperform the baselines on unseen data. The separate calibration/evaluation experiments on Mistral and cross-construction transfers are valuable, but they do not directly validate the headline Qwen advantage under the same setting. Furthermore, the main evidence comes from small PG-19-derived document pools, and the retrieval proxy shares that source. These limitations reduce confidence that the reported allocation benefits extend beyond the evaluated samples and constructions.

Questions for authors
  1. Clarification of evaluation: In the seed-8 replications, are the evaluated allocations frozen from seed-7 calibration, or selected again using seed-8 documents? Which reported results directly establish the Qwen measured-versus-static advantage with head selection and evaluation performed on disjoint books?

  2. Clarification of contribution: Beyond the reported operating points, is there evidence supporting a measurable criterion that predicts when per-head allocation will beat static layer allocation? Please distinguish experimentally supported explanations from hypotheses, and clarify the additional insight relative to the sensitivity-based allocation and final-model evaluation already discussed in Section 8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.