acceptodds
Under review as a conference paper at ICLR 2027

Gradient Norms, Not Attention Mass: Cheap Proxies for Layer-Wise KV-Cache Bit Allocation

Abstract

Layer-wise mixed-precision KV-cache quantization needs a per-layer sensitivity signal to decide where to spend bits. Search-based methods such as KVTuner measure damage for every (layer, bit-width) pair on calibration data; proxy-based methods replace the search with a cheap statistic. We evaluate three single-pass proxies—attention mass on outlier positions, key-quantization MSE, and the L2 norm of gradients with respect to the K/V projections—inside one matched harness on Phi-4-mini-instruct (3.8B) and Llama-3.2-1B, against our own re-implementation of the KVTuner sensitivity search as the published baseline. The gradient proxy is the only signal whose layer ranking correlates with measured 2-bit damage (Spearman ρ = 0.60 and 0.76 on the two backbones, vs. ≤0.21 for the others), and its allocations beat both activation-based proxies and the search-based allocation at the aggressive 3.25-bit budget, and beat the search on Phi and match it on Llama at 3.5 bits—with the advantage growing as the budget tightens—at two to three orders of magnitude lower calibration cost. This confirms, head-to-head and at matched allocation algorithm, recent rate-distortion analyses that claim gradient sensitivity dominates activation-based heuristics. We further document a failure mode of greedy water-filling on measured damage: when the candidate bit set extends above the budget (e.g. 8-bit candidates at a 4.0-bit budget), the allocator promotes insensitive layers to high precision and pushes sensitive layers into super-linearly damaging 2-bit territory, producing allocations worse than uniform quantization. A restricted-candidate control isolates the failure to the candidate-set width rather than the damage estimates themselves.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, quantization, bit allocation, KVTuner, gradient sensitivity, mixed precision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7Q1zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies cheap sensitivity signals for layer-wise key-cache bit allocation. It compares attention mass on outlier positions, key reconstruction MSE, and K/V projection gradient norms against a measured single-layer damage baseline, using a shared greedy allocator. Experiments on Phi-4-mini-instruct and Llama-3.2-1B use key-only rowwise quantization, fp16 values, and WikiText-2 decode cross-entropy. Gradient norms correlate best with measured 2-bit damage and match or improve on the search-inspired baseline at 3.25/3.5 average key bits. The paper also reports a poor allocation from a wide candidate set at a 4-bit budget, removed by restricting candidates.

Critical strengths
  1. The controlled comparison addresses a useful practical question: whether an inexpensive statistic can replace repeated sensitivity evaluations. Holding the quantizer and evaluation harness fixed, and reporting correlations as well as end quality on two backbones, makes the observed advantage of gradient norms informative.

  2. Including random/inverted rankings and a restricted-candidate control helps diagnose whether gains come from signal quality or allocation behavior. The paper also explicitly acknowledges that the gradient signal is inherited from KVmix rather than presenting it as a new method.

Critical weaknesses
  1. The controlled comparison is useful, but the conclusions about replacing search are broader than the baseline establishes. The paper describes KVTuner as using multi-objective search over precision pairs (Section 2), while its experimental baseline uses measured single-layer key damage followed by the shared greedy allocator (Sections 2-3). Sharing an allocator is a reasonable way to isolate sensitivity signals; however, matching or beating this KVTuner-inspired baseline does not by itself establish matching or beating the published method. This distinction matters because the gradient signal is inherited from KVmix and the head-to-head search comparison is a central part of the new contribution. The additional evidence is also limited to two sub-4B backbones, 24 WikiText-2 documents, and one key-only rowwise quantizer. These experiments support a useful finding within this setting, but provide a limited basis for the broader conclusion that search buys nothing or that gradient sensitivity generally dominates activation statistics.

  2. The wide-candidate experiment demonstrates a bad allocation in this harness, but does not fully establish the proposed mechanism. At an average budget of 4 bits, restricting candidates to {2,3,4} and exhausting the budget necessarily returns uniform 4-bit allocation. Its recovery therefore shows that this restriction avoids the observed failure, but does not establish the stronger claim that the damage estimates were never at fault. The analysis needs to distinguish greedy optimization error on the measured table from errors in predicting joint quantization damage using isolated-layer measurements. This matters because identifying and explaining this failure is another central contribution. The recommendation to clamp candidates to the average budget also needs qualification: the successful 3.25/3.5-bit allocations themselves use 4-bit layers.

Questions for authors
  1. What are the precise differences between the implemented search baseline and published KVTuner, particularly in the allocator and K/V precision configurations? Can the authors either validate the comparison against the published method or consistently limit the claims to the KVTuner-inspired baseline used here?

  2. For the 4-bit wide-candidate failure, how do predicted additive damage and observed joint damage compare for the greedy allocation and uniform allocation? A stronger optimizer over the same measured table (for example, dynamic programming) would be one useful diagnostic: does it avoid the failure, and does its predicted improvement carry through to actual CE? This would help distinguish optimization failure from damage-model error.

  3. Is there additional evidence that the gradient advantage persists beyond the present corpus and quantizer, or with calibration resampling? I am not requesting every combination of larger models, quantizers, and downstream tasks; a focused additional check would help establish whether the main contribution generalizes beyond this harness.

Reviewer_TuSzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares attention mass, key-quantization MSE, and gradient norms as proxies for layer-wise KV-cache bit allocation. On two models, gradient-guided allocation matches or outperforms a reimplemented search baseline at much lower calibration cost. It also identifies a greedy-allocation failure involving high-precision candidates and shows that restricting the candidate set avoids the observed failure.

Critical strengths
  1. The comparison of three inexpensive proxies under a shared quantizer and allocator provides useful evidence about signal choice.
  2. Random and inverted-attention controls help assess whether the signals provide meaningful information for bit allocation.
Critical weaknesses
  1. The presentation needs improvement. Frequent dashes and parenthetical remarks interrupt the reading experience. The use of "arm" in this paper is less intuitive in this context than “method”.
  2. Abbreviations such as CE in Table 2 should be explicitly defined at first use. The bold number in Table 2 is unclear. For Llama-3.2-1B at 3.5 bits, 2.9636 is bolded, but the search baseline achieves a lower CE of 2.9551.
  3. The KVTuner baseline replaces its original pruning, clustering, and multi-objective search with a greedy allocator, leaving it unclear whether proxy-based allocation outperforms the original KVTuner.
  4. The experimental evidence is insufficient to support the author claims. Evaluation is restricted to cross-entropy on WikiText-2, without downstream generation tasks, leaving it unclear whether the reported improvements translate into better generation quality or task performance under low-bit KV-cache quantization.
Questions for authors
  1. Do the CE improvements mean better performance on downstream generation tasks?
  2. Why does the gradient proxy outperform the measured-damage search baseline, and does this advantage hold against the original KVTuner?
  3. What are the exact measured runtimes for each proxy and one search evaluation, and what stages are included?
  4. Why is proxy calibration slower on Llama-3.2-1B than on the larger Phi-4-mini-instruct?
  5. How much total KV-cache memory is saved over uniform 4-bit, and does this justify the higher CE?
Reviewer_ENrNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In order to reduce the offline calibration cost of the mix-precision KV cache bit allocation, the paper evaluates three cheap surrogate metrics and evaluates them under the relatively small and limited models and benchmarks.

The paper uncovers the failure mode of greedy water-filling allocation, which is critical.

Critical strengths
  • Focus on the critical and cheap calibration for mixed precision KV cache compression.
  • Fair and head-to-head comparison of three single-pass sensitivity proxies for layer-wise KV cache bit allocation.
Critical weaknesses
  • No new methods nor metrics are proposed for the layer-wise KV cache bit allocation problem. The novelty is thus limited.
  • Limited evaluations including models and datasets.
  • No in-depth nor theoretical analysis of the metrics.
  • The cross-entropy reduction than KVTuner is not significant.
Questions for authors

How the KVTuner search time cost is computed or estimated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.