Gradient Norms, Not Attention Mass: Cheap Proxies for Layer-Wise KV-Cache Bit Allocation

Layer-wise mixed-precision KV-cache quantization needs a per-layer sensitivity signal to decide where to spend bits. Search-based methods such as KVTuner measure damage for every (layer, bit-width) pair on calibration data; proxy-based methods replace the search with a cheap statistic. We evaluate three single-pass proxies—attention mass on outlier positions, key-quantization MSE, and the L2 norm of gradients with respect to the K/V projections—inside one matched harness on Phi-4-mini-instruct (3.8B) and Llama-3.2-1B, against our own re-implementation of the KVTuner sensitivity search as the published baseline. The gradient proxy is the only signal whose layer ranking correlates with measured 2-bit damage (Spearman ρ = 0.60 and 0.76 on the two backbones, vs. ≤0.21 for the others), and its allocations beat both activation-based proxies and the search-based allocation at the aggressive 3.25-bit budget, and beat the search on Phi and match it on Llama at 3.5 bits—with the advantage growing as the budget tightens—at two to three orders of magnitude lower calibration cost. This confirms, head-to-head and at matched allocation algorithm, recent rate-distortion analyses that claim gradient sensitivity dominates activation-based heuristics. We further document a failure mode of greedy water-filling on measured damage: when the candidate bit set extends above the budget (e.g. 8-bit candidates at a 4.0-bit budget), the allocator promotes insensitive layers to high precision and pushes sensitive layers into super-linearly damaging 2-bit territory, producing allocations worse than uniform quantization. A restricted-candidate control isolates the failure to the candidate-set width rather than the damage estimates themselves.