- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies cheap sensitivity signals for layer-wise key-cache bit allocation. It compares attention mass on outlier positions, key reconstruction MSE, and K/V projection gradient norms against a measured single-layer damage baseline, using a shared greedy allocator. Experiments on Phi-4-mini-instruct and Llama-3.2-1B use key-only rowwise quantization, fp16 values, and WikiText-2 decode cross-entropy. Gradient norms correlate best with measured 2-bit damage and match or improve on the search-inspired baseline at 3.25/3.5 average key bits. The paper also reports a poor allocation from a wide candidate set at a 4-bit budget, removed by restricting candidates.
Critical strengths
-
The controlled comparison addresses a useful practical question: whether an inexpensive statistic can replace repeated sensitivity evaluations. Holding the quantizer and evaluation harness fixed, and reporting correlations as well as end quality on two backbones, makes the observed advantage of gradient norms informative.
-
Including random/inverted rankings and a restricted-candidate control helps diagnose whether gains come from signal quality or allocation behavior. The paper also explicitly acknowledges that the gradient signal is inherited from KVmix rather than presenting it as a new method.
Critical weaknesses
-
The controlled comparison is useful, but the conclusions about replacing search are broader than the baseline establishes. The paper describes KVTuner as using multi-objective search over precision pairs (Section 2), while its experimental baseline uses measured single-layer key damage followed by the shared greedy allocator (Sections 2-3). Sharing an allocator is a reasonable way to isolate sensitivity signals; however, matching or beating this KVTuner-inspired baseline does not by itself establish matching or beating the published method. This distinction matters because the gradient signal is inherited from KVmix and the head-to-head search comparison is a central part of the new contribution. The additional evidence is also limited to two sub-4B backbones, 24 WikiText-2 documents, and one key-only rowwise quantizer. These experiments support a useful finding within this setting, but provide a limited basis for the broader conclusion that search buys nothing or that gradient sensitivity generally dominates activation statistics.
-
The wide-candidate experiment demonstrates a bad allocation in this harness, but does not fully establish the proposed mechanism. At an average budget of 4 bits, restricting candidates to {2,3,4} and exhausting the budget necessarily returns uniform 4-bit allocation. Its recovery therefore shows that this restriction avoids the observed failure, but does not establish the stronger claim that the damage estimates were never at fault. The analysis needs to distinguish greedy optimization error on the measured table from errors in predicting joint quantization damage using isolated-layer measurements. This matters because identifying and explaining this failure is another central contribution. The recommendation to clamp candidates to the average budget also needs qualification: the successful 3.25/3.5-bit allocations themselves use 4-bit layers.
Questions for authors
-
What are the precise differences between the implemented search baseline and published KVTuner, particularly in the allocator and K/V precision configurations? Can the authors either validate the comparison against the published method or consistently limit the claims to the KVTuner-inspired baseline used here?
-
For the 4-bit wide-candidate failure, how do predicted additive damage and observed joint damage compare for the greedy allocation and uniform allocation? A stronger optimizer over the same measured table (for example, dynamic programming) would be one useful diagnostic: does it avoid the failure, and does its predicted improvement carry through to actual CE? This would help distinguish optimization failure from damage-model error.
-
Is there additional evidence that the gradient advantage persists beyond the present corpus and quantizer, or with calibration resampling? I am not requesting every combination of larger models, quantizers, and downstream tasks; a focused additional check would help establish whether the main contribution generalizes beyond this harness.