- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how calibration timing and scale granularity affect rotated uniform KV-cache quantization. It compares prefill-frozen and per-token calibration with global and grouped scales, accounting for their metadata costs. Per-token calibration substantially improves four-bit results on Llama, but all tested four-bit configurations fail on Qwen. The study highlights calibration as a confounding factor when comparing scale granularities and shows that its benefits depend on the model.
Critical strengths
-
The factorial design separates calibration timing from spatial scale granularity, helping clarify which factor drives the observed improvements.
-
The paper explicitly accounts for scale and offset storage, making clear that equal payload precision does not imply equal total memory cost.
Critical weaknesses
- The paper lacks a main figure that clearly presents the method and key observations. This weakens the presentation and makes the paper difficult to follow.
- The evaluation focuses mainly on post-prefill perplexity, with no downstream QA or reasoning experiments. SQuAD is used only as a source of text contexts. It therefore remains unclear whether the observed improvements translate into better task performance.
- The paper mainly reports observations of calibration for quantization, with limited novelty and contribution. I do not think it meets the bar for ICLR.
Questions for authors
Using perplexity as the main measure of quantization loss seems rather dated. Could you provide results on downstream benchmarks, such as QA or reasoning tasks, to validate your observations?
2 comments
Thanks for reply. I will keep my score.
The calibration-timing effect also appears in generated-answer accuracy. We completed 1,000 downstream evaluations and added K/V interventions and storage-budget controls.
We evaluated Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on 50 SQuAD v1.1 validation questions and 50 GSM8K test problems, selected by a fixed hash ordering before generation. Each model uses the same questions and prompts in all five conditions. SQuAD is now evaluated as question answering, not merely as a source of text. GSM8K adds a non-Wikipedia mathematical-reasoning setting.
All quantized arms use the same signed rotation seed, asymmetric grids, and persistent one-write QDQ; grouped scales use 32 channels. Generation is greedy, capped at 48 new tokens for QA and 384 for math. Length-capped outputs remain in the denominator. QA uses normalized exact match and token F1; math uses a fixed numeric-answer extraction rule. All 1,000 saved outputs were independently regraded, with zero discrepancies.
For Llama, changing calibration timing at fixed four-bit global granularity raises QA accuracy by 20 percentage points and math accuracy by 74 points. Task-paired bootstrap 95% intervals are [10, 32] and [60, 86] points. QA F1 increases from 81.7% to 94.7%. Per-write global and grouped calibration yield the same QA accuracy and closely matched math accuracy. Qwen retains severe four-bit degradation, while its eight-bit results remain close to the dense baseline.
On Llama math, 47/50 prefill-frozen runs hit the 384-token cap, compared with 4/50 per-write global runs under the same cap. The timing change improves both accuracy and completion behavior.
K/V interventions locate Qwen's sensitivity. On a fixed diagnostic window, rotated four-bit K-only quantization gives PPL 1284.31, whereas V-only gives 7.37 against a dense reference of 7.39. Without rotation, K-only gives 1683.98 and V-only 7.39. Key quantization reproduces the failure in both bases; value quantization alone does not.
We also tested how to spend a common total-storage budget. On Llama, symmetric per-write global four-bit quantization gives PPL 5.8866 at 4.03125 effective bits per KV element. Grouped four-bit gives 5.8541 at 5.00000 bits, but global five-bit gives 5.8146 at 5.03125 bits. At the common 5.03125-bit ceiling, spending the budget on payload precision outperforms spending it on finer scale metadata. These are modeled payload-plus-metadata costs, not packed-kernel measurements.
These experiments establish three design-relevant findings: calibration timing changes downstream correctness at fixed precision and granularity; Qwen's failure is concentrated on key quantization; and spending metadata overhead on payload precision can reverse the global-versus-grouped comparison.
We have prepared a four-panel figure showing the execution path, downstream accuracy, K/V sensitivity, and total-storage comparison. The revision will open with this figure and organize the results around three questions: when to fit scales, which cache component needs precision, and how to allocate the storage budget. The mathematical motivation will follow the experimental overview.