acceptodds
Under review as a conference paper at ICLR 2027

Calibration Timing and Model Sensitivity in Rotated KV-Cache Quantization

Abstract

Does a rotated uniform KV-cache quantizer require locally fitted scales? A comparison can conflate scale granularity with calibration timing and metadata cost. We audit these factors with non-overlapping finite windows, a dense non-evicted reference, and one-write quantize–dequantize simulation. Following development experiments, a frozen 80-condition comparison crosses model, dataset, rotation seed, calibration timing, granularity, and symmetry, with eight-bit controls. On Llama-3.1-8B-Instruct, four-bit per-write global calibration yields post-prefill perplexities of 7.508–7.557 on WikiText-2 test (reference 7.404), whereas prefill-frozen global configurations yield 125.414–39,255.244. The timing effect persists on SQuAD contexts and for two new rotations. Grouped per-write scales reduce error further but incur more metadata, preventing an equal-total-budget claim. Qwen2.5-7B-Instruct behaves differently: all tested four-bit configurations fail severely across both datasets and rotations, while eight-bit controls remain close to their dense references. Unquantized cache and rotation-roundtrip controls also remain close on a diagnostic window. These results support calibration- and model-specific conclusions rather than universal necessity of local scales. They do not establish packed-kernel speed, cross-domain generalization, or a unique mechanism for the Qwen low-bit failure.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache, quantization, calibration, language models, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FzB7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how calibration timing and scale granularity affect rotated uniform KV-cache quantization. It compares prefill-frozen and per-token calibration with global and grouped scales, accounting for their metadata costs. Per-token calibration substantially improves four-bit results on Llama, but all tested four-bit configurations fail on Qwen. The study highlights calibration as a confounding factor when comparing scale granularities and shows that its benefits depend on the model.

Critical strengths
  1. The factorial design separates calibration timing from spatial scale granularity, helping clarify which factor drives the observed improvements.

  2. The paper explicitly accounts for scale and offset storage, making clear that equal payload precision does not imply equal total memory cost.

Critical weaknesses
  1. The paper lacks a main figure that clearly presents the method and key observations. This weakens the presentation and makes the paper difficult to follow.
  2. The evaluation focuses mainly on post-prefill perplexity, with no downstream QA or reasoning experiments. SQuAD is used only as a source of text contexts. It therefore remains unclear whether the observed improvements translate into better task performance.
  3. The paper mainly reports observations of calibration for quantization, with limited novelty and contribution. I do not think it meets the bar for ICLR.
Questions for authors

Using perplexity as the main measure of quantization loss seems rather dated. Could you provide results on downstream benchmarks, such as QA or reasoning tasks, to validate your observations?

2 comments

The calibration-timing effect also appears in generated-answer accuracy. We completed 1,000 downstream evaluations and added K/V interventions and storage-budget controls.

  1. Downstream QA and mathematical reasoning

We evaluated Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on 50 SQuAD v1.1 validation questions and 50 GSM8K test problems, selected by a fixed hash ordering before generation. Each model uses the same questions and prompts in all five conditions. SQuAD is now evaluated as question answering, not merely as a source of text. GSM8K adds a non-Wikipedia mathematical-reasoning setting.

Cache configurationLlama QA EMLlama math accuracyQwen QA EMQwen math accuracy
Dense FP1688%82%88%84%
4-bit global, prefill-frozen68%10%6%2%
4-bit global, per-write88%84%0%0%
4-bit grouped, per-write88%82%18%4%
8-bit global, per-write88%82%86%86%

All quantized arms use the same signed rotation seed, asymmetric grids, and persistent one-write QDQ; grouped scales use 32 channels. Generation is greedy, capped at 48 new tokens for QA and 384 for math. Length-capped outputs remain in the denominator. QA uses normalized exact match and token F1; math uses a fixed numeric-answer extraction rule. All 1,000 saved outputs were independently regraded, with zero discrepancies.

For Llama, changing calibration timing at fixed four-bit global granularity raises QA accuracy by 20 percentage points and math accuracy by 74 points. Task-paired bootstrap 95% intervals are [10, 32] and [60, 86] points. QA F1 increases from 81.7% to 94.7%. Per-write global and grouped calibration yield the same QA accuracy and closely matched math accuracy. Qwen retains severe four-bit degradation, while its eight-bit results remain close to the dense baseline.

On Llama math, 47/50 prefill-frozen runs hit the 384-token cap, compared with 4/50 per-write global runs under the same cap. The timing change improves both accuracy and completion behavior.

  1. What the additional controls contribute

K/V interventions locate Qwen's sensitivity. On a fixed diagnostic window, rotated four-bit K-only quantization gives PPL 1284.31, whereas V-only gives 7.37 against a dense reference of 7.39. Without rotation, K-only gives 1683.98 and V-only 7.39. Key quantization reproduces the failure in both bases; value quantization alone does not.

We also tested how to spend a common total-storage budget. On Llama, symmetric per-write global four-bit quantization gives PPL 5.8866 at 4.03125 effective bits per KV element. Grouped four-bit gives 5.8541 at 5.00000 bits, but global five-bit gives 5.8146 at 5.03125 bits. At the common 5.03125-bit ceiling, spending the budget on payload precision outperforms spending it on finer scale metadata. These are modeled payload-plus-metadata costs, not packed-kernel measurements.

These experiments establish three design-relevant findings: calibration timing changes downstream correctness at fixed precision and granularity; Qwen's failure is concentrated on key quantization; and spending metadata overhead on payload precision can reverse the global-versus-grouped comparison.

  1. Main figure and presentation

We have prepared a four-panel figure showing the execution path, downstream accuracy, K/V sensitivity, and total-storage comparison. The revision will open with this figure and organize the results around three questions: when to fit scales, which cache component needs precision, and how to allocate the storage budget. The mathematical motivation will follow the experimental overview.

Thanks for reply. I will keep my score.

Reviewer_yCMUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether the benefit of grouped scales in rotated KV cache quantization comes from scale granularity or calibration timing. Calibration timing, granularity, and quantizer symmetry are controlled independently, while accounting for scale and offset metadata. On Llama-3.1-8B-Instruct, per-write global calibration substantially reduces the PPL gap to grouped calibration. However, on Qwen2.5-7B-Instruct, all tested 4-bit configurations fail severely, while 8-bit controls retain performance close to the FP16 reference. These findings are reported on WikiText-2 test and SQuAD validation contexts under simulated quantization, using randomized signed Hadamard rotations with two different seeds.

Critical strengths
  1. The paper addresses a useful question: whether the apparent benefit of grouped scales comes from scale granularity or calibration timing.
  2. The paper explicitly accounts for scale and offset metadata, distinguishing nominal bit-width from effective storage cost and modeled compression from actual memory usage.
  3. The claims are appropriately restrained. The paper reports severe failures and acknowledges the limitations of its evaluation without claiming universal improvements or deployment benefits.
Critical weaknesses

My main concern is that the paper's presentation needs significant improvement, and the empirical relevance of the theoretical propositions is not clearly established.

  1. The writing is difficult to follow. The manuscript interleaves problem definitions, experiments, and discussions, making it hard to identify the authors’ core claims and contributions.
  2. KV cache compression is particularly relevant to decode-oriented tasks with long generation, such as reasoning, coding, and agentic tasks. However, the evaluation is limited to short-window, teacher-forced perplexity, leaving its practicality unclear.
  3. The calibration-timing claim should be evaluated on existing KV cache compression methods such as KIVI [1] and KVQuant [2]. Simply ablating calibration timing and grid and reporting PPL does not establish generality beyond the tested quantizer.
  4. The theoretical analysis is not connected to the empirical results. A rotated-versus-unrotated comparison and measurements of attention-logit error ranges would help establish the relevance of Sections 2.1 and 2.2. Also, the connection between the theoretical notation and the actual quantization setting is unclear.
Questions for authors

I have several questions for the authors:

  1. What causes the Qwen failure? K-only versus V-only quantization, layer-wise analysis, and rotated-versus-unrotated comparisons would help identify it.
  2. Does the calibration-timing finding hold for existing methods such as KIVI [1] and KVQuant [2], where applicable?
  3. Can the authors compare global and grouped calibration at matched total storage, including scale and offset metadata?
  4. Do the findings hold on more recent models (such as Qwen3) and long-generation tasks (such as AIME2025 [3] and LCB v5 [4])?
  5. Can the authors clarify how the notation and assumptions in Section 2 map to the actual quantization setting, ideally with supporting experimental results?

[1] Z. Liu et al., “KIVI: A tuning-free asymmetric 2bit quantization for KV cache,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), vol. 235, 2024, pp. 32332–32344.

[2] C. Hooper et al., “KVQuant: Towards 10 million context length LLM inference with KV cache quantization,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 37, 2024, pp. 1270–1303.

[3] https://huggingface.co/datasets/opencompass/AIME2025

[4] https://huggingface.co/datasets/livecodebench/code_generation_lite, release_v5

2 comments

We added the requested K/V and rotation interventions, measured layer-wise key and attention errors, compared designs under a common storage budget, and completed 1,000 generated-answer evaluations.

  1. What causes the Qwen failure?

On a fixed 512-token window with 128-token prefill, grouped asymmetric four-bit quantization gives:

Qwen cache interventionRotated PPLUnrotated PPL
K only1284.311683.98
V only7.377.39
K+V1154.051836.73

The dense reference is 7.3912; rotation-roundtrip and rotated eight-bit K+V give 7.3936 and 7.3938. Key quantization reproduces the failure in both bases; value quantization alone does not. Llama's rotated four-bit K-only result is 5.5105 versus reference 5.4873. Qwen's layer 27 (zero-indexed) has prefill K RMS 40.76, rotated peak 204.19, and reconstruction MSE 11.66; the largest Llama K MSE is 0.0258. These layer measurements identify selective key precision as a concrete next intervention.

  1. Downstream reasoning, recent models, and long generation

We evaluated Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on 50 SQuAD validation questions and 50 GSM8K test problems, selected before generation by fixed hash ordering. Each model sees identical tasks/prompts across five conditions:

ConfigurationLlama QA/mathQwen QA/math
Dense FP1688% / 82%88% / 84%
Global4, prefill-frozen68% / 10%6% / 2%
Global4, per-write88% / 84%0% / 0%
Group4, per-write88% / 82%18% / 4%
Global8, per-write88% / 82%86% / 86%

QA uses normalized exact match; math uses fixed numeric-answer extraction. All arms use asymmetric grids, one-write QDQ, greedy decoding, and 48/384-token QA/math output limits. Capped outputs are retained. Llama's timing gains are +20 QA and +74 math percentage points, with paired bootstrap 95% intervals [10,32] and [60,86]. Regrading all 1,000 outputs produced no discrepancies. The timing effect therefore extends to generated answers. The added benchmarks are SQuAD and GSM8K; Qwen3, AIME2025, and LCB v5 remain untested.

  1. Matched total-storage budget

With FP32 symmetric scale metadata, Group4/group32 costs 5 bits per element. Global5 costs 5+32/(H_KV d), or 5.03125 on Llama. Under this common ceiling, on two 1024-token windows with prefill 256, Global5 gives PPL 5.8146 versus Group4's 5.8541 (dense 5.7816). At equal four-bit payload, Global4 gives 5.8866. Spending the additional budget on global payload precision reverses the grouped advantage. Group4 leaves 0.03125 bits/element unused. Symmetric grids store one scale; asymmetric grids additionally store an offset. The comparison accounts for modeled payload and metadata, while execution uses QDQ.

  1. KIVI and KVQuant

The timing effect has not yet been tested within KIVI or KVQuant. KIVI uses per-channel keys and per-token values; KVQuant also uses pre-RoPE key quantization, nonuniform representations, and separate outlier treatment. A timing-only comparison must preserve those choices. Our K/V results support their distinction between key and value design, while the new timing evidence concerns the controlled uniform quantizer evaluated here.

  1. Theory-to-implementation mapping and presentation

Here xx is a cached K/V vector, RR is the signed orthogonal rotation, and d=128d=128 is the head dimension. For fixed actual queries, z=qK⊤/dz=qK^\top/\sqrt{d}, η=z−zquantized\eta=z-z_{\mathrm{quantized}}, and Δ=max⁡iηi−min⁡iηi\Delta=\max_i\eta_i-\min_i\eta_i, where ii ranges over causally visible keys. We measure Δ\Delta and attention total variation directly, using the last 16 queries/head in a 512-token dense-trajectory probe. Perturbations are evaluated locally, without injecting them into subsequent layers.

For Qwen, rotated four-bit versus eight-bit keys give mean Δ\Delta 3.734 versus 0.280 and mean attention TV 0.0970 versus 0.00615 across layers/heads/queries. Layer 0's four-bit mean TV is 0.592. These measurements connect key reconstruction error to attention distortion. We use the deterministic TV bound to interpret this probe; the separate stochastic argument assumes independent, centered rounding errors. The early 15-level symmetric grid uses s=a/7s=a/7; the later 16-level grid uses s=2a/15s=2a/15, where a=max⁡j∣(Rx)j∣a=\max_j |(Rx)_j|. The revision will state these grids separately and place the prepared four-panel experimental overview before the theory.

I appreciate the authors' effort to address my concerns, especially Q1, Q3, and Q5. However, I believe the following remain unresolved.

  1. Q2: The timing finding on existing KV quantization methods.
  2. Q4: Though GSM8K and SQuAD are added, their decode length is still much shorter than the reasoning benchmarks I asked for, and recent reasoning models are untested.
  3. The presentation still needs substantial rework to make each claim clear.

I maintain my score.

Reviewer_o3a1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts a careful controlled empirical audit on rotated uniform KV‑cache quantization, disentangling calibration timing from scale granularity, which are often confounded in prior work such as QuaRot. The authors separate prefill‑frozen calibration versus per‑write calibration, global versus grouped scale parameters, while explicitly accounting metadata overhead. On Llama‑3.1‑8B‑Instruct, per‑write global calibration delivers decent perplexity, while prefill‑frozen global scales break down severely. However Qwen2.5‑7B‑Instruct shows catastrophic failure across all 4‑bit rotated quantizer configurations, even though 8‑bit controls work fine. The paper does not propose new kernels or end‑to‑end serving systems; its main output is empirical observations and clear boundary conditions rather than universal prescriptions.

Critical strengths
  1. The work clearly points out a common confounding factor in rotated KV‑cache quantization research: calibration timing and metadata cost are often mixed together when comparing local grouped scales against global scales. It makes explicit that equal payload bits does not mean equal total memory budget.
  2. Well‑structured factorial experimental design, covering two models, two datasets, multiple rotation seeds, symmetric/asymmetric quantizer settings. It includes multiple sanity controls: unquantized cache, rotation round‑trip, eight‑bit baselines, helping narrow down failure sources.
  3. Honest and restrained conclusion. The authors avoid over‑general claims. They explicitly state they cannot pinpoint the root cause behind Qwen’s 4‑bit failure and limit their claims to the tested protocol.
  4. Thorough reproducibility handling: all per‑window metrics are retained, experimental parameters are fully specified.
Critical weaknesses
  1. It remains unclear what causes Qwen’s consistent 4‑bit degradation. The paper rules out rotation‑roundtrip artifacts but does not dig into layer‑wise statistics, outlier analysis or activation characteristics to form testable hypotheses.
  2. Evaluation is limited to perplexity metrics on Wikipedia‑origin text corpora. No downstream task accuracy results are provided, so we cannot tell how observed PPL shifts translate to real task performance.
  3. No runtime or measured GPU memory numbers. All memory figures are modeled theoretical bit‑rates, without real packed kernel implementation. Therefore we cannot assess the practical system trade‑offs.
  4. Limited domain diversity: both WikiText‑2 and SQuAD contexts are derived from Wikipedia, so cross‑domain evidence is thin.
Questions for authors
  1. Have you inspected per‑layer activation statistics for Qwen under rotated basis? Are there systematic outlier patterns that could explain its 4‑bit failure?
  2. How much PPL gain from grouped scales would vanish if you enforce equal total memory (payload plus metadata) rather than equal payload bits?
  3. Could you comment on how prefill length influences the gap between prefill‑frozen and per‑write calibration? Have you varied prefill window sizes?
1 comment

The new experiments answer the three main questions: Qwen's failure is concentrated on key quantization; the grouped advantage reverses under a common storage budget; and Llama's calibration-timing gap persists across prefill lengths.

  1. Qwen: isolating the sensitive cache component

We evaluated K-only, V-only, and K+V quantization both with and without rotation. On the same 512-token WikiText diagnostic window (prefill 128; grouped asymmetric four-bit, group size 32), Qwen gives:

Quantized componentRotated PPLUnrotated PPL
K only1284.311683.98
V only7.377.39
K and V1154.051836.73

The dense reference is 7.3912, unquantized incremental inference is 7.3916, rotation round-trip is 7.3936, and rotated eight-bit K+V is 7.3938. Key quantization is sufficient to reproduce the failure in this window; value quantization alone is not. The failure also occurs without rotation. Llama's rotated K-only and V-only PPLs are 5.5105 and 5.4902 against reference 5.4873.

Qwen's key amplitudes vary sharply across layers. At layer 27 (zero-indexed), prefill K RMS is 40.76 and maximum magnitude is 419; rotation reduces that maximum to 204.19, but four-bit reconstruction MSE remains 11.66. At layer 0, K RMS is 26.72 and MSE is 3.86. The largest corresponding Llama K MSE is 0.0258. These measurements identify high-error key layers for selective-precision tests and show why reducing peak magnitude alone need not preserve attention.

A dense-trajectory probe also measures the attention effect: across Qwen layers/heads and 16 query positions, mean attention TV from local key perturbations is 0.0970 at four bits versus 0.00615 at eight bits; layer 0 reaches 0.592 at four bits.

  1. Total storage, not payload bits alone

We ran a per-write symmetric comparison on two 1024-token windows per model (prefill 256). With FP32 scale metadata, grouped four-bit/group32 costs 4+32/32=5 bits per KV element; global five-bit costs 5+32/(H_KV d). On Llama:

ConfigurationEffective bits/elementPPL
Global four-bit4.031255.8866
Grouped four-bit5.000005.8541
Global five-bit5.031255.8146

The dense reference is 5.7816. Grouping improves over global at equal four-bit payload, but global five-bit performs better under the common 5.03125-bit ceiling. Grouped four-bit leaves 0.03125 bits/element unused. Symmetric grids store one scale; asymmetric grids store both scale and offset. On Qwen, global five-bit also improves over grouped four-bit (PPL 5208.63 versus 12535.82), although both remain severely degraded.

  1. Prefill length

We varied prefill length while scoring identical target positions 513-1023 in each of two 1024-token windows (1,022 targets/condition). Global asymmetric four-bit results on Llama are:

Prefill tokensFrozen PPLPer-write PPL
128164.546.061
256120.936.059
51248.786.023

The dense reference is 5.876. Longer prefill improves aggregate frozen-scale performance but does not close the timing gap; per-write stays near the reference. Qwen remains severely degraded at all three lengths (PPL above 10,000 in all six arms).

  1. Downstream performance and domain diversity

We completed 1,000 generated-answer evaluations: two models, 50 SQuAD validation questions and 50 GSM8K test problems, and five cache configurations. Tasks were selected by fixed hash ordering, with identical prompts across conditions. Llama's global four-bit prefill-frozen versus per-write scores are 68% versus 88% QA exact match and 10% versus 84% math accuracy. Paired bootstrap 95% intervals for the increases are [10,32] and [60,86] percentage points. Qwen's per-write global/grouped four-bit math scores are 0%/4%, versus 84% dense and 86% eight-bit. This adds actual QA answers and a non-Wikipedia reasoning domain. Greedy output limits are 48/384 tokens for QA/math, with capped outputs retained.

  1. Measured runtime and memory

On Llama GSM8K, synchronized dense and global per-write four-bit QDQ runs take 4.60 and 6.60 seconds per example, with mean peak allocated memory of 15.094 and 15.093 GiB. These measurements include variable-length generation. QDQ retains FP16 cache tensors, so the measured memory is essentially unchanged. The revision will report execution costs alongside the modeled payload-plus-metadata budget; measuring compression benefits requires packed storage.

Comments on the paper

1 comment
Area_Chair_2ULDOpenReview ↗

Action Required: Respond to Author Rebuttals Before the Discussion Period Ends

Dear Reviewers,

The authors have posted their rebuttals. As the author–reviewer discussion period is coming to a close soon, please read their responses and participate in the discussion.

All reviewers are expected to respond, at minimum with a brief acknowledgment confirming that the reviewers have read and considered the authors’ rebuttals. If your concerns have been addressed or remain unresolved, please briefly indicate this in your response.

Thank you for your time and effort.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.