acceptodds
Under review as a conference paper at ICLR 2027

KVarN: Variance-Normalized KV-Cache Quantization for Improved Reasoning in LLMs

11 views

Abstract

Test-time scaling improves language-model reasoning but increases KV-cache memory during long-horizon generation. Existing KV-cache quantization methods are commonly evaluated in static prefill settings, which do not capture feedback from repeatedly attending to a quantized cache. We show that this feedback increases attention-output error over time and identify token-scale distortion as a major contributor to the largest key-quantization errors. We introduce KVarN, a calibration-free, linear KV-Cache quantization method to effectively suppress token scale errors and error accumulation. At an effective storage cost of 2.25 bits per element, KVarN achieves the best overall quality against strong quantization baselines on AIME 2024, MATH-500, HumanEval, and IFEval. It also reduces attention-output error under our pseudo-decode diagnostic. In end-to-end vLLM measurements across three models, KVarN improves decode throughput by 10–28% over the 16-bit KV-cache baseline; at a nominal 3-bit precision, it achieves 1.44–2.00× the throughput of the vLLM TurboQuant implementation. Code for the KVarN method is given in the supplementary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization, kv-cache, inference, LLM, reasoning, compression

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_BS6LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper targets KV-cache memory growth in long-horizon (test-time scaling) generation.
  • It argues that static prefill evaluations miss the error that builds up when later tokens repeatedly attend to an already-quantized cache. It introduces a blockwise “pseudo-decode” diagnostic to measure this.
  • It splits key-quantization error into a token-magnitude (multiplicative) part and an orthogonal (additive) residual. It finds that scale distortion drives the largest errors, and that a small fraction of errors (e.g., the top 5%) causes most of the end-to-end degradation.
  • It proposes KVarN, a calibration-free linear (RTN) KV-cache quantizer with two stages:
    • a channel-wise Hadamard rotation, which equalizes channel outliers;
    • online Sinkhorn-style variance normalization along both the token and channel axes, which preserves per-token magnitudes.
  • At ~2.25 bits/element, it reports the best mean scores among baselines on AIME24, MATH-500, HumanEval and IFEval. Decode throughput is 1.10–1.28× the 16-bit baseline, and 1.44–2.00× vLLM’s TurboQuant at 3 bits.
Critical strengths
  • The motivation is clear and practical. Test-time scaling makes KV-cache memory a real bottleneck, and evaluating quantizers only on static prefill misses the feedback from re-attending to a quantized cache.
  • The finding that a small fraction of large errors (top ~5%) accounts for most of the end-to-end KL-divergence, despite being a minority of MSE (Fig. 1a, Fig. 13), is interesting and useful for the community.
  • Results are strong across a wide set of baselines (KIVI, QuaRot, KVQuant, PolarQuant, TurboQuant, Kitty), and the method stays linear/RTN, so the end-to-end vLLM throughput gains are credible.
  • The method uses uniform precision with a fused dual-scale dequant kernel, and the dequant overhead vs. KIVI is small (≤1.4%, Fig. 16). This is attractive for deployment compared to mixed-precision or codebook methods.
Critical weaknesses
  • My main concern is limited novelty. Both stages of KVarN closely mirror existing work:
    • the Hadamard rotation comes directly from QuaRot [1];
    • the variance normalization is the Sinkhorn dual-scaling of SINQ [2], and Alg. 1 is explicitly “adapted from” it.
    • KVLinC [3] already pairs Hadamard rotation with a lightweight correction for key error in 2-bit KV caches (not discussed or compared against). NSNQuant [4] is also calibration-free and normalizes the KV cache along both the token and channel axes before quantization.
    • The authors argue KVarN differs from NSNQuant in two ways: its balancing is iterative rather than a fixed single pass, and it uses linear RTN rather than codebook lookups. However, the only evidence for the first is a small kurtosis difference (−0.028 vs. −0.046, Tab. 4). There is no end-to-end comparison against NSNQuant either. The paper should state the delta over [2, 3, 4] directly and compare against KVLinC and NSNQuant at matched bits.
  • Section 3.1 is important for motivating the method, but it is fairly well known that multiplicative (scale) errors are more harmful than additive ones. It is also expected that feeding them into a softmax amplifies existing error. The decomposition is a nice way to present this, but it is not a strong contribution in itself.
  • Some minor clarity issues. Section 3.3 introduces token-axis scaling and excess kurtosis without setup. The kurtosis differences used to justify iterative balancing are also very small (−0.032 → −0.028).
  • TurboQuant runs at 3/3 bits and PolarQuant at 4/2, so these are not matched-bit comparisons. TurboQuant is also a community vLLM implementation that leaves the first and last two layers in FP16. The “3-bit” throughput comparison is also not apples-to-apples, since KVarN 3-bit uses 4-bit K and 2-bit V.
Questions for authors
  1. How does KVarN differ from KVLinC [3] beyond being calibration-free? Can you add a head-to-head comparison at matched bits/element?
  2. Fig. 10 suggests Hadamard mostly helps beyond ~50k tokens, yet the benchmark traces are ≤16k tokens. What is the end-to-end accuracy of VarN-only vs. KVarN on AIME/MATH-500? In other words, does the rotation matter at benchmark lengths?
  3. Can you report vLLM throughput for KIVI/QuaRot at 2 bits? Currently TurboQuant is the only quantized throughput baseline, and it uses a different bit budget.
  4. How sensitive are the results to the pseudo-decode block size bb, and to group size GG?

References:

[1] Ashkboos et al., “QuaRot: Outlier-free 4-bit inference in rotated LLMs.” NeurIPS, 2024

[2] Müller et al., “SINQ: Sinkhorn-normalized quantization for calibration-free low-precision LLM weights.” ICML, 2026

[3] Saxena et al., “KVLinC: KV Cache Quantization with Hadamard Rotation and Linear Correction.” ArXiv, 2025

[4] Son et al., “NSNQuant: A double normalization approach for calibration-free low-bit vector quantization of KV cache.” NeurIPS, 2025

1 comment

Thank you for your time and feedback.

1. KVLinC Addition & Comparison; KVLinC + KVarN

Thank you for bringing this paper to our attention; we will cite it and add these explanations (and a discussion in Sec. 5, Related Work).

KVarN + KVLinC: KVLinC and KVarN are essentially orthogonal and can be combined to maximally prevent (KVarN) and repair (KVLinC) quantization damage.

Algorithmic Comparison:

  1. Pre-Quantization Conditioning (KVarN) vs. Post-Quantization Recurrent Adapters (KVLinC):
  • KVLinC: Uses KIVI-like RTN on KK and compensates residual errors via a trainable linear-attention branch (ϕq(Qn)ϕk(Kie)⊤\phi_q(Q_n)\phi_k(K_i^e)^\top) added to softmax numerator/denominator, modifying attention and requiring recurrent states (Sn,PnS_n, P_n) during decoding.
  • KVarN: Leaves softmax attention untouched with no auxiliary networks. Since multiplicative per-token magnitude distortions (δtLt\delta_t L_t) drive end-to-end degradation (Sec. 3.1), KVarN preconditions the cache before RTN via iterative dual-axis (Sinkhorn) variance normalization (VarN), preserving per-token norms, reducing kurtosis (Tab. 4), and needing only a fused second FP8 scale during dequantization (2.25 vs. KVLinC's 2.71 bpe).
  1. Hadamard Rotation on KK:
  • KVLinC: Notes rotating KK hurts 2-bit single-scale channel-wise quantization, so leaves KK unrotated (rotating only VV).
  • KVarN: Shows why rotation alone struggles at 2-bit (leaving extreme token-scale errors intact, Fig. 2b, 12) and resolves this by pairing it with VarN. With VarN absorbing token-scale outliers, rotating both KK and VV suppresses cross-layer error accumulation over long reasoning horizons (Sec. 3.3, Fig. 10).

End-to-end results (Qwen3-4B, IFEval): We will add this to the paper. Qualitatively, adding KVarN further improves KVLinC.

MethodPrompt strictPrompt loose
KVLinC78.56±0.83 %82.19±0.80 %
KVarN + KVLinC79.67±0.11 %82.93±0.16 %

2. VarN-only AIME24 / MATH-500

Local errors are less sensitive to error accumulation than end-to-end evaluations: on AIME24 and MATH500 (16k token budgets), KVarN outperforms VarN. Specifically, VarN shows a 5.6% (AIME24) and 1.3% (MATH500) accuracy loss compared to KVarN. We will highlight this in the paper.

3. Throughput of KIVI / QuaRot

Thank you for the suggestion; we will add these results.

Because KIVI and QuaRot lack fast kernels for vLLM (or other serving frameworks found), we ablated our KVarN kernel: 1) without normalization/rotation (KIVI-style) and 2) without normalization, with rotation (QuaRot-style). Skipping components naturally runs faster, though KVarN's minor overhead is usually worth its substantial accuracy gains.

Setup: Qwen3-14B, FP16 weights, 8,192 in + 8,192 out tokens, 16 requests, concurrency 8 (1 run, warmup excluded; GPU: 500 FP16 TFLOP, 1.8 TB/s BW). Total time = elapsed time for 16 requests (prefill + decode); E2E = per-request completion latency; Throughput = output tokens / total time.

K2/V2

MetricFP16KIVI-styleQuaRot-styleKVarN
K/V bits16/162/22/22/2
Out (tok/s)269.10302.10293.52288.30
Total time, 16 req (s)487.07433.87446.56454.63

Nominal ~3 bits: K4/V2 & TurboQuant K3/V3

MetricFP16KIVI-styleQuaRot-styleKVarNTurboQuant
K/V bits16/164/24/24/23/3
Out (tok/s)269.10287.49276.88273.59163.16
Total time, 16 req (s)487.07455.92473.39479.08803.34

4. Sensitivity to Block & Group Size

Measured via attention output MAE on Qwen3-4B (cf. Fig. 5a–b); we will add this to the appendix. Since block size typically exceeds group size, we vary block size upward and group size downward.

Block Size: Across 128, 256, and 512, KVarN is insensitive, changing by ≤2% vs. default 128 in both static and accumulated settings.

Block sizeStatic, 9kStatic, 32kAccum., 9kAccum., 32k
128 (default)0.112±0.0020.116±0.0020.123±0.0030.124±0.002
2560.114±0.0020.117±0.0020.123±0.0030.125±0.002
5120.113±0.0020.117±0.0020.123±0.0030.125±0.002

Group Size: Smaller groups use more bits/element and thus perform better for any method. To normalize this, we report (KIVI error − KVarN error) (cf. Fig. 5b). KVarN's advantage widens with larger group sizes because fewer bits/element create a harder quantization problem where a more involved method yields larger gains.

Group sizeStatic, 9kStatic, 32kAccum., 9kAccum., 32k
320.017±0.0010.023±0.0010.021±0.0010.034±0.002
640.024±0.0010.032±0.0010.029±0.0010.043±0.002
128 (default)0.035±0.0020.043±0.0010.041±0.0020.059±0.001

Mean ± SE over 8 WikiText sequences; MAE pooled over a 1024-token query window; group-size differences computed per sequence (paired).

Reviewer_E6a1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes KVarN, a calibration-free KV-cache quantization method designed to reduce quantization errors during long-form LLM generation. The method combines Hadamard rotation with iterative variance normalization along token and channel dimensions to preserve key magnitudes under low-bit quantization.

The authors also introduce a pseudo-decode diagnostic to study error propagation through quantized caches. Experiments across reasoning, coding, instruction-following, and retrieval benchmarks demonstrate strong quantization quality at approximately 2.25 bits per quantized cache element, together with improved decode throughput in vLLM.

Critical strengths
  1. The paper addresses the growing KV-cache memory bottleneck in long-form LLM reasoning. The analysis of token-scale distortion and error accumulation provides useful insights beyond conventional static prefill evaluations.
  2. KVarN combines Hadamard rotation and dual-axis variance normalization to address complementary sources of quantization error. The error decomposition and component ablations provide convincing evidence for the proposed design.
  3. The evaluation covers multiple model families and tasks. KVarN achieves competitive low-bit quantization quality, with particularly strong results on Phi-4-14B and increasing advantages over KIVI for longer reasoning traces.
  4. The authors provide an end-to-end vLLM implementation and measure actual decoding throughput and kernel overhead. The results demonstrate meaningful efficiency improvements while maintaining competitive generation quality.
Critical weaknesses
  1. The evaluated TurboQuant implementation retains several layers in FP16, resulting in a higher effective memory cost than KVarN at the same nominal precision. The lack of end-to-end comparisons with KIVI and QuaRot under identical conditions makes it difficult to isolate the efficiency gains of the proposed algorithm.
  2. The throughput experiments use a 12K-token context but generate only 512 tokens per request. Although this is a valid serving benchmark, it does not directly establish the latency and throughput benefits for long reasoning trajectories involving thousands of generated tokens.
  3. The reported 2.25 bits per element includes quantization metadata but does not appear to include the FP16 sink and recent-token regions used during inference. Reporting the actual full-cache memory footprint would make the compression ratios and efficiency comparisons more precise.
Questions for authors
  1. Could the authors provide end-to-end throughput comparisons with KIVI and QuaRot under identical vLLM configurations? A comparison at matched actual KV-cache memory footprints would also help clarify the advantage over TurboQuant.
  2. Could the authors report latency, throughput, and peak memory usage for longer generated sequences, such as 4K or 8K tokens? This would better reflect the targeted long-form reasoning workloads.
  3. Does the reported 2.25 bits per element include the FP16 sink and recent-token regions? Please clarify the actual full-cache memory footprint for the configurations used in the throughput experiments.
1 comment

Thank you for your time and feedback. We will add the suggested measurements to the paper / appendix and clarify the bit-per-element definition.

1. Throughput Comparison to KIVI and QuaRot

Could the authors provide end-to-end throughput comparisons with KIVI and QuaRot under identical vLLM configurations? A comparison at matched actual KV-cache memory footprints would also help clarify the advantage over TurboQuant.

Notably, KIVI and QuaRot do not supply fast kernels for vLLM (or other serving frameworks as far as we could find). We implemented a variant of our KVarN kernel 1) without normalization and rotation yielding KIVI and 2) without normalization, but with rotation yielding QuaRot. Of course, using our kernel, while skipping some of its components it becomes faster. For many applications, the small time overhead is worth it for the sometimes substantial accuracy improvement KVarN brings.

We measure Qwen3-14B, FP16 weights, 8,192 input + 8,192 output tokens, 16 requests, concurrency 8; one run, warmup excluded. Total time = elapsed time for all 16 requests, including prefill and decoding; E2E = per-request completion latency. Throughput = output tokens / total time. Measurements are run on a GPU with 500 TFLOP at fp16 and 1.8 TB/s memory bandwidth.

K2/V2

MetricFP16 baselineKIVI-styleQuaRot-styleKVarN
K/V bits16/162/22/22/2
Output throughput (tokens/s)269.10302.10293.52288.30
Total time, 16 requests (s)487.07433.87446.56454.63

Nominal average of 3 bits: K4/V2 and TurboQuant K3/V3

MetricFP16 baselineKIVI-styleQuaRot-styleKVarNTurboQuant
K/V bits16/164/24/24/23/3
Output throughput (tokens/s)269.10287.49276.88273.59163.16
Total time, 16 requests (s)487.07455.92473.39479.08803.34

2. Latency / Throughput / Peak Memory Usage

Could the authors report latency, throughput, and peak memory usage for longer generated sequences, such as 4K or 8K tokens? This would better reflect the targeted long-form reasoning workloads.

Output tokensMethodOutput throughput (tokens/s)Median TTFT (s)Median TPOT (ms/token)Median E2E (s)Peak GPU memory (GiB)Allocated KV-cache memory (GiB)
4,096FP16272.854.22228.28120.0145.8115.02
4,096KVarN K2V2298.985.18025.48109.5236.113.95
8,192FP16270.334.32429.06242.3550.3320.02
8,192KVarN K2V2289.765.21226.97226.1036.884.69

The GPU and model are the same as above. We do four runs: 8,192 input tokens and either 4,096 or 8,192 generated tokens, with 8 requests at concurrency 8. Fixed output lengths (EOS stopping disabled), no prefix caching; one measured repetition after a full-length warmup.

3. BPE definition

All compared methods leave a fixed amount of initial and final tokens at fp16; we will further highlight this in the paper. We report the BPE of the quantized part only, otherwise the BPE would be a function of the context length (the non-quantized part is fixed in size, the quantized gets longer). Alternatively the reported BPE is the asymptotic BPE at long context. To get a more intuitive feel for the overhead this implies, we give a table here of context-dependent bits per element (CDBPE) for some context lengths.

With S = 128 sink tokens and R = 128 recent tokens in FP16 and a quantized body at bb bits per element (incl. scales/zero-points, as reported in Tab. 1), the full-cache footprint at context length LL is

CDBPE(L)=b+(S+R)(16−b)L.\text{CDBPE}(L) = b + \frac{(S+R)(16-b)}{L}.

MethodK/Vbb1k8k64k512k
KIVI2/22.255.692.682.302.26
QuaRot2/22.255.692.682.302.26
KVQuant-1%2/22.45.802.832.452.41
PolarQuant4/23.36.483.703.353.31
TurboQuant3/34.67.454.964.644.61
Kitty2/22.45.802.832.452.41
KVarN (ours)2/22.255.692.682.302.26

Please let us know if you have further questions.

Reviewer_aCSROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes KVarN, a calibration-free KV-cache quantization method designed to improve the efficiency and reliability of long-horizon LLM generation. The authors identify token-scale distortion as a major contributor to quantization errors and argue that conventional static-prefill evaluations fail to capture error accumulation during autoregressive decoding.

KVarN combines channel-wise Hadamard rotation with iterative dual-axis variance normalization to balance token- and channel-wise distributions before low-bit quantization. The paper further introduces a pseudo-decode evaluation procedure to characterize quantization-error accumulation. Experiments on AIME 2024, MATH-500, HumanEval, IFEval, and synthetic retrieval benchmarks demonstrate improvements over several quantization baselines. The authors also provide Triton GPU implementations and report improved end-to-end decoding throughput in vLLM, achieving 10–28% higher throughput than FP16 KV-cache inference at approximately 2.25 bits per quantized element.

Critical strengths
  1. Well-Motivated Quantization Method. The paper provides a systematic analysis of KV-cache quantization errors, highlighting the importance of token-scale distortion and error accumulation during decoding. The combination of Hadamard rotation and dual-axis variance normalization is technically motivated and supported by ablation experiments.

  2. Promising Quantization Quality and Implementation Results. KVarN achieves competitive accuracy at low bit widths across mathematical reasoning, code generation, and instruction-following benchmarks. The authors also implement GPU kernels and integrate the method into vLLM, providing useful evidence of practical throughput improvements.

Critical weaknesses
  1. Insufficient Wall-Clock Latency Evaluation. Although the paper reports normalized decoding throughput under 256 concurrent requests, it does not provide absolute output throughput (tokens/s), total generation time, time to first token (TTFT), inter-token latency (ITL/TPOT), or full request completion time including prefill and decoding. Furthermore, batch-size-1 performance and throughput scaling across different batch sizes are not evaluated. These omissions make it difficult to assess the practical inference efficiency of KVarN, particularly in latency-sensitive deployment scenarios.

  2. Limited Evaluation of General-Purpose and Agentic Workloads. The experiments primarily cover mathematical reasoning, code generation, instruction following, and synthetic retrieval. General-purpose question-answering tasks are not evaluated, limiting evidence of the method's applicability beyond the selected benchmarks. More importantly, the paper does not investigate agentic workloads, where task completion and behavioral consistency are important evaluation dimensions. Consequently, the generalizability of KVarN to interactive, multi-step applications remains insufficiently established.

Questions for authors
  1. Absolute Wall-Clock Performance. Could the authors report absolute output throughput (tokens/s), total generation time, TTFT, ITL/TPOT, and end-to-end request completion time (including prefill and decoding) for KVarN and the evaluated baselines? These measurements are important for assessing the practical efficiency gains beyond normalized throughput.

  2. Batch-Size Sensitivity. How does KVarN perform across different batch sizes and concurrency levels, particularly batch size 1? Could the authors provide absolute throughput and latency measurements across representative serving configurations to demonstrate whether the reported efficiency gains generalize beyond high-concurrency inference?

  3. General-Purpose Question Answering. Could the authors extend the evaluation to non-mathematical, non-reasoning, and non-retrieval-oriented workloads, particularly general-purpose question-answering benchmarks such as Natural Questions or TriviaQA? This would help establish whether the reported improvements generalize to conventional language-model applications.

  4. Agentic Task Performance. How does KVarN perform on agentic benchmarks involving multi-step planning, tool use, and interactive execution? Could the authors report task completion rates and other relevant metrics compared with FP16 inference?

  5. Agent Behavioral Consistency. Does KV-cache quantization affect agent behavior, including tool selection, action sequences, and intermediate decisions? Could the authors evaluate behavioral consistency relative to FP16 inference under controlled experimental settings?

5 comments

Thank you for your time and feedback.

1. Absolute Performance

We will add these to the paper. KVarN is faster than the baselines (fp16 and TurboQuant). Ablations based on our own kernel (KIVI and QuaRot) show the overhead of the hadamard transform and the VarN. The TurboQuant implementation is that of vLLM at 3 BPE (2 BPE is not implemented). Our kernels only support 2 and 4 bits, so we use 4/2 to BPE-match TurboQuant's 3/3.

We use Qwen3-14B, at FP16, 8k in/out tokens, 16 requests, concurrency 8. Total time = elapsed time for all 16 requests; E2E = per-request completion latency. Throughput = output tokens / total time. Measured on a GPU with 500 TFLOPS at fp16 and 1.8 TB/s memory bandwidth

K2/V2

MetricFP16 baselineKIVI-styleQuaRot-styleKVarN
K/V bits16/162/22/22/2
Output throughput (tokens/s)269.10302.10293.52288.30
Total time, 16 requests (s)487.07433.87446.56454.63
Median TTFT (s)3.4823.5573.5903.708
Median ITL (ms)28.9425.5526.0626.46
Median E2E (s)243.46216.88223.22227.26

Matched BPE comparison: K4/V2 and TurboQuant K3/V3

MetricFP16 baselineKIVI-styleQuaRot-styleKVarNTurboQuant
K/V bits16/164/24/24/23/3
Output throughput (tokens/s)269.10287.49276.88273.59163.16
Total time, 16 requests (s)487.07455.92473.39479.08803.34
Median TTFT (s)3.4823.6944.3884.0533.540
Median ITL (ms)28.9427.0527.6728.0448.30
Median E2E (s)243.46227.90236.63239.47401.60

2. Concurrency

We will add these results to the appendix. At concurrency 1, KVarN is only marginally faster; the advantage over fp16 grows with increasing concurrency. KV-Cache management is generally less of an issue at lower concurrency, so the speed-up is also less.

The setup is similar to above, but with 8k in/out tokens, 8 requests per configuration; concurrency 1, 4 and 8.

ConcMethodOut (tok/s)Med TTFT (s)ITL p90 (ms)Med E2E (s)E2E p90 (s)Rel KV Cap
1FP1648.940.80520.67167.41167.421.00×
1KVarN K2V249.100.81020.47166.89166.906.23×
4FP16164.162.34425.24199.59200.331.00×
4KVarN K2V2169.553.11224.02193.24193.576.23×
8FP16270.014.24231.52242.64242.701.00×
8KVarN K2V2289.905.17628.08225.99226.056.23×

3. General Purpose QA

Following Dutta et al. ('Accuracy is not all you need', NeurIPS 2024) we avoided QA benchmarks, which can be misleading and noisy when evaluating quantization methods. For completeness, we add the suggested TriviaQA (Wikipedia validation subset); we will add these to the appendix. Here are the measurements for Qwen3-4B.

MethodEM %F1 %
FP1664.62±0.0573.35±0.04
TurboQuant-K3V357.41±0.0867.91±0.07
KVarN-K2V260.22±0.1169.72±0.06

4. Agentic Performance

We use τ²-bench Retail with 114 interactive customer-service tasks. The agent talks to a simulated customer and calls tools to inspect and modify a database. Task completion measures whether the final database state and any specified natural-language requirements satisfy the criteria.

Setup: Qwen3-14B with FP16 weights; DeepSeek V4.1 Flash 1017 for the user simulator. Avg. over 3 runs.

MetricFP16KVarN K2V2
Task completion55/114 (48.2%)54/114 (47.4%)
Mean tool calls/task7.608.12
Tool error response rate12.0%15.6%
Mean generated tokens/task6,9277,364
Mean interaction time (s)160.1174.9

Interaction time includes the agent, user simulator and tool execution, excluding final judging. Tool error rate counts responses beginning with "Error:" across all tool responses, including recoverable errors.

5. Agentic Behavioral Consistency

Overall, the FP16 and KVarN baseline are about equally capable, but the trajectories taken do differ: In 31 of the 37 tasks completed by both the sequence of which tools were called differs. For example, both complete task 0, while KVarN uses 6 tool calls versus 7 and generates 4,879 tokens versus 5,535.

The most interesting observation we have made on the agent traces is a weak signal for detail recall issues with the quantized KV-Cache: The largest increase in error category is in the "User not found" error from 47 to 71, presumably because an incorrect user string was entered in the tool-call. However, the model seems to be able to recover through further reasoning / tool-calling. This may be the underlying reasons for the slightly higher number of generated tokens per task.

We will add the agentic evaluation in the appendix and highlight in the paper that many-turn agentic settings can observe a minor latency increase due to additional tool-calls.

Please let us know if there are further questions.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns regarding absolute inference performance, general-purpose QA, and agentic evaluation. The new results are informative; however, several concerns remain unresolved.

  1. Performance Relative to Quantization Baselines. The newly reported measurements show that KVarN is consistently slower than both the KIVI-style and QuaRot-style implementations. At K2/V2, KVarN achieves 288.30 tokens/s, compared with 302.10 for KIVI and 293.52 for QuaRot, with longer end-to-end completion times (227.26 s versus 216.88 s and 223.22 s, respectively). The same trend persists at K4/V2. Although KVarN outperforms FP16 and TurboQuant in throughput, its additional normalization does not improve runtime over these simpler quantization baselines. The authors should clarify the quality–latency trade-off and demonstrate whether the accuracy improvements justify the additional computational cost.

  2. Incomplete Concurrency Evaluation. The additional concurrency experiments compare KVarN only against FP16, without including KIVI or QuaRot. Given their superior throughput in the absolute performance measurements, comparisons against these quantization baselines across concurrency levels are necessary. Furthermore, KVarN provides only marginal throughput improvement at concurrency 1, while TTFT increases at higher concurrency. A comprehensive comparison across serving configurations would better establish its practical efficiency.

  3. Agentic Performance and Behavioral Consistency. The additional agentic evaluation is valuable, but the results raise further concerns. Although task completion remains similar (48.2% versus 47.4%), the tool error response rate increases from 12.0% to 15.6%, mean generated tokens increase from 6,927 to 7,364, and mean interaction time increases from 160.1 s to 174.9 s. The reported increase in "User not found" errors (47 to 71) is also concerning. These observations suggest that comparable final task completion may conceal differences in execution efficiency and behavioral reliability. Could the authors provide a more detailed analysis of these differences, including variability across runs and whether the increased error rates are statistically meaningful? The reported trajectory differences should also be interpreted against the variability of repeated FP16 runs.

Overall, the rebuttal provides useful additional evidence and partially addresses my original concerns. However, the runtime disadvantage relative to simpler quantization methods, the missing cross-baseline concurrency comparisons, and the increased agentic execution errors remain important limitations. I therefore maintain my rating of 2 (Weak Reject) at this stage, pending further clarification and evidence addressing these issues.

Thank you for the further feedback.

1. Quality–latency trade-off

VarN and the rotation are quality mechanisms: they cost throughput relative to plain RTN (KIVI) and we report this cost. Across our measurements at K2/V2, KVarN is between 0% (within noise) and 4.6% slower than the KIVI-style kernel: 4.6% in the single-run setting of Tab. 9, and 2.4%, 0.8% and within noise at concurrency 1, 4 and 8 with repeated measurements, see below. What this buys:

  • MATH-500 on Phi-4: QuaRot loses 7.9 points and KIVI 10.5 points relative to FP16, KVarN 0.1 points, see Tab. 1.
  • HumanEval on Phi-4: KIVI loses 14.3 points, KVarN 0.7, see Tab. 2.
  • τ²-bench: KIVI completes 28.9% and QuaRot 40.4% of tasks, KVarN 47.4% (FP16: 48.2%), see Sec. 3 below.

At task level on τ²-bench the kernel cost does not show: KVarN is on par with both in generated tokens and interaction time (174.9 s vs. 176.1 s and 177.4 s) and has the lowest tool error rate of the three.

Note that before this work no fast serving kernel existed for KIVI or QuaRot: the comparison is only possible because all three are configurations of our kernels. A user can thereby choose the operating point: plain RTN for maximal speed, or at most 4.6% less throughput for near-FP16 accuracy. We will release all three and state this explicitly in the contributions.

2. Concurrency

We complete Tab. 11 with the KIVI-style and QuaRot-style kernels (throughput: mean ± std. err. over 10 repetitions).

Conc.MethodK/VThroughput (tokens/s) ↑Median TTFT (s) ↓ITL p90 (ms) ↓Median E2E (s) ↓E2E p90 (s) ↓
1FP1616/1648.94 ± 0.020.80520.67167.41167.42
1KIVI-style2/250.28 ± 0.080.99620.06162.94162.95
1QuaRot-style2/249.41 ± 0.071.04320.41165.80165.81
1KVarN2/249.10 ± 0.060.81020.47166.89166.90
4FP1616/16164.16 ± 0.232.34425.24199.59200.33
4KIVI-style2/2170.94 ± 0.312.61823.67191.66192.11
4QuaRot-style2/2169.90 ± 0.192.67023.83192.82192.90
4KVarN2/2169.55 ± 0.203.11224.02193.24193.57
8FP1616/16270.01 ± 0.324.24231.52242.64242.70
8KIVI-style2/2290.12 ± 0.314.29327.85225.81225.88
8QuaRot-style2/2288.23 ± 0.924.31928.02227.31227.36
8KVarN2/2289.90 ± 0.935.17628.08225.99226.05

The cost of VarN and the rotation shrinks with concurrency: 2.4% relative to the KIVI-style kernel at concurrency 1, 0.8% at 4, and within noise at 8. Relative to FP16, KVarN gains 0.3%, 3.3% and 7.4%. At concurr. 1 the KV-cache traffic is small relative to the weights, so no variant is much faster than FP16; the benefit there is memory (4.69 vs. 20.02 GiB allocated, see Tab. 10).

TTFT is a cost: at concurrency 8 the median TTFT of KVarN is 0.93 s higher than for FP16 (5.18 s vs. 4.24 s), because the prompt cache is quantized during prefill. For the long generations we target this is 0.4% of the end-to-end time, which remains 6.9% lower than for FP16 (226.0 s vs. 242.6 s). For short outputs where TTFT dominates, KV-cache quantization is less attractive.

3. Agentic evaluation

We add standard errors and report KIVI and QuaRot as well (mean ± std. err. over 3 runs).

MetricFP16KVarN (ours)KIVI-styleQuaRot-style
K/V bits16/162/22/22/2
Task completion (%) ↑48.2 ± 2.047.4 ± 3.128.9 ± 2.440.4 ± 2.2
Mean tool calls/task7.60 ± 0.028.12 ± 0.178.10 ± 0.178.29 ± 0.12
Tool error response rate (%) ↓12.0 ± 0.715.6 ± 0.619.7 ± 0.817.9 ± 1.0
Mean generated tokens/task6,927 ± 117,364 ± 1747,612 ± 1737,499 ± 60
Mean interaction time (s) ↓160.1 ± 0.3174.9 ± 4.3176.1 ± 3.6177.4 ± 1.1

Task completion. KVarN is within noise of FP16 (47.4 ± 3.1% vs. 48.2 ± 2.0%). It completes 18.4 points more tasks than KIVI and 7.0 points more than QuaRot.

Tool errors. The increase over FP16 is 3.6 points, about 4 standard errors: it is real, and we do not claim that 2-bit quantization is free. Every quantized cache raises the error rate; KVarN raises it least (3.6 points vs. 5.9 for QuaRot and 7.7 for KIVI), and the model recovers without losing completions. The price of the recovery is 6.3% more generated tokens and 9.2% more interaction time than FP16, in exchange for a KV-cache that is up to 6.9× smaller at long context.

Trajectories. Sampling is stochastic (temp 0.6), so trajectories differ between repeated FP16 runs; the std. err. above quantify this (e.g. ± 2.0 points in FP16 task completion). We did not measure trajectory agreement between FP16 runs and do not claim behavioral equivalence: our claim is limited to task completion on par with FP16 and the measured increase in tool errors.

Please let us know, if there are further questions.

Reviewer_x5afOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies 2-bit KV-cache quantization for long generations such as reasoning traces. It makes three empirical arguments about per-channel (KIVI-style) key quantization:

  • the largest key-quantization errors are predominantly token-magnitude distortions;
  • these large errors drive much of the end-to-end degradation;
  • quantization error accumulates during decoding, because later tokens' keys and values are computed while attending to an already-quantized cache.

The paper introduces a block-wise "pseudo-decode" diagnostic to measure this accumulation.

The proposed method, KVarN, is calibration-free. For each 128-token chunk, it applies a Hadamard rotation and then a Sinkhorn-style alternating row/column variance normalization adapted from SINQ, before round-to-nearest quantization. This stores one additional per-token scale, for about 2.25 bits per element.

Critical strengths

S1. Practical efficiency and system-level integration.

  • KVarN is calibration-free and uses linear round-to-nearest quantization with a uniform 2-bit precision.
  • Its dual-scale dequantization kernel stays within 1.4% of KIVI's kernel across all evaluated context lengths.
  • The authors provide a ready-to-use vLLM integration that reaches 1.10–1.28× the decode throughput of the 16-bit KV baseline.
  • Why it matters: efficient KV-cache quantization is a timely and important problem for LLM inference. Beyond the methodological contribution, the optimized kernel and the vLLM integration make the approach readily deployable, which strengthens the paper's technical contribution and practical relevance.

S2. Strong empirical performance.

  • Among the quantized methods, KVarN achieves the best score in all 12 reported quality columns
  • It remains close to FP16 in most cases.
  • AIME24 and MATH-500 are averaged over three independent runs, which improves the reliability of the evaluation.
  • Why it matters: these results show that the method can substantially reduce KV-cache precision while largely preserving model quality. Combined with the optimized kernel and the vLLM integration, this makes it a promising solution for efficient LLM serving.
Critical weaknesses

W1. Limited novelty and incomplete baseline comparisons.

  • Overlap with existing methods.
    • The two main ingredients come from prior work: a Hadamard rotation as in QuaRot, and Sinkhorn-style variance normalization adapted from SINQ (Müller et al., 2025). The paper acknowledges this (§3.3; App. N).
    • NSNQuant (Son et al., NeurIPS 2025) already applies token-wise normalization with a Hadamard transform for calibration-free low-bit KV quantization.
    • The paper describes how KVarN differs from NSNQuant (iterative normalization and linear RTN instead of a codebook; L437–441) but does not compare against it empirically.
  • Error accumulation has been studied before.
    • PM-KVQ (ICLR 2026) identifies the cumulative error caused by quantizing the KV cache at every decoding step in long-CoT models.
    • GEAR notes that autoregressive decoding "can compound the approximation error at every step".
    • Neither work is cited or discussed.
  • Baseline coverage and reproducibility.
    • TurboQuant is evaluated with the community implementation merged into vLLM, because no official implementation is available (L947–948).
    • KVQuant is represented by a "KVQuant-1%" configuration: 2-bit group quantization plus the top-1% of channels kept in FP16 (L954–956). Its correspondence to the original method needs justification.
    • NSNQuant is not compared.
    • The paper does not state which inference stack produced the non-TurboQuant results in Tables 1–3.

Why it matters: the substantial overlap with existing research makes it difficult to identify the paper's distinct technical contributions and assess its novelty. The ambiguities in baseline configurations and the omission of relevant competing methods also weaken the fairness and completeness of the evaluation. This makes it hard to determine whether the method offers meaningful improvements over existing approaches.

W2. Limited efficiency evaluation and insufficient ablations.

  • Narrow throughput setting.
    • Throughput (Table 7) is measured in a single configuration: 256 concurrent prompts, a 12K-token context and 512 generated tokens per prompt. No repeated measurements are reported.
    • 512 generated tokens is far shorter than the reasoning traces that motivate the paper (up to 16,384 new tokens in the AIME/MATH-500 setup).
    • The GPU memory capacity and the achieved batch sizes are not reported.
  • Memory-reduction figure. The stated 6.9× memory reduction (L375, L485) corresponds to 16/2.3 bits. It appears to exclude the FP16 sink (S = 128) and recent-token (R = 128) regions. By my estimate, including them lowers the effective reduction to about 6.3× at the 12K-token context of Table 7.
  • Ablations.
    • The contributions of the Hadamard rotation and the variance normalization are analyzed only through error-level diagnostics: error distributions (Fig. 8) and attention-output error (Figs. 10–12).
    • There are no end-to-end task-level ablations of each component.
    • The sensitivity of task accuracy to G, S, R and the number of Sinkhorn iterations is not reported. The iteration count is studied only through kurtosis (Table 4).

Why it matters: broader efficiency evaluations and task-level ablations are needed to substantiate the claimed improvements and to separate the paper's new contributions from techniques adopted from prior work. Without isolating each component and examining its sensitivity to key design choices, it is hard to tell which innovations drive the gains and whether they are robust across settings.

Presentation (reflected in my Presentation score). Several figures illustrating the method and the results are simplistic and lack the detail needed to communicate the key ideas. The text inside the figures is too small, and large unused areas make the layout unbalanced. Overall, the paper does not yet appear fully polished.

Overall. The technical contributions and empirical results could still constitute a meaningful contribution. My assessment depends on the authors clarifying the experimental setup, adding the missing baseline comparisons, and distinguishing their contributions more clearly from existing approaches.

Questions for authors
  1. Efficiency setup and variance (clarification).
    • What GPU memory capacity and batch sizes were used for Table 7?
    • Do the throughput gains come mainly from reduced memory bandwidth or from increased capacity (larger batches)?
    • What is the run-to-run variance of the throughput and accuracy results? Several margins over the best baseline (0.2–0.7 points on Phi-4 HumanEval, all IFEval columns and Qwen3-4B MATH-500) are comparable to the reported ± values.
    • How does the method behave for longer generations?
  2. NSNQuant (clarification / limited experiment). Why is NSNQuant, the closest calibration-free normalization approach, not included as a baseline? Please also clarify whether the KVQuant-1% configuration includes the other components of KVQuant, such as pre-RoPE key quantization and non-uniform datatypes.
  3. Related work (discussion). How does this work relate to PM-KVQ (ICLR 2026), GEAR, RotateKV (IJCAI 2025) and "Quantization Hurts Reasoning?", particularly regarding error accumulation in long reasoning traces?
3 comments

Thank you for the detailed review.

1. NSNQuant

We will add NSNQuant. KVarN is better or within noise in all cases.

ModelMethodAIME24 Acc. (%)AIME24 # TokensMATH500 Acc. (%)MATH500 # Tokens
Qwen3-4BFP1661.1 ± 1.812,477 ± 24382.6 ± 0.33,857 ± 13
NSNQuant60.0 ± 3.312,926 ± 12078.9 ± 0.33,844 ± 18
KVarN60.0 ± 0.612,408 ± 19379.2 ± 0.23,925 ± 22
Phi-4-14BFP1662.2 ± 0.911,747 ± 20084.9 ± 0.52,839 ± 20
NSNQuant61.1 ± 4.811,884 ± 15776.5 ± 0.12,927 ± 34
KVarN61.7 ± 1.011,598 ± 13684.8 ± 0.42,984 ± 39
ModelMethodHumanEval Acc. (%)HumanEval # Tokens
Qwen3-4BFP1688.8 ± 1.02,764 ± 162
NSNQuant88.6 ± 0.72,876 ± 70
KVarN88.4 ± 0.23,192 ± 48
Phi-4-14BFP1688.9 ± 0.63,015 ± 30
NSNQuant84.3 ± 1.93,178 ± 54
KVarN88.2 ± 0.23,220 ± 43
ModelMethodIFEval Prompt StrictIFEval Prompt Loose
Qwen3-4BFP1681.2 ± 0.2%86.1 ± 0.4%
NSNQuant80.3 ± 0.3%83.9 ± 0.7%
KVarN80.6 ± 0.4%85.9 ± 0.2%
Llama3.1-8BFP1672.8 ± 0.3%81.4 ± 0.4%
NSNQuant71.9 ± 0.6%75.9 ± 0.8%
KVarN72.4 ± 0.1%80.9 ± 0.2%
Phi4-14BFP1663.8 ± 0.3%70.2 ± 0.1%
NSNQuant63.6 ± 0.1%69.1 ± 0.1%
KVarN63.7 ± 0.1%69.9 ± 0.2%

2. Experimental setup

Bandwidth or capacity? See revised App. Q. With 8 concurrent requests the 16-bit cache fits comfortably, so capacity does not limit the baseline: the gain here is not solely capacity driven (+0.3% at concurrency 1, +3.3% at 4 and +7.4% at 8), see Tabs. 10 and 11.

Longer generations. With 8k input and 8k output tokens on Qwen3-14B, KVarN generates 288.3 tokens/s vs. 269.1 for FP16 (+7.1%); see new Tabs. 9 and 10.

Hardware and batch. Tab. 8 uses 256 concurrent prompts on one GPU with 96 GB VRAM (2× 96 GB for GLM-4.5); the batch is chosen to fill the memory.

Variance. Over 10 runs the throughput with std. is; in the setup of new Tab. 9:

MetricFP16KVarN
K/V bits16/162/2
Output throughput (tokens/s) ↑269.10 ± 0.41288.30 ± 0.26

For accuracy, we agree that some margins are within noise (± is the standard error over 3 runs); this is now stated in the limitations. Still, KVarN is always better or within noise.

Memory reduction. You are right: 6.9× is the long-context limit. E.g. at 12k, in practice this is 6.3×. We now report context-dependent values, see Eq. 3 and Tab. 5.

KVQuant. wherever available, we ran the official code, here including pre-RoPE key quantization and non-uniform datatypes; will add to appendix H.

3. Contribution vs. prior work

The key novelty is our diagnosis and what it enables:

  • Token-scale (multiplicative) errors dominate the largest key errors and are about 4× more harmful than additive errors at equal energy, see Figs. 1b and 2a.
  • Rotation alone leaves these errors intact, which is why QuaRot-style K rotation struggles at 2 bits, see Fig. 8.
  • Direct token scaling as in NSNQuant raises kurtosis (−0.032 to −0.028); the iterative variant lowers it to −0.046, see Tab. 4. This is what permits linear RTN instead of a codebook, and thereby end-to-end throughput gains.

Error accumulation. PM-KVQ and GEAR observed accumulation before us; we will cite both and soften our wording. Our contribution here is the mechanism, the pseudo-decode diagnostic and a calibration-free remedy:

WorkWhat accumulatesFixKey difference vs. KVarN
GEARPer-step error, compounded by autoregressionLow-rank + sparse residual after quantizationReduces reconstruction error incl. entry-wise outliers; KVarN targets token-scale outliers, which drive KL
PM-KVQErrors from quantizing every decode stepProgressive bit-width reduction + per-block bit allocation + RoPE-aware calibrationAllocates bits over time and blocks (calibrated); KVarN reduces the error each bit carries, calibration-free
RotateKV(not framed as accumulation)Calibrated outlier-aware Hadamard rotation, pre-RoPE, sink-awareProtects a few sink tokens at high precision; KVarN normalizes all token scales without calibration
Quantization Hurts Reasoning?Speculated, mechanism left openNone proposed (empirical; recommends QuaRot, KV4 near-lossless)No mechanism for the low-bit failure; KVarN gives mechanism + diagnostic + remedy
KVarNFeedback loop: quantized cache corrupts the next layer's K,VK,V before they are quantizedDual-axis variance normalization + rotation, calibration-freeIsolates accumulation via pseudo-decode and separates immediate error (VarN) from long-horizon compounding (rotation)

I thank the authors for their detailed reply and the additional experiments. The NSNQuant comparison, the clarified experimental setup (hardware, concurrency, longer generations and run-to-run variance), the corrected memory-reduction figure, and the added discussion of PM-KVQ and GEAR address most of my concerns. However, my concern about overlapping contributions remains. KVarN combines a Hadamard rotation, as in QuaRot, with Sinkhorn-style variance normalization adapted from SINQ. The main argued distinction from NSNQuant, iterative rather than single-pass normalization, rests on a modest kurtosis difference (−0.046 vs. −0.028, Tab. 4). In my view, the iterative normalization therefore does not constitute a sufficiently novel contribution on its own. I therefore maintain my score.

Thank you for the follow-up; we are glad that most concerns are resolved.

On the remaining point, we agree in part: the kurtosis difference is modest, and iterative normalization on its own would not carry a paper. This is not our claim. The role of the iterative step is that it makes linear round-to-nearest sufficient, and this is what makes the method fast. We see the contribution in four points that are not in QuaRot, SINQ or NSNQuant:

  1. Speed. KVarN is, to our knowledge, the first 2-bit KV-cache quantizer that decodes faster than the 16-bit cache in a production serving stack: 1.10–1.28× FP16 throughput in vLLM across three models, see Tab. 8, and still +7.1% for 8k-token generations. In contrast, the codebook-based method available in vLLM (TurboQuant) runs at 0.59–0.75× FP16; KVarN is 1.44–2.00× faster at matched nominal bits. NSNQuant is also codebook-based and does not offer fast kernels (only a function-oriented HF implementation). As you note in S1, this is what makes the approach deployable.

  2. Quality without a codebook. With linear RTN, KVarN matches or exceeds NSNQuant at task level: on Phi-4, 84.8% vs. 76.5% on MATH-500 and 88.2% vs. 84.3% on HumanEval; on Llama-3.1-8B, 80.9% vs. 75.9% on IFEval loose. This, not the kurtosis, is the evidence of a relevant difference.

  3. Diagnosis. We show that token-scale (multiplicative) errors dominate the largest key errors and are about 4× more harmful than additive errors at equal energy, see Figs. 1b and 2a. We further explain why rotating K fails at 2 bits, which prior work observed but did not explain.

  4. The combination is not additive. On AIME24 with Qwen3-4B, rotation alone gains 1.2 points over KIVI and VarN alone gains nothing, while together they gain 4.5 points. Neither ingredient paper predicts this.

In SINQ the normalization serves a different purpose (approximating calibration data in weights); here there is no calibration data to approximate, and it helps for an unrelated reason.

We respect that the weighting of novelty is a judgment call and thank you for the constructive discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Window Closing

Dear Reviewers,

The discussion window is closing soon. We have posted individual responses to all reviewers; please take a look. We would be happy to provide further clarification if you have any open questions.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.