- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper targets KV-cache memory growth in long-horizon (test-time scaling) generation.
- It argues that static prefill evaluations miss the error that builds up when later tokens repeatedly attend to an already-quantized cache. It introduces a blockwise “pseudo-decode” diagnostic to measure this.
- It splits key-quantization error into a token-magnitude (multiplicative) part and an orthogonal (additive) residual. It finds that scale distortion drives the largest errors, and that a small fraction of errors (e.g., the top 5%) causes most of the end-to-end degradation.
- It proposes KVarN, a calibration-free linear (RTN) KV-cache quantizer with two stages:
- a channel-wise Hadamard rotation, which equalizes channel outliers;
- online Sinkhorn-style variance normalization along both the token and channel axes, which preserves per-token magnitudes.
- At ~2.25 bits/element, it reports the best mean scores among baselines on AIME24, MATH-500, HumanEval and IFEval. Decode throughput is 1.10–1.28× the 16-bit baseline, and 1.44–2.00× vLLM’s TurboQuant at 3 bits.
Critical strengths
- The motivation is clear and practical. Test-time scaling makes KV-cache memory a real bottleneck, and evaluating quantizers only on static prefill misses the feedback from re-attending to a quantized cache.
- The finding that a small fraction of large errors (top ~5%) accounts for most of the end-to-end KL-divergence, despite being a minority of MSE (Fig. 1a, Fig. 13), is interesting and useful for the community.
- Results are strong across a wide set of baselines (KIVI, QuaRot, KVQuant, PolarQuant, TurboQuant, Kitty), and the method stays linear/RTN, so the end-to-end vLLM throughput gains are credible.
- The method uses uniform precision with a fused dual-scale dequant kernel, and the dequant overhead vs. KIVI is small (≤1.4%, Fig. 16). This is attractive for deployment compared to mixed-precision or codebook methods.
Critical weaknesses
- My main concern is limited novelty. Both stages of KVarN closely mirror existing work:
- the Hadamard rotation comes directly from QuaRot [1];
- the variance normalization is the Sinkhorn dual-scaling of SINQ [2], and Alg. 1 is explicitly “adapted from” it.
- KVLinC [3] already pairs Hadamard rotation with a lightweight correction for key error in 2-bit KV caches (not discussed or compared against). NSNQuant [4] is also calibration-free and normalizes the KV cache along both the token and channel axes before quantization.
- The authors argue KVarN differs from NSNQuant in two ways: its balancing is iterative rather than a fixed single pass, and it uses linear RTN rather than codebook lookups. However, the only evidence for the first is a small kurtosis difference (−0.028 vs. −0.046, Tab. 4). There is no end-to-end comparison against NSNQuant either. The paper should state the delta over [2, 3, 4] directly and compare against KVLinC and NSNQuant at matched bits.
- Section 3.1 is important for motivating the method, but it is fairly well known that multiplicative (scale) errors are more harmful than additive ones. It is also expected that feeding them into a softmax amplifies existing error. The decomposition is a nice way to present this, but it is not a strong contribution in itself.
- Some minor clarity issues. Section 3.3 introduces token-axis scaling and excess kurtosis without setup. The kurtosis differences used to justify iterative balancing are also very small (−0.032 → −0.028).
- TurboQuant runs at 3/3 bits and PolarQuant at 4/2, so these are not matched-bit comparisons. TurboQuant is also a community vLLM implementation that leaves the first and last two layers in FP16. The “3-bit” throughput comparison is also not apples-to-apples, since KVarN 3-bit uses 4-bit K and 2-bit V.
Questions for authors
- How does KVarN differ from KVLinC [3] beyond being calibration-free? Can you add a head-to-head comparison at matched bits/element?
- Fig. 10 suggests Hadamard mostly helps beyond ~50k tokens, yet the benchmark traces are ≤16k tokens. What is the end-to-end accuracy of VarN-only vs. KVarN on AIME/MATH-500? In other words, does the rotation matter at benchmark lengths?
- Can you report vLLM throughput for KIVI/QuaRot at 2 bits? Currently TurboQuant is the only quantized throughput baseline, and it uses a different bit budget.
- How sensitive are the results to the pseudo-decode block size , and to group size ?
References:
[1] Ashkboos et al., “QuaRot: Outlier-free 4-bit inference in rotated LLMs.” NeurIPS, 2024
[2] Müller et al., “SINQ: Sinkhorn-normalized quantization for calibration-free low-precision LLM weights.” ICML, 2026
[3] Saxena et al., “KVLinC: KV Cache Quantization with Hadamard Rotation and Linear Correction.” ArXiv, 2025
[4] Son et al., “NSNQuant: A double normalization approach for calibration-free low-bit vector quantization of KV cache.” NeurIPS, 2025
Thank you for your time and feedback.
1. KVLinC Addition & Comparison; KVLinC + KVarN
Thank you for bringing this paper to our attention; we will cite it and add these explanations (and a discussion in Sec. 5, Related Work).
KVarN + KVLinC: KVLinC and KVarN are essentially orthogonal and can be combined to maximally prevent (KVarN) and repair (KVLinC) quantization damage.
Algorithmic Comparison:
VarN), preserving per-token norms, reducing kurtosis (Tab. 4), and needing only a fused second FP8 scale during dequantization (2.25 vs. KVLinC's 2.71 bpe).VarN. WithVarNabsorbing token-scale outliers, rotating both K and V suppresses cross-layer error accumulation over long reasoning horizons (Sec. 3.3, Fig. 10).End-to-end results (Qwen3-4B, IFEval): We will add this to the paper. Qualitatively, adding KVarN further improves KVLinC.
2. VarN-only AIME24 / MATH-500
Local errors are less sensitive to error accumulation than end-to-end evaluations: on AIME24 and MATH500 (16k token budgets), KVarN outperforms VarN. Specifically, VarN shows a 5.6% (AIME24) and 1.3% (MATH500) accuracy loss compared to KVarN. We will highlight this in the paper.
3. Throughput of KIVI / QuaRot
Thank you for the suggestion; we will add these results.
Because KIVI and QuaRot lack fast kernels for vLLM (or other serving frameworks found), we ablated our KVarN kernel: 1) without normalization/rotation (KIVI-style) and 2) without normalization, with rotation (QuaRot-style). Skipping components naturally runs faster, though KVarN's minor overhead is usually worth its substantial accuracy gains.
Setup: Qwen3-14B, FP16 weights, 8,192 in + 8,192 out tokens, 16 requests, concurrency 8 (1 run, warmup excluded; GPU: 500 FP16 TFLOP, 1.8 TB/s BW). Total time = elapsed time for 16 requests (prefill + decode); E2E = per-request completion latency; Throughput = output tokens / total time.
K2/V2
Nominal ~3 bits: K4/V2 & TurboQuant K3/V3
4. Sensitivity to Block & Group Size
Measured via attention output MAE on Qwen3-4B (cf. Fig. 5a–b); we will add this to the appendix. Since block size typically exceeds group size, we vary block size upward and group size downward.
Block Size: Across 128, 256, and 512, KVarN is insensitive, changing by ≤2% vs. default 128 in both static and accumulated settings.
Group Size: Smaller groups use more bits/element and thus perform better for any method. To normalize this, we report (KIVI error − KVarN error) (cf. Fig. 5b). KVarN's advantage widens with larger group sizes because fewer bits/element create a harder quantization problem where a more involved method yields larger gains.
Mean ± SE over 8 WikiText sequences; MAE pooled over a 1024-token query window; group-size differences computed per sequence (paired).