acceptodds
Withdrawn from ICLR 2027

Consistency over Accuracy: Outlier-Bypass Layer-Adaptive Quantization for Codec-Based KV Cache Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reusing key–value (KV) caches can eliminate repeated prefill computation in long-context large language model serving, but practical reuse requires the cache to be compressed without significantly altering the model response. Practical lossy KV cache compression raises two challenges: how to measure reuse fidelity and how to mitigate quantization distortion. For the former, aggregate task accuracy is insufficient to reliably characterize answer consistency before and after compression, as compression-induced perturbations may spuriously turn an incorrect answer into a correct one. To address this issue, we propose replacing accuracy with answer consistency as the metric for evaluating compression-induced distortion. For the latter, existing codec-based systems compress the dense representation effectively, yet sparse outliers are not adequately handled. To be specific, a few extreme values can consequently dominate the shared group scale, reducing the effective resolution of normal values and compromising subsequent related allocations. We refer to this effect as outlier-scale pollution. Accordingly, we propose outlier-bypass layer-adaptive quantization. The method detects group-wise extremes by gap prominence, stores them exactly in a separate sparse bypass. Then, the remaining values are quantized using layer-specific level where the corresponding thresholds are customized for each model through offline cumulative-distribution fitting (CDF) calibration. By the calibrated CDF, we derive two presets corresponding to distinct consistency–compression operating points. Across five models and three long-context benchmarks, the high-fidelity preset (Ours-HF) attains 82.33% consistency at 7.41× compression, simultaneously outperforming KVCodec and CacheGen by +1.96 percentage points in consistency with +5.26% and +26.24% in compression ratio, respectively. For workloads that prioritize storage efficiency, the high-compression preset (Ours-HC) further increases the compression ratio to 8.35×, a gain of 18.61% over KVCodec and 42.25% over CacheGen, at a modest consistency cost of 1.45 percentage points.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache compression, long-context LLM serving, quantization, outlier bypass, layer-adaptive quantization, video coding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VcdZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper argues that benchmark accuracy, being an aggregate over many samples, fails to capture compression-induced output changes, and proposes consistency with the FP16 output as the evaluation metric instead. It also proposes outlier-bypass layer-adaptive quantization for KV cache compression.
Critical strengths
  • The paper points out a limitation of the benchmark accuracy used by prior methods and proposes consistency as an alternative metric.
  • It identifies the negative impact of KV cache outliers and stores them separately in FP16, minimizing the effect of quantization on model performance.
  • Experiments are conducted across various models and benchmarks.
Critical weaknesses
  • The novelty of this paper is limited. It is well known that outliers degrade quantization performance, and preserving outliers in FP16 while quantizing the rest is a widely used approach. What is the key finding of this paper that distinguishes it from prior work?
  • The paper lacks a comparison of the fidelity–latency trade-off. The TTFT results in the Appendix show that the proposed method is slower than KVCodec. To judge whether the consistency gain justifies this cost, the method should be compared with prior methods in terms of consistency vs. latency (or throughput).
  • The practicality of benchmark-specific threshold calibration is questionable. In real-world serving, the input domain is not known in advance. Cross-domain experiments are needed, applying thresholds calibrated on one domain to other benchmarks.
Questions for authors
  • Please see the weaknesses above.
Reviewer_QmUqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors argue that task accuracy hides compression damage as the right and wrong flips cancel out, so consistency with FP16 output should be the metric for lossy KV cache reuse. They propose an outlier bypass that stores a few extreme values per group in FP16 and layer-wise precision with CDF-calibrated thresholds on top of KVCodec. HF preset reaches 82% consistency at 7x compression vs 80% at 7x for KVCodec across five models and three benchmarks.

Critical strengths
  1. Consistency metric is demonstrated clearly. up-flips equal down-flips on LVEval in all configs.
  2. Outlier bypass adds almost no storage overhead.
  3. The authors report full per-model tables and calibration stability
Critical weaknesses
  1. Gains over KVCodec are thin.
  2. Allocation ablation only compares against a random shuffle
  3. Thresholds are calibrated on samples from each test benchmark. The authors do not test calibration on one benchmark and evaluating on another
Questions for authors
  1. What is the compression ratio of the uniform 4 bit rows in Table 4?
  2. Why is TTFT slower than KVCodec in Table 13? The authors do not discuss this in the main text.
Reviewer_nKTCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies lossy KV-cache compression for reuse, emphasizing agreement with FP16 outputs. It combines gap-based outlier detection, exact sparse preservation, and layer-adaptive quantization with CDF-calibrated thresholds and a video-coding backend. Experiments on five models and three benchmarks report improved aggregate consistency–compression trade-offs over CacheGen and KVCodec.

Critical strengths

The method separates sparse-extreme preservation from bulk quantization. Exact bypass retains extreme values while reducing the scale used for the remaining entries. Equations 4 and 7 provide a clear local rationale: reducing the scale tightens the rounding-error bound for retained, unclipped values. This addresses bulk resolution without discarding the extremes themselves.

The allocation design connects distribution-aware quantization with codec constraints. Layer-level bounds preserve a simple coding structure, while separate Key/Value ladders and tier quotas offer control over precision allocation. Using relative scale statistics allows allocation to respond to the observed cache rather than relying solely on layer depth (§4.3, Tables 5–6).

Critical weaknesses

The distinctive benefit of the outlier detector is not established. KVQuant and GEAR already separate outliers before quantizing the remaining values, although their broader designs differ. The relevant methodological distinction is therefore the gap-prominence rule. Table 4 compares bypass with no bypass, which does not show why this detector is preferable to a percentile-based alternative. A comparison using the same backend and comparable complete-bitstream budgets is needed to establish this contribution.

The allocation signal and calibration procedure are insufficiently validated. The maximum cleaned group scale is a plausible heuristic for precision allocation, but its connection to downstream sensitivity is not directly established. Moreover, Equation 8 calibrates thresholds using raw maxima, whereas Equation 5 applies them to cleaned maxima. The fit and stability results in Tables 7–8 do not validate this transfer. Comparing calibrated placement with one random permutation only establishes that placement matters; raw-versus-cleaned allocation and a simple empirical-quantile control would clarify the value of the proposed procedure.

Questions for authors

Does gap-prominence detection outperform percentile-based bypass under comparable complete-bitstream budgets?

What supports calibrating on raw scales and allocating on cleaned scales? How do raw-scale allocation and direct empirical-quantile thresholds perform?

In Table 10, Qwen3-14B/LongChat drops from 100.00 to 97.40 while only 0.50% of samples reportedly decrease in score. Are different aggregation weights or subsets used? Please reconcile these statistics and specify the decoding and randomness controls for consistency evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.