acceptodds
Under review as a conference paper at ICLR 2027

Never Evict Always Taper: A Static Logarithmic Bit-Width Schedule for KV Cache Quantization

Abstract

Long-context inference relies on an ever-growing key-value (KV) cache, making precision allocation a key challenge for memory-efficient generation. We study how to distribute precision across cached tokens while preserving access to older information, without requiring online attention tracking. Under an exponential distortion surrogate and a prescribed precision floor, reverse water-filling yields an optimal allocation in which precision depends logarithmically on token importance; with power-law age weighting, this naturally produces a precision schedule that decreases with token age. A complementary perturbation analysis establishes a sufficient precision floor for preserving the highest-scoring attention key. Motivated by these results, we propose TaperKV, a token-preserving KV cache with a static, age-indexed integer-precision schedule that is compatible with a broad range of quantizers. Experiments on retrieval, language modeling, and mathematical reasoning show that TaperKV substantially reduces cache storage while maintaining strong task performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache quantization, long-context inference, memory-efficient inference, quantization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_5ZJMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TaperKV, a mixed-precision KV cache compression method that the authors claim is novel. It retains all KV cache entries and statically assigns numerical precision based on token positions, with theoretical justification for this allocation strategy. Through an experimental evaluation with certain limitations, the paper provides evidence to support its claims.

Critical strengths

None. Its only merit is that Section 7 of this paper thoroughly discusses its own limitations of its experiments.

Critical weaknesses
  1. The paper contains several writing and presentation errors, including the unresolved reference “Figure ??” on line 76 and the incorrect cropping of Figure 2(a). Moreover, Section 3 lacks intuitive explanations of what the theoretical results imply, and their connection to the method in Section 4 is weak, making the paper difficult to follow.

  2. To my understanding, the core method assigns different numerical precisions to key and value cache entries based on their tokens’ distance from the current position. However, no ablation studies justify the specific bit-width hyperparameters or establish their effectiveness. Regarding the theoretical justification, the authors explicitly state on line 178 of Section 3: “They do not certify the equal-width implementation below.” Thus, these results do not provide theoretical guarantees for the method subsequently introduced.

  3. The experimental evaluation is severely lacking. As a paper on KV cache compression, it reports neither speedups nor actual memory usage, and the maximum context length evaluated on RULER is only 16K. The limitations acknowledged in Section 7 concern experiments that are essential to substantiate the paper’s claims, rather than optional extensions that can be deferred to future work.

Questions for authors

None

Reviewer_VVncOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies age-based mixed-precision quantization of KV caches, progressively reducing the precision of older entries while retaining all tokens. Under prescribed distortion and importance models, the authors derive a logarithmic bit-allocation rule and a conditional precision floor for preserving the highest-scoring attention key. The implemented method, TaperKV, uses equal-width age bands, while TurboTaper incorporates a rotated quantizer. Experiments cover retrieval and mathematical reasoning tasks and report encoded-cache storage ratios.

Critical strengths
  1. A simple scheduling rule. The method allocates precision using only position and age, without calibration or online attention tracking. This provides a straightforward approach to retaining all historical positions.

  2. Some configurations show a useful compression–accuracy trade-off. For example, TaperKV-4bit reports the same NIAH and MATH-subset scores as FullKV on GPT-OSS-20B while reducing encoded-cache storage.

Critical weaknesses
  1. A weak connection between theory and implementation, with limited novelty. The analysis yields a logarithmic precision allocation, whereas the implementation uses equal-width age bands without controlled comparisons to justify this choice. Relative to existing mixed-precision and progressive quantization methods, the distinct value of the age-based schedule remains insufficiently established.

  2. Inconsistent configurations and storage accounting. The default ladder in the main text implies an asymptotic nominal-code fraction of 8/32, whereas the appendix states 9/32. Combining the appendix's code-bit lower bound, the main text's metadata specification, and the normalization in Eq. (29) gives a storage fraction of at least approximately 0.2344 for TurboTaper-3bit, yet Table 1 reports 0.219 for the R1 model. Incomplete configuration details and run records make these discrepancies difficult to resolve.

  3. Insufficient validation of baseline comparisons. StreamingLLM and ThinKV score near zero on several tasks, but the paper provides insufficient configuration details and failure analysis to establish whether these results are representative. PM-KVQ, a directly relevant progressive quantization method, is also absent from the experiments.

  4. Practical efficiency benefits remain unverified. The evaluation uses simulated quantization without measuring actual cache memory usage, re-encoding overhead, or serving throughput. The cost of periodically scanning and requantizing the cache remains unclear, so the extent to which encoded-storage reductions translate into deployment benefits is not established.

Questions for authors

Please address the four concerns above, particularly the bit-width configurations, storage-accounting conventions, and baseline settings.

Reviewer_b6n9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • The paper proposes TaperKV, a training-free, calibration-free KV cache quantization scheme. It keeps every token but lowers its bit width as the token ages. It does not track attention online.
  • Under an exponential distortion surrogate with a precision floor, reverse water-filling gives optimal bit widths that grow logarithmically with token importance (Theorem 3.1). If importance follows a power law in token age, precision then decreases logarithmically with age (Corollary 3.2).
  • The deployed schedule uses equal-width age bands with protected sink and recent tokens, and separate ladders for keys and values. TurboTaper pairs the same schedule with a rotated Lloyd–Max codec.
  • Evaluation covers NIAH, AIME24/25 and MATH-500 on GPT-OSS-20B and DeepSeek-R1-Distill-Qwen-7B, plus a key-floor sweep on Llama3-8B.
Critical strengths
  • The motivation is reasonable. A static, position-only precision rule that needs no attention statistics, calibration or phase detection is attractive for deployment. It is also naturally compatible with paged attention.
  • The “taper before you evict” framing is intuitive. The comparison against eviction makes the point that keeping a low-precision version of a token beats deleting it.
  • The floor sensitivity result (Figure 2a) is a useful practical finding. An overly low key floor is the main cause of degradation.
Critical weaknesses
  • My main concern is the lack of experiments and the near-absence of ablations. This domain expects extensive evaluation from a quantization paper, and the current one is thin:
    • only two main models;
    • small evaluation sets (NIAH n=42–45, the first 100 MATH-500 problems, AIME with 8 samples);
    • no confidence intervals;
    • NIAH only up to 16k context;
    • no standard long-context suites such as LongBench [1], RULER [2] or InfiniteBench [3].Cases where TaperKV beats FullKV (e.g., AIME24 0.788 vs. 0.758, NIAH 0.600 vs. 0.567) are very likely within noise.
  • The paper proposes both TaperKV and TurboTaper, but TurboTaper performs poorly. On R1-Qwen-7B it collapses: NIAH 0.072 and AIME24 0.013 at 3 bits. This is never explained. It also undercuts the claim that the schedule “accepts any codec that takes an integer width”, since rotated codecs [4, 5] usually reduce distortion.
  • The contribution is mainly theoretical, and I find the theory itself lacking (I note this is not my core area, so I am less confident on this point). Much of it applies classical reverse water-filling to a stylized surrogate.
    • The theory is also disconnected from the deployed method. The paper concedes that the equal-width ladder is not the logarithmic optimum.
    • The power-law age-importance assumption is never checked against measured attention (Figure 3 is illustrative only).
    • The floor condition in Eq. 11 is never measured either.
  • At comparable memory, the gains over KIVI are marginal or absent. KIVI-INT3 matches or beats TaperKV-4bit on most GPT-OSS columns. No equal-memory comparison is given. The closest related work is missing: mixed-precision and importance-aware baselines such as MiKV [6], ZipCache [7], KVTuner [8] and PM-KVQ [9].
  • Quantization is simulated, and there are no measurements of latency, throughput, peak memory or requantization overhead (re-encoding every r=16 steps).
Questions for authors
  1. Why does TurboTaper degrade so badly on R1-Qwen-7B relative to TaperKV, despite using a rotated codec that should reduce distortion? Is this an implementation issue, or an interaction between progressive requantization and the rotated codec?
  2. Can you provide equal-memory comparisons against KIVI and mixed-precision baselines (MiKV, ZipCache, KVTuner, PM-KVQ), with confidence intervals across seeds?
  3. What is the wall-clock overhead of periodic requantization in a real packed-cache implementation, and what are the actual memory savings?

References:

[1] Bai et al., “LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding.” ACL, 2024

[2] Hsieh et al., “RULER: What’s the Real Context Size of Your Long-Context Language Models?” COLM, 2024

[3] Zhang et al., “Infinity-Bench: Extending Long Context Evaluation Beyond 100K Tokens.” ACL, 2024

[4] Ashkboos et al., “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.” NeurIPS, 2024

[5] Zandieh et al., “TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate.” ICLR, 2026

[6] Yang et al., “No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization.” ArXiv, 2024

[7] He et al., “ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification.” NeurIPS, 2024

[8] Li et al., “KVTuner: Sensitivity-Aware Layer-wise Mixed Precision KV Cache Quantization.” ICML, 2025

[9] Liu et al., “PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs.” ICLR, 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.