acceptodds
Under review as a conference paper at ICLR 2027

Quantize, Evict, or Both? When Mixed KV-Cache Allocation Pays, and Why its Distortion Proxy Misleads

Abstract

A key-cache budget can be spent by quantizing every token, by evicting tokens, or by mixing zero and finite precisions across tokens; values stay exact. Hybrid systems exist. We ask when this mixed interior beats both endpoints, and whether an answer measured on attention-output distortion carries over to end tasks. If eviction is treated as a zero-rate tier, a finite-bit tier is dominated when its logitnoise cost exceeds the cost of zero rate. The share of heads whose 2-bit tier is dominated orders the interior’s exactly recomputed output-error advantage across six LLMs and 25 model–context cells (Spearman ρ = −0.978; −0.94 controlling for model). Where the interior stops paying, however, depends on the architecture, and longer contexts move a model toward eviction. Over 4,096 decode tokens a head’s best policy family is stable, but its token allocation goes stale within a few steps. On retrieval tasks we compare against three dense quantizers (TurboQuant, KIVI, KVQuant) and six eviction methods (H2O, SnapKV, DropKV, Ada-KV, LaProx, OBCache). Routed per KV head, the allocator that is optimal under a local surrogate of output distortion beats every eviction method (mean score 0.77 vs. 0.60), partly by sending 1–25% of heads to dense quantization; its mixed interior alone scores 0.47. Yet dense quantization at the same key code bits does better on all three end-task models, and the best quantizer depends on the model. TurboQuant wins on Llama-3.1-8B (0.99 vs. 0.65 for the allocator) and Mistral-7B (0.95 vs. 0.70). On Qwen3-8B TurboQuant loses to the allocator (8k–32k: 0.87 vs. 0.96), and per-channel KIVI or KVQuant win (0.99). Output distortion ranks methods only weakly and overstates the cost of rotated quantization noise. The allocator mainly fails by truncating multi-token answers; smoothing its token scores over neighboring positions largely removes this failure (Llama, 8k–32k: about 40% to 10% of answers). At both levels, the choice of baselines and of when the cache is compressed moves verdicts more than prompt sampling does.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, llm inferences, Quantization, eviction policies

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ARm4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a measurement study of how a key-cache compression budget should be spent when three actions are available: quantize every token, evict tokens, or mix zero-rate (eviction) and finite-precision tiers across tokens. The paper does not propose a better compressor. It builds SIEVE, a per-KV-head, per-token allocator that is optimal under a local first-order attention-output-distortion surrogate, with eviction as the zero-rate tier and values kept exact, and uses it as an instrument to ask three questions: when does the mixed interior beat both endpoints under the distortion objective, does the answer transfer to end-task accuracy, and does the proxy behind such allocators predict what users observe.

Critical strengths
  1. The truncation analysis (F6) explains why token-separable allocation fails on multi-token answers, since a kept token protects only itself and a multi-token needle value is then destroyed, shows that the eviction methods avoiding this all smooth scores over neighboring positions, and demonstrates that pooling the allocator's scores recovers most of the gap (interior mean 0.458 to 0.756 on Llama, past OBCache-K + Ada-KV). F5 adds the counterexample that makes the proxy failure vivid, TurboQuant's high-error/high-accuracy profile. Between them these give the field a diagnosis and two corrected rules of thumb, allocate spans rather than tokens, and use distortion for budget sizing but not for method choice.
  2. The study quantifies how much each protocol choice moves a verdict (Table 3): the eviction corner (5.1-8.6 band points), the information contract (the STOP crossing moves from 46.9% to 57.4%), question visibility at compression time (SnapKV goes from 0.11-0.65 to 1.00), and the choice of dense baseline (0.87 to 0.99 on Qwen), against a prompt-sampling standard deviation of 0.4-3.0 points that changed no verdict. F7 is directly actionable for anyone benchmarking cache compression, and the paired-rerun design with a measured noise threshold is something I would like to see other papers in this area copy. The disclosure apparatus (pre-registration, a detailed AI-use statement, spread estimation) is well above community norms.
Critical weaknesses
  1. The generality of the headline recommendations exceeds the measured scope. Implication 1, report dense low-bit quantization next to any eviction or mixed method, rests on four retrieval-style tasks, three models, 2-3 key code bits, values uncompressed, and the workload family where dense quantization is known in advance to be strongest. Eviction methods are generally expected to do relatively better on generative or aggregative workloads, summarization, long-document QA beyond needle lookup, reasoning traces, and none of these is measured. The attempt to widen the task base, LongBench-v2, was abandoned when it could not separate policies at these budgets. The paper scopes its claims honestly (Table 1, Section 8), but the conclusion most readers will carry away is broader than the evidence. The recommendation would be more defensible if stated as conditional on retrieval-style evaluation, or supported by one non-retrieval workload at budgets where policies differ.

  2. Accuracy-only measurement of an accuracy-motivated problem, with an instrument of unknown deployability. The paper motivates the problem by batch size and decode throughput, but reports no throughput, latency, or memory measurements, and its own appendix notes the eager attention path means accuracy, not cost, is measured. SIEVE needs per-token weights a_i^2 * ||v_i - o||^2, per-KV-head routing, re-budgeting every few decode steps (F3), and cascade re-scoring. Table 5 prices these as regret multiples, not in compute or memory, so a reader cannot judge whether the router's partial accuracy advantages survive any realistic serving constraint. Since dense quantization already wins on all three end-task models, this gap does not change the paper's central negative verdict. It does leave Implication 5, re-budget often, unpriced.

Questions for authors
  1. Has the pre-registered decisive test (positional pooling at Llama-3.1-8B, 128k, B=2) now been run? If so, what is the result? If not, why does the paper still call it "the decisive test" while reporting only the 8k-32k evidence?

  2. Do the F4/F5 verdicts hold on any non-retrieval workload at budgets where policies differ, for example a larger LongBench-v2 sample or a summarization task? If measurement cost makes this infeasible, please state in Implication 1 that the dense-beats-all recommendation is conditional on retrieval-style evaluation.

  3. How do the conclusions change when values are also compressed, say 2-bit KIVI-style value quantization, which is the practical regime? Does the mixed interior become more competitive when both K and V are lossy, or does its advantage remain confined to the key-only setting?

  4. Given the 0.15 tie threshold and single full-grid runs, how stable are SIEVE's wins over OBCache-K + Ada-KV (+0.163) and over the best per-cell eviction method (+0.127)? Would 3-5 reruns of these specific pairs, or 100 prompts on a subset of cells, move these means across the threshold in either direction?

  5. What is the measured serving overhead of SIEVE (weight computation, routing, re-budgeting, cascade scoring) relative to TurboQuant at matched accuracy? Even one end-to-end throughput measurement on a single cell would calibrate whether the router is deployable and whether Implication 5 is affordable.

  6. H2O is implemented with raw accumulated attention, no pooling, and no recent-token budget beyond the 32-token window shared by all methods (Table 7), and scores 0.35, well below commonly reported behavior. Does giving H2O its standard sink plus recent-window configuration change any F4 verdict?

Reviewer_hRRPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when mixed per-token KV cache allocation, which considers token eviction and finite-bit quantization, outperforms two endpoint strategies under the same key-bit budget: uniform quantization that retains all tokens and eviction that retains a subset of tokens at 8-bit precision. It also examines whether improvements in attention-output distortion translate into better downstream task performance.

The authors formulate eviction as a zero-bit option within a locally separable distortion surrogate and introduce SIEVE as a diagnostic allocator for studying mixed allocation. They also propose a tier-extinction metric, D2. Through exact recomputation of attention outputs across six models and 25 model–context configurations, along with retrieval and variable-tracking evaluations on three models, the paper finds that D2 strongly predicts the relative distortion benefits of mixed allocation. Furthermore, lower attention-output distortion does not reliably translate into higher task accuracy. Token-level allocation can lead to truncated multi-token answers, which can be partially mitigated by smoothing token importance scores over neighboring positions.

Critical strengths

The unified analytical framework and controlled measurements provide explanatory value. The paper clearly distinguishes the exact set-eviction error from its separable zero-rate approximation, and recomputes attention outputs after actual quantization or eviction, rather than evaluating performance using the surrogate objective employed during optimization. Across six models and 25 model–context configurations, (D_2) achieves a Spearman correlation of -0.978 with the gains from mixed allocation. The authors also control the information available to both sides of the comparison and report the impact of replacing the endpoint baselines. This makes the results more interpretable than a simple ranking of methods. (§§2–4, Figs. 1–2)

The evidence for the failure of the distortion metric is concrete and provides useful design insights. In Table 9, for Llama-3.1-8B at 128k under a 2-bit key budget, the SIEVE router achieves an output error of 0.094 and a task score of 0.442, while TurboQuant achieves 0.706 and 0.942, respectively. This counterexample shows that lower actual attention-output error does not necessarily lead to better downstream task performance, and that the issue is not merely an inaccurate surrogate approximation. Including multiple quantizers avoids mistaking the failure of a single quantizer for a failure of the entire family. The analysis of answer truncation and the smoothing intervention further identify the granularity of information retention as a key factor. (§5, Tables 2 and 9)

Critical weaknesses

The scope and generalizability of the characterization remain unclear. The main experiments use TurboQuant and show a strong correlation between D2 and the attention-output distortion gains of mixed allocation under this setting. However, D2 is defined based on the logit perturbation cost introduced by a specific quantizer at 2-bit precision, so its value and predictive relationship may depend on the quantization method. Meanwhile, SIEVE minimizes a local separable distortion surrogate, and its performance does not necessarily characterize the potential benefits of the broader mixed-allocation space under exact output error or downstream task accuracy. I think this distinction is important, because the paper itself shows that output-error gains do not reliably translate into task-level improvements. Therefore, the experiments characterize the behavior of a particular surrogate-based allocator under TurboQuant, but provide limited evidence for when mixed allocation is beneficial more generally. In my opinion, validation with an alternative quantization scheme or a small-scale exact-error allocation baseline would strengthen the central claims.

The validation of the main failure mechanism remains incomplete in the critical setting. The paper identifies the smoothing intervention on Llama-3.1-8B at 128k with B=2 as the decisive test, but the reported improvements mainly come from the 8k and 32k settings, while the results of 128k experiment have not been reported. The improvement on Mistral is also limited: the truncation rate decreases from 39% to 24%, while the SIEVE interior score increases from 0.070 to 0.172. The existing evidence demonstrates that attention-output distortion and downstream task performance can diverge, but the proposed explanation that the failure is mainly caused by token-level allocation and can be mitigated by protecting neighboring token spans is only partially supported. The critical intervention should report actual attention-output distortion, task scores, and answer truncation rates together to distinguish a mismatch in the optimization objective from limitations of the specific scoring and allocation implementation.

Questions for authors

Q1. Equation (3) and Appendix C.1 solve the discrete bit allocation problem by searching for lambda. Could the authors clarify whether "optimal" refers to the global optimum of the original discrete budget-constrained problem or the optimum of the Lagrangian subproblem? How are budget discontinuities, ties, and unused budget handled? Could the authors provide the conditions under which optimality is guaranteed, or a small-scale comparison against exact solutions? Q2. Are the predictive results for D2 established only using TurboQuant? Does the ranking relationship still hold with an alternative token-wise quantization scheme, while keeping the budget, available information, and allocation granularity consistent? Moreover, to what extent should the resulting regime map be interpreted as a characterization of SIEVE under its particular distortion surrogate, rather than mixed allocation in general?

Q3. Are the results of the pre-registered smoothing experiment on Llama-3.1-8B at 128k with (B=2) available? Could the authors report the actual attention-output distortion, task scores, and answer truncation rates for the unsmoothed allocator, the smoothed allocator, and the reference eviction method on the same set of prompts? Please also clarify whether the results meet the pre-specified criterion of recovering at least half of the performance gap.

Reviewer_m6azOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

KV-cache compression is important for reducing GPU memory consumption during long-context LLM inference. Existing approaches primarily rely on quantization, token eviction, or combinations of both. However, it remains unclear 1) under what conditions mixed quantization–eviction strategies outperform simpler alternatives and 2) whether minimizing attention-output distortion reliably improves downstream task performance.

To investigate these questions, this paper introduces SIEVE, a token-level rate-allocation framework that jointly considers quantization and eviction under a fixed key-cache memory budget . Specifically, the authors formulate eviction as a zero-bit allocation option within a local attention-output distortion surrogate, introduce a tier-extinction statistic D2D_2 to describe when mixed allocation is beneficial, and employ a per-KV-head router to select among mixed allocation, uniform quantization, and eviction.

The authors evaluate their approach across six LLMs and 25 model–context configurations for attention-output distortion, with additional downstream evaluation on three models using RULER-style retrieval tasks. The experiments compare SIEVE against three dense quantization methods and six eviction methods. The results show a strong correlation between D2D_2 and the attention-output distortion advantage of mixed allocation. However, while the routed SIEVE method generally outperforms eviction baselines, dense quantization achieves better downstream accuracy on the evaluated models. The paper further identifies multi-token answer truncation as a failure mechanism and investigates positional smoothing as a potential remedy.

Critical strengths

S1. Important and well-motivated research problem.

The paper investigates an important problem in efficient LLM inference. Understanding the trade-offs between KV-cache quantization and eviction is practically relevant, particularly for long-context serving. Rather than proposing another compression algorithm solely to improve benchmark performance, the paper examines when different compression strategies are beneficial and whether commonly used distortion objectives accurately reflect downstream task quality.

S2. Interesting empirical characterization of mixed allocation.

The proposed D2D_2 statistic provides an interpretable way to characterize when low-bit quantization becomes less attractive than eviction under the local distortion surrogate. The strong correlation between D2D_2 and the measured distortion advantage across different models and context lengths is an interesting empirical observation. The analysis of how compression regimes change with context length also provides useful insights.

S3. Relatively comprehensive experimental comparisons.

The paper evaluates multiple model architectures and context lengths, and includes both quantization and eviction baselines. The authors also carefully examine the effects of information availability, baseline selection, and compression timing. These controlled comparisons strengthen the experimental analysis and help distinguish the benefits of mixed allocation from those arising from evaluation choices.

S4. Insightful failure analysis and practical implications.

The paper goes beyond observing that attention-output distortion does not always predict downstream accuracy. It investigates specific failure mechanisms, including the relatively high distortion but strong task accuracy of rotated quantization and the truncation of multi-token answers caused by token-level allocation. These analyses make the paper more informative than a straightforward empirical comparison of compression methods.

Critical weaknesses

W1. How general is the D2D_2 diagnostic?

The strong correlation between D2D_2 and the benefit of mixed allocation is compelling, but all reported D2D_2 results are based on TurboQuant-MSE. It is therefore unclear whether D2D_2 captures a general property of the model or the rate–distortion behavior of this particular quantizer. Repeating the analysis with one substantially different quantizer would clarify this point. If the ordering across contexts remains stable but the thresholds change, D2D_2 would be better described as a quantizer-dependent diagnostic. If both remain stable, the paper’s mechanism claim would become considerably stronger.

W2. The downstream evidence could be broader and more directly comparable.

The downstream evaluation covers three models and four synthetic retrieval tasks, with 20 held-out prompts per setting. These experiments provide good evidence for retrieval, but it is not yet clear whether the same conclusions apply to other long-context tasks, such as document question answering, summarization, or reasoning. One non-retrieval benchmark on which the compression methods produce meaningfully different results would help establish the scope of the findings. The Qwen comparison could also be made more conclusive: KIVI and KVQuant achieve slightly higher accuracy than SIEVE but use somewhat more effective key bits. Reporting paired uncertainty and, if possible, comparing the methods at matched effective memory would support a fairer practical ranking.

W3. Practical cost and the proposed failure mechanism need further validation.

The paper does not report end-to-end GPU memory, latency, or throughput. In practice, mixed allocation may introduce additional costs from importance scoring, metadata, irregular bit packing, and nonuniform memory access. A complete memory breakdown, together with even a small serving benchmark, would help readers understand whether the observed accuracy and distortion benefits translate into practical gains. If such an implementation is outside the paper’s scope, the conclusions should be framed accordingly. In addition, positional pooling substantially reduces answer truncation at 8k and 32k, but the corresponding Llama-3.1-8B 128k result is not reported. Since this is the setting with the clearest failure, reporting its accuracy, truncation rate, and output distortion would provide a particularly informative test of the paper’s explanation.

Questions for authors

Q1. How does the proposed D2D_2 statistic perform when evaluated with different quantization methods or distortion models? Does the strong correlation with the benefit of mixed allocation persist beyond the particular quantizer used in the main experiments?

Q2. Could the authors provide additional results with exactly matched effective memory budgets, particularly for the comparison between SIEVE, KIVI, and KVQuant? It would also be helpful to report the results of positional smoothing at the 128k context length, where SIEVE exhibits its most significant truncation failure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.