acceptodds
Under review as a conference paper at ICLR 2027

Same Memory, Better Answers: Choosing KV Caches by Their Outputs

Abstract

KV-cache compression research asks which tokens to keep, and judges each compressor by the one cache it builds. This misses a larger lever. At a fixed memory budget, different compressors succeed on different questions: retrospectively choosing the best of their caches for each question matches or beats the uncompressed model while keeping one-tenth of the document. Common eviction scores—attention mass, reconstruction error, and hidden-state distance—measure what a cache preserves inside the model, not its next-token distribution. On question answering over research papers, retained attention mass ranks candidate caches worse than a coin flip. We select caches by their outputs instead: each candidate is scored by how far it moves the next-token distribution from the full cache's. The geometry of the unembedding explains what the output score measures: it weights internal changes by their effect on next-token probabilities, information that internal distances discard. With the same candidate caches and scoring passes, output-based selection picks better caches than internal-state features on question answering. It also beats fixed policies tuned on training data and transfers without retraining from 7B to 70B models and to 128K-token documents. Internal features lead on the synthetic needle-retrieval task, where the answer is a verbatim copy.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache compression, long-context inference, efficient LLM inference, cache selection, KL divergence, token eviction, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q7FmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies selecting the best KV cache among different compression methods instead of relying on a single compression policy. The authors propose using output-based features, including KL divergence and top-token overlap, to select a compressed cache. The experiments show better results compared to internal-state-based selection on QA tasks, while state-based methods perform better on synthetic retrieval. The evaluation includes multiple models, up to 70B parameters, and long-context inputs.

Critical strengths
  • The idea of selecting among different compressed KV caches based on their outputs is interesting and provides a useful direction for improving KV-cache compression.

  • The paper provides extensive experiments across multiple models and tasks. The comparison between output-based and internal-state-based features uses the same candidates and forward passes, which makes this particular comparison fair.

Critical weaknesses

Please refer to the questions below.

Questions for authors
  • The proposed method requires significantly more computation compared to existing KV-cache compression methods and still needs access to the full KV cache. While the comparison between output and state features is fair, the overall method is not compared under the same computation and peak memory budgets. Could the authors provide such a comparison? For example, would using a larger compressed cache with a single compression method achieve better results with the same computational resources?

  • The paper reports that retained attention mass performs worse than a coin flip, with a 54.38% ranking error. However, reversing the ranking would result in 45.62% error, which is better than random. This suggests that the metric still contains useful information. Have the authors considered reversing or learning the direction of this metric? How would this affect the conclusions?

  • Have the authors compared the proposed learned selector with simply selecting the cache that minimizes KL divergence from the full-cache output distribution? It would be interesting to understand how much improvement comes from learning the selector versus using the output-based metric directly.

Reviewer_2tpNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a solution related to choosing KV caches, and presents empirical results on question-answering datasets across a range of model scales. Unfortunately, the quality of writing in this paper is not conducive to me understanding enough of it to write a detailed summary.

Critical strengths

None

Critical weaknesses
  1. Despite my best efforts to read this paper, the writing is very hard to parse and understand: there are too many invented jargons, ambiguous references, result paragraphs that are filled with numbers and little insights, and most critical of all, incoherent arguments. Regardless of the research content, I think coherent presentation is non-negotiable for consideration of publication.
  2. To go somewhat deeper into the quality of arguments, I would like to point out some examples that in my opinion do not contribute the best to the logical integrity of the paper:
  • The concept of "candidate" is reused heavily in the paper, but there is no explanation as to what kind of object a candidate is.
  • The experimental setup section mentions a "main grid" and an "executed grid", but there is no justification as to how the latter is supposed to be different, and why.
  • The usage of "F1" as a unit for all numerical arguments is quite non-standard, since it is a metric rather than a unit, as far as I know. There is no justification as to what F1 is an appropriate metric for the task at hand, to begin with.
  • When the authors write "a neural ranker over all fourteen features", it is unclear what "neural ranker" means and what its specifications are.
  • "KV-cache compression has treated the compressor as the unit of progress" is an unverifiable claim.
  • There is no clear conceptual comparison to previous work that may be used to solve the same problem.
  • There are plenty more of such, or worse, instances, which I do not think me pointing out each of them is a productive pursuit, as much as the authors reflecting on the paper, and fundamentally improving its clarity.
Questions for authors

I would like not to ask any question, since I do not think any answer from the authors can change my assessment.

Reviewer_sT1FOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that, at a fixed KV budget, choosing among candidate compressed caches per question is a better than building a single compressor. Different candidate masks, such as, ExpectedAttention, SnapKV, StreamingLLM-style, query-conditioned, document-shared, random, are scored before decoding, either by eight hidden-state distances to the full-cache pass or by two output features (next-token KL, top-10 overlap). The authors train a lightweight linear model on output features to select to select the best cache. The linear cache selector performs better on Qasper and 2WikiMQA datasets but poorer on the RULER dataset compared to the any single compressor. A second-order KL expansion is offered as the explanation of why the output feature might perform bette. The selectors trained on 7–8B models transfer to 32B/70B models.

Critical strengths
  1. The experimental setup involves comparison of internal state features and output features which isolates the usefulness of the output feature signals.
  2. Shows transferability across different model sizes without retraining and reports both positive and negative results highlighting datasets on which the method performs well and on which it lags.
Critical weaknesses
  1. The inference stage comparison of the full caches with all the candidate caches adds decent latency which can impact the practical usefulness of the method. A more detailed end-to-end evaluation of quality improvement and the latency incurred should be discussed.

  2. A single budget (10%), uniform per-head allocation, three extractive-QA-style workloads with token-F1, and a hand-built pool. Claims such as "cache selection is a first-order lever" may be budget- or task-specific. Papers such as AdaKV and RCStat have already shown that adapting the KV budget per different layers and heads can achieve better compression, without having to do multiple types of compressions.

[AdaKV] Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference [RCStat] RCStat: A Statistical Framework for using Relative Contextualization in Transformers

Questions for authors
  1. Can authors discuss more about using different KV cache budgets for the method ?
  2. Can the authors include datasets for more open ended generation like summarisation as well which can help to establish if the chosen output features are general enough.
  3. Authors can add additional cache compression method in the candidate pool to increase generalisability of the method and can discuss about why TOVA was not included in the 11 candidate pool.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Any other concerns or experiments

Dear Reviewers, Area Chairs,

If there are any other concerns or comments, we are happy to clarify and answer specific questions, including any additional concerns you may have after our rebuttal.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.