acceptodds
Under review as a conference paper at ICLR 2027

Information-Aware KV Cache Compression for Long Reasoning

Abstract

Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages. Existing KV cache compression methods mainly rely on attention weights to estimate token importance. While attention effectively captures contextual relevance, it overlooks complementary information-theoretic signals related to predictive uncertainty and token informativeness. In this paper, we revisit token importance from a forward-looking perspective and introduce Forward Influence, a metric that measures how compressed tokens affect future contexts. Our analysis reveals that tokens selected by attention scores mainly influence nearby contexts, whereas tokens associated with high predictive uncertainty exhibit substantially stronger influence on distant future contexts. Based on the observation, we propose InfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals. It combines token-level predictive uncertainty with layer-wise representation evolution and integrates the resulting entropy scores with attention scores during reasoning. Experiments on long-context reasoning benchmarks with Llama-3.1, Llama-3.2, and DeepSeek-R1 demonstrate that InfoKV consistently outperforms existing attention-based KV compression methods in both long prefilling and decoding scenarios.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, KV Compression, Long Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6RPGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a KV cache compression method called InfoKV. It introduces entropy and cross-layer representation evolution as signals to evaluate token importance compared to traditional methods that only use attention weights. The experiments cover long prefilling and long decoding scenarios, and the performance of InfoKV is competitive across multiple models, tasks and budgets.

Critical strengths
  1. The paper focuses on the problem that attention fails to evaluate token importance accurately under some specific contexts. It introduces entropy based on forward influence analysis as an importance signal, and the strategy is aligned with the intuition.

  2. The experiments cover multiple settings, especially in terms of model size and KV budgets.

Critical weaknesses
  1. Although main hyperparameter ablation studies and sensitivity experiments are included, some design choices and parameter settings do not have clear explanations or justification, such as the lack of an ablation study on cross-layer representation, and the lack of evaluation on some latest model families, e.g. Qwen3/3.5.

  2. It should include more recent baselines, e.g. KeyDiff and R-KV.

  3. The efficiency experiment only uses 128-token inputs, and does not quantify the extra computation introduced by entropy and cross-layer representation evolution under long prefilling.

Questions for authors
  1. How does InfoKV perform under long prefilling? For example, how does it compare with baselines in terms of TTFT, peak GPU memory, etc.?

  2. The authors may consider adding KeyDiff and R-KV as baselines. KeyDiff uses key similarity as a signal, while R-KV focuses on long-decoding scenarios.

  3. Currently, the sensitivity analysis of (\tau) does not isolate the effect of cross-layer representation distance.

  4. What are the parameter settings of the 14B model? Do they align with those of the 7B model?

  5. The model DeepSeek-R1-Distill-Qwen-7B is based on Qwen2.5, which is relatively old. The authors may consider verifying the performance on some more recent model families. The Qwen3.5 series utilizes hybrid attention, which may make adaptation difficult; perhaps Qwen3 could be considered.

Reviewer_GnYeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes InfoKV, which scores tokens for KV cache compression by combining attention with predictive entropy. The entropy term is reweighted per layer using the distance between intermediate and final hidden states. The motivation comes from a study of how removing cached tokens affects predictions much later in the sequence. Experiments cover both long-prompt and long-generation settings (LongReason, AIME2024, LiveCodeBench, IFEval). The appendix adds LongBench, a 14B model and repeated IFEval runs.

Critical strengths

1.The paper gives a good reason to look beyond recent-window attention. The forward-influence experiment asks whether a kept token still matters far down the sequence, which is the right question for long reasoning. On the tested documents, tokens chosen by entropy have a longer-lasting effect than those chosen by attention (Figures 1–2). Combining the two signals is simple and targets a concrete weakness of attention-only scoring.

2.The evaluation covers both prompt compression and compression during generation. Gains are clearest on several LongReason CoT settings and on IFEval. For example, Llama-3.1-8B at 40% cache reaches 52.53 average CoT accuracy vs 51.09 for SnapKV. The three-run IFEval results and the 14B results make the evidence stronger. It is also good that the extra runtime and memory over RPC are reported.

Critical weaknesses

1.The explanation of why the score works is not well supported. Predictive entropy measures how uncertain the model is before seeing a token. It does not directly measure how much information the actual token carries. In the same way, a large hidden-state change in some layers doesn't by itself mean the token will be useful later. The forward-influence study helps, but it uses scientific documents and a single model, while the paper's claim is about reasoning traces. Also, the ablations vary tau, k and alpha, but never remove the hidden-state-distance term while keeping the rest. So it is unclear whether this component is needed at all.

2.The "consistent improvement" claim is stronger than the results, and hyperparameter selection is unclear. In Figure 4, InfoKV is below LazyEviction on Qwen-7B/LiveCodeBench at 25% and below RPC on Llama-8B/LiveCodeBench at 25% (36.20 vs 36.56). Several other gaps are small. At the same time, Appendix A.2 uses different tau, k and alpha for each model–task pair, with no mention of a separate tuning split or search budget. If these were tuned on the test sets, the small gains on math and code could just come from tuning. The repeated IFEval runs don't address this.

Questions for authors

1.Is the entropy for token x_i computed from the distribution that predicts x_i, or from the logits after reading x_i? These measure different things. For top-k entropy, are the top-k probabilities renormalized? 2.What happens if you drop the hidden-state-distance term and keep entropy, attention and everything else the same? 3.How were the task-specific settings in Appendix A.2 chosen? Were any evaluation examples used? How does a single fixed setting do across the three decoding tasks?

1 comment

Dear Reviewer GnYe,

As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.

Thank you.

AC

Reviewer_Rr1MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InfoKV, a training-free KV eviction method combining recent attention scores, predictive entropy, and cross-layer representation distance. It introduces Forward Influence to analyze how deleting cached tokens affects future predictive distributions, motivating entropy as a complementary signal for preserving information needed during long reasoning.

Critical strengths
  1. The motivation is relevant and supported by a useful diagnostic. Recent attention may overlook information needed later in a reasoning trace. The Forward Influence analysis provides evidence that entropy-selected tokens can have persistent effects on future predictions.
  2. The method is relatively simple and demonstrates useful improvements. It modifies the selection score without requiring model training or synthesized cache states. Experiments show improvements across several long-prefill and long-decoding settings, supporting the value of incorporating signals beyond attention.
Critical weaknesses
  1. The proposed signals remain heuristic proxies for long-term utility. Predictive entropy and cross-layer representation changes may correlate with future importance, but the connection is not sufficiently established to show that they reliably address evolving dependencies during long reasoning. The method still modifies the metric used to rank and evict KV entries; once relevant entries are removed, they cannot be recovered. The contribution therefore appears primarily to be an incremental scoring improvement, while the underlying challenge of preserving information for unknown future needs remains unresolved.

  2. Practical integration require further investigation. Computing predictive entropy and cross-layer distances requires collecting additional distributions and intermediate representations, introducing computation, storage, and bookkeeping costs. The paper does not establish how the resulting selection policy integrates with shared prefix caches, including whether compressed prefixes can be reused across requests or require request-specific cache management. Evaluation in a serving system with prefix caching enabled would help establish whether the accuracy improvements translate into practical benefits.

Questions for authors

N/A

1 comment

Dear Reviewer Rr1M,

As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.