acceptodds
Under review as a conference paper at ICLR 2027

Keys Decide Attention, Values Decide Contribution: Value-Gated KV Cache Eviction at 20× Compression

Abstract

Long documents are queried many times, so serving systems keep a document's key–value (KV) cache and compress it once, before any question arrives. Existing query-agnostic eviction methods rank cache entries by their keys alone and collapse under severe compression: with 5% of the cache retained, the best of them scores 34 on LongBench passage retrieval, against 100 for the full cache. Key-only scores ignore half of attention. Keys decide whether an entry is attended; values decide what an attended entry contributes, and a distinctive key with a small projected value contributes almost nothing. We propose VG-Evict (Value-Gated Eviction), a training-free rule that multiplies KeyDiff's normalized key-geometry score by the normalized magnitude of each entry's output-projected value, with a small floor and a positional reserve, at cost linear in context length. At 5%/7.5% retention, VG-Evict scores 50.5/70.5 on Qwen2.5-7B, 16.5/11.5 points above Compactor and 7.5/7.0 above CriticalKV applied to KeyDiff, and gains 14.0/17.0 points over Compactor on Mistral-7B; every paired bootstrap interval excludes zero. Controlled ablations attribute the gain to token-aligned value information: gating adds 18.0/23.5 points over KeyDiff at a matched reserve, shuffling the gates among tokens erases the gain, and a single ranking stage matches the best of CriticalKV's two-stage configurations without a stage-ratio hyperparameter. One compressed cache also serves a whole session: on 100 RULER documents at 10% retention it answers 97.5% of 400 independent questions with a tenth of the KV memory. The gains concentrate on passage identification and sparse lookup; for free-form answer extraction, leverage-score selection remains preferable. Code: https://anonymous.4open.science/status/VG-Evict-Anonymous-Code-706D.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, KV cache eviction, query-agnostic compression, long-context inference, LLM inference efficiency, value-aware token selection, key-value attention, passage retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WGX5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper suffers from very limited novelty, highly selective experimental evaluation, and a complete lack of latency measurements. I strongly encourage the authors to thoroughly reassess the quality and completeness of their experimental evaluation before submitting to a top-tier conference, rather than leaving reviewers to identify such fundamental shortcomings.

Critical strengths

 Practical motivation. Compressing the KV cache before queries arrive and reusing the same compressed cache for multiple queries is a meaningful and practically relevant problem.

Critical weaknesses
  1. Very limited novelty. The proposed method is highly similar to VATP and CriticalKV, essentially combining KeyDiff with existing value-aware eviction strategies. Table 2 further shows that single-stage CriticalKV achieves nearly identical performance, making the novelty of VG-Evict highly questionable.
  2. Highly selective and insufficient evaluation. The main experimental results (Table 1 and Figures 1 and 3) focus almost exclusively on Passage Retrieval, just one of the 16 English datasets in LongBench. I think it unacceptable for a KV cache compression paper to build its main claims around a single favorable dataset rather than providing comprehensive benchmark evaluations.
  3. No latency evaluation. The paper provides no compression latency, inference latency, or throughput measurements, despite proposing a method intended to improve inference efficiency. Without these results, its practical efficiency remains unsubstantiated.
  4. Narrowly selected RULER evaluation. Table 4 evaluates only a specific subset of 100 RULER documents with four questions each, rather than providing a comprehensive evaluation. This again raises concerns about selective experimental reporting, especially given the very small improvement over Compactor.
  5. Poor performance on broader benchmarks. Table 11 clearly shows that VG-Evict underperforms Compactor and substantially trails KVzip on the broader 13-task RULER evaluation at both 16K and 32K context lengths. These unfavorable results are relegated to the appendix, while the main paper emphasizes results from narrowly selected tasks.
Questions for authors

see weakness

Reviewer_eWJXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VG-Evict, a training-free, query-agnostic KV cache eviction method that combines KeyDiff's key-geometry score with output-projected value magnitude. The method aims to preserve important KV entries under aggressive compression without requiring future queries. Experiments on LongBench passage retrieval show substantial improvements over existing baselines at 5% and 7.5% KV retention. Ablation studies examine the effects of value gating, positional reservation, and single-stage selection.

Critical strengths
  1. The paper addresses a practical problem in query-agnostic KV cache compression. Compressing a document cache before future queries arrive is relevant to repeated-query serving scenarios. The proposed value-gated selection rule is simple, training-free, and compatible with standard KV eviction.

  2. The main experimental gains are substantial and supported by controlled ablations. VG-Evict consistently improves passage-retrieval accuracy over KeyDiff and Compactor at very low retention rates on two model families.

Critical weaknesses
  1. Limited novelty relative to CriticalKV. The proposed method largely combines KeyDiff's key-geometry score with value weighting already explored by CriticalKV. More importantly, Table 2 shows that removing CriticalKV's key-only selection stage (r = 0) achieves nearly identical accuracy to VG-Evict. The additional normalization and floor also provide no significant improvement over simpler product-based variants. This raises concerns about whether the proposed method offers a sufficiently distinct technical contribution beyond an existing value-aware selection rule.

  2. Limited generality and insufficient evidence of practical efficiency. The strongest results are concentrated on passage retrieval and sparse lookup tasks. Section 4 shows that VG-Evict performs substantially worse than Compactor on free-form answer extraction, while broader RULER, scientific-document, and code evaluations also show limited advantages. Furthermore, despite targeting inference efficiency, the paper reports analytical KV memory savings without measuring compression overhead, end-to-end latency, or throughput. These limitations make the practical benefits difficult to assess beyond the selected workloads.

Questions for authors
  1. Table 2 shows that CriticalKV with r = 0 achieves essentially the same accuracy as VG-Evict. What is the key technical contribution beyond applying a single-stage value-weighted ranking to KeyDiff? Are there settings where VG-Evict consistently outperforms this configuration?

  2. Equation (5) considers the contribution of removed entries before softmax renormalization, whereas actual eviction also changes the attention weights of retained entries. Could the authors clarify how this affects the justification for the proposed selection rule, particularly when normalized KeyDiff scores are used as a surrogate for attention weights?

  3. Do the improvements on Mistral-7B persist when CKV(KeyDiff) uses the same positional reserve and the single-stage configuration? Comparisons at the existing 5% and 7.5% retention rates would help clarify whether VG-Evict provides benefits beyond the optimized baseline across model families.

  4. What are the compression time, peak temporary memory, and end-to-end latency of VG-Evict compared with raw-value gating and the closest baselines? Given that output projection provides only modest additional accuracy, does its computational cost justify its use, particularly when the compressed cache is reused across multiple queries?

Reviewer_aYvUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Summary

  • The authors propose VG-Evict, a method that quantifies the importance of KV pairs by combining key distinctiveness score and value magnitude. The key distinctiveness score was proposed by KeyDiff, and the value magnitude score was used by VATP and CriticalKV, and the score they propose is the product of these two, and they propose to use this combined score.
Critical strengths

Strength

  • They propose a new method for measuring the score for KV pair importance in query-agnostic KV cache eviction. They evaluate on diverse settings.
Critical weaknesses

Weakness

  • VATP and CriticalKV already incorporate value magnitude into attention-based importance scores. This method appears to replace the attention score with KeyDiff’s key-distinctiveness score, with normalization. Although the authors provide a bound motivating the product, the substitution of key distinctiveness for attention remains heuristic, with empirical benefits concentrated in low-budget settings.
  • The proposed method doesn’t show consistent benefit over baseline methods across different settings. For instance, in Table 3, in the passage retrieval task with Qwen-14B and Mistral-7B 15% - 20%, the method is dominated by KVzip performance. In the natural-answer extraction (Appendix D.2), the method is also consistently dominated by Compactor, and in the scientific-document and code retrieval (Appendix E.1), it is also outperformed by Compactor, and in the RULER evaluation (Appendix E.2), by both Compactor and KVzip. I am concerned about whether this is usable across different applications. Furthermore, the trend seems to show that the method seems to be losing gains in the KV storage beyond 15% and more.
  • They do not report the practical compression costs, such as compression latency or the additional memory introduced by their method, compared to the query-free baselines.
Questions for authors

See the weakness section please.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.