acceptodds
Under review as a conference paper at ICLR 2027

DeltaKV: Beyond Attention Magnitude for KV Cache Compression

17 views

Abstract

The growing memory footprint and computation cost of the key–value (KV) cache pose a longstanding bottleneck for efficient long-context processing. KV eviction mitigates these costs by retaining a subset of cached entries, and most existing methods decide which entries to retain based on attention scores. However, our analysis shows that attention scores exhibit both query-relevant and query-invariant components; attention over the same context exhibits substantial shared patterns across different queries. Under a limited KV budget, these shared patterns may bias KV selection toward entries that are less relevant to the current query. Motivated by this observation, we propose DeltaKV, a KV eviction method that scores cached entries by contrasting attention under the actual query with attention under a generic proxy query. This simple contrast discounts shared structure and emphasizes query-specific changes to guide KV selection, with minimal computational overhead. Experiments on LongBench-E, RULER, and MileBench show that DeltaKV degrades more gracefully than competing methods as the compression ratio increases. Across these three benchmarks, evicting 90% of KV entries costs less than 7% relative performance. Even with only 2% of the original KV cache, DeltaKV retains over 70% of full-cache performance and outperforms all evaluated baselines in every model–benchmark combination.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, efficient inference, long-context language models, multimodal large language models, attention mechanisms

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tMPTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DeltaKV is a training-free KV-cache eviction method that contrasts attention under the actual task query with attention under a generic proxy query. The difference is intended to suppress shared attention patterns and emphasize query-specific information. Experiments across text, image, and video tasks show strong performance under aggressive compression, although the benefits vary across task types.

Critical strengths
  1. The paper questions whether high attention magnitude reliably indicates task relevance. Contrasting actual-query and proxy-query attention provides a simple, interpretable way to investigate this distinction.
  2. The method requires no training or model-weight changes. Reusing the context cache and jointly processing the two queries provides a practical implementation that limits additional scoring computation.
  3. The evaluation spans multiple modalities and model families, with additional analyses of proxy choice, pooling, and quantization compatibility. These experiments provide useful evidence beyond a single benchmark or task setting.
Critical weaknesses
  1. The proxy robustness analysis does not adequately test semantic overlap between the proxy and the actual task. A generic summarization proxy may attend to precisely the information needed for a summarization request. If the proxy closely matches the real query, subtraction can cancel the useful signal rather than isolate it.
  2. The comparisons do not fully isolate the contribution of attention subtraction. Differences in pooling granularity and query observation windows can independently affect performance, particularly when retaining neighboring evidence is important. A direct no-subtraction control using the otherwise identical pipeline is needed.
  3. The demonstrated applicability is limited to compression conditioned on a known query before generation. Information selected for the initial question may become insufficient as a long answer develops or later turns introduce different requirements. The evaluation does not establish robustness under a continuously enforced cache budget.
Questions for authors

How does DeltaKV behave when the proxy is semantically similar or identical to the actual query, and what prevents useful attention from being cancelled?

Does the advantage remain when actual-query attention alone uses exactly the same aggregation, pooling, sink protection, and budget-allocation procedure?

How does DeltaKV perform during long generation with repeated eviction, or when subsequent questions require information discarded for the initial query?

3 comments

[W1] Robustness to proxy choice

Table 3 reports a proxy-choice ablation on LongBench-E, comparing the default summarization instruction with unrelated statements and a content-free newline proxy at 90% and 95% eviction. Aggregate performance remains stable across these choices. To examine semantic overlap more directly, we provide dataset-level summarization results below and separately discuss the identical-query case.

[Q1] Semantic overlap and attention cancellation

Our default proxy requests summarization, making GovReport and MultiNews relevant cases of task-level semantic overlap. Results on Qwen3-8B at kernel size 32 and 90% eviction are:

ProxyGovReportMultiNewsSAMSum
Default summarization28.1520.5537.58
World Cup statement26.9920.5837.10
Pacific Ocean statement28.9921.1636.82
Twenty newlines28.9820.9237.13

Scores are ROUGE-L, averaged equally across the three dataset length buckets. The default summarization proxy achieves scores comparable to those of the other tested proxies, suggesting that task-level semantic overlap does not necessarily cause performance collapse. The newline results also show that a semantically meaningful summarization instruction is not required for the proxy to be useful in these settings.

These experiments do not establish robustness to arbitrarily similar or identical queries. If the actual and proxy queries produce identical processed attention scores, subtraction yields zero scores and cannot distinguish entries by query-specific relevance; the current method has no explicit fallback for this case. Our intended use is a fixed generic reference. We will clarify this degenerate case and limit our robustness claim to the tested proxy choices.

[Q2, W2] Isolating the contribution of proxy attention subtraction

We added an actual-query-only control within the DeltaKV pipeline, preserving query tokens, aggregation, sink protection, context KV budget, and the global selection procedure. Both scoring methods use identical pooling settings within each comparison. Results on Qwen3-8B and RULER 4K are:

Eviction ratioReal-only, K1DeltaKV, K1Real-only, K32DeltaKV, K32
0.9066.4674.2287.2990.97
0.9548.8556.4777.4682.58
0.9819.2632.4161.6668.31

Without neighborhood pooling (K1), subtraction improves accuracy by 7.76, 7.62, and 13.15 points at 90%, 95%, and 98% eviction, respectively. With K32 fixed, the corresponding gains are 3.68, 5.12, and 6.65 points.

These results support our central premise: query-specific attention contrast provides useful information for KV selection beyond actual-query attention magnitude alone. The gains are observed both without pooling and with pooling held constant. We will incorporate these controls into the revised paper.

[Q3, W3] Query-conditioned selection and long generation

DeltaKV targets query-conditioned selection: under a tight context KV budget, it aims to retain the entries most useful for the available question. The intended usage is to perform selection for each new query. Reselection requires access to the original context for reprocessing or to a separately retained full KV cache, allowing information excluded for one query to be selected for another. It cannot recover discarded evidence from the compressed cache alone.

This represents a different usage pattern from selecting a cache without knowing future questions: DeltaKV prioritizes selection quality for the current request under aggressive compression. Our multiple-question evaluation assumes that all questions are available before selection; it does not evaluate sequential reselection across turns.

For long generation, our summarization evaluations include several hundred generated tokens following one-shot selection. Generated KV entries accumulate during decoding, so the current implementation does not repeatedly evict entries to enforce a fixed total budget. These results evaluate generation from a compressed context, but do not establish robustness under repeated eviction. We will clarify both the intended query-conditioned usage and the distinction between pre-generation selection and continuous cache-budget enforcement.

We have addressed your comments in our response including additional clarifications and experimental results. We would appreciate it if you could review our response and, if it adequately addresses your concerns, reconsider your score.

We would like to kindly follow up on our response. We would appreciate it if you could review our clarifications and additional results and reconsider your score if your concerns have been addressed.

Thank you for your time and consideration.

Reviewer_2DmCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

DeltaKV studies how to reduce KV cache memory and decoding cost while keeping information needed for a given query. The paper finds that different queries share attention patterns over the same context. It proposes scoring cached entries by subtracting attention from a generic proxy query from attention under the actual query. Scores are pooled over nearby positions, and selected entries are kept alongside attention sinks. Experiments cover LongBench-E, RULER, MileBench, and Video-MME with both text and vision-language models. The authors report strong performance at small cache budgets, and also examine proxy choice, model size, quantization, decoding speed, and memory use.

Critical strengths

1.The motivation has useful evidence. On 100 HotpotQA examples, unrelated proxies share 75.2% of selected entries at 10% retention (Section 3, Table 1). This supports looking beyond attention magnitude when choosing which entries to keep.

2.With 90% eviction, Qwen3-8B scores 49.79 against 50.78 with full KV on LongBench-E, and 90.97 against 95.35 on RULER (Tables 3–4). Keeping most performance at this budget is a useful result.

3.In the 32K test, 90% eviction improves decoding throughput by 75.0% and cuts total latency by 20.3% versus full KV, with 6.3% higher first-token latency (Figure 4). This shows a useful speed tradeoff.

Critical weaknesses

1.DeltaKV uses pooling size 32, while SnapKV uses 5 (Section 5.1, Appendix A). Figure 6 shows that pooling strongly affects RULER. Without a matched comparison removing subtraction, the separate contribution of Eq. (5) remains unclear.

2.Section 5.1 describes RULER as covering different context lengths, but Appendix D does not state the tested lengths or report scores by length. This makes the scope of the long-context results difficult to judge.

3.On LCC, DeltaKV scores 41.93 against SnapKV’s 66.11 at 90% eviction (Table 5), with a similar gap in Table 7. The query tokens for code completion are unspecified, making these differences difficult to interpret.

Questions for authors

1.Could the authors report DeltaKV without proxy subtraction, keeping pooling, query tokens, sinks, and cache allocation unchanged? This comparison would help establish how much subtraction adds on its own.

2.Which context lengths were tested on RULER, and how were their scores combined? Results for each tested length would help establish the range over which the gains in Figure 2 hold.

3.How are context and real-query tokens separated for LCC, and do both methods use the same query window? Clarifying this setup and the losses in Tables 5 and 7 would help establish the method’s limits.

2 comments

[W1] Effect of the pooling configuration

To examine whether the different pooling sizes explain the performance gap, we additionally evaluated SnapKV with kernel size 32. On Qwen3-8B and RULER 4K at 90% eviction:

MethodPooling kernel sizeScore
SnapKV544.26
SnapKV3226.51
DeltaKV3290.97

Increasing SnapKV’s pooling size from 5 to 32 reduces its score from 44.26 to 26.51, indicating that a larger pooling kernel alone does not explain DeltaKV’s advantage. To isolate subtraction specifically, we report a matched control below.

[Q1] Isolating the contribution of proxy attention subtraction

Our central hypothesis is that actual-query attention contains both shared attention patterns and query-specific information. Subtracting proxy attention helps identify relevance that attention magnitude alone does not fully capture.

We added an actual-query-only control within the DeltaKV pipeline, preserving query tokens, aggregation, sink protection, context KV budget, and the global selection procedure. Both scoring methods use identical pooling settings within each comparison. Results on Qwen3-8B and RULER 4K are:

Eviction ratioReal-only, K1DeltaKV, K1Real-only, K32DeltaKV, K32
0.9066.4674.2287.2990.97
0.9548.8556.4777.4682.58
0.9819.2632.4161.6668.31

Without neighborhood pooling (K1), subtraction improves accuracy by 7.76, 7.62, and 13.15 points at 90%, 95%, and 98% eviction, respectively. With K32 fixed, the corresponding gains are 3.68, 5.12, and 6.65 points.

These results support the usefulness of query-specific attention contrast beyond raw attention magnitude, both without pooling and with pooling held constant. We will incorporate these controls into the revised paper.

[Q2, W2] RULER context lengths and score aggregation

The main RULER comparisons use the nominal 4,096-length configuration. Scores are unweighted averages across 13 tasks, with 500 examples per task (6,500 total); they are not averages across context lengths.

We additionally evaluated Qwen3-8B at a nominal context length of 16,384 using the same sample counts and evaluation settings. The full-cache score is 93.05; parentheses indicate the percentage of that score retained. CR denotes the fraction of context KV entries evicted.

MethodCR=0.50CR=0.70CR=0.90CR=0.95
DeltaKV, K3289.92 (96.6%)88.18 (94.8%)86.80 (93.3%)84.49 (90.8%)
SnapKV (query-aware)89.63 (96.3%)83.77 (90.0%)70.58 (75.9%)60.52 (65.0%)
KVzip93.14 (100.1%)92.93 (99.9%)78.15 (84.0%)39.20 (42.1%)

These additional results are consistent with the 4K findings: DeltaKV limits performance degradation even under aggressive compression, retaining 93.3% and 90.8% of the full-cache score at 90% and 95% eviction, respectively.

We will clarify the evaluated lengths in Section 5.1 and Appendix D, report scores separately by length, and include the full evaluated compression range. Figure 4’s 32K setting concerns efficiency measurements, separately from these accuracy evaluations.

[Q3, W3] LCC query construction and performance gap

For LCC, the code prefix forms the context, and DeltaKV’s real-query segment consists of the chat-template suffix followed by Next line of code:\n. The ground-truth continuation is excluded from scoring. The observation windows differ: DeltaKV uses this query segment, whereas SnapKV observes the final 64 context tokens containing recent code.

Our results suggest that query-specific attention contrast is particularly effective when an explicit question directs the model to retrieve and combine relevant information from a long context. In LCC, however, the generic completion instruction leaves the required variables, functions, and dependencies implicit in the code. Attention shared with the proxy may therefore remain useful for continuation, while the query provides fewer cues for distinguishing relevant information. SnapKV’s observation of recent code may also help.

This interpretation is consistent with the LCC gaps in Tables 5 and 7, although the existing comparison does not isolate their cause. We will clarify the query construction and observation windows, and discuss how the information conveyed by the query may affect the usefulness of query-specific attention contrast.

Thanks for the additional experiments and explanations. The matched controls and 16K results address my main concerns. The LCC clarification is also helpful. Please include these updates in the revised paper. I raise the soundness score from 2 to 3 and keep my overall rating as 3 (weak acceptance).

Reviewer_fRBnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeltaKV is a KV cache compression method that suppresses repetitive, query-agnostic attention patterns by subtracting the attention weights associated with a generic "proxy" query from those of the actual query. Without requiring additional training, it computes the difference in attention between the two queries and employs techniques such as pooling and the preservation of "attention sinks" to select specific KV states within a constrained budget. The method is evaluated on benchmarks including LongBench-E, RULER, MileBench, and Video-MME, with an analysis of how compression ratios affect both model accuracy and inference efficiency.

Critical strengths
  1. The paper proposes a simple and intuitive method for estimating query-specific KV importance. At 10% KV retention, Table 1 reports 75.2% overlap between Proxy 1 and Proxy 2, versus 44.9% between the actual query and Proxy 1. Gold-document KV retention is 16.8% under the actual query, compared with 7.4% under Proxy 1. These observations motivate contrasting actual-query and proxy-query attention.

  2. DeltaKV maintains high performance across various models and modalities, even under aggressive compression. For Qwen3-8B, with 90% of KV pairs removed, the model achieves a score of 49.79 on LongBench-E—nearly matching the 50.78 score obtained with the full KV cache—and maintains a strong performance of 90.97 on RULER, compared to 95.35 with the full KV cache (Tables 3–4). Additionally, as illustrated in Figure 4, this level of compression yields a 75.0% improvement in decoding throughput and a 20.3% reduction in total request latency, demonstrating its practical efficiency.

Critical weaknesses
  1. The effect of the core contribution—proxy attention subtraction—has not been evaluated independently of the pooling configuration. In Appendix A, DeltaKV employs a pooling kernel size of 32, whereas SnapKV uses the default value of 5. According to Figure 6(b), increasing the kernel size from 4 to 32 in DeltaKV alone significantly improves the RULER performance retention rate—from approximately 56% to 95%—under a 90% token-removal condition. However, the study fails to provide a direct comparison between using only the attention from the actual query tokens and using the proxy attention subtraction method while keeping the pooling configuration constant. Consequently, it remains unclear whether the reported performance gains stem specifically from the core mechanism of attention subtraction.

  2. Certain evaluation details necessary for reproducibility are insufficiently specified. The RULER accuracy experiments do not clearly state the evaluated context lengths, sample counts, or generation seeds. The 32K configuration in Figure 4 concerns efficiency measurements and cannot be assumed to describe the accuracy evaluation. The reduction used to aggregate attention scores from query heads sharing the same KV head under GQA, as well as the numerical value of the sink-protection parameter s, should also be specified.

Questions for authors
  1. Regarding Weakness 1: Could you compare the performance of selecting tokens based solely on actual query attention versus using the proposed subtraction method (subtracting proxy attention), while keeping the pooling kernel size (32), query aggregation, sink token preservation, and KV budget constant? This comparison is essential for evaluating the specific contribution of the proposed subtraction mechanism.

  2. Regarding Weakness 2: Could you specify the context lengths, sample counts, and generation seeds used for the RULER accuracy evaluation? Please also clarify the reduction used to aggregate attention scores from query heads sharing the same KV head under GQA and the value of the sink-protection parameter s

3 comments

[Q1, W1] Isolating the contribution of proxy attention subtraction

Our central hypothesis is that high attention alone does not fully identify query-specific relevance: some tokens receive high attention across different queries. DeltaKV uses the contrast between actual-query and proxy attention to distinguish these shared patterns from attention specific to the current query.

To test this hypothesis, we added an actual-query-only control within the DeltaKV pipeline, preserving query tokens, aggregation, sink protection, context KV budget, and the global selection procedure. We evaluated both scoring methods at matched kernel sizes, including K1, which performs no neighborhood pooling.

Results on Qwen3-8B and RULER are:

Compression ratioReal-only, K1DeltaKV, K1Real-only, K32DeltaKV, K32
0.9066.4674.2287.2990.97
0.9548.8556.4777.4682.58
0.9819.2632.4161.6668.31

Even without neighborhood pooling (K1), DeltaKV improves accuracy by 7.76, 7.62, and 13.15 points at 90%, 95%, and 98% compression ratio, respectively. These results support our central premise: the difference between actual-query and proxy attention provides useful information for KV selection beyond actual-query attention magnitude alone.

The benefit also persists with the reviewer-requested kernel size of 32: DeltaKV improves accuracy by 3.68, 5.12, and 6.65 points at 90%, 95%, and 98% eviction, respectively. We will incorporate these controls and clarify this distinction in the revised paper.

[Q2, W2] RULER evaluation settings, GQA aggregation, and sink protection

The main RULER comparisons use the nominal 4,096-length configuration, with 500 examples per task across 13 tasks (6,500 total). We use greedy decoding with the evaluation random seed set to 42. Overall scores are unweighted averages of the 13 task scores, computed separately for each context length. The 32K configuration in Figure 4 is used for efficiency measurements and is separate from the RULER accuracy evaluations reported here.

We additionally evaluated Qwen3-8B at a nominal context length of 16,384 using the same sample counts and evaluation settings. The full-cache score is 93.05; parentheses indicate the percentage of that score retained. CR denotes compression ratio, the fraction of context KV entries evicted.

MethodCR=0.50CR=0.70CR=0.90CR=0.95
DeltaKV89.92 (96.6%)88.18 (94.8%)86.80 (93.3%)84.49 (90.8%)
SnapKV89.63 (96.3%)83.77 (90.0%)70.58 (75.9%)60.52 (65.0%)
KVzip93.14 (100.1%)92.93 (99.9%)78.15 (84.0%)39.20 (42.1%)

These additional results are consistent with the 4K findings: DeltaKV limits performance degradation even under aggressive compression, retaining 93.3% and 90.8% of the full-cache score at 90% and 95% eviction, respectively.

For GQA, we use max aggregation across query heads sharing the same KV head, as well as across query tokens. Thus, each context token receives the maximum attention value over both axes. We then apply mean pooling along the context-token axis. This procedure is performed separately for the actual and proxy queries, after which the proxy scores are subtracted from the actual-query scores. We protect the first four context positions in every layer/head (s=4). Sinks count toward the context KV budget; query and generated-token KV entries are additional.

We will document these settings, report results separately by context length, and clarify that nominal dataset lengths can differ from actual token counts after model-specific tokenization and chat formatting.

We have addressed your comments in our response including additional clarifications and experimental results. We would appreciate it if you could review our response and, if it adequately addresses your concerns, reconsider your score.

Thank you for the clarifications and additional experiments, which have addressed my concerns. I will raise my score accordingly.

Comments on the paper

1 comment
Area_Chair_CPYTOpenReview ↗

Please Discuss

The authors have responded to the reviews. Given the differing scores, please take another look at the rebuttal and discuss any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.