- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DeltaKV is a training-free KV-cache eviction method that contrasts attention under the actual task query with attention under a generic proxy query. The difference is intended to suppress shared attention patterns and emphasize query-specific information. Experiments across text, image, and video tasks show strong performance under aggressive compression, although the benefits vary across task types.
Critical strengths
- The paper questions whether high attention magnitude reliably indicates task relevance. Contrasting actual-query and proxy-query attention provides a simple, interpretable way to investigate this distinction.
- The method requires no training or model-weight changes. Reusing the context cache and jointly processing the two queries provides a practical implementation that limits additional scoring computation.
- The evaluation spans multiple modalities and model families, with additional analyses of proxy choice, pooling, and quantization compatibility. These experiments provide useful evidence beyond a single benchmark or task setting.
Critical weaknesses
- The proxy robustness analysis does not adequately test semantic overlap between the proxy and the actual task. A generic summarization proxy may attend to precisely the information needed for a summarization request. If the proxy closely matches the real query, subtraction can cancel the useful signal rather than isolate it.
- The comparisons do not fully isolate the contribution of attention subtraction. Differences in pooling granularity and query observation windows can independently affect performance, particularly when retaining neighboring evidence is important. A direct no-subtraction control using the otherwise identical pipeline is needed.
- The demonstrated applicability is limited to compression conditioned on a known query before generation. Information selected for the initial question may become insufficient as a long answer develops or later turns introduce different requirements. The evaluation does not establish robustness under a continuously enforced cache budget.
Questions for authors
How does DeltaKV behave when the proxy is semantically similar or identical to the actual query, and what prevents useful attention from being cancelled?
Does the advantage remain when actual-query attention alone uses exactly the same aggregation, pooling, sink protection, and budget-allocation procedure?
How does DeltaKV perform during long generation with repeated eviction, or when subsequent questions require information discarded for the initial query?
3 comments
We have addressed your comments in our response including additional clarifications and experimental results. We would appreciate it if you could review our response and, if it adequately addresses your concerns, reconsider your score.
We would like to kindly follow up on our response. We would appreciate it if you could review our clarifications and additional results and reconsider your score if your concerns have been addressed.
Thank you for your time and consideration.
[W1] Robustness to proxy choice
Table 3 reports a proxy-choice ablation on LongBench-E, comparing the default summarization instruction with unrelated statements and a content-free newline proxy at 90% and 95% eviction. Aggregate performance remains stable across these choices. To examine semantic overlap more directly, we provide dataset-level summarization results below and separately discuss the identical-query case.
[Q1] Semantic overlap and attention cancellation
Our default proxy requests summarization, making GovReport and MultiNews relevant cases of task-level semantic overlap. Results on Qwen3-8B at kernel size 32 and 90% eviction are:
Scores are ROUGE-L, averaged equally across the three dataset length buckets. The default summarization proxy achieves scores comparable to those of the other tested proxies, suggesting that task-level semantic overlap does not necessarily cause performance collapse. The newline results also show that a semantically meaningful summarization instruction is not required for the proxy to be useful in these settings.
These experiments do not establish robustness to arbitrarily similar or identical queries. If the actual and proxy queries produce identical processed attention scores, subtraction yields zero scores and cannot distinguish entries by query-specific relevance; the current method has no explicit fallback for this case. Our intended use is a fixed generic reference. We will clarify this degenerate case and limit our robustness claim to the tested proxy choices.
[Q2, W2] Isolating the contribution of proxy attention subtraction
We added an actual-query-only control within the DeltaKV pipeline, preserving query tokens, aggregation, sink protection, context KV budget, and the global selection procedure. Both scoring methods use identical pooling settings within each comparison. Results on Qwen3-8B and RULER 4K are:
Without neighborhood pooling (K1), subtraction improves accuracy by 7.76, 7.62, and 13.15 points at 90%, 95%, and 98% eviction, respectively. With K32 fixed, the corresponding gains are 3.68, 5.12, and 6.65 points.
These results support our central premise: query-specific attention contrast provides useful information for KV selection beyond actual-query attention magnitude alone. The gains are observed both without pooling and with pooling held constant. We will incorporate these controls into the revised paper.
[Q3, W3] Query-conditioned selection and long generation
DeltaKV targets query-conditioned selection: under a tight context KV budget, it aims to retain the entries most useful for the available question. The intended usage is to perform selection for each new query. Reselection requires access to the original context for reprocessing or to a separately retained full KV cache, allowing information excluded for one query to be selected for another. It cannot recover discarded evidence from the compressed cache alone.
This represents a different usage pattern from selecting a cache without knowing future questions: DeltaKV prioritizes selection quality for the current request under aggressive compression. Our multiple-question evaluation assumes that all questions are available before selection; it does not evaluate sequential reselection across turns.
For long generation, our summarization evaluations include several hundred generated tokens following one-shot selection. Generated KV entries accumulate during decoding, so the current implementation does not repeatedly evict entries to enforce a fixed total budget. These results evaluate generation from a compressed context, but do not establish robustness under repeated eviction. We will clarify both the intended query-conditioned usage and the distinction between pre-generation selection and continuous cache-budget enforcement.