- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SelKV is a KV cache compression method that requires no training and compresses once after prefill. It targets two problems in existing token merging methods: every evicted token receives the same merge or drop decision regardless of compatibility with its merge target, and a merged entry representing several tokens still gets roughly the attention of one token, a problem that count-based corrections worsen under aggressive compression. SelKV answers these with a soft cosine gate on value vectors that interpolates between full merge and drop, and with a decode-time logit bias derived from prefill attention ratios, bounded by the redistributed attention mass.
The method is evaluated on 16 LongBench datasets across three MHA and GQA models at 25% retention, and on RULER up to 64k tokens. As I read it, the contributions are the two mechanisms and a bootstrap-validated account of when merging helps: blind merging incurs a significant loss that the full method recovers, SelKV outperforms prior merging baselines, and it reaches parity, rather than superiority, with strong eviction. The paper also reports a 3.3x decoding speedup over the full cache at 100k tokens.
Critical strengths
-
The central claims rest on paired bootstrap tests over ablations where all configurations share the same token selection and code path, and the eviction comparison is reported as not significant rather than buried.
-
The paper shows that blind merging is a measured failure mode, not an assumption: under identical selection, eviction alone beats unconditional merging by 0.69 points (p below 0.001), and the gate plus compensation recovers essentially all of that loss. The staged ablation separates what each component contributes.
Critical weaknesses
Weakness 1: Merging is justified against a dichotomy that no longer holds, and the paper's own statistics show it reaches at best parity with simpler eviction
-
The paper motivates merging by contrasting it with eviction: eviction destroys information irreversibly, while merging retains it. I had trouble accepting this framing. Sparse attention with an indexer over the full cache, as in DeepSeek-V3.2, selects tokens for each query and discards nothing, and the full cache can be offloaded instead of compressed. Recent production architectures (e.g., Deepseek-v4 HCA and CSA) learn compression during training, and their attention operates on the compressed entries directly. With these alternatives in place, the choice between destroying tokens and merging them is not exhaustive. The under-weighting of merged positions that the paper corrects is a cost of applying merging after training, not an unavoidable property of compression. I looked for a discussion or a baseline covering either alternative and could not find one.
-
The paper's own results remove the reason to adopt merging even within the framing it chose. The paired bootstrap comparison against eviction gives +0.12 points with p = 0.20, and the paper correctly does not claim superiority. Parity is also the best case on offer: at 31.5k tokens SelKV ties SnapKV exactly at 48.46, at 10% retention SnapKV leads (Table 10, Appendix K), PyramidKV leads on the MHA model, and SnapKV leads on code tasks. The one setting where merging could hold an advantage, QA where the evidence is spread across documents, is described by the authors as suggestive rather than conclusive. What the paper establishes is that merging can be made no worse than eviction, at the cost of a second prefill pass, routing, and a logit bias during decoding. The design space is thin as well: the selection rule is standard and contributes almost nothing (Appendix H), the choice of gate signal barely moves the average (Table 6), and the compensation refines KeepKV's logit bias. I do not see the case for preferring this machinery over the simpler baseline it matches.
Weakness 2: The aggressive compression claim conflicts with Table 10, several reported numbers have no table behind them, and RULER is reported without an eviction baseline
-
Several reported numbers do not reconcile with the paper's own tables. Section 4.4 states that at 10% retention SelKV drops only 1.75%, from 39.81 to 38.06. Table 10 reports 37.56 for SelKV at 10%. The 38.06 in that sentence is the SnapKV number. With the table value the drop is 2.25%, and SnapKV is ahead. The same section cites 10% scores for KVMerger and WeightedKV and points to Appendix K, but the tables in Appendix K contain neither method. Table 2 and Table 6 report different scores for the same configurations: 39.01 against 38.96 with no gate and no compensation, and 39.57 against 39.59 with the value cosine gate alone. Appendix M states that decoding is greedy and deterministic, so identical configurations should give identical scores. The gaps for individual tasks quoted in Section 4.4, such as +4.89 on Qasper against KVMerger, appear in no table.
-
The omitted comparisons are the ones the abstract relies on. RULER is reported against the full cache only, so the claim of retaining 95% quality at 32k and 64k tokens has no eviction comparison point. D2O is the only prior method that partitions tokens into drop and merge sets, exactly the design choice the gate is positioned against, and it is not evaluated. The main LongBench table runs two of the three models on inputs of 3.5k tokens, short for a long context compression paper. The speedup at 100k tokens in the abstract comes with no quality measurement at that length. The limitations section discloses each of these gaps, but the abstract still carries the corresponding numbers as contributions.
Questions for authors
Question 1: Section 4.4 reports SelKV at 38.06 at 10% retention, but Table 10 reports 37.56, and 38.06 is SnapKV's score. Which number is correct, and if Table 10 is, how should the abstract's claim that margins widen under aggressive compression be read when SnapKV leads at the most aggressive setting? Please also point to where the cited 10% scores for KVMerger (32.98) and WeightedKV (33.71) appear, since the tables in Appendix K contain neither method.
Question 2: Table 2 and Table 6 report different scores for the same configurations: 39.01 against 38.96 with no gate and no compensation, and 39.57 against 39.59 with the value cosine gate alone. Appendix M states that decoding is greedy and deterministic. What exactly differs between the two experiments?
Question 3: D2O is excluded on the ground that its official implementation supports only LLaMA-2, yet Appendix E reports your own results on LLaMA-2-7B. What prevents evaluating D2O on LLaMA-2-7B, the one model where both methods are available? It is the comparison most directly matched to the gate's claim, since D2O is the only prior method that partitions tokens into drop and merge sets.
Question 4: RULER is reported against the full cache only, so the 95% retention headline at 32k and 64k has no eviction comparison point. Can SnapKV be added to Table 4 at these two lengths? Appendix M puts a full evaluation on 16 datasets at 1 to 2 GPU hours, so the cost appears modest.
Question 5: The related work stops around the middle of 2025 and omits the directions that most directly bear on the paper's framing: DeepSeek-V3.2's sparse attention, where a lightning indexer selects from the full cache and discards nothing, and DeepSeek-V4's CSA, which learns compression during training. Why are these not discussed, and under what serving conditions is this form of lossy compression after prefill preferable to either?
Question 6: The aggregate gap over eviction is +0.12 and not significant, and the paper locates merging's remaining advantage on tasks where evidence is spread across documents. On which individual tasks or prompts does SelKV beat the eviction only configuration of Table 2, and does the win concentrate where evidence is distributed?
4 comments
Q3: D2O comparison
We ran D2O on all three models, including LLaMA-2-7B as the reviewer requested. We implemented D2O's core algorithm (key-cosine nearest-neighbor matching, mean-similarity threshold, similarity-weighted merging) within our framework for a controlled comparison (same token selection, same retention budget, same evaluation code):
| Method | LLaMA-2-7B | LLaMA-3.1-8B | Gemma-2-9B |
|---|---|---|---|
| SelKV | 31.78 | 39.90 | 42.31 |
| SnapKV (eviction) | 32.77 | 39.77 | 42.16 |
| D2O | 31.89 | 39.48 | 42.13 |
On LLaMA-3.1, SelKV leads D2O by +0.42 (paired bootstrap, one-sided p<0.001, 95% CI [+0.20, +0.63]), winning 13/16 tasks. On Gemma-2, SelKV leads by +0.18. On LLaMA-2 (MHA), D2O slightly leads SelKV (31.89 vs 31.78), though both trail SnapKV (32.77). This is consistent with the paper's finding that merging is weaker under MHA where union-based selection does not apply.
See Reviewer fg8h Q1 for per-task breakdown and implementation details.
Q4: RULER with SnapKV
We added SnapKV and 5 other methods to RULER (LLaMA-3.1-8B, 25% retention):
| Method | 4k | 32k | 64k |
|---|---|---|---|
| Full Cache | 92.1 | 84.4 | 81.2 |
| SnapKV (eviction) | 80.7 | 82.2 | 78.2 |
| SelKV (gate+comp) | 79.9 | 80.5 | 77.0 |
| Merge-all (no gate) | 79.6 | 75.4 | 75.7 |
| H2O | 40.6 | 77.9 | 75.8 |
| LOOK-M | 28.1 | 70.6 | 72.0 |
SnapKV leads SelKV by 0.8 at 4k, 1.7 at 32k, and 1.2 at 64k, consistent with the aggregate LongBench finding. RULER's tasks are predominantly exact-match retrieval where token identity matters, the regime where eviction has a natural advantage. We present this transparently. SelKV outperforms all other merging methods; compensation drives the largest gain at 32k (+5.2: gate-only 75.3→80.5) while the gate contributes at 64k (+1.4).
Q5: DeepSeek-V3.2/V4 in related work
See CW1 above. We will add these to the related work with a discussion of when post-prefill compression remains the right choice.
Q6: Per-task analysis of merging vs eviction
Using SnapKV as the eviction baseline (same token selection, eviction-only), full per-task breakdown on LLaMA-3.1-8B (25% retention):
| Task | Category | SnapKV | SelKV | Δ |
|---|---|---|---|---|
| trec | Few-shot | 65.50 | 66.50 | +1.00 |
| triviaqa | Few-shot | 89.24 | 90.00 | +0.76 |
| musique | Multi-doc QA | 18.89 | 19.55 | +0.66 |
| multifieldqa_en | Single-doc QA | 49.88 | 50.27 | +0.39 |
| hotpotqa | Multi-doc QA | 39.75 | 39.94 | +0.19 |
| narrativeqa | Single-doc QA | 21.38 | 21.56 | +0.18 |
| samsum | Few-shot | 42.10 | 42.19 | +0.09 |
| 2wikimqa | Multi-doc QA | 35.86 | 35.91 | +0.05 |
| passage_retrieval_en | Synthetic | 34.00 | 34.00 | +0.00 |
| multi_news | Summarization | 23.91 | 23.91 | +0.00 |
| passage_count | Synthetic | 4.08 | 4.03 | −0.05 |
| gov_report | Summarization | 28.09 | 28.00 | −0.09 |
| repobench-p | Code | 58.48 | 58.31 | −0.17 |
| qmsum | Summarization | 21.56 | 21.38 | −0.18 |
| qasper | Single-doc QA | 38.92 | 38.62 | −0.30 |
| lcc | Code | 64.69 | 64.17 | −0.52 |
| AVERAGE | 39.77 | 39.90 | +0.13 |
SelKV wins on 8 tasks, ties on 2, SnapKV wins on 6. Note: SnapKV uses the same importance scoring (Eq. 1) as SelKV but evicts rather than merges, so this is a direct eviction-vs-merging comparison. Per-task deltas are indicative (all methods were re-run from a single checkpoint; we do not have per-task significance tests). The pattern aligns with the design:
- SelKV wins on all 3 multi-document QA tasks (musique +0.66, hotpotqa +0.19, 2wikimqa +0.05), where answers require synthesizing evidence from multiple locations and merging preserves information that eviction discards.
- SnapKV wins on code (lcc −0.52) and retrieval (qasper −0.30), where tokens carry unique structural identity that averaging corrupts.
Merging helps when compatible information is distributed, and hurts when token identity is critical. The gate limits the worst per-task delta to −0.52 (lcc).
I recognize the authors for the additional experiments and detailed response.
But the new results do not quite restore the case for the method. The revised positioning puts SelKV in the same setting as SnapKV, and the simpler eviction baseline matches or beats SelKV. Since the contribution reads as weaker after the rebuttal, I will keep my rating at 1. After all, I thank the author's honest disclosure.
We appreciate the reviewer's acknowledgement and fair reading of the results. We want to clarify one point about framing, since the rebuttal may have put too much weight on the SelKV-vs-SnapKV comparison.
We agree that merging does not beat eviction in aggregate, and that is not the claim we are making. What we show is that blind merging loses 0.65 points vs eviction (p<0.001), that prior merging methods all operate in this regime, and that the gate and compensation recover the loss entirely. The ablations show why: value-space compatibility turns out to be the right merging criterion (+0.75 over EMS-style key×value, p<0.001), and attention-ratio compensation replaces KeepKV's count-based correction which becomes harmful at high compression. These findings apply to any merging pipeline, not just ours.
The per-task picture also matters: SelKV wins 8/16 tasks (all three multi-doc QA), SnapKV wins 6 (code, retrieval). Merging and eviction fail on different tasks for different reasons, and knowing when each applies is useful even if the averages tie.
We respect that the reviewer may still consider this insufficient for acceptance. We have revised the abstract and claims accordingly.
We recognize this review raises fundamental questions about the paper's positioning, and we address them directly.
CW1: Merging motivation and DeepSeek-V3.2/V4
The reviewer raises an important point about the evolving landscape. We agree the drop-vs-merge dichotomy is not exhaustive and will revise the introduction.
DeepSeek-V3.2's sparse attention uses a full-cache indexer that selects tokens per query without discarding anything. This is powerful but has different requirements: it maintains the full KV cache in memory (or on secondary storage with offloading) and adds a retrieval step per query. SelKV targets the setting where the cache must physically fit in GPU memory, e.g., serving multiple concurrent requests on commodity hardware where offloading is too slow.
DeepSeek-V4's CSA/HCA learns compression during training, producing superior compressed representations, but requires training-time integration and architecture-specific design. SelKV is training-free and applies post-hoc to any existing model. CSA/HCA is for frontier model design; SelKV is for compressing the cache of already-deployed models that cannot be retrained.
We will add a "Positioning" subsection to the introduction distinguishing three compression paradigms:
SelKV contributes to paradigm 3, which remains relevant for serving existing models under memory constraints.
CW2: Merging vs eviction, why prefer the machinery?
We accept the reviewer's assessment that SelKV achieves parity with eviction, not superiority, in aggregate. The case for merging rests on three points:
Safety: Without the gate, merging loses 0.65 points (p<0.001). The gate makes merging safe. Among post-prefill compression strategies, merging is the only one that preserves evicted information within the compressed cache (as opposed to sparse retrieval or offloading, which maintain a full copy); if it can be made reliable, it opens a path to better future methods.
Per-task advantages: On all three multi-document QA tasks, SelKV outperforms SnapKV: musique +0.66, hotpotqa +0.19, 2wikimqa +0.05. These are the tasks where compressed information matters, since the answer requires synthesizing evidence from multiple locations and eviction permanently loses some of it. SelKV wins on 8 of 16 tasks (2 ties, 6 losses; see Q6 below for the full table).
Merging-method comparison: SelKV outperforms all prior merging methods by large margins (KVMerger +1.70, WeightedKV +1.39, LOOK-M +1.19, all p<0.001) and the eviction baseline H2O by +1.59. For practitioners who choose merging for the reasons above, SelKV is the best option.
Regarding the "thin design space" concern: we agree the token selection rule is standard (Appendix H), but the gate signal choice matters more than the aggregate suggests. Value cosine outperforms both key cosine (+0.29 without compensation) and an EMS-style key×value gate (+0.75 with compensation, p<0.001), a meaningful difference supported by the ablation: the value-only gate empirically outperforms alternatives that incorporate key information. The compensation mechanism is also fundamentally different from KeepKV's merge-count bias: attention-ratio compensation is bounded by the actual redistributed attention mass, while merge-count compensation uses raw counts that become unstable under aggressive compression (KeepKV scores 38.90 vs SelKV's 39.90).
We acknowledge this is a narrower contribution than the original framing suggested, and will revise the abstract and introduction accordingly.
Q1: Section 4.4 numerical error
The reviewer is correct. Table 10 reports SelKV=37.56 and SnapKV=38.06 at 10% retention. The Section 4.4 text incorrectly cites 38.06 for SelKV, which comes from a different experiment. SnapKV leads at 10% retention.
The abstract's claim that "margins widen under aggressive compression" refers to the comparison against other merging methods (KVMerger=32.98, WeightedKV=33.71), not against eviction. We will clarify this and correct the text.
Q2: Table 2 vs Table 6 discrepancy
Tables 2 and 6 came from separate experimental runs, both using the code's default hyperparameters rather than the values specified in Section 4.1. This affected Tables 2 and 6 (LLaMA-3.1 ablation); Tables 1 and 4 used correct settings and have been re-verified. We re-ran all LLaMA-3.1 configurations at 25% from a single checkpoint with the correct settings (see the unified 2×2 factorial in our response to Reviewer WE9x Q6). All baseline methods were also re-run. All numbers in the revision will come from this unified run.