- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a KV cache eviction strategy with learned gating weights. This approach enables a query-inependent gate that can capture redundancy and maintans a recovery mechanism for the discarded entires. The key insight they identify is that these learned weights in softmax when trained on a fixed model is similar to dense attention model collapse to predictions of dense attn and the training approach proposed denies the gating mechanism dense attention and forces it to match dense attention predictions. The discarded entries are also retained with centroid recovery to get the residual contributions. This duality of selection process and maintaining a summarized evicted cache is an extension of similar idea proposed in indexmem paper. The results presented in the paper are promising and it can maintain performance at long context benchmarks with up to 75% evictions.
Critical strengths
The paper proposes gating for KV cache eviction and also highlights key challenge with collapsing score when used inside softmax calcutaion. The propose a clean solution to this by forcing it to estimate the dense attention. They propose using distillation rather than standard cross enropy loss which mixes the contributions from eviction loss from the frozen backbone. This is a clean separation to only train on changes due to eviction. The ablation studies clearly show the benefits of this formulation. The results for RULER benchmark and longbench accuracy tests maintain for upto 75% compression and shows advantage over other KV eviction approaches. The decode throughput also shows good performance with increasing context length.
Critical weaknesses
Generalization of the approach on broader class of MoE models is not presented. The accuracy result on Ruler and longbech are promising, but to be broadly application benchmarks such as GMS8K, AIME etc. are missing (noted in next section). The throughput results are not conclusive because baseline and indexmem++ use different implementation approaches. Discrepancy in training and inference gates are highlighted, but need further ablation to showcase the impact and also OPD is proposed as future work but it is unclear how this impact generalization of this approach
Questions for authors
The paper mainly focuses on dense models, how well does this scale for MoE model architectures. The measurement setup uses two different implementation eager with FAv2 and cudagraphed for indexmem++, this maked the comparison results on latency difficult to determine the improvements from KV eviction. Can we match results with similar cuagraph implementation for baseline as well. Can authors add other standard benchmarks like GSM8K, AIME, MMLU to understand impact of generalization how does the difference in gates trained from dense teacher v/s deployment / inference on evictions without dense teacher guidance impact RULER, GMS8k benchmarks? Can we add an ablation study where we evict based on the teacher to isolate accuracy impact.
3 comments
Q4. The gap between training with a dense teacher and deploying hard eviction; an ablation using teacher-guided eviction.
A4.
The dense teacher is used only during training, to provide the target output distribution; at inference the scorer runs on its own. The gap between the two is not in the scorer's input or its output, which are identical, but in the conditions it is trained under. There are two concrete differences.
- Soft gating vs. hard selection. During training the gate only scales an entry's attention weight, so a low-scored entry still contributes a little and its gradient remains informative. At inference the same scores pick a top-B set, and every entry outside it is deleted outright. Training therefore optimizes a smooth relaxation of the decision that is actually deployed: an error near the selection boundary costs little during training but removes the entry entirely at inference.
- Teacher-forced vs. self-generated states. During training every position conditions on the reference text with a dense cache, so the gate never sees a state produced by its own earlier evictions. At inference, each eviction changes the hidden states of later tokens, which in turn are scored for the next eviction. Errors can therefore compound over long generations, a mismatch that training on fixed text does not expose.
The deployed results suggest these gaps are modest in practice. At 50% compression IndexMem++ reaches 93.79 on RULER 8K against 93.86 for Full-KV, and on Math500 it generates almost the same number of tokens as Full-KV (5,226 vs. 5,258), whereas the other methods generate substantially longer outputs at the same budget. A policy whose errors compounded during decoding would be expected to drift into longer reasoning chains.
On-policy distillation (OPD) targets both gaps directly. Instead of teacher-forced text, the student generates continuations with hard eviction active, and the dense teacher then scores those same student-generated prefixes. The gate would thus be trained on the states its own evictions produce, under the selection rule it actually uses, and the teacher's distribution would supply a dense correction at every position the student visits.
Thanks for the responses, most of my comments are addressed. I will maintain my score.
Q1. Generalization to MoE architectures.
A1.
IndexMem++ operates only on the attention KV cache. The scorer reads the hidden state that enters each attention layer and decides which keys and values to keep; it never touches the feed-forward layers. MoE models replace the dense feed-forward layers with experts but keep standard GQA attention, so the method applies to them without modification, and the KV cache it compresses has the same structure.
Q2. A fair throughput comparison with CUDA graphs for Full-KV as well.
A2.
We agree that the previous comparison mixed eager Full-KV with CUDA-graph IndexMem++. We added a CUDA-graph Full-KV baseline and reran both methods on the same H20 GPUs with identical inputs and settings. Both capture the complete decode step; prefill and graph capture are excluded. IndexMem++ uses 2,048 entries per KV head, including 64 centroid slots. Throughput is total generated tokens per second; the ratio is IndexMem++ over Full-KV.
Full-KV is faster at short contexts and small batches, the two are on par at 8K with batch 32, and IndexMem++ pulls ahead as context grows (up to 3.00× at 128K). We will revise the paper to report this operating range rather than a uniform speedup.
The bounded cache also runs configurations where dense KV does not fit on one GPU:
† Rerun with
expandable_segments:Truefor both methods, after IndexMem++ hit prefill OOMs under the default allocator. We assign no speedup where Full-KV cannot run.Q3. Additional benchmarks: GSM8K, AIME, and MMLU.
A3
AIME25 is already in the paper (Table 6): all 30 problems of AIME 2025 I and II.
We prioritized long-context and long-reasoning benchmarks because eviction only matters once the cache exceeds its budget. GSM8K and MMLU have short prompts and short answers, so without long reasoning or many-shot prompts the whole sequence often fits within the budget, nothing is evicted, and every retention policy behaves like Full-KV.