acceptodds
Desk rejected at ICLR 2027

The Gating Mechanism of KV Cache Eviction

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

KV cache eviction must decide which entries carry information that the rest of the cache can no longer supply. A natural way to estimate this is a learnable gate: give each cached entry a gate on its attention weight and train the gate on the model's own objective. The gate belongs inside the normalizing softmax, where suppressing one entry hands its mass to the others rather than discarding it. That same normalizer absorbs any per-query constant, so a constant gate cancels and reproduces the frozen model exactly. Trained on the language-modeling loss the gate stays near a constant score while its loss falls throughout: optimization succeeds, but a frozen backbone was fit for dense attention, so its loss prefers the no-op, and a near-constant score ranks nothing. The no-op is not a floor the gate improves from but what its objective asks for. A second obstacle survives once the gate moves, since the same gradient charges the gate for errors the frozen model already makes, which no reweighting of positions separates out. Much of the machinery in learnable eviction, from sparsity penalties to relaxations of the discrete decision, works around the first obstacle rather than removing it; both are removed by putting dense attention beyond the gate's reach and making the dense model's predictions its target. Under these two conditions capacity is never the missing ingredient: neither a larger scorer, nor explicit access to the past, nor a trainable backbone helps. The same holds for what eviction unavoidably discards, which is recovered by parameter-free centroids read inside the same softmax far better than by any memory read outside it, since the mass they must return is a share of that normalizer and available nowhere else. At 75% KV cache compression, this gate reaches 90.14 on RULER at 8K and 46.03 on LongBench with Qwen3-8B, 3.72 and 1.36 points below full-cache inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache eviction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8PRsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a KV cache eviction strategy with learned gating weights. This approach enables a query-inependent gate that can capture redundancy and maintans a recovery mechanism for the discarded entires. The key insight they identify is that these learned weights in softmax when trained on a fixed model is similar to dense attention model collapse to predictions of dense attn and the training approach proposed denies the gating mechanism dense attention and forces it to match dense attention predictions. The discarded entries are also retained with centroid recovery to get the residual contributions. This duality of selection process and maintaining a summarized evicted cache is an extension of similar idea proposed in indexmem paper. The results presented in the paper are promising and it can maintain performance at long context benchmarks with up to 75% evictions.

Critical strengths

The paper proposes gating for KV cache eviction and also highlights key challenge with collapsing score when used inside softmax calcutaion. The propose a clean solution to this by forcing it to estimate the dense attention. They propose using distillation rather than standard cross enropy loss which mixes the contributions from eviction loss from the frozen backbone. This is a clean separation to only train on changes due to eviction. The ablation studies clearly show the benefits of this formulation. The results for RULER benchmark and longbench accuracy tests maintain for upto 75% compression and shows advantage over other KV eviction approaches. The decode throughput also shows good performance with increasing context length.

Critical weaknesses

Generalization of the approach on broader class of MoE models is not presented. The accuracy result on Ruler and longbech are promising, but to be broadly application benchmarks such as GMS8K, AIME etc. are missing (noted in next section). The throughput results are not conclusive because baseline and indexmem++ use different implementation approaches. Discrepancy in training and inference gates are highlighted, but need further ablation to showcase the impact and also OPD is proposed as future work but it is unclear how this impact generalization of this approach

Questions for authors

The paper mainly focuses on dense models, how well does this scale for MoE model architectures. The measurement setup uses two different implementation eager with FAv2 and cudagraphed for indexmem++, this maked the comparison results on latency difficult to determine the improvements from KV eviction. Can we match results with similar cuagraph implementation for baseline as well. Can authors add other standard benchmarks like GSM8K, AIME, MMLU to understand impact of generalization how does the difference in gates trained from dense teacher v/s deployment / inference on evictions without dense teacher guidance impact RULER, GMS8k benchmarks? Can we add an ablation study where we evict based on the teacher to isolate accuracy impact.

3 comments

Q1. Generalization to MoE architectures.

A1.

IndexMem++ operates only on the attention KV cache. The scorer reads the hidden state that enters each attention layer and decides which keys and values to keep; it never touches the feed-forward layers. MoE models replace the dense feed-forward layers with experts but keep standard GQA attention, so the method applies to them without modification, and the KV cache it compresses has the same structure.


Q2. A fair throughput comparison with CUDA graphs for Full-KV as well.

A2.

We agree that the previous comparison mixed eager Full-KV with CUDA-graph IndexMem++. We added a CUDA-graph Full-KV baseline and reran both methods on the same H20 GPUs with identical inputs and settings. Both capture the complete decode step; prefill and graph capture are excluded. IndexMem++ uses 2,048 entries per KV head, including 64 centroid slots. Throughput is total generated tokens per second; the ratio is IndexMem++ over Full-KV.

ModelContextBatchFull-KV + graphIndexMem++ + graphRatio
4B8K1120.7354.650.45×
4B8K32741.37744.931.00×
4B16K16371.70439.381.18×
4B16K32411.29683.061.66×
4B128K458.11174.603.00×
4B256K124.8852.592.11×
8B8K195.2048.950.51×
8B8K32701.18691.420.99×
8B16K16349.95397.871.14×
8B16K32398.76630.311.58×
8B128K455.70158.092.84×
8B†256K123.4546.992.00×

Full-KV is faster at short contexts and small batches, the two are on par at 8K with batch 32, and IndexMem++ pulls ahead as context grows (up to 3.00× at 128K). We will revise the paper to report this operating range rather than a uniform speedup.

The bounded cache also runs configurations where dense KV does not fit on one GPU:

ModelContextBatchFull-KV + graphIndexMem++ + graph
4B256K4 / 16 / 32exceeds capacity174.81 / 348.08 / 533.44
8B†256K4 / 16 / 32exceeds capacity158.11 / 318.23 / 498.21

† Rerun with expandable_segments:True for both methods, after IndexMem++ hit prefill OOMs under the default allocator. We assign no speedup where Full-KV cannot run.


Q3. Additional benchmarks: GSM8K, AIME, and MMLU.

A3

AIME25 is already in the paper (Table 6): all 30 problems of AIME 2025 I and II.

We prioritized long-context and long-reasoning benchmarks because eviction only matters once the cache exceeds its budget. GSM8K and MMLU have short prompts and short answers, so without long reasoning or many-shot prompts the whole sequence often fits within the budget, nothing is evicted, and every retention policy behaves like Full-KV.

Q4. The gap between training with a dense teacher and deploying hard eviction; an ablation using teacher-guided eviction.

A4.

The dense teacher is used only during training, to provide the target output distribution; at inference the scorer runs on its own. The gap between the two is not in the scorer's input or its output, which are identical, but in the conditions it is trained under. There are two concrete differences.

  • Soft gating vs. hard selection. During training the gate only scales an entry's attention weight, so a low-scored entry still contributes a little and its gradient remains informative. At inference the same scores pick a top-B set, and every entry outside it is deleted outright. Training therefore optimizes a smooth relaxation of the decision that is actually deployed: an error near the selection boundary costs little during training but removes the entry entirely at inference.
  • Teacher-forced vs. self-generated states. During training every position conditions on the reference text with a dense cache, so the gate never sees a state produced by its own earlier evictions. At inference, each eviction changes the hidden states of later tokens, which in turn are scored for the next eviction. Errors can therefore compound over long generations, a mismatch that training on fixed text does not expose.

The deployed results suggest these gaps are modest in practice. At 50% compression IndexMem++ reaches 93.79 on RULER 8K against 93.86 for Full-KV, and on Math500 it generates almost the same number of tokens as Full-KV (5,226 vs. 5,258), whereas the other methods generate substantially longer outputs at the same budget. A policy whose errors compounded during decoding would be expected to drift into longer reasoning chains.

On-policy distillation (OPD) targets both gaps directly. Instead of teacher-forced text, the student generates continuations with hard eviction active, and the dense teacher then scores those same student-generated prefixes. The gate would thus be trained on the states its own evictions produce, under the selection rule it actually uses, and the teacher's distribution would supply a dense correction at every position the student visits.

Thanks for the responses, most of my comments are addressed. I will maintain my score.

Reviewer_N3NNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

For the KV cache eviction problem, the authors propose a query-independent KV eviction gate algorithm that focuses on redundancy rather than relevance. Authors implement a differentiable gating mechanism as an additive logit bias within the softmax operation. To prevent the no-op when naively training with LM loss, author additionally devise a way to avoid reproducing dense attention, attention sinks and a local window at one and force the remaining entries to share a small-valued fixed gate mass. They further distill the dense model’s output distribution using reverse KL, rather than next-token prediction. Moreover, for inference time, the authors introduce centroid KVs and an head-budget allocation strategy based on attention mass. The resulting method, IndexMem++, shows strong performance on RULER-8K and LongBench at certain compression ratios compared with existing baselines.

Critical strengths
  1. I believe The motivation for focusing on redundancy rather than relevance in KV cache eviction is sound, and the proposed gating mechanism and its key design choices are well justified. The approach to addressing the no-op issue and the use of distillation is reasonable and well-motivated, and the proposed techniques seem effective.

  2. The authors provide detailed ablation studies to evaluate the effectiveness of individual components, showing that several components play critical roles in the overall performance.

  3. In certain scenarios like compression ratio 50-75%, the proposed method shows strong performance compared to others. The paper also provides sufficient details about the proposed method and experimental setup.

Critical weaknesses
  1. I believe the presentation could be significantly improved. While some parts of the motivation and methodology are easy to follow, many of the remaining sections are difficult to understand due to the excessive textual explanations and lack of intuitive figures. For example,
  • I believe Section 3.2 include too much details and could be shortened or moved to the implementation section or appendix.
  • Some technically complex parts are difficult to follow through text alone, particularly Lines 326–340 in Section 3.3 and Lines 73–94 in the Introduction. More intuitive figures or step-by-step explanations would help clarify these parts.
  • The final method is named IndexMem++, but the name first appears in Line 89 without any explanation. The relationship between IndexMem and IndexMem++, including the key contributions introduced in IndexMem++, is also unclear. Furthermore, IndexMem is not used as the baseline.
  1. Limited Experimental Settings and Results
  • The key idea for preventing the no-op would be formulated in Equation (4). However, Table 2 only examines different choices of protected sets. To better support the central claim, the authors should evaluate the impact of beta and compare other variants.
  • I believe the compression ratio is too narrow (only 50%, 75%). Evaluations across a broader range of compression ratios, including more aggressive ones, would provide a more comprehensive assessment of the method.
  • The closest baselines are missing, and the baseline set changes across benchmarks: IndexMem / KVzip / Fast-KV zip/ KV-zap/ Attention-Gate/ StreamingLLM. Not all are consistently compared across the experiments. Fast-KVzip and KV-Zap include only in specific tables (e.g., Table 6).
  • Since the proposed method involves a training procedure, I believe its generalization capability would be the critical point. It would be needed to understand its robustness under distribution shifts or domain shifts, as well as whether its performance generalizes beyond the training data.
  • Performance in diverse context length (~ 128k) needs to be reported.
  • Details on training/inference cost might be needed to include
  1. Others
  • The paper emphasizes the query-independent nature of the proposed gating mechanism, but I'm little confused about whether indexMem++ itself is also query-agnostic. When using head-budget allocation strategy based on attention mass, could we say that IndexMem++ is query agnostic??
  • What clustering algorithm produces the centroids, at what cost, and are centroids updated during decoding?
  • Would it be possible for reporting main results on multiple seeds for more robust evaluation?
Questions for authors

Wrote above.

3 comments

W1. Section 3.2 contains too much detail.

A1.

We agree that Section 3.2 should make the design choices easier to follow. The paper starts from a framework shared by learnable eviction methods: a small scorer assigns a retention score to each KV entry, the score gates its attention during training, and the highest-scoring entries are retained at inference. Our aim is to identify which choices within this framework matter.

We will shorten Section 3.2 around three findings. First, with a frozen backbone, the training objective admits a constant gate that reproduces dense attention without learning to evict. Second, protecting the sinks and local window while giving the remaining entries a small fixed gate mass removes this no-op solution. Third, the dense model's output distribution provides a training signal for preserving the model's behavior after compression. The main text will state these findings and connect each to its ablation; the derivations, implementation details and secondary scorer variants will move to the appendix. This will make the role of each component clear before introducing the full method.


W2. The Introduction and Section 3.3 need intuitive figures or step-by-step explanations.

A2.

We will add a method overview and use it to organize the explanations at Lines 73–94 and 326–340. The figure will follow one context through four steps: score each entry from its own hidden state, divide the cache budget across heads, retain the selected entries and summarize the evicted ones as centroid KVs, then attend to both within one softmax. It will show that the context is compressed before a downstream question is supplied, so the resulting cache can serve different questions.

A second panel will explain the training gate. It will contrast a constant gate, which leaves attention unchanged, with Equation (4), where the sinks and local window have gate one and the other entries share a fixed mass β\beta. The accompanying text will explain one operation at a time and point to the corresponding part of the figure. We will also show the distinction between soft gates during training and hard selection at inference, rather than introducing both through equations alone.


W3. The relationship between IndexMem and IndexMem++ is unclear, and IndexMem is missing as a baseline.

A3.

IndexMem and IndexMem++ share the same core framework: evict KV entries and reorganize information from the evicted entries into a form the model can reuse. Both have an eviction gate and a recovery branch. IndexMem++ rethinks how these two parts should be designed.

The paper derives these choices from the requirements of eviction, which is why we discussed a broader family of learnable eviction methods before explaining the relationship to IndexMem. We agree that this left the name and the specific differences unclear. We will explain the relationship when IndexMem++ first appears in the Introduction.

  • A cache shared by future queries needs a query-independent eviction score. IndexMem computes query-to-key scores and rescores every 128 decoded tokens. IndexMem++ instead scores each entry from its own hidden state when it is created. The gate learns which KV entries are redundant under the training objective; it does not require the future question to select a reusable cache. The gate acts inside the softmax, allowing attention removed from one entry to be redistributed to those that remain.
  • Recovery should participate in the same attention normalization. IndexMem reads a fast-weight linear memory and adds its output outside the attention softmax, with a separately learned scale. IndexMem++ represents evicted entries with parameter-free centroid keys and values. These centroids compete with retained entries inside the same softmax, with cluster size included in their logits.

We have added direct comparisons with IndexMem. On RULER 16K with Qwen3-8B, the results are:

MethodFull-KV50% KV removed75% KV removed
IndexMem92.9680.0072.20
IndexMem++92.9690.8780.93

At 75% compression on LongBench, using the same inputs, evaluation protocol and scoring script, IndexMem scores 41.05 and IndexMem++ scores 46.03. The largest differences are on LCC (41.5 versus 63.8) and PassageRetrieval (52.3 versus 94.4). We will include these comparisons in the revision alongside the explicit description of the two methods.

W4. The impact of β\beta in Equation (4), and other variants.

A4.

We added a sweep of β\beta and an ablation of the two parts of Equation (4). The setup is that of Table 1 (Qwen3-0.6B, RULER 8K, cache budget 2048 of 8192 entries, 100 training steps, reverse KL). Each number in these two tables is the mean over two training seeds. The dense backbone without eviction scores 70.01. We also added a separate comparison with four alternative training gates below.

Gate mass β\beta. Equation (4) makes the evictable entries share a total gate mass β\beta. The paper uses β=256\beta = 256.

Qwen3-0.6B, RULER 8Kβ=16\beta = 1664256 (paper)10244096∣Ei∣\lvert E_i \rvert
Avg.65.765.767.159.117.78.5

From β=16\beta = 16 to 256256 the score changes by less than 1.5 points. Larger values hurt, and β=∣Ei∣\beta = \lvert E_i \rvert gives 8.5. This matches the requirement β≪∣Ei∣\beta \ll \lvert E_i \rvert in Appendix E.1. At 8K, ∣Ei∣\lvert E_i \rvert is about 8064, so β/∣Ei∣\beta / \lvert E_i \rvert is 3% at β=256\beta = 256, 13% at β=1024\beta = 1024 and 51% at β=4096\beta = 4096. As this ratio grows, the pressure away from the no-op solution weakens, and it vanishes at β=∣Ei∣\beta = \lvert E_i \rvert.

The two parts of Equation (4). The always-attended entries are the attention sinks and the causal local window (the protected set PiP_i), and they keep a gate of one. The remaining evictable entries share the fixed gate mass β\beta. The table below removes each part during training.

Always-attendedGate mass β\betaRULER 8K Avg.
sinks + windowyes67.1
sinks + windowno10.8
noneyes15.6
noneno14.3

Removing either part leaves the score below 16, as Appendix E.1 argues. With always-attended entries but no gate mass, the remaining gates can all equal one, which is dense attention (second row). Without always-attended entries, the normalizer adds the same constant to every attention logit and cancels inside the softmax, so the last two rows are mathematically the same model. These ablations change training only. Evaluation always keeps the sinks and the local window. The 4.72 for the unprotected gate in Table 2 used the LM-loss objective; the 15.6 and 14.3 here use reverse KL and should be labeled as separate experiments.

Other training gates. We compared the normalized gate with sigmoid + L1, a soft target-mass penalty, straight-through (STE) hard top-BB, and stochastic hard-concrete with an expected-L0L_0 penalty. This is a separate controlled Qwen3-0.6B pilot: a frozen backbone, 100 reverse-KL optimizer steps, and a global batch of eight 8K sequences, totaling 6,553,600 training tokens per run. Initialization and training-document order are matched across methods within each of two training seeds (1000 and 1001).

Training gateSeed 1000Seed 1001Mean ± sample SDNo-op rows
Normalized gate, Equation (4)68.1766.4767.32 ± 1.211.39%
Budget-offset sigmoid + L160.0962.7161.40 ± 1.861.10%
Budget-offset soft target-mass penalty59.4966.4562.97 ± 4.922.36%
STE causal hard top-BB37.0943.6940.39 ± 4.673.17%
Budget-offset hard-concrete56.4260.9258.67 ± 3.181.62%

For each tunable alternative, we tested the same coefficient grid (0.03, 0.1, 0.3) for ten steps per candidate and selected by budget adherence and reverse KL on separate corpus documents, without using RULER. The budget offsets are applied on every forward pass. The target-mass method uses a soft quadratic penalty, not an exact constraint: the final historical soft gate mass divided by B=256B=256 is 0.442 for sigmoid + L1 and 1.053 for the target penalty, versus exactly 1 for Equation (4). Hard-concrete has expected active count/B=0.831B=0.831 and soft mass/B=0.460B=0.460; these are different quantities. STE uses exact causal top-BB selection with a sigmoid backward surrogate. The alternatives keep histories with at most BB evictable entries dense; the original gate retains its short-history amplification.

The normalized gate has the highest mean RULER score in this pilot, exceeding the alternatives by 4.35–26.93 points. No-op rows are measured on four common heldout documents, using attention total-variation distance from dense attention below 10−310^{-3}; hard-concrete uses deterministic zero-noise gates for this diagnostic. The low no-op rates for all methods show that avoiding near-dense attention alone does not guarantee good hard-selection accuracy. These results support Equation (4) under this recipe and these implementations; two training seeds do not establish a universal ranking. The fresh 67.32 baseline belongs to this comparison and does not replace 67.1 in the separate ablations above.

W5. Generalization under distribution or domain shift, beyond the training data.

A5.

The gate is trained only on long documents from LongMino, a long-context pretraining corpus. The main Qwen3-8B checkpoint records the subsets 2e16 and 2e17; we will correct Appendix G to match this configuration. We tested it on two kinds of input that differ from such documents, SWE-agent trajectories (code, tool outputs and error logs) and AIME25 reasoning traces, with eight of each. For each one, the first 8K tokens are prefilled and compressed to a cache budget of 2048. We then measure the NLL of the next 256 assistant tokens and the KL divergence from the Full-KV output distribution. The baseline is recency, which fills the same cache budget with the most recent entries and uses the same 64 centroid slots.

Qwen3-8B, budget 2048SWE-agent NLLSWE-agent KLAIME25 NLLAIME25 KL
IndexMem++1.2210.1200.2070.021
Recency1.5510.6300.2790.082

IndexMem++ has the lower NLL on every one of the 16 trajectories. Outside the always-attended entries, the entries kept by the gate overlap with a recency window of the same size by only 48% (SWE-agent) and 53% (AIME25), so the gate has not learned a sliding window.


W6. Is IndexMem++ query-agnostic when head budgets use attention mass?

A6.

Yes. IndexMem++ compresses a context into a cache that can be reused for different downstream questions without repeating compression. The gate learns the redundancy of the current KV entries from the training objective, and each entry's score is computed from its own hidden state. It does not score relevance to a future question or take that question as an input to compression.

The attention mass used for head-budget allocation is computed from the final context position during prefill. Here, “query” refers to an attention query vector belonging to the context itself, not a future user question. The allocator uses this context-derived statistic to divide a fixed budget across heads; the resulting allocation and compressed context cache are then reused. Centroid construction also depends only on the evicted context entries. Thus, both retention and recovery can be prepared before any downstream question is known. We will make this distinction explicit in the paper.


W7. Which clustering algorithm produces the centroids, at what cost, and whether they are updated during decoding.

A7.

Algorithm. After prefill, each KV head in each layer runs k-means (15 iterations) on the keys of its evicted entries, giving 64 clusters. Each cluster is averaged into a centroid key and value and stored with its size. The centroids enter the same softmax as the retained entries, with logit q⋅kˉ/d+log⁡nq \cdot \bar{k} / \sqrt{d} + \log n, where kˉ\bar{k} is the centroid key and nn is the cluster size.

Cost and number of centroid slots. The table below is for RULER 8K with Qwen3-8B at 75% compression, with the budget of A5. The time is the total over the 36 layers for one 8K input on one H20 GPU. Clustering runs once, after prefill.

Centroid slots0163264128
RULER 8K Avg.91.3491.4091.4091.5991.60
Clustering time (s)00.310.330.370.45

Using 64 slots scores 0.25 points higher than using none. Going from 64 to 128 slots gives no further gain.

Updates during decoding. The RULER and LongBench accuracy evaluations build centroids after prefill and hold them fixed while processing the question and answer. The math evaluations do not use centroids. The physical-eviction implementation used for the decode-throughput experiment supports incremental updates: a newly evicted entry fills an empty slot if available, or joins its nearest centroid with an update to the key/value means and count, without rerunning k-means. We will distinguish these execution paths explicitly; the frozen-centroid accuracy tables do not evaluate the incremental update rule.


W8. Results on multiple seeds.

A8.

For Qwen3-0.6B, in the setup of the last row of Table 1 (67.60 in the paper), we trained the gate with three training seeds. The RULER 8K scores are 67.96, 66.64 and 67.47, with a mean of 67.36 and a standard deviation of 0.67.

The separate comparison in A4 adds two matched training seeds for each of five gates and reports every seed score and the sample standard deviation. Its normalized-gate baseline scores 68.17 and 66.47 (67.32 ± 1.21). We keep these separate from the three-seed experiment above. These 0.6B results do not constitute a multiple-training-seed evaluation of the main Qwen3-8B model. We will also clarify that the 0.28 in the Table 2 caption is the difference between evaluation seeds 42 and 43 for the same checkpoint (67.60 and 67.88), not variation across training seeds.

Reviewer_CeRyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a streamlined technique for quickly learning a KV cache eviction mechanism end-to-end from a frozen LLM. First, the fixed-budget eviction decision is relaxed to a set of learnable gates of fixed total mass, which decay slowly and asymptotically over time. Evicted entries are then compressed into a fixed quantity of centroids that act as additional kv entries, re-inserting the lost information at lower granularity. At inference time, eviction thresholds are set such that the target prefill eviction rate is achieved. Performance on information retrieval benchmarks recovers most of the original at moderate compression levels, outperforming prior techniques.

Critical strengths

The idea is sensible, sound, straightforward and elegant. Exhaustive ablations justify the overall design. Results appear strong, and the expected speedups from smaller KV cache sizes are demonstrated in practice.

Critical weaknesses

The main issue is that this work reads as a streamlining of the cited IndexMem cache eviction method (it’s even named IndexMem++), but no actual comparison is ever made to this baseline. Rather than discuss the design deltas in the methods section, the entire IndexMem++ is derived from first principles and presented as entirely original. Likewise, the results do not include the original IndexMem as a baseline, despite the fact that the direct comparison I was able to make based on published results (Qwen3-8B on RULER 16K) was heavily favorable toward IndexMem++. Without some discussion of what the differences are between IndexMem and IndexMem++, and why they matter, I have to consider this an incremental improvement to an existing method rather than a novel contribution. Based on the results I’m ready to be convinced that this isn’t really the case, but the comparison needs to be addressed and discussed!

The following three points all fall under issues of presentation, and are less important (though still issues) compared to above:

  • The language comes off to me as unnatural and hard to follow. Things like “The only difference from the outer gate is the denominator, and it is decisive:” in L154, “The reason is structural and it comes from the same normalizer that gives the gate its meaning” in L67, “This section follows that consequence through, to what has to be taken away from the gate, what has to be given to it, and the one question a retention score is unable to answer” in L176-178, and the heavy overuse of colons (I count 2 in the abstract, and 11 in the intro text, including two in back to back sentences in L50-52) are all overly verbose and abstract, and inhibit the reader. Please simplify the verbiage to make it more generally direct.

  • The exhaustive ablations in section 3 should be presented as separate ablations, rather than as the method itself. As-is, the paper is presenting what authors didn’t do rather than what they did.

  • The conclusion and abstract summaries – and to a lesser extent, the methods section – all read as a recitation of the thought process behind the model design, rather than as a summary of the actual contributions being presented in the paper. Begin with what the model accomplishes, and where its novelty lies, before trying to justify the reasoning behind it in general terms.

Combined with the missing discussion comparing and contrasting to the IndexMem baseline, I don’t think this paper is ready for publication. The idea (on its own terms) is sound and the results appear strong, but a serious overhaul is needed to presentation, context, and structure.

Questions for authors
  • Is per-head varying KV capacity feasible from the standpoint of efficient implementation? To my knowledge, efficient attention kernels that handle varying cache sizes per-head don’t yet exist. If authors created these, that’s a major contribution in its own right.
  • How are evicted-entry centroids updated at inference time? Are they iteratively updated?
  • Appendix states that IndexMem++ is trained over 600 optimization steps. How does that compare to your various baselines? If any involve longer training periods, that’s worth mentioning in the experiments section, and if none of them do, that’s worth mentioning as well.
3 comments
AuthorsOpenReview ↗

Response to weakness

W1. Design differences of IndexMem and IndexMem++.

A1.

IndexMem and IndexMem++ share the same core framework: evict KV entries without permanently discarding what they carry, and reorganize the evicted content into a form the model can reuse.

IndexMem++ keeps this framework, with a gate that performs eviction and a recovery branch that restores information after eviction. Relative to IndexMem, the core difference is that we rethink the design of learnable eviction and the design of recovery.

We did not frame IndexMem++ as a modification of IndexMem because the paper derives each design choice from what eviction requires, rather than from an existing method. Two requirements decide the design, and IndexMem answers both differently:

  • Eviction is irreversible, so the score cannot depend on the query. A deleted entry can never be recovered, so the decision to delete it must hold for every future query. IndexMem's indexer is query-dependent: it computes a dense query-to-key score matrix and rescores every 128 decoded tokens, so an entry it deletes may be one a later query needs. IndexMem++ scores each entry once, from its own hidden state, when the entry is created. For the same reason the gate must sit inside the softmax, so that removing an entry hands its attention to the entries that remain.
  • Recovery has to restore a share of the same softmax. What eviction loses is a fraction of the attention normalizer, and that fraction is only available inside the softmax. IndexMem reads a fast-weight linear memory and adds the result to the attention output outside the softmax, so its scale must be learned separately. IndexMem++ inserts parameter-free centroids of the evicted keys and values into the cache, where the same softmax weights them against the retained entries. We find that keeping recovery inside the same softmax is crucial.

Derived this way, IndexMem is one instance of a broader family, together with TrimKV, the Forgetting Transformer and the other learnable eviction methods, which share the same paradigm but fix these choices differently. We therefore discussed the family as a whole instead of singling IndexMem out. We agree this left the relationship to IndexMem unclear, and we will state it explicitly and add IndexMem as a baseline in the revision.

Direct comparison.

Qwen3-8B, RULER 16KFull-KV50%75%
IndexMem92.9680.072.2
IndexMem++92.9690.8780.93

W2. The writing (e.g., L67, L154, L176–178).

A2.

We agree and will rewrite the paper in plainer, more direct language, and remove most colons from the abstract and introduction. For the examples cited:

  • L154: "The two gates differ only in the denominator. Inside the softmax, a suppressed entry's attention goes to the other entries."
  • L67: "The softmax cancels any constant added to all scores, so a constant gate has no effect."
  • L176–178: "This section explains how we fix this."

W3. The ablations in Section 3

A3.

Our paper does not aim to propose yet another learnable eviction architecture. It starts from what all of them share. TrimKV, DMC/DMS, SP-KV, Locret, Attention-Gate and the Forgetting Transformer all instantiate one framework: a small scorer maps each token's hidden state to a retention score, the score gates that entry's attention during training, and at inference the top-scoring entries are kept. They differ only in details, and each paper fixes those details differently without isolating which ones matter.

Working from this framework, we ablated those details and found that many of them are unnecessary:

  • More expressive scorers do not help. A larger MLP, or a scorer that reads past hidden states through a convolution, a recurrent state or prefix attention, all stay within 1.2 points of a two-layer MLP on the current hidden state.
  • Training the backbone does not help. Unfreezing it, as in SP-KV, leaves the model intact but decalibrates the gate.
  • Sparsity penalties and relaxations of the discrete decision work around the real problem instead of removing it. The real problem is that, with a frozen backbone, the language-modeling loss prefers a constant gate, which changes nothing.
  • Gating outside the softmax, as in DMA, removes attention mass instead of redistributing it.

What we adopt is therefore minimal: a gate inside the softmax; a protected set of entries with a fixed gate mass, which rules out the constant gate and acts as the budget without any regularizer; distillation from the dense model; a two-layer MLP scorer on a frozen backbone; head budgets allocated by retained attention mass; and centroids of evicted entries read inside the same softmax.

AuthorsOpenReview ↗

Response to question

W4. Efficient attention kernels that handle varying cache sizes per-head don’t yet exist.

A4. Yes, and it does not require a new attention kernel. We reuse FlashAttention's paged KV-cache kernel (flash_attn_with_kvcache) by changing what counts as a sequence.

  • Each KV head is its own sequence. We flatten every (layer, batch, KV head) into one row of the kernel's batch axis. A row has a single KV head, and the query heads that share it under GQA are passed as that row's query heads. The kernel already supports a different length per sequence (cache_seqlens), so a different length per KV head comes for free.
  • Paging makes the ragged budgets cheap. Each row maps to its own list of 256-entry pages through a block table, so a head stores only the pages its budget needs. Padding every head to the largest budget in its layer would cost 2.12× the logical slots; paging costs 6.1%.
  • Decoding reads the budget, not the context. Every retained entry is already causally past and selected, so a decode step is one dense paged-attention call over each head's budget, with no mask and no gather. The new token's own key and value are attended separately and merged by log-sum-exp, as are the centroid slots.

W5. How are evicted-entry centroids updated at inference time?

A5. The centroids are built once and then updated incrementally, without re-clustering.

  • Initialization. After prefill, the entries evicted from each (layer, KV head) are clustered by k-means (15 iterations) into 64 slots. Each slot stores the mean key, the mean value and the number of entries nrn_r it represents.
  • Update during decoding. Evicted entries are physically deleted, so k-means cannot be rerun over them. Instead, each newly evicted entry is assigned to the nearest centroid by key distance, and that centroid is updated as a running mean, cr←cr+(x−cr)/nrc_r \leftarrow c_r + (x - c_r)/n_r after incrementing nrn_r; an empty slot is taken over by the new entry. This keeps every centroid equal to the exact mean of its members, at O(R d)O(R\,d) cost per evicted entry and with no extra pass over the cache.
  • Read. Each centroid enters the same softmax as the retained entries with logit q⊤kc/d+log⁡nrq^\top k_c/\sqrt{d} + \log n_r, so a slot that summarizes more entries receives proportionally more attention mass.

W6. Baselines optimization step

A6. No baseline was trained for less than IndexMem++ under a comparable recipe; the released baselines were trained considerably longer.

MethodTrainingSource
IndexMem++600 steps (300 at 8K, 300 at 16K), global batch 8, gate onlyours
DMA600 steps (300 at 8K, 300 at 16K), global batch 8, gate onlytrained by us
TrimKVone full epoch over OpenR1-Math-220k, up to 16K contextreleased checkpoint
KVpopreleased variantreleased checkpoint
KVzap, Fast-KVzipreleased scorersreleased checkpoints
Reviewer_CeRyOpenReview ↗

Response to authors

I thank the authors for their response, new results, and clarifications. I consider my concerns partially addressed. On the IndexMem baseline: a comparison is provided for RULER at 16k, delivering the expected positive results. That said, to "add IndexMem as a baseline in the revision," presumably in tables 4 and 5, authors would need comparisons in 3 additional RULER settings, plus an entirely separate benchmark (LongBench). I of course do not expect authors to obtain all of these new results during the rebuttal window, but can authors confirm that an IndexMem update to tables 4 and 5 is intended for the revision?

Similarly, regarding the presentation and framing, the provided recontextualization is sensible and compelling. But the fact remains that I gained a clearer understanding of the paper's motivations and contributions from author responses A1 and A3, than I did from the actual submitted abstract and introduction. This indicates the need for a pretty substantial rewrite, which is hard to judge sight unseen. Can authors provide an updated abstract or outline (apologies for the late ask), and/or elaborate on the scope of intended revision? Do authors intend to reframe the motivation and contributions under this new framework? (It occurs to me that moving Related Work to somewhere before Section 3 might help with this)

Score currently unchanged, pending further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.