acceptodds
Under review as a conference paper at ICLR 2027

GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

Abstract

Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key–value (KV) cache grows with the context and dominates serving memory. In this paper, we ask whether the recurrent memory already knows what the attention memory will need. We first show that which GDN heads of Qwen3.8-27B keep long memories is fixed by the weights rather than by the document. Next, we define the future utility of a KV entry by how much removing it changes the attention output over the model's own continuations, and find that the write gate, decay and write magnitude that GDN layers emit during prefill anticipate this utility before any query is seen. Because the relation is spread across many GDN heads and layers, and a linear map recovers its trend but less of its high-utility tail than a nonlinear one, we propose GateSight, which learns it with lightweight per-head multilayer perceptrons that read these signals and no attention score. Their predicted utility decides which entries each attention layer keeps, and how much each layer keeps is set by the model's native attention output gate, whose mean differs by more than an order of magnitude across layers. Finally, at serving time, GateSight scores the cache once after prefill and physically releases discarded KV entries. Experimental results on SCBench and RULER-4096 with Qwen3.8-27B and, with the same recipe, the sparse-attention model Qwen3.8-Flash-Next, up to the 262K-token limit of the models, show that with a tenth of the attention KV cache, GateSight is the most accurate of five query-agnostic methods on three of the four model–benchmark pairs and leads KVzip on all four, by 9.5 to 47.2 points. Notably, it scores a full-length context with negligible overhead relative to prefill.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache eviction, Hybrid attention models, Linear attention, Long-context inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iiKSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed query-agnostic KV eviction for the full attention layer of a hybrid Transformer. The authors found three observations about the recurrent memory in these hybrid Transformers as follows. 1) The forgetting speed of GDN differs across heads, and this behavior is consistent across contexts. 2) The write and forget signals of GDN can predict the future importance of KVs. 3) The relationship between KV importance and the GDN signal is distributed and nonlinear. Based on these three observations, the author proposed introducing an MLP model that predicts future utility from entry information. Then, the proposed method allocates the heterogeneous budgets across layers, but uniformly across heads within the given layer to ensure compatibility with vLLM PagedAttention. Experimental results claim that their method has superior performance when the compaction ratio is 0.1.

Critical strengths
  • The proposed method is compatible with PagedAttention, and the implementation is conducted with vLLM.
  • KV eviction on a hybrid LLM remains underexplored, and this paper investigates it with an observation that can be captured only in the GDN.
Critical weaknesses
  • One major concern is the selection of baselines. The author claims that 1) GateSight outperforms the other baseline methods (SnapKV, TriAttention, and KeyDiff), as shown in Figure 4. 2) Although KVzip exhibits better accuracy than GateSight until p=0.4 (Table 3), KVzip takes 1.5-2.2 times the prefill time, so directly comparing KVzip with GateSight is unfair (Section 5.3). 3) KVzip allocates KVs using global score across layers and heads, which is not compatible with the authors' vLLM protocol (Section 5.1 Baselines). However, the idea of introducing MLPs for each head to predict the importance score of each KV entry has already been explored in KVzap (https://arxiv.org/abs/2601.07891). Specifically, the source of the low wall-clock scoring time of both KVzap and GateSight is the same, so many of the claims of this paper are questionable. Compared to KVzap, GateSight performs worse, despite both being "no attention score" methods and having similar wall-clock times. Furthermore, KVzip and KVzap can also allocate the budget uniformly across heads, as provided in the ablation studies in their papers. Hence, the authors should revisit existing KV eviction methods, including current SOTA methods (https://huggingface.co/spaces/nvidia/kvpress-leaderboard), and provide comparisons against strong baselines.
  • The method should also be compared with the alternatives, such as smaller models. On RULER 4K, the proposed GATESIGHE exhibits Qwen3.8-Flash-Next (93.4%, p=0.7) and Qwen3.8-27B (91.9 %, p=0.4), respectively, which are lower than Qwen3-4B (95.1%, p=1). This comparison is not just about KV cache size itself, but about the practical value of the achieved accuracy after compression.
  • In Abs., the main claim about the experimental results is as follows: "GATESIGHT is the most accurate of five query-agnostic methods on three of the four model–benchmark pairs and leads KVzip on all four, by 9.5 to 47.2 points". However, the numbers 9.5 and 47.2 are meaningless because these numbers are from p=0.1, and the performance of GATESIGHT in this stage is even worse than or competitive with Qwen3-0.6B.
  • If the authors claim that comparing head-wise and layer-wise budget allocation is unfair, they should provide a clear explanation, including experimental evidence that shows the trade-off between accuracy and inference speed. The original PagedAttention block was organized with the shape (Layer, Head, block size, head dim), whereas the layer dimension was modified to support group-wise paging for Hybrid Transformer. This suggests that implementing vLLM PagedAttention compatible with head-wise budget is not technically impossible. Such complex memory management may reduce GPU utilization and increase inference latency. However, the authors should quantify the resulting latency overhead through experiments rather than just stating that "Each uses the budget allocation its paper defines (KVzip’s global score-sorted allocation, flat per-head budgets for the others); the gate-based per-layer allocation belongs to GATESIGHT only. To ensure a fair comparison, all methods run in the same vLLM pipeline under the prefix-only protocol of Appendix A, with every setting listed in Appendix G."
Questions for authors
  • What is the advantage of using Qwen3.8-27B (91.9 %, p=0.4) instead of Qwen3-4B (95.1%, p=1)?
  • Recent models, such as DeepSeek V3.2 and HySparse2, already have a top-k indexer in their model. What is the advantage of post-hoc KV cache eviction methods compared to those inherent top-k indexer modules?
  • The original implementation of KVzip in kvpress uses a non-uniform budget for both layers and heads, but this paper seems to use a uniform budget across heads. Could you explain the vLLM implementation of KVzip, including PagedAttention?
  • Could you provide the TPOT before and after compaction?
  • Do you think that Ov@10 = 41% is useful?
  • How accurate is the future KV utility metric? Have the authors compared it with shapley value to validate whether the utility scores reliably reflect the token contributions?
5 comments

Thank you for the review. We number the four weaknesses W1–W4 and the six questions Q1–Q6 in the order they appear in the review.

W1: GateSight's contribution is the predictive relationship between recurrent signals and attention-KV utility. KVzap (arXiv:2601.07891) addresses a different question. On dense Transformers (Qwen3-8B, Llama-3.1-8B-Instruct, Qwen3-32B), its per-layer surrogates read each attention layer's input hidden states, imitate KVzip+ scores, and prune by threshold. GateSight reads the write and forget signals of GDN layers and learns rollout utility from self-generated continuations. Both use a small learned scorer; that is not our claimed novelty.

Beyond the paper's age and leave-one-corpus-out controls, MLPs trained on the lightweight attention-side signals of TriAttention and KeyDiff, with the same data, utility labels, and training procedure, reach Ov@10 of 28.83% and 18.06%, versus 38.23% for GDN signals. Scores reported on different models do not rank KVzap and GateSight on a common checkpoint.

GateSight is specific to hybrid models: it reads GDN signals absent from dense Transformers. The kvpress leaderboard evaluates dense models, not hybrid ones. Among its methods, the paper compares with KVzip (Table 3 and scoring cost), KeyDiff, and SnapKV. RestoreKV (arXiv:2608.01247, posted 2 August 2026; training code released October 2026) does not score entries: it keeps a base method's eviction, e.g., KVzip's, and fills part of the same KV budget with a restore cache built by an extra LoRA-adapted pass over the full cache.

KVzip's quality advantages remain valid: at p=.4, Table 3 reports GateSight/KVzip SCBench scores of 51.7/60.4 on 27B and 45.3/60.9 on Flash. We report reconstruction cost separately. Part 2 clarifies our allocation port and adds head-wise KVzip.

W2/W3/Q1: Compression retains a selected checkpoint while reducing its KV footprint. On RULER-4096, GateSight keeps 27B at 97.4% at p=.7 and Flash at 97.0% at p=.9 (uncompressed: 97.5%/96.8%), above the cited 95.1% for Qwen3-4B; the figures quoted in W2 are at lower budgets. Qwen3-4B's context is 32,768 tokens natively (131,072 with YaRN) and Qwen3-0.6B's is 32,768 (model cards), so neither covers the SCBench contexts evaluated here, which extend to the 262K-token limit.

The abstract already limits the 9.5–47.2-point gains to “a tenth of the attention KV cache”; we will clarify that these use the layer-wise KVzip port. With global head-wise KVzip, GateSight still leads the 13-task RULER-4K mean at p=.1, by 32.5 points (76.62 vs. 44.08; Part 2).

At higher-quality operating points, 27B/p=.7 and Flash/p=.9 retain 97.49%/98.59% of the all-12-task SCBench mean. In a separate 260,096-token decoding workload, maximum batches rise 4→6 and 9→10. Peak throughput ratios are 1.708×/1.856× for 27B eager/graphs and 1.068× for Flash.

For seven Appendix E tasks—qa_eng, qa_chn, choice_eng, many_shot, summary, mf, and summary_with_needles—the p=.1 mean retains 98.02%/98.36% of the uncompressed mean for 27B/Flash. The fixed workload yields 10.25×/12.16× throughput for 27B eager/graphs and 5.24× for Flash; Part 2 reports batch sizes and timings.

Q2: Native indexing and physical eviction can be combined. An indexer selects entries to read; an unselected entry's page is not automatically released. Eviction reduces the entries and pages retained.

Flash-Next already uses native QSA. GateSight rebuilds four-token index blocks after compaction and continues using native QSA selection and attention. Maximum batch at 260,096 tokens rises from 9 to 10/86 at p=.9/.1.

We will clarify the contribution and KVzap relationship, baseline allocation, quality-constrained operating points, and QSA adaptation.

Q6: We added permutation-sampled Shapley values. The diagnostic uses 12 training-corpus documents, 512 context tokens each, 16 layers × 4 KV heads, and 2,048 permutations per document/layer/head. The game is the negative ungated layer-output distance from full context, with protected entries retained; φ_mean is its Shapley value averaged over continuation queries.

U_max versus φ_mean gives Spearman .9006, Ov@10 69.18%, and Cov@10 88.23%. Half-sample rankings have median Spearman .9995. The deployed MLP's Spearman with φ_mean is .4866. Retaining top-p entries gives these mean relative layer-output errors after joint removal:

pU_max selectionShapley selection
.1.21472.20360
.3.08117.07997
.5.03568.03879

U_max selection stays within .012 of Shapley selection at every p and is lower at p=.5. We will add the metric definitions and the Shapley/joint-removal validation.

Continued from Part 1 (W4, Q3–Q5).

W4/Q3: Head-wise budgets are implementable in vLLM. Our original KVzip port globally selects each layer's total by reconstruction score, then splits it equally across four KV heads; each head selects tokens separately and compacts them into that layer's block-table prefix. We will distinguish this from global selection across layers and heads.

We added global head-wise KVzip with the same scores and total prunable-KV budget; the protected prefix is additional. Below are equal-weight means (%) over all 13 RULER-4K tasks, 100 examples each, as in the paper. We will add this comparison to the revised appendix.

Retained fractionOriginal GateSightLayer-wise KVzipGlobal head-wise KVzip
.176.6229.4244.08
.282.4654.1771.10
.388.1283.8486.62
.491.8695.5096.93
.594.9797.3997.24
.797.4497.6697.43

Head-wise allocation improves KVzip by up to 16.93 points over layer-wise KVzip (p=.2). GateSight leads at p=.1 by 32.54 points and at p=.2 by 11.36 points; it also leads at p=.3 by 1.50 points. Head-wise KVzip leads at p=.4–.5 by 2.27–5.07 points, and the two are essentially tied at p=.7. On the other ten tasks, GateSight is at least as accurate at every budget.

The variants use the same total prunable-KV budget, but global head-wise KVzip reallocates it across layers and heads, so its gain reflects both cross-layer and within-layer allocation. The resulting layer budgets are non-uniform: at p=.1 the four highest-share layers hold 45.8% of the mean prunable budget, versus 25% under uniform layers. Layer budgets also matter for GateSight: fixing its scores and total budget on cwe, niah_multiquery, and niah_multivalue at p=.1 gives 98.58/95.45/52.05 with gate/flat/PyramidKV rules; here PyramidKV only changes allocation.

Costs use actual per-head block pools, with separate lengths and block indices per request/layer/head. On 27B/H200 with common eager/single-split execution, a 4,096-token prefix, batches 1/2/4/8/16/32/64, and p=.1/.2/.3/.4/.5/.7, all 42 matched configurations show 9.37%–11.68% higher TPOT and 8.57%–10.46% lower throughput than layer-wise KVzip.

At 260,096 prefix tokens and batch 1:

pLayer-wise ms/tokenHead-wise ms/tokenTPOT increaseThroughput decrease
.169.710577.072710.56%9.55%
.371.677887.965122.72%18.52%
.577.7576100.725629.54%22.80%

Each point uses 128 decode tokens and the median of three repetitions, excluding prefill, scoring, top-B selection, and cloning. The 4K workload adds an 18-token query. Thus, head-wise allocation improves KVzip by up to 16.93 points at p=.2, while GateSight remains ahead by 11.36 points; the head-wise implementation has 9.37%–11.68% higher TPOT at 4K and 10.56%–29.54% at 260K.

Q4: TPOT at equal batch and at each budget's throughput peak. The fixed decoding workload uses 260,096 input tokens, 128 decode tokens, and medians of three repetitions: one H200 for 27B, two H200s/TP2 for Flash. p=1 uses the same implementation and execution mode.

Model / modepMaximum batchBatch at peakTPOT (ms/token)System tokens/s
27B / eager144117.02334.181
27B / eager.766102.76558.386
27B / eager.14141117.000350.427
27B / graphs144116.20234.423
27B / graphs.76693.90863.892
27B / graphs.1413276.459418.522
Flash / eager199121.53374.054
Flash / eager.91010126.44179.089
Flash / eager.18686221.557388.161

Each row uses its throughput-maximizing batch, so the throughput gain includes the capacity released by compaction. At equal batch 1, p=.1 lowers 27B TPOT from 112.33 to 67.16 ms/token (eager) and from 111.57 to 35.08 (graphs). Flash TPOT at p=.9 and .1 stays within 2.2% of p=1 at batches 1 and 4 (p=.1: 109.72→110.16 and 116.09→118.63 ms/token); its gain comes from the larger maximum batch (9→86).

We will report both quality and cost and clarify the trade-off behind adaptive layer budgets with equal within-layer head counts.

Q5: Ov@10=41% exceeds random and age controls. Ov@10 measures overlap with the oracle top-10% set. Random selection, the age-feature MLP, and GDN MLP score 10.0%/17.9%/41.0%; Cov@10 is 19.0%/29.6%/58.4%. Coverage divides selected utility by oracle top-10% utility. Downstream, GateSight keeps 98.02% of the uncompressed mean on the seven Appendix E tasks at p=.1 on 27B (Part 1).

Thanks for the detailed rebuttal. Among my concerns, some have been resolved, while others haven't.

  • W1: KVzap or KVpress leaderboard was evaluated only on dense Transformers, but this fact does not justify omitting it as a baseline in this study. The full-attention module in the Hybrid Transformer is identical to the one in the Dense Transformer, so most of the eviction methods can be straightforwardly applied without any effort (including KVzap, KVgrad, etc.), and this paper should set them as baselines. More broadly, from a pracctitioner's perspective, KV compression, including KV quantization and compaction, serves the same purpose and result: reducing memory, speeding up inference, and maintaining accuracy. RestoreKV should also be compared, because it is far closer to eviction than quantization or compaction.

  • W2, etc.: I believe your justification in fact supports my initial concern. I cited the smaller models to illustrate how large the accuracy drop is (I apologize if my intention is not well written). Hence, the point of this concern is what performance drop is acceptable in practice, and how much the proposed paper can compress the KV cache within these bounds compared to the other baselines. Although there is no agreed threshold, I suggest a tight bound (1%) and a loose bound (5%) from the uncompressed model as reasonable bounds. It would be great if both the authors and I could agree on these bounds.

  • W4/Q3: This includes a serious reporting issue. I could not believe that the paper states, "Each uses the budget allocation its paper defines (KVzip’s global score-sorted allocation, flat per-head budgets for the others);" in L373-374, but the actual implementation split each layer's budget equally across heads, which is disclosed in the rebuttal.

To summarize, 1) within both the tight and loose bounds, GateSignt is worse than baselines 2) the range where the authors claim their advantage is not practical (even worse than 0.6B fullKV model), 3) several criticisms regarding the KVzip are overclaimed; 4) the authors' response supports my original claims remain unchallenged. Taking these together, I believe keeping my current score is appropriate unless the algorithm is substantially redesigned to deliver a significant improvement.

Reviewer_JSgHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GateSight, which studies KV cache compression in hybrid LLMs. It shows that signals from GDN layers can indicate which tokens will be useful to attention layers later, and uses these signals to keep important KV entries. GateSight performs well even when only 10% of the KV cache is retained.

Critical strengths
  1. The idea of using GDN signals to predict which KV entries will be useful later is interesting.

  2. The paper is well written and easy to follow.

  3. The experimental results are strong. GateSight still performs well when only 10% of the KV cache is kept.

Critical weaknesses
  1. One concern is how the training target is constructed. It is based on rollout utility measured on model-generated continuations, which may not reflect the information needed for real user queries.

  2. There is a mismatch between training and inference. Utility is measured by removing one KV entry at a time during training, whereas many entries are removed together at inference. This may overlook interactions between KV entries.

Questions for authors
  1. Although GateSight is evaluated on contexts up to 262K tokens, its utility prediction quality is directly validated only up to 8K. Could you report the prediction accuracy at longer context lengths?
1 comment

Response to JSgH: Supervision, joint removal, and longer contexts — W1–W2/Q1

Thank you for the feedback.

W1: Self-generated utility is a proxy when future questions are unknown; downstream questions test its usefulness. SCBench reuses the same compacted context for multiple questions, and predictor training uses neither those questions nor benchmark data.

We also tested a target that uses no self-generated continuation, reconstruction attention, with the same GDN features, MLP, and training (table in our response to Reviewer AaKr, W2/Q2). At p=.4 it lowers accuracy on KV and prefix/suffix and raises it on RepoQA+KV, so neither target is better on all tasks.

On GSM8K with the question in the compressed cache (response to Reviewer AaKr, W3/Q3), GateSight matches uncompressed accuracy down to p=.5, with responses of 162–1,024 tokens, all longer than the 128-token training continuations.

We will clarify the proxy target and add these task-specific supervision and reasoning results.

W2: Under joint removal, selecting by the single-removal label gives nearly the same error as selecting by Shapley values. Equation (2) is the exact ungated layer-output change for one removal, holding the query, other entries, and remaining layer computation fixed; it does not itself measure interactions, so we tested them.

We added Shapley and joint-removal analyses. For retained set S, define v_t(S)=−‖r_t(S)−r_t(N)‖, where r_t is the ungated layer output and N the full context. Protected entries remain. An entry's marginal contribution when added last equals its leave-one-out perturbation; Shapley averages over different positions and retained sets.

We used 12 training-corpus documents, 512 context tokens each, 16 layers × 4 KV heads, and 2,048 permutations per document/layer/head (768 combinations). φ_mean is the Shapley value of the game averaged over continuation queries.

Score vs. φ_meanSpearman ρOv@10Cov@10
Paper label U_max.900669.18%88.23%
U_mean.950483.84%95.55%
Deployed MLP.486638.48%57.23%
Attention max.909470.33%89.08%
Attention mean.948582.20%95.25%

Half-sample rankings have median Spearman .9995 (minimum .9921).

Retaining top-p entries gives the following mean relative layer-output errors after joint removal:

pU_maxφ_meanAttention maxMLPRandom
.1.21472.20360.21301.31187.42682
.3.08117.07997.08134.16690.27828
.5.03568.03879.03602.09515.18984

U_max and Shapley selection give similar errors at every p, and MLP selection lowers the error of random selection by 27–50%.

Q1: Direct prediction now extends to 64K, using the original scorer without long-text retraining. SCBench cohorts cover 16 attention layers × 4 KV heads; labels use four 64-token self-generated continuations.

Context tokensDocumentsSpearman ρOv@10Cov@10
16,38416.4468434.774%61.068%
32,76812.4330932.449%54.119%
65,53610.4033131.457%52.892%

On the same cohorts, an untrained age ranking gives Ov@10 of 19.12%/15.86%/15.84%, and random selection about 10%, so the scorer keeps 1.8–2.0× the age ranking's Ov@10 at every length. Cov@10 divides selected utility by oracle top-10% utility.

We will add the Shapley definition, sampling stability, joint-removal results, and expanded prediction table, specifying these scopes.

Reviewer_AaKrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GateSight, a query-agnostic KV-cache eviction method for hybrid recurrent–attention language models. It uses lightweight MLPs to predict KV importance from native Gated DeltaNet signals and allocates per-layer budgets using attention output gates. Experiments on two hybrid models demonstrate low scoring overhead and competitive accuracy under aggressive compression.

Critical strengths

S1: Using recurrent-layer signals to guide attention-cache eviction is an interesting and well-motivated idea.

S2: The empirical analysis supports the predictive value of these signals, including through leave-one-corpus-out evaluation and an age-only control.

S3: The implementation supports physical cache compaction and reuse. Results at a 10% retained-cache budget are strong relative to the evaluated eviction baselines.

Critical weaknesses

W1: Compression gains are limited when near-orignal-moel quality is required. Figure 4 suggests that retaining at least 95% of the uncompressed SCBench model score requires approximately 70% of the KV cache for Qwen3.8-27B and 90% for Flash-Next. These operating points offer significantly smaller savings than the highlighted 90% cache reduction. The paper does not establish the resulting end-to-end serving benefits under such a quality constraint with original models.

W2: Large performance decrease on exact-retrieval tasks exposes an important limitation. On Flash-Next at a 40% retained-cache budget, Appendix E reports gaps relative to KVzip of 88.6, 71.5, and 51.3 percentage points on kv, prefix_suffix, and repoqa_and_kv, respectively. These failures are particularly relevant because preserving retrieval capability motivates retaining attention layers in hybrid models. They raise concerns that utility learned from short self-generated continuations may overlook information needed by future questions. The paper does not sufficiently explain whether these decrease arise from the training target, budget allocation, or sparse-cache restructuring.

W3: Robustness to common reasoning workloads remains untested. Permanent eviction prevents later queries from recovering discarded information. This may be consequential during long reasoning, where information needs change across generation steps. Although SCBench evaluates shared-context reuse, it does not establish robustness to long reasoning trajectories. Additional evaluation is needed, particularly because training utilities are derived from continuations of at most 128 tokens.

W4: Generalizability beyond the Qwen family is unclear. Both evaluated models belong to the Qwen family, and the estimator is retrained for each checkpoint. The results therefore demonstrate applicability to two related architectures rather than transfer across model families. Evaluation on another hybrid family would help establish whether the predictive relationship between recurrent signals and attention utility is broadly applicable.

W5: The contribution of recurrent signals is not fully isolated. GateSight combines learned scoring with a distinct budget allocation strategy, whereas the baselines use different allocations. Moreover, the age-only control does not establish whether recurrent signals outperform other inexpensive features in a comparably trained predictor. Matched-allocation comparisons and a similarly sized predictor using attention-side features would strengthen the contribution.

Questions for authors
  1. What memory savings and serving-throughput improvements are achievable while retaining at least 95% of the uncompressed model’s score?

  2. What causes the large exact-retrieval deficits, and can the effects of scoring, allocation, and sparse-cache restructuring be separated?

  3. How does performance vary with generation length, and does the method generalize to hybrid models outside the Qwen family?

2 comments

Thank you for the feedback.

W1/Q1: Benefits at ≥95% quality. Across all 12 SCBench tasks, 27B/p=.7 retains 97.49% of the uncompressed mean (63.34/64.97), and Flash/p=.9 retains 98.59% (63.80/64.71). The protected prefix is additional to budget p.

On a fixed decoding workload with 260,096 input tokens and 128 decode tokens (median of three repetitions; one H200 for 27B, two H200s/TP2 for Flash), matched implementations give:

Model/modepMaximum batchTokens/s at max batchRatio
27B/eager1→.74→634.181→58.3861.708×
27B/graphs1→.74→634.423→63.8921.856×
Flash/eager1→.99→1074.054→79.0891.068×

Per request, attention-KV cache shrinks by 29.9% on 27B and attention+QSA cache by 9.9% on Flash.

The budget that retains ≥95% is task-dependent. For the seven Appendix E tasks—qa_eng, qa_chn, choice_eng, many_shot, summary, mf, and summary_with_needles—the mean score at p=.1 retains 98.02%/98.36% of the uncompressed mean for 27B/Flash. At the same budget, peak throughput on the fixed decoding workload is 10.252×/12.158× (27B eager/graphs) and 5.242× (Flash) that of the uncompressed reference.

W2/Q2: The retrieval deficits come from entry selection, not from allocation or restructuring alone. On Flash, we compared self-generated utility (SU) with teacher-forced reconstruction-attention (RA) supervision, fixing GDN features, MLP architecture, training, flat allocation, and native QSA. A/B are disjoint sets of 1,090 training documents from the same nine sources, with a shared 248-document validation set; downstream seed is 0. Each task uses its first 50 contexts at p=.4 (250 questions; 400 for RepoQA+KV). Accuracy (%):

TaskA: SUA: RAB: SUB: RA
KV21.22.028.80.8
Prefix/suffix9.62.44.00.8
RepoQA+KV33.7557.5030.7547.25

RA does not improve KV or prefix/suffix but improves RepoQA+KV: the target contributes to that deficit and does not explain the other two.

Separately, fixing the original scorer, inputs, p=.4 budget, and QSA, gate→flat allocation gives KV 20.8→14.4, prefix/suffix 7.2→4.8, and RepoQA+KV 31.25→29.00, so the gate allocation is not the cause.

Restructuring is shared by all methods: KVzip uses the same compaction, four-token regrouping, and native QSA and scores 98.0/78.1/82.9 on KV/prefix-suffix/RepoQA+KV at p=.4 (Appendix E). Restructuring alone therefore does not cause the deficits; they arise from which entries the scorer selects.

W3/Q3: With generations up to 1,024 tokens, GateSight matches uncompressed GSM8K accuracy down to p=.5. On the first 100 official GSM8K test questions, 27B uses zero-shot CoT, compresses the cache containing the question, and generates up to 1,024 tokens. Uncompressed accuracy is 95/100. Correct answers:

Methodp=.9.7.5.3.2.1
GateSight959696906412
KVzip959595764411
TriAttention97958144259
KeyDiff959782493011
SnapKV916224844

Uncompressed generations take 162–1,024 tokens (median 351.5). In the 129–256/257–512/513–1,024-token reference-length bins (13/66/21 questions), p=.5 answers 13/65/18 correctly versus 13/65/17 uncompressed, so the longest generations lose no accuracy at p=.5. GateSight also has the highest count of the five methods at p=.3 and .2; all methods fail at p=.1.

W4/Q3: Beyond Qwen. On official Nemotron-H-8B-Base-8K, 240 documents from nine sources give:

PredictorSpearman ρOv@10Cov@10
Age.242925.49%37.18%
Recurrent-signal MLP.484836.20%55.02%

With a predictor trained on this checkpoint, the relationship between recurrent signals and attention utility holds outside the Qwen family.

W5: Signal and allocation controls. As controls, we trained MLPs on the lightweight attention-side signals of TriAttention and KeyDiff, with the same data, utility labels, and training as for GDN signals. Leave-one-source-out results on 240 documents/nine sources, with no age or combined features added:

InputSpearman ρOv@10Cov@10
TriAttention.389028.83%43.99%
KeyDiff.213018.06%30.28%
GDN.515738.23%55.46%

GDN signals give the highest ρ, Ov@10, and Cov@10.

Separately, fixing GateSight scores and total budget at p=.1 on cwe/niah_multiquery/niah_multivalue gives 98.58/95.45/52.05 for gate/flat/PyramidKV layer-budget rules; PyramidKV changes only allocation.

We will add these results and protocols to the paper and clarify the signal controls in Observation 3.

My main concern remains the practical benefit at near-original quality. The throughput gains on 27B are useful, but preserving quality still requires retaining 70–90% of the cache, limiting the memory savings and end to end benefits. The reasoning evaluation is also limited: GSM8K tests relatively simple multi-step arithmetic, and generation is capped at 1,024 tokens. A harder benchmark such as AIME, with a larger generation budget, would better stress long reasoning. I therefore maintain my score.

Comments on the paper

1 comment

Dear reviewers,

A reminder that the discussion period will end on November 18 at 11:59pm AoE. I encourage you to interact with the authors' rebuttal to clarify any remaining issue (and how well the rebuttal addressed your concerns) before the next phase.

Thanks, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.