- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed query-agnostic KV eviction for the full attention layer of a hybrid Transformer. The authors found three observations about the recurrent memory in these hybrid Transformers as follows. 1) The forgetting speed of GDN differs across heads, and this behavior is consistent across contexts. 2) The write and forget signals of GDN can predict the future importance of KVs. 3) The relationship between KV importance and the GDN signal is distributed and nonlinear. Based on these three observations, the author proposed introducing an MLP model that predicts future utility from entry information. Then, the proposed method allocates the heterogeneous budgets across layers, but uniformly across heads within the given layer to ensure compatibility with vLLM PagedAttention. Experimental results claim that their method has superior performance when the compaction ratio is 0.1.
Critical strengths
- The proposed method is compatible with PagedAttention, and the implementation is conducted with vLLM.
- KV eviction on a hybrid LLM remains underexplored, and this paper investigates it with an observation that can be captured only in the GDN.
Critical weaknesses
- One major concern is the selection of baselines. The author claims that 1) GateSight outperforms the other baseline methods (SnapKV, TriAttention, and KeyDiff), as shown in Figure 4. 2) Although KVzip exhibits better accuracy than GateSight until p=0.4 (Table 3), KVzip takes 1.5-2.2 times the prefill time, so directly comparing KVzip with GateSight is unfair (Section 5.3). 3) KVzip allocates KVs using global score across layers and heads, which is not compatible with the authors' vLLM protocol (Section 5.1 Baselines). However, the idea of introducing MLPs for each head to predict the importance score of each KV entry has already been explored in KVzap (https://arxiv.org/abs/2601.07891). Specifically, the source of the low wall-clock scoring time of both KVzap and GateSight is the same, so many of the claims of this paper are questionable. Compared to KVzap, GateSight performs worse, despite both being "no attention score" methods and having similar wall-clock times. Furthermore, KVzip and KVzap can also allocate the budget uniformly across heads, as provided in the ablation studies in their papers. Hence, the authors should revisit existing KV eviction methods, including current SOTA methods (https://huggingface.co/spaces/nvidia/kvpress-leaderboard), and provide comparisons against strong baselines.
- The method should also be compared with the alternatives, such as smaller models. On RULER 4K, the proposed GATESIGHE exhibits Qwen3.8-Flash-Next (93.4%, p=0.7) and Qwen3.8-27B (91.9 %, p=0.4), respectively, which are lower than Qwen3-4B (95.1%, p=1). This comparison is not just about KV cache size itself, but about the practical value of the achieved accuracy after compression.
- In Abs., the main claim about the experimental results is as follows: "GATESIGHT is the most accurate of five query-agnostic methods on three of the four model–benchmark pairs and leads KVzip on all four, by 9.5 to 47.2 points". However, the numbers 9.5 and 47.2 are meaningless because these numbers are from p=0.1, and the performance of GATESIGHT in this stage is even worse than or competitive with Qwen3-0.6B.
- If the authors claim that comparing head-wise and layer-wise budget allocation is unfair, they should provide a clear explanation, including experimental evidence that shows the trade-off between accuracy and inference speed. The original PagedAttention block was organized with the shape (Layer, Head, block size, head dim), whereas the layer dimension was modified to support group-wise paging for Hybrid Transformer. This suggests that implementing vLLM PagedAttention compatible with head-wise budget is not technically impossible. Such complex memory management may reduce GPU utilization and increase inference latency. However, the authors should quantify the resulting latency overhead through experiments rather than just stating that "Each uses the budget allocation its paper defines (KVzip’s global score-sorted allocation, flat per-head budgets for the others); the gate-based per-layer allocation belongs to GATESIGHT only. To ensure a fair comparison, all methods run in the same vLLM pipeline under the prefix-only protocol of Appendix A, with every setting listed in Appendix G."
Questions for authors
- What is the advantage of using Qwen3.8-27B (91.9 %, p=0.4) instead of Qwen3-4B (95.1%, p=1)?
- Recent models, such as DeepSeek V3.2 and HySparse2, already have a top-k indexer in their model. What is the advantage of post-hoc KV cache eviction methods compared to those inherent top-k indexer modules?
- The original implementation of KVzip in kvpress uses a non-uniform budget for both layers and heads, but this paper seems to use a uniform budget across heads. Could you explain the vLLM implementation of KVzip, including PagedAttention?
- Could you provide the TPOT before and after compaction?
- Do you think that Ov@10 = 41% is useful?
- How accurate is the future KV utility metric? Have the authors compared it with shapley value to validate whether the utility scores reliably reflect the token contributions?
5 comments
Continued from Part 1 (W4, Q3–Q5).
W4/Q3: Head-wise budgets are implementable in vLLM. Our original KVzip port globally selects each layer's total by reconstruction score, then splits it equally across four KV heads; each head selects tokens separately and compacts them into that layer's block-table prefix. We will distinguish this from global selection across layers and heads.
We added global head-wise KVzip with the same scores and total prunable-KV budget; the protected prefix is additional. Below are equal-weight means (%) over all 13 RULER-4K tasks, 100 examples each, as in the paper. We will add this comparison to the revised appendix.
| Retained fraction | Original GateSight | Layer-wise KVzip | Global head-wise KVzip |
|---|---|---|---|
| .1 | 76.62 | 29.42 | 44.08 |
| .2 | 82.46 | 54.17 | 71.10 |
| .3 | 88.12 | 83.84 | 86.62 |
| .4 | 91.86 | 95.50 | 96.93 |
| .5 | 94.97 | 97.39 | 97.24 |
| .7 | 97.44 | 97.66 | 97.43 |
Head-wise allocation improves KVzip by up to 16.93 points over layer-wise KVzip (p=.2). GateSight leads at p=.1 by 32.54 points and at p=.2 by 11.36 points; it also leads at p=.3 by 1.50 points. Head-wise KVzip leads at p=.4–.5 by 2.27–5.07 points, and the two are essentially tied at p=.7. On the other ten tasks, GateSight is at least as accurate at every budget.
The variants use the same total prunable-KV budget, but global head-wise KVzip reallocates it across layers and heads, so its gain reflects both cross-layer and within-layer allocation. The resulting layer budgets are non-uniform: at p=.1 the four highest-share layers hold 45.8% of the mean prunable budget, versus 25% under uniform layers. Layer budgets also matter for GateSight: fixing its scores and total budget on cwe, niah_multiquery, and niah_multivalue at p=.1 gives 98.58/95.45/52.05 with gate/flat/PyramidKV rules; here PyramidKV only changes allocation.
Costs use actual per-head block pools, with separate lengths and block indices per request/layer/head. On 27B/H200 with common eager/single-split execution, a 4,096-token prefix, batches 1/2/4/8/16/32/64, and p=.1/.2/.3/.4/.5/.7, all 42 matched configurations show 9.37%–11.68% higher TPOT and 8.57%–10.46% lower throughput than layer-wise KVzip.
At 260,096 prefix tokens and batch 1:
| p | Layer-wise ms/token | Head-wise ms/token | TPOT increase | Throughput decrease |
|---|---|---|---|---|
| .1 | 69.7105 | 77.0727 | 10.56% | 9.55% |
| .3 | 71.6778 | 87.9651 | 22.72% | 18.52% |
| .5 | 77.7576 | 100.7256 | 29.54% | 22.80% |
Each point uses 128 decode tokens and the median of three repetitions, excluding prefill, scoring, top-B selection, and cloning. The 4K workload adds an 18-token query. Thus, head-wise allocation improves KVzip by up to 16.93 points at p=.2, while GateSight remains ahead by 11.36 points; the head-wise implementation has 9.37%–11.68% higher TPOT at 4K and 10.56%–29.54% at 260K.
Q4: TPOT at equal batch and at each budget's throughput peak. The fixed decoding workload uses 260,096 input tokens, 128 decode tokens, and medians of three repetitions: one H200 for 27B, two H200s/TP2 for Flash. p=1 uses the same implementation and execution mode.
| Model / mode | p | Maximum batch | Batch at peak | TPOT (ms/token) | System tokens/s |
|---|---|---|---|---|---|
| 27B / eager | 1 | 4 | 4 | 117.023 | 34.181 |
| 27B / eager | .7 | 6 | 6 | 102.765 | 58.386 |
| 27B / eager | .1 | 41 | 41 | 117.000 | 350.427 |
| 27B / graphs | 1 | 4 | 4 | 116.202 | 34.423 |
| 27B / graphs | .7 | 6 | 6 | 93.908 | 63.892 |
| 27B / graphs | .1 | 41 | 32 | 76.459 | 418.522 |
| Flash / eager | 1 | 9 | 9 | 121.533 | 74.054 |
| Flash / eager | .9 | 10 | 10 | 126.441 | 79.089 |
| Flash / eager | .1 | 86 | 86 | 221.557 | 388.161 |
Each row uses its throughput-maximizing batch, so the throughput gain includes the capacity released by compaction. At equal batch 1, p=.1 lowers 27B TPOT from 112.33 to 67.16 ms/token (eager) and from 111.57 to 35.08 (graphs). Flash TPOT at p=.9 and .1 stays within 2.2% of p=1 at batches 1 and 4 (p=.1: 109.72→110.16 and 116.09→118.63 ms/token); its gain comes from the larger maximum batch (9→86).
We will report both quality and cost and clarify the trade-off behind adaptive layer budgets with equal within-layer head counts.
Q5: Ov@10=41% exceeds random and age controls. Ov@10 measures overlap with the oracle top-10% set. Random selection, the age-feature MLP, and GDN MLP score 10.0%/17.9%/41.0%; Cov@10 is 19.0%/29.6%/58.4%. Coverage divides selected utility by oracle top-10% utility. Downstream, GateSight keeps 98.02% of the uncompressed mean on the seven Appendix E tasks at p=.1 on 27B (Part 1).
Thanks for the detailed rebuttal. Among my concerns, some have been resolved, while others haven't.
-
W1: KVzap or KVpress leaderboard was evaluated only on dense Transformers, but this fact does not justify omitting it as a baseline in this study. The full-attention module in the Hybrid Transformer is identical to the one in the Dense Transformer, so most of the eviction methods can be straightforwardly applied without any effort (including KVzap, KVgrad, etc.), and this paper should set them as baselines. More broadly, from a pracctitioner's perspective, KV compression, including KV quantization and compaction, serves the same purpose and result: reducing memory, speeding up inference, and maintaining accuracy. RestoreKV should also be compared, because it is far closer to eviction than quantization or compaction.
-
W2, etc.: I believe your justification in fact supports my initial concern. I cited the smaller models to illustrate how large the accuracy drop is (I apologize if my intention is not well written). Hence, the point of this concern is what performance drop is acceptable in practice, and how much the proposed paper can compress the KV cache within these bounds compared to the other baselines. Although there is no agreed threshold, I suggest a tight bound (1%) and a loose bound (5%) from the uncompressed model as reasonable bounds. It would be great if both the authors and I could agree on these bounds.
-
W4/Q3: This includes a serious reporting issue. I could not believe that the paper states, "Each uses the budget allocation its paper defines (KVzip’s global score-sorted allocation, flat per-head budgets for the others);" in L373-374, but the actual implementation split each layer's budget equally across heads, which is disclosed in the rebuttal.
To summarize, 1) within both the tight and loose bounds, GateSignt is worse than baselines 2) the range where the authors claim their advantage is not practical (even worse than 0.6B fullKV model), 3) several criticisms regarding the KVzip are overclaimed; 4) the authors' response supports my original claims remain unchallenged. Taking these together, I believe keeping my current score is appropriate unless the algorithm is substantially redesigned to deliver a significant improvement.
Thank you for the review. We number the four weaknesses W1–W4 and the six questions Q1–Q6 in the order they appear in the review.
W1: GateSight's contribution is the predictive relationship between recurrent signals and attention-KV utility. KVzap (arXiv:2601.07891) addresses a different question. On dense Transformers (Qwen3-8B, Llama-3.1-8B-Instruct, Qwen3-32B), its per-layer surrogates read each attention layer's input hidden states, imitate KVzip+ scores, and prune by threshold. GateSight reads the write and forget signals of GDN layers and learns rollout utility from self-generated continuations. Both use a small learned scorer; that is not our claimed novelty.
Beyond the paper's age and leave-one-corpus-out controls, MLPs trained on the lightweight attention-side signals of TriAttention and KeyDiff, with the same data, utility labels, and training procedure, reach Ov@10 of 28.83% and 18.06%, versus 38.23% for GDN signals. Scores reported on different models do not rank KVzap and GateSight on a common checkpoint.
GateSight is specific to hybrid models: it reads GDN signals absent from dense Transformers. The kvpress leaderboard evaluates dense models, not hybrid ones. Among its methods, the paper compares with KVzip (Table 3 and scoring cost), KeyDiff, and SnapKV. RestoreKV (arXiv:2608.01247, posted 2 August 2026; training code released October 2026) does not score entries: it keeps a base method's eviction, e.g., KVzip's, and fills part of the same KV budget with a restore cache built by an extra LoRA-adapted pass over the full cache.
KVzip's quality advantages remain valid: at p=.4, Table 3 reports GateSight/KVzip SCBench scores of 51.7/60.4 on 27B and 45.3/60.9 on Flash. We report reconstruction cost separately. Part 2 clarifies our allocation port and adds head-wise KVzip.
W2/W3/Q1: Compression retains a selected checkpoint while reducing its KV footprint. On RULER-4096, GateSight keeps 27B at 97.4% at p=.7 and Flash at 97.0% at p=.9 (uncompressed: 97.5%/96.8%), above the cited 95.1% for Qwen3-4B; the figures quoted in W2 are at lower budgets. Qwen3-4B's context is 32,768 tokens natively (131,072 with YaRN) and Qwen3-0.6B's is 32,768 (model cards), so neither covers the SCBench contexts evaluated here, which extend to the 262K-token limit.
The abstract already limits the 9.5–47.2-point gains to “a tenth of the attention KV cache”; we will clarify that these use the layer-wise KVzip port. With global head-wise KVzip, GateSight still leads the 13-task RULER-4K mean at p=.1, by 32.5 points (76.62 vs. 44.08; Part 2).
At higher-quality operating points, 27B/p=.7 and Flash/p=.9 retain 97.49%/98.59% of the all-12-task SCBench mean. In a separate 260,096-token decoding workload, maximum batches rise 4→6 and 9→10. Peak throughput ratios are 1.708×/1.856× for 27B eager/graphs and 1.068× for Flash.
For seven Appendix E tasks—qa_eng, qa_chn, choice_eng, many_shot, summary, mf, and summary_with_needles—the p=.1 mean retains 98.02%/98.36% of the uncompressed mean for 27B/Flash. The fixed workload yields 10.25×/12.16× throughput for 27B eager/graphs and 5.24× for Flash; Part 2 reports batch sizes and timings.
Q2: Native indexing and physical eviction can be combined. An indexer selects entries to read; an unselected entry's page is not automatically released. Eviction reduces the entries and pages retained.
Flash-Next already uses native QSA. GateSight rebuilds four-token index blocks after compaction and continues using native QSA selection and attention. Maximum batch at 260,096 tokens rises from 9 to 10/86 at p=.9/.1.
We will clarify the contribution and KVzap relationship, baseline allocation, quality-constrained operating points, and QSA adaptation.
Q6: We added permutation-sampled Shapley values. The diagnostic uses 12 training-corpus documents, 512 context tokens each, 16 layers × 4 KV heads, and 2,048 permutations per document/layer/head. The game is the negative ungated layer-output distance from full context, with protected entries retained; φ_mean is its Shapley value averaged over continuation queries.
U_max versus φ_mean gives Spearman .9006, Ov@10 69.18%, and Cov@10 88.23%. Half-sample rankings have median Spearman .9995. The deployed MLP's Spearman with φ_mean is .4866. Retaining top-p entries gives these mean relative layer-output errors after joint removal:
U_max selection stays within .012 of Shapley selection at every p and is lower at p=.5. We will add the metric definitions and the Shapley/joint-removal validation.