- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies decoding-time KV-cache eviction for the full-attention layers of hybrid models (Qwen3.5-4B, Nemotron-Nano-9B-v2, Kimi-Linear-48B-A3B). It builds on the periodic RPC schedule, which keeps a fixed fraction of the generated history. It adds a retention signal from linear layers (the magnitude of each token's recurrent-state update) and age-aware block quotas. Based on the findings (F) and methods (M) below, it matches or outperforms four decoding-time baselines across three models, five benchmarks, and five retention ratios. Its overhead over RPC is small.
Finding
- (F.1) Tokens with low recurrent-update magnitude are "often overlooked by attention-based KV selection", and keeping them improves accuracy. Their overlap with attention selection is lower than for high-magnitude tokens (Fig. 2 left), and they extend to earlier positions (Fig. 2 middle). Adding them improves PolyMath ave@4 by 1.5/1.2 points, while High-Mag, Random, Low-Attn, and Pos-Matched do not match this gain (Table 1). They contain 17–22 pp more formula-like tokens than attention/RPC selections (App. C).
- (F.2) The nearest preceding linear layer is the best score source among the tested layers and aggregations (Fig. 2 right, App. B).
- (F.3) Under repeated eviction, older history becomes sparser. This is derived under an idealized model (App. D), not measured. With attention scores only, finer blocks for older candidates (64/128/256) work best, and the reversed schedule falls below global top-K (Table 1).
Method
- (M.1) Rank-based hybrid selection within each block. Attention fills K_b − ⌊αK_b⌋ slots first. The lowest update magnitudes from the nearest preceding linear layer fill the rest (Eq. 4).
- (M.2) Age-aware block quotas. Block width shrinks linearly from s_max (newest) to s_min (oldest), and each block keeps a length-proportional, positive quota (Eqs. 5–6).
Main Results
- (R.1) StacKV is best or tied against RPC, R-KV, VaSE, and LazyEviction in all 3 models × 5 benchmarks × 5 retention ratios (Fig. 4, App. G).
- (R.2) On AIME 2024 at ρ = 1/4, each component improves over attention-only, and the full method is best (Table 2).
- (R.3) At ρ = 1/4 in Nano-vLLM, compression allows a larger batch (192 vs. 77) and gives 1.69× FullCache throughput. RPC reaches a similar speedup at the same batch size (Table 3). Compared with RPC, StacKV's throughput is at most 1.63% lower, and its extra memory is at most 1.80 MiB (Sec. 5.4, App. H).
Critical strengths
-
The gains are consistent. StacKV is best or tied in all settings, across three different recurrent modules (gated delta rule, Mamba2, KDA). Kimi uses transferred hyperparameters (R.1).
-
The method is cheap. It stores one scalar per token, and the measured overhead over RPC is small (R.3).
-
Using the recurrent state to guide full-attention eviction is a new and natural idea for hybrid models. The controls (Pos-Matched, reversed blocks) are well chosen. However, the link between the signal and what should be retained is not yet supported (W1, W3).
Critical weaknesses
Weaknesses
My main concerns are twofold.
(1) The problems the paper identifies are not separated from the method. They are defined by, or evidenced only through, the gains of the method itself (W1)*.
(2) Read as an accuracy heuristic, the gain seems to come largely from block quotas, a decode-level design that does not depend on the hybrid structure. The hybrid-specific part may reduce to Low-Mag, whose advantage over simpler signals and whose rationale are both unclear (W2).
The method may still work well in practice, but these issues make the contribution much weaker than claimed. Additionally, the problem setting, decode-time compression in hybrid models with a ratio-based budget, is defined only in Sec. 4.3. This makes it hard to tell which parts of the method address hybrid models and which address decoding (W3).
W1. The problem statement and its evidence are not separated from the method*.
- The first problem is defined as the absence of the proposed signal. The Intro states that existing pruning is "completely detached from the concurrent recurrent states (S_t)". Not using S_t is not a failure by itself. A failure would be something attention-only selection does wrong, such as evicting tokens that are needed later. The main evidence that this is a problem is that adding S_t-based scores improves accuracy (Table 1). This shows that S_t helps, not what attention-only selection misses. App. C shows that attention keeps fewer formula-like tokens than Low-Mag, but not that the missing tokens are needed later (see also W2).
- The second problem can be defined without the method, but it is never measured. The Intro states that global top-K carves "contiguous temporal voids that sever context dependency and degrade multi-step reasoning performance". Gap lengths on the original timeline could be measured for any eviction policy, but they are not. The only evidence is again the accuracy of the method's own block variants (Table 1).
- The supporting analyses are run inside the method. F.1 compares second signals inside the attention-first α-mixture (Eq. 4), F.2 compares source layers inside the same mixture, and F.3 compares block schedules inside the same quota rule (Eqs. 5–6). The only analysis outside the method, the overlap and position plots (Fig. 2, left and middle), shows that Low-Mag selects different tokens from attention, but not that these tokens are needed.
These statements are not wrong if the goal is accuracy alone. In that case, however, the contribution reduces to an empirical heuristic that improves accuracy on the tested settings. The paper claims more: it presents these as problems of hybrid models that the method diagnoses and fixes. That stronger claim needs a method-independent diagnosis. For example, the paper could first show which tokens attention-only eviction discards and later needs, and how long the gaps on the original timeline become, and then show that Low-Mag and blocks fix these failures.
W2. Even as a heuristic, the gain seems to come more from a decode-level block design than from the hybrid structure.
- Blocks seem to address a decode problem, not a hybrid one. The Intro frames both problems as "misalignments when applied to hybrid models". But the mechanism the paper describes for the second problem, repeated thinning of older history, does not involve the recurrent state. It should therefore also arise in non-hybrid models under repeated decode-time eviction. Blocks are not tested on such models. If blocks help only hybrid models, the paper's framing is supported empirically, and an explanation would strengthen it. If blocks also help Transformers, the hybrid-specific contribution reduces to Low-Mag alone.
- Blocks may account for much of the main gain. In the only reported ablation (AIME 2024, ρ = 1/4), "Attn + blocks" alone, without any recurrent signal, is already above every baseline (Table 2 vs. Tables 8–10). This rests on one setting, so it is not conclusive. But it suggests that the improvement over prior work may come mainly from block quotas, which could be added to any decode-time method. This makes the paper look weaker as a contribution to hybrid models.
- Low-Mag's advantage is shown only against attention. It consistently improves over attention alone, including on top of blocks (Table 2). But it is not compared with other signals that also complement attention: R-KV's redundancy, VaSE's value magnitude, or simpler signals that need only the KV cache (key/value norm, accumulated attention). Table 1 compares it only with random and attention-based fillers. So it is not shown that the recurrent state gives a better signal than the KV cache alone.
- Low-Mag has no rationale beyond accuracy. The direction (lowest, not highest) is chosen by accuracy alone (Table 1). In delta-rule layers (Eqs. 8 and 10), a small update can mean that the token was not written (small β_t) or that the state already knew it (small v_t − S̃_t^⊤k_t). These are opposite readings, and Mamba2 layers (Eq. 9) allow only the first. The only explanation offered is post hoc: Low-Mag keeps 17–22 pp more formula-like tokens (App. C), with no base rate among the candidates, plus one case study (App. I). With 90–92% formula-like tokens on AIME and 81–84% on GPQA, this may simply be a token-type prior, which a rule could reproduce.
So the part of the method that is specific to hybrid models is the least supported, while the part with the clearest gain may not be specific to hybrid models.
W3. The problem setting is not defined clearly, which makes the paper hard to follow.
- KV-cache compression covers very different settings: one-shot prefill compression (SnapKV, PyramidKV), fixed-budget decoding (H2O, R-KV), and ratio-based periodic decoding (RPC). Section 2.1 lists them together without saying which setting this paper targets.
- This paper targets a specific combination: hybrid models, and periodic compression during decoding that keeps a fixed fraction of the generated history while protecting the prompt and recent tokens. This definition appears only in Sec. 4.3 (Eq. 7). The Abstract reports "a 25% retention budget" and Section 3 uses "ρ = 1/4" without saying what the ratio is relative to.
- The setting matters for the claims. Because the paper does not separate the hybrid part from the decode part of its setting, it is hard to tell which component addresses which (W2). F.3, for example, seems to depend on repeated compression during decoding, not on the hybrid structure.
A problem-setup section before Section 3 that defines the setting, ρ, P, R, and the protected tokens, and states which component targets the hybrid structure and which targets decoding, would resolve most of these issues.
* Why this is a problem. If a problem is shown only through the gain of the proposed method, the evidence is circular: the method defines the problem and also confirms it. An accuracy gain shows that a design choice helps on the development set. It does not show what fails without it, so readers cannot predict when the method will transfer to other tasks, models, or budgets. The paper is still valid as an accuracy improvement, but its contribution is then smaller than claimed: an effective heuristic, not an understanding of what attention-only eviction misses in hybrid models.
Questions for authors
Q1. (W2) Do age-aware blocks also help non-hybrid models, e.g., a pure Transformer with RPC? This would clarify whether blocks address a hybrid or a decode problem.
Q2. Is "Attn only" in Table 2 identical to RPC?
3 comments
Thank you for your thoughtful follow-up and for acknowledging that our previous response addressed part of your concerns.
- Tokens reused later in decoding should be kept. We define this objective at the token level.
- These tokens have lower update magnitudes on average, especially those missed by RPC (Table 6).
- At the same budget, attention + Low-Mag retains more of these tokens (Table 7) and achieves higher end-to-end accuracy than attention alone (Tables 1 and 2).
Q1. What was the smallest unit the annotators labeled?
The annotators labeled text spans. On each fixed Qwen3.5-4B trace, they marked earlier content used later, such as an equation used in a subsequent substitution. Each model token covered by a marked span inherited its evidence label. All full-attention layers share these labels, and each layer makes its own KV retention decisions. Annotators were blind to update magnitudes and retention decisions.
Q2. Did you average the magnitudes across source layers for Table 6?
Yes. We average over the eight linear source layers, one immediately preceding each full-attention layer.
Q3. Does the ordering in Table 6 hold for each layer separately?
Yes. We repeated the Table 6 analysis separately for each layer. Each row gives the mean update magnitude in the indicated source layer for each token population. For the last column, we determine whether RPC missed each evidence position in the corresponding full-attention layer. Layer indices start at 0.
| Full-attention layer | Linear source layer | All tokens | Reused evidence | Reused evidence missed by RPC |
|---|---|---|---|---|
| 3 | 2 | 0.12723 | 0.11327 | 0.09726 |
| 7 | 6 | 0.11555 | 0.10663 | 0.09258 |
| 11 | 10 | 0.11710 | 0.10715 | 0.09295 |
| 15 | 14 | 0.13147 | 0.11633 | 0.10093 |
| 19 | 18 | 0.08558 | 0.07056 | 0.06114 |
| 23 | 22 | 0.16940 | 0.14639 | 0.12448 |
| 27 | 26 | 0.26965 | 0.24036 | 0.20194 |
| 31 | 30 | 0.15906 | 0.14291 | 0.12192 |
All eight layers show the same ordering. Mean magnitude is lower for reused evidence than for all candidates, and lower still for reused evidence missed by RPC.
Q4. When is recall measured in Table 7?
At each compression round on the fixed traces, we measure recall for each policy. The numerator counts evidence tokens retained after compression, and the denominator counts all evidence tokens in the policy's candidate set just before compression, excluding those evicted in earlier rounds. Table 7 reports the mean of these per-round recalls, giving every round equal weight.
We also measured recall over successive compression rounds using the same denominator for all three policies at each round. For each fixed trace and annotator, the denominator counts every evidence token in the original trace that has become eligible for compression by that round, including those evicted in earlier rounds. Each policy's numerator counts how many of those tokens remain in its cache immediately after compression. We again weight each round equally.
| Dataset | Annotator | Attention | Attention + Low-Mag | Attention + High-Mag |
|---|---|---|---|---|
| AIME 2025 | GPT-6.1 Sol | 22.83% | 27.59% | 20.99% |
| AIME 2025 | GPT-6.1 Terra | 22.69% | 27.84% | 20.58% |
| LiveCodeBench | GPT-6.1 Sol | 23.89% | 27.91% | 22.03% |
| LiveCodeBench | GPT-6.1 Terra | 23.94% | 28.23% | 22.15% |
With this shared denominator, attention + Low-Mag exceeds attention alone by 4.02–5.15 percentage points across the four dataset–annotator pairs. Attention + High-Mag has the lowest recall in all four. The ordering we observed in Table 7 holds under both recall definitions.
Age-aware blocks and other decoding methods
(1) R-KV and VaSE. In Fig. 4, StacKV outperforms R-KV in all 75 settings and VaSE in 74 of 75, with one tie. Their selection signals are orthogonal to block allocation. Table 8 tests these signals with identical age-aware block quotas and a fixed 50/50 split between attention and the supplementary signal. In all four model–benchmark settings, pairing the blocks with Low-Mag yields higher accuracy than pairing them with either R-KV-style redundancy or VaSE-style value norm.
(2) ThinKV. ThinKV progressively prunes earlier thought segments as the reasoning trajectory changes. StacKV assigns quotas by position and uses finer blocks for older candidates. ThinKV (ICLR 2026)
We evaluated StacKV and ThinKV's TBE pruning on Qwen3.5-4B using the same Nano-vLLM backend. KV quantization is orthogonal to StacKV's eviction rule. To compare the eviction policies at the same KV precision and retention budget, we disabled ThinKV's TBQ quantization and adapted TBE to StacKV's RPC schedule. At each 2,048-token checkpoint, both methods retained exactly 25% of the cumulative eligible history while protecting the full prompt and the most recent 128 generated tokens. We also extended Table 2's evaluation of RPC attention with age-aware blocks from AIME 2024 to all five benchmarks.
| Accuracy (%) | AIME 2024 (ave@64) | AIME 2025 (ave@64) | HMMT-90 (ave@16) | GPQA-Diamond (ave@10) | LiveCodeBench (ave@4) |
|---|---|---|---|---|---|
| ThinKV TBE (RPC schedule, TBQ off) | 66.25 | 51.56 | 38.82 | 74.49 | 62.40 |
| Attn(RPC)+ blocks | 67.92 | 51.93 | 39.31 | 74.49 | 62.75 |
| StacKV | 68.96 | 53.80 | 40.14 | 75.05 | 63.20 |
Using RPC attention scores alone, age-aware blocks improve accuracy over adapted TBE by an average of 0.58 percentage points. across the five benchmarks. Adding Low-Mag to the same blocks improves accuracy by 0.45–1.87 percentage points across the five benchmarks. Full StacKV exceeds adapted TBE by 0.56–2.71 points on these benchmarks.
We hope these clarifications and new results resolve your remaining concerns, and we would be grateful if you could reconsider your assessment.
Thank you for the rebuttal. The rebuttal addresses part of my concerns, but my major weaknesses (W1, W2) are not fully addressed. I leave the following questions.
For Low-Mag, I now understand your argument as below. Is this correct? (1) Tokens reused later in decoding should be kept (at the token level, not per-layer cache entries). (2) Such tokens have low update magnitude, especially those missed by RPC (Table 6). (3) Keeping them improves accuracy.
The key details I would like to ask about are:
These details would help understand your assumption and analyses.
For age-aware blocks, the larger gain on the hybrid model is interesting. However, as you agreed, blocks are a general decoding component. Please compare them with other decoding-level methods, such as ThinKV (ICLR 2026).