acceptodds
Under review as a conference paper at ICLR 2027

StacKV: Spatio-Temporally Aware Compression of KV Caches in Hybrid Language Models

Abstract

Hybrid language models, which interleave linear recurrent layers with full-attention layers, offer a promising balance between training efficiency and expressive power. However, during autoregressive generation, the key-value (KV) cache of the attention layers also poses a severe memory bottleneck. Existing KV cache eviction strategies, primarily designed for pure Transformers, are sub-optimal for hybrid architectures. They rely exclusively on signals from full-attention layers, without accounting for recurrent-state dynamics, and employ unconstrained global pruning, which disrupts the contiguous reasoning chains required for complex tasks. In this paper, we introduce StacKV, a training-free KV cache compression framework tailored for hybrid models, addressing both what to retain and where to retain it. First, we identify a complementary retention signal from recurrent-state dynamics: tokens with low-magnitude recurrent updates are often overlooked by attention-based KV selection, yet retaining them alongside high-attention tokens improves performance. Second, to prevent the compounding loss of local context during rolling eviction, we propose an age-aware non-uniform spatial blocking strategy. Extensive experiments on 5 tasks demonstrate that StacKV outperforms existing baselines, maintaining over 95% of full-cache performance on average for each model at a 25% retention budget, while achieving a 1.69 serving throughput speedup in Nano-vLLM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient reasoning, Hybrid language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_URy9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies decoding-time KV-cache eviction for the full-attention layers of hybrid models (Qwen3.5-4B, Nemotron-Nano-9B-v2, Kimi-Linear-48B-A3B). It builds on the periodic RPC schedule, which keeps a fixed fraction of the generated history. It adds a retention signal from linear layers (the magnitude of each token's recurrent-state update) and age-aware block quotas. Based on the findings (F) and methods (M) below, it matches or outperforms four decoding-time baselines across three models, five benchmarks, and five retention ratios. Its overhead over RPC is small.

Finding

  • (F.1) Tokens with low recurrent-update magnitude are "often overlooked by attention-based KV selection", and keeping them improves accuracy. Their overlap with attention selection is lower than for high-magnitude tokens (Fig. 2 left), and they extend to earlier positions (Fig. 2 middle). Adding them improves PolyMath ave@4 by 1.5/1.2 points, while High-Mag, Random, Low-Attn, and Pos-Matched do not match this gain (Table 1). They contain 17–22 pp more formula-like tokens than attention/RPC selections (App. C).
  • (F.2) The nearest preceding linear layer is the best score source among the tested layers and aggregations (Fig. 2 right, App. B).
  • (F.3) Under repeated eviction, older history becomes sparser. This is derived under an idealized model (App. D), not measured. With attention scores only, finer blocks for older candidates (64/128/256) work best, and the reversed schedule falls below global top-K (Table 1).

Method

  • (M.1) Rank-based hybrid selection within each block. Attention fills K_b − ⌊αK_b⌋ slots first. The lowest update magnitudes from the nearest preceding linear layer fill the rest (Eq. 4).
  • (M.2) Age-aware block quotas. Block width shrinks linearly from s_max (newest) to s_min (oldest), and each block keeps a length-proportional, positive quota (Eqs. 5–6).

Main Results

  • (R.1) StacKV is best or tied against RPC, R-KV, VaSE, and LazyEviction in all 3 models × 5 benchmarks × 5 retention ratios (Fig. 4, App. G).
  • (R.2) On AIME 2024 at ρ = 1/4, each component improves over attention-only, and the full method is best (Table 2).
  • (R.3) At ρ = 1/4 in Nano-vLLM, compression allows a larger batch (192 vs. 77) and gives 1.69× FullCache throughput. RPC reaches a similar speedup at the same batch size (Table 3). Compared with RPC, StacKV's throughput is at most 1.63% lower, and its extra memory is at most 1.80 MiB (Sec. 5.4, App. H).
Critical strengths
  1. The gains are consistent. StacKV is best or tied in all settings, across three different recurrent modules (gated delta rule, Mamba2, KDA). Kimi uses transferred hyperparameters (R.1).

  2. The method is cheap. It stores one scalar per token, and the measured overhead over RPC is small (R.3).

  3. Using the recurrent state to guide full-attention eviction is a new and natural idea for hybrid models. The controls (Pos-Matched, reversed blocks) are well chosen. However, the link between the signal and what should be retained is not yet supported (W1, W3).

Critical weaknesses

Weaknesses

My main concerns are twofold.

(1) The problems the paper identifies are not separated from the method. They are defined by, or evidenced only through, the gains of the method itself (W1)*.

(2) Read as an accuracy heuristic, the gain seems to come largely from block quotas, a decode-level design that does not depend on the hybrid structure. The hybrid-specific part may reduce to Low-Mag, whose advantage over simpler signals and whose rationale are both unclear (W2).

The method may still work well in practice, but these issues make the contribution much weaker than claimed. Additionally, the problem setting, decode-time compression in hybrid models with a ratio-based budget, is defined only in Sec. 4.3. This makes it hard to tell which parts of the method address hybrid models and which address decoding (W3).

W1. The problem statement and its evidence are not separated from the method*.

  • The first problem is defined as the absence of the proposed signal. The Intro states that existing pruning is "completely detached from the concurrent recurrent states (S_t)". Not using S_t is not a failure by itself. A failure would be something attention-only selection does wrong, such as evicting tokens that are needed later. The main evidence that this is a problem is that adding S_t-based scores improves accuracy (Table 1). This shows that S_t helps, not what attention-only selection misses. App. C shows that attention keeps fewer formula-like tokens than Low-Mag, but not that the missing tokens are needed later (see also W2).
  • The second problem can be defined without the method, but it is never measured. The Intro states that global top-K carves "contiguous temporal voids that sever context dependency and degrade multi-step reasoning performance". Gap lengths on the original timeline could be measured for any eviction policy, but they are not. The only evidence is again the accuracy of the method's own block variants (Table 1).
  • The supporting analyses are run inside the method. F.1 compares second signals inside the attention-first α-mixture (Eq. 4), F.2 compares source layers inside the same mixture, and F.3 compares block schedules inside the same quota rule (Eqs. 5–6). The only analysis outside the method, the overlap and position plots (Fig. 2, left and middle), shows that Low-Mag selects different tokens from attention, but not that these tokens are needed.

These statements are not wrong if the goal is accuracy alone. In that case, however, the contribution reduces to an empirical heuristic that improves accuracy on the tested settings. The paper claims more: it presents these as problems of hybrid models that the method diagnoses and fixes. That stronger claim needs a method-independent diagnosis. For example, the paper could first show which tokens attention-only eviction discards and later needs, and how long the gaps on the original timeline become, and then show that Low-Mag and blocks fix these failures.

W2. Even as a heuristic, the gain seems to come more from a decode-level block design than from the hybrid structure.

  • Blocks seem to address a decode problem, not a hybrid one. The Intro frames both problems as "misalignments when applied to hybrid models". But the mechanism the paper describes for the second problem, repeated thinning of older history, does not involve the recurrent state. It should therefore also arise in non-hybrid models under repeated decode-time eviction. Blocks are not tested on such models. If blocks help only hybrid models, the paper's framing is supported empirically, and an explanation would strengthen it. If blocks also help Transformers, the hybrid-specific contribution reduces to Low-Mag alone.
  • Blocks may account for much of the main gain. In the only reported ablation (AIME 2024, ρ = 1/4), "Attn + blocks" alone, without any recurrent signal, is already above every baseline (Table 2 vs. Tables 8–10). This rests on one setting, so it is not conclusive. But it suggests that the improvement over prior work may come mainly from block quotas, which could be added to any decode-time method. This makes the paper look weaker as a contribution to hybrid models.
  • Low-Mag's advantage is shown only against attention. It consistently improves over attention alone, including on top of blocks (Table 2). But it is not compared with other signals that also complement attention: R-KV's redundancy, VaSE's value magnitude, or simpler signals that need only the KV cache (key/value norm, accumulated attention). Table 1 compares it only with random and attention-based fillers. So it is not shown that the recurrent state gives a better signal than the KV cache alone.
  • Low-Mag has no rationale beyond accuracy. The direction (lowest, not highest) is chosen by accuracy alone (Table 1). In delta-rule layers (Eqs. 8 and 10), a small update can mean that the token was not written (small β_t) or that the state already knew it (small v_t − S̃_t^⊤k_t). These are opposite readings, and Mamba2 layers (Eq. 9) allow only the first. The only explanation offered is post hoc: Low-Mag keeps 17–22 pp more formula-like tokens (App. C), with no base rate among the candidates, plus one case study (App. I). With 90–92% formula-like tokens on AIME and 81–84% on GPQA, this may simply be a token-type prior, which a rule could reproduce.

So the part of the method that is specific to hybrid models is the least supported, while the part with the clearest gain may not be specific to hybrid models.

W3. The problem setting is not defined clearly, which makes the paper hard to follow.

  • KV-cache compression covers very different settings: one-shot prefill compression (SnapKV, PyramidKV), fixed-budget decoding (H2O, R-KV), and ratio-based periodic decoding (RPC). Section 2.1 lists them together without saying which setting this paper targets.
  • This paper targets a specific combination: hybrid models, and periodic compression during decoding that keeps a fixed fraction of the generated history while protecting the prompt and recent tokens. This definition appears only in Sec. 4.3 (Eq. 7). The Abstract reports "a 25% retention budget" and Section 3 uses "ρ = 1/4" without saying what the ratio is relative to.
  • The setting matters for the claims. Because the paper does not separate the hybrid part from the decode part of its setting, it is hard to tell which component addresses which (W2). F.3, for example, seems to depend on repeated compression during decoding, not on the hybrid structure.

A problem-setup section before Section 3 that defines the setting, ρ, P, R, and the protected tokens, and states which component targets the hybrid structure and which targets decoding, would resolve most of these issues.

* Why this is a problem. If a problem is shown only through the gain of the proposed method, the evidence is circular: the method defines the problem and also confirms it. An accuracy gain shows that a design choice helps on the development set. It does not show what fails without it, so readers cannot predict when the method will transfer to other tasks, models, or budgets. The paper is still valid as an accuracy improvement, but its contribution is then smaller than claimed: an effective heuristic, not an understanding of what attention-only eviction misses in hybrid models.

Questions for authors

Q1. (W2) Do age-aware blocks also help non-hybrid models, e.g., a pure Transformer with RPC? This would clarify whether blocks address a hybrid or a decode problem.

Q2. Is "Attn only" in Table 2 identical to RPC?

3 comments

Thank you for the rebuttal. The rebuttal addresses part of my concerns, but my major weaknesses (W1, W2) are not fully addressed. I leave the following questions.

For Low-Mag, I now understand your argument as below. Is this correct? (1) Tokens reused later in decoding should be kept (at the token level, not per-layer cache entries). (2) Such tokens have low update magnitude, especially those missed by RPC (Table 6). (3) Keeping them improves accuracy.

The key details I would like to ask about are:

  • Q1. Were evidence labels given as text spans or as tokens? (i.e., what is the smallest unit the LLM annotators labeled?)
  • Q2. How is "Mean magnitude" in Table 6 calculated? Did you average u_{t,l} over the linear layers that precede each full-attention layer?
  • Q3. (If yes to Q2) The method selects tokens per full-attention layer, each with its own source layer. Does the pattern in Table 6 hold for each layer separately?
  • Q4. When is recall in Table 7 measured: averaged over compression rounds, on the final cache, or just before each evidence token is reused? (Quite confused how this can get measured)

These details would help understand your assumption and analyses.

For age-aware blocks, the larger gain on the hybrid model is interesting. However, as you agreed, blocks are a general decoding component. Please compare them with other decoding-level methods, such as ThinKV (ICLR 2026).

Thank you for your thoughtful follow-up and for acknowledging that our previous response addressed part of your concerns.

  1. Tokens reused later in decoding should be kept. We define this objective at the token level.
  2. These tokens have lower update magnitudes on average, especially those missed by RPC (Table 6).
  3. At the same budget, attention + Low-Mag retains more of these tokens (Table 7) and achieves higher end-to-end accuracy than attention alone (Tables 1 and 2).

Q1. What was the smallest unit the annotators labeled?

The annotators labeled text spans. On each fixed Qwen3.5-4B trace, they marked earlier content used later, such as an equation used in a subsequent substitution. Each model token covered by a marked span inherited its evidence label. All full-attention layers share these labels, and each layer makes its own KV retention decisions. Annotators were blind to update magnitudes and retention decisions.

Q2. Did you average the magnitudes across source layers for Table 6?

Yes. We average ut,ℓu_{t,\ell} over the eight linear source layers, one immediately preceding each full-attention layer.

Q3. Does the ordering in Table 6 hold for each layer separately?

Yes. We repeated the Table 6 analysis separately for each layer. Each row gives the mean update magnitude in the indicated source layer for each token population. For the last column, we determine whether RPC missed each evidence position in the corresponding full-attention layer. Layer indices start at 0.

Full-attention layerLinear source layerAll tokensReused evidenceReused evidence missed by RPC
320.127230.113270.09726
760.115550.106630.09258
11100.117100.107150.09295
15140.131470.116330.10093
19180.085580.070560.06114
23220.169400.146390.12448
27260.269650.240360.20194
31300.159060.142910.12192

All eight layers show the same ordering. Mean magnitude is lower for reused evidence than for all candidates, and lower still for reused evidence missed by RPC.

Q4. When is recall measured in Table 7?

At each compression round on the fixed traces, we measure recall for each policy. The numerator counts evidence tokens retained after compression, and the denominator counts all evidence tokens in the policy's candidate set just before compression, excluding those evicted in earlier rounds. Table 7 reports the mean of these per-round recalls, giving every round equal weight.

We also measured recall over successive compression rounds using the same denominator for all three policies at each round. For each fixed trace and annotator, the denominator counts every evidence token in the original trace that has become eligible for compression by that round, including those evicted in earlier rounds. Each policy's numerator counts how many of those tokens remain in its cache immediately after compression. We again weight each round equally.

DatasetAnnotatorAttentionAttention + Low-MagAttention + High-Mag
AIME 2025GPT-6.1 Sol22.83%27.59%20.99%
AIME 2025GPT-6.1 Terra22.69%27.84%20.58%
LiveCodeBenchGPT-6.1 Sol23.89%27.91%22.03%
LiveCodeBenchGPT-6.1 Terra23.94%28.23%22.15%

With this shared denominator, attention + Low-Mag exceeds attention alone by 4.02–5.15 percentage points across the four dataset–annotator pairs. Attention + High-Mag has the lowest recall in all four. The ordering we observed in Table 7 holds under both recall definitions.

Age-aware blocks and other decoding methods

(1) R-KV and VaSE. In Fig. 4, StacKV outperforms R-KV in all 75 settings and VaSE in 74 of 75, with one tie. Their selection signals are orthogonal to block allocation. Table 8 tests these signals with identical age-aware block quotas and a fixed 50/50 split between attention and the supplementary signal. In all four model–benchmark settings, pairing the blocks with Low-Mag yields higher accuracy than pairing them with either R-KV-style redundancy or VaSE-style value norm.

(2) ThinKV. ThinKV progressively prunes earlier thought segments as the reasoning trajectory changes. StacKV assigns quotas by position and uses finer blocks for older candidates. ThinKV (ICLR 2026)

We evaluated StacKV and ThinKV's TBE pruning on Qwen3.5-4B using the same Nano-vLLM backend. KV quantization is orthogonal to StacKV's eviction rule. To compare the eviction policies at the same KV precision and retention budget, we disabled ThinKV's TBQ quantization and adapted TBE to StacKV's RPC schedule. At each 2,048-token checkpoint, both methods retained exactly 25% of the cumulative eligible history while protecting the full prompt and the most recent 128 generated tokens. We also extended Table 2's evaluation of RPC attention with age-aware blocks from AIME 2024 to all five benchmarks.

Accuracy (%)AIME 2024 (ave@64)AIME 2025 (ave@64)HMMT-90 (ave@16)GPQA-Diamond (ave@10)LiveCodeBench (ave@4)
ThinKV TBE (RPC schedule, TBQ off)66.2551.5638.8274.4962.40
Attn(RPC)+ blocks67.9251.9339.3174.4962.75
StacKV68.9653.8040.1475.0563.20

Using RPC attention scores alone, age-aware blocks improve accuracy over adapted TBE by an average of 0.58 percentage points. across the five benchmarks. Adding Low-Mag to the same blocks improves accuracy by 0.45–1.87 percentage points across the five benchmarks. Full StacKV exceeds adapted TBE by 0.56–2.71 points on these benchmarks.

We hope these clarifications and new results resolve your remaining concerns, and we would be grateful if you could reconsider your assessment.

Reviewer_YRyEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StacKV is a training-free KV-cache eviction method for hybrid LLMs. It combines recent-query attention with low-magnitude recurrent-state updates from the nearest preceding linear layer, and applies age-aware block quotas to improve positional coverage. Experiments across three hybrid architectures and five benchmarks show consistent improvements over the evaluated baselines.

Critical strengths
  1. Exploiting recurrent-state dynamics for attention-cache retention is a promising direction for hybrid models.
  2. Random, low-attention, and position-matched selection controls, source-layer comparisons, and component ablations provide useful evidence for the proposed design.
  3. Qwen-selected hyperparameters transfer to Kimi without retuning, while shared defaults remain competitive across models.
  4. The paper includes optimized kernels and reports both throughput and memory measurements.
Critical weaknesses
  1. The paper does not explain why low-magnitude updates should indicate useful tokens. Appendix C shows that 80–92% of Low-Mag selections are formula-like tokens. A simple rule-based baseline (e.g., retain numbers and math symbols at the same α share) is missing, and it may explain much of the gain.
  2. DeltaS [1] also uses similar signal, gated linear attention state change but favors high-drift content. This opposite conclusion deserves discussion. A normalized high-drift variant should also be compared.
  3. Several improvements are around 0.5–1.5 percentage points. Confidence intervals or paired tests are needed, particularly for small benchmarks. Repeated sampling reduces generation noise but does not replace broader problem coverage.
  4. Block quotas constrain single-round deletion gaps in the current candidate sequence. They do not guarantee cumulative coverage in original token positions or preservation of complete reasoning steps.
  5. The 1.69× throughput gain uses a larger feasible batch, demonstrating capacity-driven serving benefits without isolating fixed-batch latency improvements. StacKV is slightly slower than RPC at matched settings. Moreover, the nominal retention ratio excludes protected regions, Qwen is evaluated in non-thinking mode, and generation-length and compression-round distributions are missing.

[1] Kwon, Taeyoun, et al. "DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video." arXiv preprint arXiv:2609.27470 (2026).

Questions for authors
  1. How do separate decay/write scores and state-normalized update magnitudes compare with the proposed score?
  2. Does Low-Mag outperform a numbers-and-symbols heuristic, or controls jointly matched for token type, position, and attention?
  3. Authors need to clarify their finding with DeltaS’s high-drift retention.
  4. Can the authors report paired confidence intervals and cumulative coverage statistics in original token positions?
  5. Can the authors provide fixed-batch latency, actual KV occupancy, and generation-length/compression-round distributions?
2 comments

Thank you for the detailed response. The additional clarifications have made the paper's contributions much clearer. I will maintain my initial rating of weak accept.

Thank you for your thoughtful feedback and for taking the time to review our responses and updated materials. We are pleased that our additional clarifications have helped make the paper’s contributions clearer. We sincerely appreciate your continued support and the constructive suggestions you have provided throughout this process, which have helped us strengthen the paper.

Reviewer_xZTeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes StacKV, a training-free KV-cache compression method for hybrid language models. It supplements recent-query attention scores with low-magnitude recurrent-state updates from the nearest preceding linear layer. Selection operates within block-wise retention budgets, using finer blocks for older candidates to preserve positional coverage.

Critical strengths
  1. Using recurrent-state updates to guide retention in full-attention layers is an interesting contribution tailored to hybrid models. The comparisons with high-magnitude, random, low-attention, and position-matched selection provide useful evidence that the proposed signal contributes beyond attention scores and coarse positional coverage.
  2. The evaluation is reasonably thorough. Component ablations support both the recurrent signal and block allocation. Hyperparameters are selected on a separate development set, and their transfer from Qwen to Kimi without retuning strengthens the results. Improvements over four compression baselines are consistent across the reported settings, although some margins are modest.
  3. The implementation demonstrates practical value with little additional overhead. At 25% retention, StacKV achieves 1.69× FullCache throughput in Nano-vLLM by supporting a larger batch size. Its throughput is slightly below RPC at the same budget and batch size, so the main benefit is improved quality at a comparable serving cost.
Critical weaknesses
  1. The explanation for why low-magnitude updates are useful remains incomplete. A small state change does not necessarily indicate information that needs additional explicit storage; it can also arise when the existing state already represents the input well. The controls establish the signal’s usefulness, but do not fully explain its relationship to recurrent-memory limitations. Its preference for formula-like tokens also leaves open how much of the benefit comes from token type.
  2. The spatial analysis supports a narrower claim than preserving reasoning chains. As Appendix D acknowledges, the bounds apply to deletion runs within the current candidate sequence in a single compression round. They do not bound cumulative gaps on the original token timeline. More direct evidence connecting these coverage constraints to retained reasoning dependencies would strengthen the motivation.
  3. Some improvements over the strongest baseline are below one percentage point. Although the experiments average multiple sampled responses, confidence intervals would help assess these smaller gains.
Questions for authors
  1. Does Low-Mag still improve retention when the comparison also matches the selected tokens’ types, especially numbers and mathematical symbols?
  2. Can the authors measure gaps on the original token timeline across successive compression rounds, comparing age-aware blocks with uniform blocks and global selection?
  3. How are LazyEviction’s delayed eviction and recurrence tracking preserved under the shared compression schedule? Could the authors also report paired confidence intervals for the smaller improvements over the strongest baselines?
1 comment

The designs of exploiting recurrent-state changes and the age-aware non-uniform spatial blocking strategy make sense. However, some design choices remain heuristic. Could the author elaborate when the designs may be less effective?

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your thoughtful review and constructive feedback. We have also revised the manuscript in response to your comments and suggestions. As the discussion period draws to a close, we would be grateful if you could consider revisiting your assessment and score if you feel our rebuttal and revisions have addressed your concerns. Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.