acceptodds
Under review as a conference paper at ICLR 2027

HybriKV: KV Cache Eviction for Hybrid Linear-Attention Language Models

Abstract

Recent frontier language models increasingly adopt hybrid linear-attention architectures, replacing full attention with linear attention in most layers to reduce long-context memory overhead. However, even when retained in only a few layers, full attention still incurs KV caches that grow linearly with context length, posing a major memory bottleneck for long-context inference. KV-cache eviction provides a complementary way to alleviate this memory pressure, but existing methods are primarily designed for attention-only architectures and do not fully account for the distinct information pathways introduced by hybrid models. We identify two architecture-specific characteristics: the remaining full-attention layers exhibit less concentrated attention, and the linear pathway provides complementary token-level information beyond that available from the full-attention pathway. Motivated by these observations, we introduce HybriKV, a hybrid-aware KV-cache eviction method. To account for the less concentrated attention in the full-attention pathway, HybriKV aggregates evidence from multiple salient responses rather than relying primarily on the strongest one. To exploit complementary token-level information from the linear pathway, it reserves a small portion of the cache budget for positions highlighted by linear-layer signals. Across RULER, LongBench, and SCBench, HybriKV achieves near-lossless performance under approximately - KV-cache compression and reduces decoding-time attention latency by 45% compared with FlashAttention. It also consistently outperforms existing KV-eviction methods across hybrid models ranging from 7B to 48B parameters. Code is available at https://anonymous.4open.science/r/ICLR-Anonymous-397E.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hybrid Linear-Attention Models, KV Cache Eviction, Long-Context Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_env4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses an important practical problem in long-context inference: full-attention KV caches can remain a substantial memory bottleneck even when much of a model uses linear attention. Targeting this bottleneck is relevant to realizing the memory benefits of hybrid architectures. The proposed approach operates through cache selection without retraining the backbone, and its combination of reconstruction responses and linear-layer activations offers a concrete attempt to adapt eviction to these models. The reported improvements in several evaluated settings provide preliminary evidence of the combined policy’s practical value. Evaluation across multiple architectures and task families offers useful observations about its behavior under different conditions. In addition, the learned-gate variant and efficiency measurements reflect attention to the computational overhead of making eviction decisions, an important consideration for practical deployment. This inclusion of quality and computational-cost evaluations helps readers assess the approach as an inference method.

Critical strengths

The paper addresses a practically relevant problem: the memory cost of full-attention KV caches in hybrid linear–attention models. Its focus on cache selection without retraining the backbone makes the approach relevant to long-context deployment. The reported improvements in several evaluated settings provide preliminary evidence of the combined policy’s practical value. The presentation separates the motivation, method, and evaluation, and distinguishes the main procedure from its learned-gate variant. Qualitative examples provide an accessible illustration of the intended token-selection behavior. The appendix also collects pseudocode, hyperparameter settings, and detailed benchmark results, making these supporting materials convenient to consult. In addition, the learned-gate variant and efficiency measurements demonstrate attention to the computational overhead of cache management, extending the evaluation beyond prediction quality to practical inference costs.

Critical weaknesses

The paper’s main weaknesses are its insufficiently substantiated hybrid-specific rationale and an evaluation that does not fully establish the proposed components’ added value. Figure 1 shows greater query-wise attention dispersion, but this does not directly justify MRA’s aggregation across queries. Similarly, low linear-layer output norms do not establish that the corresponding tokens require protection in full attention, since output magnitude depends on the current query, recurrent state, and projections. The claimed connection between these proxies and eviction-induced degradation therefore remains unclear. Although the paper includes some ablations, LATP lacks matched random or uniform protection controls, leaving open whether its gains reflect complementary memory or simply a generic retention safeguard. Complete MRA × LATP comparisons are also missing on Olmo and Kimi, and the absence of KVzip+ leaves competitiveness against a stronger reconstruction-based scoring baseline unresolved. The breadth of the reported benefits also requires qualification: approximately 70% of the average improvement on Olmo/RULER comes from MK3 alone, while SCBench performance drops from 54.01 to 50.42 at a 70% eviction rate, limiting the generality of the “near-lossless” claim. These limitations materially weaken confidence in both the proposed mechanistic explanation and the generality of the empirical advantage.

Questions for authors

The following six issues are the most consequential for the review. Item 5 is a verifiable reporting error; the others are substantive gaps in the reasoning or experimental evidence.

  1. LATP’s central premise remains unsubstantiated: linear-layer output norms do not establish memory importance. Section 4.3 protects tokens with low output norms, but these norms depend on the current query, recurrent state, and projections. They do not directly indicate which information requires compensation from full attention. Without comparisons against random protection, uniform retention, and structural-token protection, the gains could simply come from reserving additional token positions rather than identifying “complementary memory.”
  2. There is a logical gap between MRA’s motivation and its scoring mechanism. Figure 1 shows that attention distributions are more diffuse for individual queries, whereas MRA changes how each cached token’s responses are aggregated across queries. Higher row-wise entropy does not directly justify replacing column-wise maximum aggregation with a Top-K mean. The analysis neither establishes this connection nor controls for confounders such as model size and training differences.
  3. The mathematical specification does not fully define the actual cache operations, leaving budget fairness difficult to verify. Equations 7–8 and Algorithm 1 do not sufficiently specify candidate sets, integer budgets, or how additionally retained tokens are deduplicated and counted. The position set PP and the tuple set RR also cannot be directly combined by a set union as written. Moreover, the comparability of raw scores used for selection across layers and heads is unverified. These ambiguities affect how much cache is retained, where it is allocated, and whether competing methods use equivalent budgets. They establish an incomplete specification, not a demonstrated implementation bug.
  4. The experiments do not adequately attribute the gains to the proposed mechanisms. Although the paper includes some ablations, it lacks a complete MRA × LATP factorial comparison on Olmo and Kimi, as well as comparisons with important scoring baselines such as KVzip+. The Fast variant also changes both the scoring target and the protection mechanism, requiring separate comparisons under matched training conditions. The results support the effectiveness of the combined method, but do not sufficiently establish the necessity or contribution of each component.
  5. Table 1 contains directly verifiable category-reporting errors. Several Single-Doc QA and Multi-Doc QA entries are inconsistent with Table 6. For example, Qwen-HybriKV should report 50.95 for Single-Doc QA and 45.84 for Multi-Doc QA, but Table 1 reverses these values. Several Olmo rows exhibit the same issue. Because the reversal does not affect every row, swapping the column headers would not resolve the problem; the entries require individual verification against the original results.
  6. Key claims exceed what the results support. On Olmo/RULER, approximately 70% of the 8.03-point average improvement comes from a single subtask, MK3, which does not support the claim that the improvement is “not driven by a single category.” On SCBench, performance decreases from 54.01 to 50.42 at a 70% eviction rate, so “near-lossless” requires explicit qualification. These results remain useful, but do not establish consistently broad, near-lossless improvements.
Reviewer_7RQxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HybriKV, a query-agnostic KV-cache eviction method for hybrid linear-attention language models. It introduces Mixed Reconstruction Aggregation to combine multiple salient reconstruction responses and Linear-Aware Token Protection to preserve tokens selected using linear-layer output norms. Experiments on RULER, LongBench, and SCBench across several hybrid models demonstrate consistent improvements over existing KV-eviction methods.

Critical strengths
  1. The paper addresses an important memory bottleneck in hybrid language models. HybriKV consistently improves upon KVzip, particularly under aggressive cache eviction. For example, it improves the RULER score from 76.72 to 84.75 on Olmo-Hybrid-7B at 70% eviction (Table 1).

  2. The authors identify differences in attention concentration and information pathways between hybrid and attention-only models, providing a reasonable motivation for adapting existing eviction strategies.

  3. The evaluation covers multiple architectures, benchmarks, and compression ratios. The component ablations in Figure 3 and the learned-gate extension in Table 3 further support the effectiveness of the proposed method.

Critical weaknesses
  1. MRA is essentially a weighted combination of maximum and top-k mean aggregation, while LATP uses a simple norm-based token protection heuristic. Although both components are effective, the proposed method appears to be an incremental extension of reconstruction-based eviction. The paper should better establish the novelty and necessity of these particular designs.

  2. Figure 1(c) shows that reconstruction-based eviction does not preferentially select tokens with extreme linear-output norms, but this does not necessarily establish their importance for full-attention KV retention. Although Figure 3 demonstrates the effectiveness of LATP, comparisons against random protection and simple position-based protection under identical budgets are missing. These controls would help determine whether the improvements originate from the linear-aware signal or simply from protecting a shared subset of tokens.

  3. Figure 4 reports context-preparation overhead and average full-attention-layer latency, but not complete generation latency or throughput. Since hybrid models spend considerable computation outside their full-attention layers, the reported attention speedup may not translate into comparable end-to-end acceleration. The practical benefit of reconstruction-based eviction also depends on how frequently the compressed cache is reused.

  4. Appendix B.2 indicates that LATP provides limited benefits on the sliding-window hybrid model, while MRA accounts for most of the improvement. This suggests that the effectiveness of LATP may depend strongly on the architecture. Moreover, the default hyperparameters appear to be motivated by performance on the evaluated benchmarks, and the sensitivity to the top-k response fraction is not reported.

Questions for authors
  1. How does LATP compare with random or position-based token protection under the same cache budget and cross-head reservation strategy?

  2. Can the authors report end-to-end generation latency or throughput, including the overhead of reconstruction-based eviction?

  3. Were the default values of α_max, β, and the top-k response fraction selected using a separate validation set? How robust are these settings across different models?

  4. Could the authors clarify why LATP provides limited improvement on the sliding-window hybrid model, and whether MRA alone also benefits attention-only models?

Reviewer_7ZemOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents HybriKV, a KV cache eviction framework designed for hybrid linear/full-attention language models. The authors observe two architecture-specific characteristics: the remaining full-attention layers in hybrid models tend to exhibit less concentrated attention distributions, and the linear-attention pathway provides complementary token-level signals that are not directly captured by conventional attention-based eviction criteria.

Motivated by these observations, HybriKV introduces two components. First, Mixed Reconstruction Aggregation (MRA) combines the maximum reconstruction response with the mean of multiple salient responses to obtain more robust KV importance estimates. Second, Linear-Aware Token Protection (LATP) uses the output norms of linear-attention layers to identify and explicitly preserve a small subset of token positions, while allocating the remaining cache budget according to reconstruction-based importance scores.

The authors evaluate HybriKV on several hybrid architectures, including Qwen3.5-9B, Olmo-Hybrid-7B, and Kimi-Linear-48B-A3B, across RULER, LongBench, and SCBench. The results show consistent improvements over existing KV eviction baselines, particularly under aggressive compression. The paper further introduces Fast HybriKV, which combines the proposed retention strategy with learned importance gates to reduce reconstruction overhead and support decoding-time eviction.

Critical strengths

S1. Relevant motivation and meaningful novelty.

The paper addresses a timely and practically relevant problem by revisiting KV cache eviction for modern hybrid linear/full-attention architectures, rather than focusing exclusively on conventional attention-only models. The observation that the linear pathway can provide complementary information for KV retention is interesting, and the proposed hybrid-aware design offers a meaningful extension to existing reconstruction-based eviction methods.

S2. Comprehensive evaluation across diverse architectures and model scales.

The experimental evaluation is relatively comprehensive, covering different hybrid attention architectures, model sizes ranging from 7B to 48B, and several long-context benchmarks. The authors also provide component-level ablations, longer-context evaluations, and a learned-gate extension to reduce reconstruction overhead. These experiments provide useful evidence for the effectiveness and generality of the proposed method.

Critical weaknesses

W1. Limited evidence of end-to-end practical benefits in realistic serving scenarios.

Although HybriKV demonstrates promising KV-cache compression and attention-level latency improvements, the evaluation does not fully establish its practical benefits in modern LLM serving systems. Since hybrid linear/full-attention architectures already substantially reduce the KV-cache footprint, it remains unclear how much additional end-to-end benefit eviction can provide after accounting for reconstruction or gate-prediction overhead, particularly under continuous batching and concurrent requests. The reported attention-kernel improvements do not necessarily translate directly into proportional serving-level speedups.

Questions for authors

Q1. Practical serving benefits.

Given that hybrid architectures already reduce the proportion of full-attention layers, could the authors further quantify the incremental benefits of HybriKV under realistic serving workloads? In particular, how would HybriKV affect end-to-end token latency, request throughput, and GPU memory utilization under continuous batching, compared with full-cache inference and existing eviction methods?

Q2. Applicability to agentic workloads.

The paper includes SCBench evaluations, which partially cover shared-context and multi-task settings. However, modern agentic workloads often involve long-running interactions, dynamically evolving contexts, tool outputs, and changing information requirements. Could the authors discuss or evaluate whether HybriKV remains effective in such settings, particularly when information that appears unimportant during eviction becomes relevant in subsequent reasoning or tool-use steps?

Q3. Interpretation of linear-pathway signals.

The proposed Linear-Aware Token Protection relies on low-norm linear-attention outputs, but the relationship between these signals and the importance of the corresponding full-attention KV entries is not entirely clear. Could the authors provide further analysis of why low-norm positions are particularly valuable, and whether the improvements persist when compared with random or structure-aware token protection under the same budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.