acceptodds
Under review as a conference paper at ICLR 2027

TridentKV: Optimizing KV Cache Eviction with Diversity-Aware Budget Allocation and Spatial-Temporal Score Aggregation

18 views

Abstract

Long-context inference in large language models (LLMs) is increasingly bottlenecked by the growing memory and computation cost of the key-value (KV) cache. While KV cache eviction provides a training-free solution, existing methods typically optimize cache allocation or importance aggregation without explicitly accounting for behavioral diversity across layers and attention heads or localized variations in token importance. We introduce TridentKV, a training-free KV cache eviction framework that improves cache utilization through diversity-aware budget allocation and spatial-temporal score aggregation. TridentKV consists of three complementary components: (1) uniqueness-based layer budget allocation, which assigns larger cache budgets to layers exhibiting distinctive attention behaviors; (2) behavior-guided head budget allocation, which clusters attention heads by behavioral similarity and restricts budget competition within each cluster to preserve diverse attention patterns; and (3) spatial-temporal token score aggregation, which jointly captures importance variations across head clusters and query window to preserve locally salient tokens. Experiments on LongBench, RULER, and Needle-in-a-Haystack across Llama, Mistral, and Qwen models show that TridentKV robustly preserves long-context performance under constrained cache budgets. With a cache budget of 512, it retains over 99.6% of full-cache performance on LongBench and outperforms the strongest evaluated baselines on 32K RULER by 2.7%–5.4%, while achieving up to 2.76 decoding speedup and a 36.2% reduction in peak memory at 128K context length. We provide our code in the supplementary material to facilitate straightforward reproduction of our method.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long context, Efficient decoding, KV cache compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qJ83OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TridentKV, a training-free KV cache eviction framework for efficient long-context LLM inference. The method combines uniqueness-based layer budget allocation, behavior-guided head budget allocation, and spatial-temporal token score aggregation. It aims to preserve diverse attention behaviors while improving token retention under constrained cache budgets. Experiments on LongBench, RULER, and Needle-in-a-Haystack across Llama, Mistral, and Qwen models demonstrate competitive performance compared with existing KV cache eviction methods. The paper also reports decoding acceleration and memory savings under long-context settings.

Critical strengths
  1. The paper addresses an important practical problem in long-context LLM inference and proposes a training-free solution.
  2. The method combines layer-level and head-level budget allocation with spatial-temporal token importance aggregation.
  3. The experiments cover multiple LLMs and benchmarks, with competitive results and ablation studies supporting the proposed components.
Critical weaknesses
  1. The proposed budget allocation relies on attention-pattern diversity, but its relationship with the functional importance of individual attention heads is not sufficiently established.
  2. Although prefill and decoding latency are reported separately, the end-to-end inference benefits over existing KV eviction methods remain unclear.
  3. The reported LongBench results differ between Table 1 and Table 10 despite using the same stated cache budget and model configurations.
Questions for authors
  1. Could the authors provide a small-scale analysis of whether attention-pattern similarity reflects the functional importance of attention heads, for example, through head-level output perturbation measurements?
  2. Could the authors clarify whether the additional prefill overhead affects the overall inference efficiency compared with existing KV eviction methods?
  3. Could the authors explain the differences in the LongBench results between Table 1 and Table 10 under the same stated experimental settings?
Reviewer_xpaaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TridentKV, a training-free KV-cache eviction framework that jointly optimizes layer-level budget allocation, head-level budget allocation, and token importance aggregation. It uses cross-layer attention dissimilarity to allocate cache budgets, behavioral clustering to preserve diverse attention heads, and spatial-temporal pooling to improve token selection. Experiments on LongBench, RULER, and Needle-in-a-Haystack demonstrate competitive performance across several language models.

Critical strengths
  1. The paper addresses an important challenge in long-context LLM inference. TridentKV consistently outperforms several representative KV-eviction methods. For example, at a budget of 512 entries per KV head on average, it improves RULER performance over DefensiveKV from 80.08 to 82.27 on Llama and from 82.87 to 85.19 on Qwen (Table 2).

  2. Considering behavioral diversity across layers and attention heads is an interesting perspective. The three components are reasonably motivated and organized into a coherent cache allocation and eviction framework.

  3. The experiments cover multiple models, benchmarks, and cache budgets. Table 3 provides useful ablations of the individual components, clustering strategies, uniqueness measures, and pooling configurations.

Critical weaknesses
  1. Eq. (3) directly compares attention tensors across layers using Frobenius similarity, implicitly assuming that heads with the same indices across different layers are comparable. However, attention-head indices do not necessarily have consistent semantic meanings across layers. A function-preserving permutation of attention heads could change the resulting uniqueness scores and budget allocation. The authors should justify this assumption or evaluate a permutation-invariant alternative.

  2. Algorithm 1 requires attention information from all layers before computing their cache budgets, whereas Section 4.1 describes eviction during layer-wise prefilling. It is unclear how earlier layers can be compressed before the global allocation is determined. The authors should clarify whether full KV caches must be retained temporarily, whether additional computation is required, and whether these costs are included in the reported peak memory and prefill latency in Figure 7.

  3. The framework combines attention similarity, head clustering, and 2D max pooling, all of which are relatively established techniques. While the combination is effective, the paper could better explain the distinctive methodological contribution beyond integrating these existing strategies for KV-cache eviction.

  4. Although Figure 7 reports promising efficiency results, the relationship between the global allocation procedure and the measured memory overhead remains unclear. Moreover, the improvements over the strongest baselines are relatively modest on some benchmarks. Additional analysis of longer contexts and performance variability would help establish the robustness of the reported gains.

Questions for authors
  1. How robust are the uniqueness scores and budget allocations in Eq. (3) to function-preserving permutations of attention heads across layers? Could a permutation-invariant similarity measure address this issue?

  2. Could the authors clarify the exact execution order of global budget allocation and layer-wise eviction? Does the efficiency evaluation in Figure 7 account for any additional cache storage, computation, or recomputation required by this procedure?

  3. Could the authors provide additional evidence of performance robustness at longer context lengths and clarify whether the improvements over the strongest baselines are consistent across individual tasks?

Reviewer_c2rEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents TridentKV, a training-free KV cache eviction framework that improves cache utilization through diversity-aware budget allocation and spatial-temporal token importance aggregation. The authors identify two limitations of existing approaches: insufficient consideration of behavioral diversity across layers and attention heads during budget allocation, and the potential loss of locally salient importance signals during score aggregation.

TridentKV introduces three complementary components: (1) uniqueness-based layer budget allocation, which prioritizes layers with distinctive attention patterns; (2) behavior-guided head budget allocation, which clusters heads according to attention similarity and restricts budget competition within clusters; and (3) spatial-temporal token score aggregation, which applies cluster-local 2D max pooling to preserve locally salient importance signals.

Experiments across three model families (Llama, Mistral, and Qwen) on LongBench, RULER, and Needle-in-a-Haystack demonstrate consistent improvements over existing KV cache eviction methods under constrained budgets. The method also preserves near-full-cache performance on LongBench while reducing decoding latency and memory consumption compared with full-cache inference.

Critical strengths

S1. Clear motivation and excellent presentation.

The paper is well motivated and clearly presented. The authors provide intuitive observations and visualizations to illustrate the limitations of existing budget allocation and score aggregation strategies. In particular, the connection between behavioral diversity and cache allocation is explained convincingly. The overall methodology is well organized, and the figures effectively communicate the motivation and design of the three proposed components, making the paper easy to follow.

S2. Comprehensive experimental evaluation.

The experimental evaluation is comprehensive and well organized. The authors evaluate TridentKV across multiple model families, benchmarks, and cache budget settings, with comparisons against a broad set of representative baselines. The extensive ablation studies further examine the contributions of individual components and key design choices. Additionally, the efficiency analysis considers both decoding latency and memory consumption, providing useful evidence of the method's effectiveness in the evaluated settings.

Critical weaknesses

W1. Limited conceptual novelty relative to existing methods.

While the proposed framework is reasonably designed and empirically effective, several of its underlying ideas have already been explored in prior work. For example, CAKE considers spatial-temporal attention characteristics for cache allocation, ReST-KV introduces spatial-temporal smoothing, and DefensiveKV incorporates robust importance aggregation across query positions and shared query heads under GQA. Although TridentKV combines these ideas differently, the conceptual distinction and additional benefits of its proposed aggregation mechanism are not sufficiently established.

W2. Limited evidence of practical benefits in modern LLM serving scenarios.

The efficiency evaluation focuses on isolated inference with conventional GQA-based models, without examining realistic serving workloads involving continuous batching and concurrent requests. It remains unclear whether the reported latency and memory improvements translate into meaningful serving-level benefits, especially considering the potential overhead of non-uniform cache allocation. Furthermore, the applicability and incremental benefits of TridentKV for modern sparse-attention and hybrid linear/full-attention architectures remain unexplored, limiting the assessment of its broader practical relevance.

Questions for authors

Q1: Could the authors further clarify the key conceptual novelty of TridentKV relative to CAKE, ReST-KV, and DefensiveKV? In particular, what specific benefits arise from cluster-local 2D max pooling compared with existing importance aggregation strategies? A controlled comparison under matched budget allocation and importance scoring settings would help isolate its contribution.

Q2: Could the authors discuss or evaluate TridentKV under realistic serving workloads, such as continuous batching with concurrent requests? How would the method affect request throughput, latency, and actual GPU memory utilization compared with baseline methods under matched workloads?

Q3: Given the increasing adoption of native sparse attention and hybrid linear/full-attention architectures (e.g., DeepSeek-V3.2 or Qwen3.5 series), how broadly applicable is TridentKV to these models?

I would be open to revising my rating upward if the authors can provide convincing clarification or additional evidence addressing these questions, particularly regarding the method's practical serving benefits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.