acceptodds
Withdrawn from ICLR 2027

Beyond Linear Surrogates: Softmax-Aware Information Capacity for KV Cache Eviction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models (LLMs). However, existing policies predominantly rely on empirical heuristics, lacking a principled characterization of token utility under the inherently nonlinear softmax attention mechanism. In this work, we revisit KV cache eviction through the lens of local information geometry, modeling attention as a nonlinear Gaussian communication channel. By locally linearizing the attention mapping through a first-order Taylor expansion, we derive the Jacobian Information Capacity, an information-theoretic objective that jointly captures query relevance, softmax sensitivity, and structural diversity. Guided by this formulation, we introduce Jacap, a capacity-aware eviction method that combines softmax-aware importance weighting with statistical leverage scores for subset selection. Extensive experiments across diverse model architectures and benchmarks show that Jacap strong and consistent improvements over existing eviction methods across most settings, particularly under high compression ratios.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV-Cache Eviction, KV-Cache, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fKMfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors argue that existing KV-cache eviction policies are mostly heuristic, while the Information-Bottleneck-based CapKV relies on a linear surrogate that ignores softmax nonlinearity. To this end, they propose JaCap, which analyzes KV-cache eviction from the perspective of local information geometry: by applying a first-order Taylor expansion to softmax attention, they derive the Jacobian Information Capacity and extract from it the three ingredients most relevant to token selection, namely local query relevance, softmax sensitivity, and output-space diversity, which are then used to score and evict tokens. Experiments show that on LongBench across three models, JaCap outperforms the strongest baseline by about 5 points on average at a compression ratio of 0.9, and remains competitive on NIAH and on online decoding with AIME25.

Critical strengths
  1. The paper provides an information-theoretic derivation of KV-cache eviction that accounts for the softmax nonlinearity overlooked by prior methods, giving the method a clear theoretical motivation.
  2. Experimental results show that the method performs well, especially under high compression ratios.
Critical weaknesses
  1. The derivation relies on many assumptions and approximations that may not hold in practice, and the paper does not verify whether these assumptions are reasonable.
  2. Limited model coverage. The main experiments involve only two model families (Qwen3 and Llama3.1) at 8B–14B scale, lacking validation on larger models as well as on newer attention mechanisms and architectures (e.g., hybrid models, Multi-head Latent Attention (MLA), and DeepSeek Sparse Attention (DSA)).
  3. Limited datasets. The main experiments are conducted almost entirely on LongBench, while NIAH and AIME25 are each evaluated on only a single model, and evaluations on a broader range of benchmarks are missing.
  4. Insufficient efficiency evaluation. As an efficiency-oriented method, the paper only reports the total time for generating 100 tokens in the appendix, without reporting the overhead of the eviction procedure itself, memory usage, or metrics such as TTFT and TPOT.
Questions for authors
  1. How does JaCap perform on larger models, as well as on hybrid models or models with newer attention mechanisms such as MLA and DSA?
  2. How does JaCap perform on more benchmarks?
  3. Could the authors report the time overhead of the eviction procedure itself, memory usage, and metrics such as TTFT and TPOT?
Reviewer_XZRqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper treats KV cache eviction as an information bottleneck problem. Specifically, authors define an output mapping function as in Eq. (3), and assume that the queries follow a Gaussian distribution. By applying a first-order Taylor approximation around this mean and evaluating the mutual information, they derive the Jacobian Information Capacity, which incorporates the Jacobian matrix and noise covariance terms. To reduce the computational overhead, the authors introduce several approximations, such as pre-eviction attention score, diagonal sensitivity, and diagonal query response covariance. In experiments, Jacap achieves consistently better performance across the evaluated baseline methods.

Critical strengths

Jacap extends CapKV by reflecting the nonlinear mapping function into an information bottleneck formulation and introducing practical approximations that achieve promising empirical results.

Critical weaknesses
  • The major concern with this paper is that the baseline methods are outdated. The experiments are conducted under kvpress frameworks, but the compared methods exhibit poor performance in the kvpress leaderboard (https://huggingface.co/spaces/nvidia/kvpress-leaderboard). The paper should compare their methods with SOTA methods, such as KVzip, KVzap, KVgrad, and RestoreKV. The current SOTA methods are almost lossless until the compression ratio is 0.875. Achieving SOTA is not necessary, but in this case, the novelty of extending CapKV seems to be not enough to overcome the performance gap.
  • The approximation of the KV cache eviction function introduced in L152 does not precisely capture the influence on the final model output, as it focuses on the local layer output instead. This locally obtained importance may be flipped through the successive layers.
Questions for authors
  • L41 claims that “two individually important tokens may carry highly redundant information”. This is a key limitation of token-wise importance scores and one of the major motivations for IB-based methods. Is there any experimental analysis demonstrating that Jacap effectively excludes redundant KVs?

  • The results in Figure 4 are suspicious. In my experience, the eviction time of KeyDiff, EA, etc., is around 1% compared to the prefill time. Can you separately measure and present the prefill, scoring, and decoding time?

  • Could you conduct experiments on the RULER dataset to compare Jacap with the KVpress leaderboard?

  • L310 mentions that µ_Q and Λ_Q are estimated from historical queries observed during decoding. Does this mean that the method requires several decoding steps with the full KV cache before eviction can be performed? If so, how stable are these estimates, and how sensitive is the method's performance to the number of observed queries?

  • Is this method comparable to LLM inference engines, such as vLLM or SGLang? Specifically, can it be integrated with paged attention, continuous batching, and radix attention, without requiring substantial modifications to the serving infrastructure?

  • Typos in L240: Eq. equation 2

Reviewer_Ajf4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JACAP, a Jacobian-capacity-aware KV-cache eviction method that extends the information-theoretic formulation of CapKV from a linear-Gaussian attention surrogate to a locally linearized nonlinear softmax-attention model.

The authors formulate attention as a nonlinear communication channel from future queries to attention outputs. By applying a first-order Taylor expansion around the mean of a Gaussian query distribution, they derive a local Jacobian Information Capacity objective involving query statistics, key representations, value-induced output directions, and the softmax Jacobian.

Since directly optimizing the resulting subset-dependent log-determinant objective is computationally expensive, the authors introduce several approximations. Specifically, they replace the full softmax Jacobian and query-response covariance with diagonal approximations, construct softmax-sensitive token weights, and combine these weights with statistical leverage scores to obtain a practical one-shot eviction policy.

The method is evaluated on LongBench using Qwen3-8B, Qwen3-14B, and Llama-3.1-8B, on Needle-in-a-Haystack retrieval tasks, and on AIME25 using Nemotron-7B. The reported results show competitive performance against CapKV and several heuristic eviction baselines, with larger improvements in some high-compression settings. The paper also includes component ablations, comparisons with a full-Jacobian variant, temperature sensitivity experiments, and runtime measurements.

Overall, the paper addresses an important problem and presents a potentially useful extension of capacity-based KV-cache eviction. However, the practical scoring rule is obtained through approximations that substantially weaken its connection to the original nonlinear information-capacity objective. Several theoretical and empirical questions therefore remain unresolved.

Critical strengths

S1. An interesting extension of information-theoretic KV-cache eviction.

The paper extends the linear-Gaussian formulation of CapKV by explicitly considering the nonlinear softmax attention mechanism.

The introduction of the softmax Jacobian into the local information-capacity objective offers a potentially useful perspective on the interaction between query relevance, attention sensitivity, and output-space diversity.

S2. A mathematically clear local Jacobian formulation.

The derivation of the attention Jacobian is straightforward and technically reasonable under the stated smoothness and local Gaussian assumptions.

The resulting formulation explicitly incorporates the softmax competition matrix, which is absent from the linear attention surrogate used by CapKV.

This formulation provides a useful starting point for investigating how local changes in future queries affect attention outputs under a restricted KV cache.

S3. Encouraging performance under aggressive cache compression.

The LongBench experiments cover three model configurations and four compression ratios.

At a compression ratio of 0.9, JACAP achieves average scores of 40.91, 44.59, and 35.98 on Qwen3-8B, Qwen3-14B, and Llama-3.1-8B, respectively.

These results outperform CapKV by 4.34, 5.06, and 5.50 absolute points, suggesting that softmax-aware importance weighting may be particularly beneficial when the available KV-cache budget is severely constrained.

S4. Useful component ablations.

The paper examines the contributions of softmax sensitivity, query covariance, and leverage-score-based diversity estimation.

At a compression ratio of 0.9 on Qwen3-8B, removing the sensitivity term reduces the reported average score from 40.69 to 33.67, while removing leverage scoring reduces it to 32.15.

The comparison with a full-Jacobian variant also provides evidence that modeling additional softmax interactions can improve retention quality, although at greater computational cost.

S5. Evaluation across several types of inference workloads.

In addition to LongBench, the authors include Needle-in-a-Haystack retrieval experiments and online decoding evaluations on AIME25.

These experiments provide useful preliminary evidence that the method may generalize beyond standard prefill-time compression.

S6. Practical implementation within an existing KV-cache compression framework.

The proposed method is implemented using KVPress and does not require retraining the backbone model.

The paper also reports runtime measurements over multiple context lengths, indicating that the additional matrix computations can be accommodated within the evaluated inference configurations.

Critical weaknesses

W1. The practical JACAP score is not sufficiently justified as an approximation to the proposed Jacobian Information Capacity.

The central theoretical contribution is the exact softmax Jacobian, which contains both diagonal sensitivity terms and off-diagonal competition terms.

However, the practical algorithm replaces this matrix with its diagonal approximation and further diagonalizes the query-response covariance. These approximations remove the cross-token interactions that are central to the motivation for introducing the nonlinear softmax formulation.

More importantly, the resulting approximation does not necessarily preserve fundamental properties of the exact Jacobian. For example, when all retained value-induced output vectors are identical, the attention output is independent of the query, and the exact Jacobian Information Capacity is zero. Nevertheless, the diagonal approximation can assign positive capacity contributions to individual tokens.

This counterexample suggests that the practical score is not uniformly faithful to the proposed information-capacity objective.

The authors should clarify the assumptions under which these approximations are valid and characterize their approximation errors. A direct empirical comparison between the proposed proxy and the exact local capacity on small candidate subsets would be particularly helpful.

W2. Maximizing retained information capacity is not necessarily equivalent to preserving the original attention computation.

Section 3.1 formulates KV-cache eviction as minimizing the expected discrepancy between full-cache and compressed-cache attention outputs.

However, the subsequent information-theoretic formulation maximizes the mutual information between future queries and the output of the retained attention channel.

These objectives are not generally equivalent. A retained subset may induce an attention mapping with high sensitivity to query variations, and therefore high local information capacity, while producing outputs that deviate substantially from those of the original full-cache model.

The paper currently provides insufficient theoretical or empirical evidence connecting the proposed capacity objective to the original attention-output approximation problem.

I would encourage the authors to investigate whether maximizing Jacobian Information Capacity correlates with lower attention-output reconstruction error, reduced output perturbation, or improved downstream accuracy across different retained subsets.

W3. The leverage-score selection procedure is not equivalent to greedy optimization of the subset-dependent objective.

The paper describes its selection procedure as being motivated by marginal contributions to a log-determinant objective.

However, JACAP constructs a single capacity matrix from the full candidate pool and ranks all tokens using fixed leverage scores computed from that matrix.

This differs from greedy log-determinant maximization, where marginal gains are recomputed after each selection using the currently selected subset.

Consequently, the proposed procedure should be characterized as a one-shot leverage-score heuristic rather than direct greedy optimization of the original subset objective.

The authors should clarify the relationship between full-pool leverage scores and true subset-dependent marginal gains. Comparisons with iterative greedy selection or approximate marginal-gain updates would help establish whether the proposed approximation preserves the intended diversity objective.

W4. The experimental evaluation lacks several strong recent KV-cache eviction baselines.

The main LongBench experiments compare JACAP primarily against CapKV, Expected Attention, KeyDiff, and KNorm.

However, stronger recent eviction methods such as CriticalKV and DefensiveKV are not included. These methods specifically investigate the relationship between token importance and attention-output preservation and are particularly relevant to the paper's proposed theoretical motivation.

Additionally, although SnapKV is mentioned among the experimental baselines and included in the NIAH evaluation, it does not appear in the main LongBench comparison table.

The absence of these baselines makes it difficult to assess the competitiveness of JACAP against the broader state of the art.

I would encourage the authors to include at least CriticalKV and DefensiveKV under matched retention budgets, model configurations, and evaluation protocols.

W5. The empirical superiority over CapKV is not consistent across compression ratios, and some claims are overstated.

The strongest improvements are concentrated at aggressive compression ratios.

At a compression ratio of 0.25, JACAP underperforms CapKV on all three evaluated models. Its advantage is also relatively small or inconsistent at a compression ratio of 0.5.

Furthermore, Section 4.1 states that JACAP exceeds the strongest baseline by approximately five points on average at a compression ratio of 0.9.

However, comparing against the strongest baseline for each model in Table 1 yields improvements of approximately 4.34, 5.06, and 1.09 points, corresponding to an average improvement of approximately 3.50 points.

The paper should distinguish improvements over CapKV from improvements over the strongest available baseline and avoid overstating the consistency of its results.

W6. Limited statistical validation and insufficiently isolated efficiency analysis.

The AIME25 evaluation contains only 30 problems, making the reported accuracy differences sensitive to a small number of questions.

For example, at a budget of 8192 tokens, JACAP achieves 0.53 accuracy compared with 0.73 for CapKV, a substantial difference that is not investigated further.

The paper does not provide sufficient multi-seed evaluation or uncertainty estimates to establish the robustness of the reported reasoning improvements.

The runtime evaluation also reports total generation time for 100 tokens, without separately measuring eviction-score computation, cache-update overhead, prefill latency, and decoding latency.

Since the proposed algorithm has a complexity of O(Nd_h² + d_h³) per eviction step, compared with the lower theoretical costs of several heuristic baselines, a more detailed efficiency breakdown is needed to establish its practical tradeoffs.

W7. The impact of the query-distribution approximation and temperature scaling remains insufficiently understood.

The theoretical formulation assumes a local Gaussian distribution of future queries, while the implementation estimates its statistics from historical queries.

The paper does not sufficiently investigate how accurately these statistics represent the future query distribution, particularly under distribution shifts or long reasoning trajectories.

Moreover, the practical method introduces a temperature parameter into the attention prior, with the reported optimum around τ=10. This substantially changes the attention distribution relative to the original softmax mechanism used in the theoretical derivation.

It remains unclear whether the observed gains primarily come from the proposed Jacobian-based sensitivity interpretation or from empirically smoothing the attention prior.

Additional controls using temperature-scaled CapKV, alternative query priors, and different covariance estimators would help clarify the mechanism.

Questions for authors

Q1. Under what conditions does the practical JACAP score faithfully approximate the exact Jacobian Information Capacity?

The proposed algorithm diagonalizes both the softmax Jacobian and the query-response covariance.

Could the authors provide approximation-error bounds or sufficient conditions under which these simplifications preserve the ranking of candidate subsets?

In particular, how does the proposed approximation behave in degenerate cases where the exact attention Jacobian is zero, such as identical output-value directions or identical key vectors?

Q2. What is the theoretical relationship between Jacobian Information Capacity and full-attention output preservation?

The original cache-eviction objective minimizes the discrepancy between full and compressed attention outputs, whereas the proposed capacity objective maximizes information transmission through the retained attention mapping.

Could the authors clarify under what assumptions these objectives are aligned?

An empirical analysis relating exact or approximate Jacobian capacity to attention-output reconstruction error and downstream performance would be valuable.

Q3. Why is one-shot full-pool leverage scoring an appropriate approximation to subset-dependent log-determinant maximization?

The proposed algorithm computes leverage scores using a capacity matrix constructed from all candidate tokens, rather than updating marginal gains after each token selection.

Could the authors compare this approach with greedy log-determinant maximization or a computationally efficient iterative alternative?

It would also be helpful to report the actual log-determinant objective values achieved by the selected subsets.

Q4. How does JACAP compare with stronger recent KV-cache eviction methods?

Could the authors add evaluations against CriticalKV and DefensiveKV under identical model configurations, compression ratios, and evaluation protocols?

Additionally, could the authors include SnapKV in the main LongBench comparison for completeness?

These comparisons are important for positioning the method within the current KV-cache eviction literature.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.