acceptodds
Under review as a conference paper at ICLR 2027

DISENKV: DISENTANGLING SHARED AND QUERYSPECIFIC EVIDENCE FOR KV CACHE RETENTION

18 views

Abstract

Existing observation-based key-value (KV) cache eviction methods mainly optimize importance scoring, evidence aggregation, or budget allocation while largely taking the observation evidence itself as given. We show that recent queries contain strong shared directional structure rather than independent retention signals. This structure can repeatedly reinforce the same historical tokens and obscure evidence expressed by fewer observations, inducing common-demand bias. We introduce DisenKV, a training-free method built on pre-aggregation query-side evidence disentanglement. Query-specific residuals supply the primary evictionrisk evidence, while raw queries retain shared demand for contextual coverage and late complementary rescue. Leave-one-out projection implements this principle by removing each query’s component along a shared direction estimated from the remaining observations. The raw and residual views are aggregated separately before a one-sided correction supplements residual risk. Across 16 LongBench tasks and four model families from 3.8B to 24B, DisenKV improves the strongest reproduced non-layer-aware baseline on every model at 20% KV retention and closes 32.5% of the remaining average gap to Full Cache. Under an identical global layer-wise allocation rule and total cache budget, Layer-DisenKV again improves all four models and closes about 44% of the corresponding gap. Controlled counterfactual interventions show that amplifying shared query structure systematically increases support-associated retention dominance under raw scoring, whereas leave-one-out residualization suppresses this response. These results support disentangling retention evidence before aggregation as a complementary approach to improving importance scoring and budget allocation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Query Disentanglement, Common-Demand Bias, Observation-Based Eviction, Long-Context Language Models, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BRkbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DisenKV, a training-free KV-cache eviction method designed to address potential biases arising from shared directional structures among observation queries.

The authors observe that recent observation queries exhibit substantial directional similarity. When multiple queries contain similar shared components, conventional attention-based importance estimation may repeatedly reinforce certain historical tokens while underrepresenting distinctive evidence expressed by fewer queries. The authors refer to this phenomenon as common-demand bias.

To address this issue, DisenKV introduces a leave-one-out (LOO) query residualization mechanism that estimates a shared direction from the remaining observation queries and removes its projection from each query. The resulting query-specific residuals provide the primary eviction-risk evidence, while the original queries preserve broadly relevant contextual information and contribute complementary evidence through asymmetric late fusion.

The method is evaluated across four model families ranging from 3.8B to 24B parameters, covering 16 LongBench tasks and additional RULER evaluations. At 20% KV-cache retention, DisenKV consistently improves over DefensiveKV across all four evaluated models. The authors further investigate tighter retention budgets, layer-wise allocation, observation-window sensitivity, component ablations, cross-method transferability, controlled counterfactual interventions, and inference efficiency.

Overall, this paper presents an interesting perspective on KV-cache retention by examining the dependence structure of the observation evidence itself. The proposed approach is lightweight and empirically effective. However, the relationship between shared query directions and semantically redundant evidence, as well as the causal connection between changes in retention scoring and downstream accuracy improvements, requires further clarification.

Critical strengths

This paper proposes DisenKV, a training-free KV-cache eviction method designed to address potential biases arising from shared directional structures among observation queries.

The authors observe that recent observation queries exhibit substantial directional similarity. When multiple queries contain similar shared components, conventional attention-based importance estimation may repeatedly reinforce certain historical tokens while underrepresenting distinctive evidence expressed by fewer queries. The authors refer to this phenomenon as common-demand bias.

To address this issue, DisenKV introduces a leave-one-out (LOO) query residualization mechanism that estimates a shared direction from the remaining observation queries and removes its projection from each query. The resulting query-specific residuals provide the primary eviction-risk evidence, while the original queries preserve broadly relevant contextual information and contribute complementary evidence through asymmetric late fusion.

The method is evaluated across four model families ranging from 3.8B to 24B parameters, covering 16 LongBench tasks and additional RULER evaluations. At 20% KV-cache retention, DisenKV consistently improves over DefensiveKV across all four evaluated models. The authors further investigate tighter retention budgets, layer-wise allocation, observation-window sensitivity, component ablations, cross-method transferability, controlled counterfactual interventions, and inference efficiency.

Overall, this paper presents an interesting perspective on KV-cache retention by examining the dependence structure of the observation evidence itself. The proposed approach is lightweight and empirically effective. However, the relationship between shared query directions and semantically redundant evidence, as well as the causal connection between changes in retention scoring and downstream accuracy improvements, requires further clarification.

Critical weaknesses

W1. The semantic relationship between shared query structure and common-demand bias remains insufficiently established.

The central motivation of this paper is based on the observation that a substantial fraction of observation-query energy aligns with a shared direction estimated from neighboring queries.

However, strong geometric alignment does not necessarily imply semantically redundant or undesirable retention evidence. Shared directional components may also arise from representation anisotropy, biases introduced by query projections, or genuinely useful contextual information repeatedly requested by the task.

Although the authors appropriately acknowledge that geometric alignment is not equivalent to semantic relevance, the paper does not sufficiently establish which shared components actually contribute to incorrect token eviction.

I would encourage the authors to provide additional null controls and investigate the relationship between shared-component strength, retention errors, and downstream generation failures.

W2. The counterfactual experiments do not fully establish a causal connection between scoring bias and downstream performance.

The controlled interventions in Appendix E.6 demonstrate that increasing shared-component strength changes retention-score dominance under raw scoring, while residualization suppresses this response.

This supports the claim that query residualization changes the sensitivity of importance estimation to shared query structure.

However, the interventions are primarily evaluated through retention scores and ranking statistics. They are not propagated through actual KV-cache eviction and subsequent generation to directly evaluate downstream accuracy.

Consequently, the current results do not establish to what extent suppressing common-demand dominance directly reduces the probability of evicting task-relevant tokens.

A stronger evaluation would apply the same interventions to the complete eviction and generation pipeline while measuring relevant-token survival and downstream accuracy.

W3. Comparisons with simpler common-component removal methods are still insufficient.

The paper includes full-window mean and norm-matched raw-query controls, which are useful. However, it remains unclear whether the proposed LOO residualization mechanism provides a substantial advantage over more general representation-centering or decorrelation techniques.

For example, direct query mean-centering, PCA-based dominant-direction removal, and suitable random-direction projection controls would help establish whether the observed improvements specifically arise from the proposed LOO shared-direction estimation or from a more general query debiasing effect.

This distinction is important for assessing the algorithmic novelty of the method, since projection-based removal of common representation components has been extensively studied in prior work.

W4. Generalization across task types and more diverse inference settings requires further validation.

Although LongBench evaluations cover four model families, experiments involving more aggressive retention budgets, RULER, and most mechanism ablations are primarily conducted on Llama-3.1-8B.

Moreover, the performance regression on the RULER CWE task indicates that suppressing shared query evidence can be harmful when repeated or distributed information is genuinely important for the downstream task.

This suggests that query residualization may introduce a tradeoff between preserving distinctive evidence and retaining repeated task-relevant information.

A more systematic analysis of different task categories, particularly those requiring frequency aggregation or distributed evidence, would strengthen the generality of the proposed interpretation.

W5. Statistical robustness and the independent contribution of layer-wise allocation require additional analysis.

Several reported component-level improvements are relatively small. For example, the difference between LOO estimation and the full-window mean is approximately 0.41 points, while the contribution of late correction is approximately 0.32 points on the reported LongBench average.

Without sufficient uncertainty estimates or repeated experiments, it is difficult to determine the robustness of these relatively small differences.

Furthermore, although Layer-DisenKV and Layer-DefensiveKV share the same global allocation rule and total cache budget, changes in importance scores can still alter the resulting allocation across layers.

A comparison with fixed per-layer budgets would help distinguish improvements arising directly from query evidence construction from those associated with changes in layer-wise capacity distribution.

Questions for authors

Q1. How can the authors distinguish semantically redundant shared query components from genuinely useful shared task demand?

Could the authors quantify the relationship between shared-component energy and actual token-eviction errors?

In particular, it would be valuable to analyze whether examples with stronger shared query structure exhibit higher rates of incorrect eviction under raw scoring, and whether residualization preferentially preserves gold-answer-supporting tokens.

Q2. Can the authors provide end-to-end counterfactual interventions connecting scoring bias to downstream performance?

Appendix E.6 demonstrates that manipulating shared-component strength changes retention-score behavior.

Could the authors extend these interventions to actual KV-cache eviction and subsequent generation, reporting gold-token survival rates, attention-output perturbations, and final task accuracy?

Such experiments would help establish whether suppressing common-demand dominance causally improves downstream performance rather than merely modifying scoring statistics.

Q3. Why is LOO projection preferable to simpler query-centering or dominant-direction removal methods?

Although the comparison with full-window mean estimation supports the usefulness of LOO, its advantage over alternative common-component removal techniques remains unclear.

Could the authors compare LOO residualization against direct query mean-centering, PCA-based dominant-component removal, and appropriate random-direction controls under identical scoring and retention budgets?

These experiments would help clarify which aspects of the proposed evidence disentanglement are essential.

Q4. How can the method avoid suppressing useful repeated evidence in tasks requiring distributed information aggregation?

The performance decrease on the RULER CWE task suggests that shared demand can sometimes carry genuinely important task information.

Could the authors further investigate the differences between CWE, multi-document QA, and retrieval-oriented tasks in terms of query geometry and relevant-token survival?

Would it be possible to adapt the residualization strength or raw-residual fusion coefficient based on the structure of the observation queries?

Q5. Can the authors provide additional evidence of statistical robustness and cross-model generalization?

It would be helpful to report results across multiple random seeds and reproduce the key norm-matched, LOO, and counterfactual experiments on at least one additional model family.

For the layer-wise variant, a fixed per-layer budget comparison would also help isolate improvements from query evidence construction versus changes in budget redistribution.

6 comments
AuthorsOpenReview ↗

Response to BRkb (1/5) — Shared Query Structure and Evidence Eviction

Q1. Does shared query structure reflect redundant evidence or useful shared demand?

(1) DisenKV improved coverage of the least-retained gold supporting passage on HotpotQA by 0.78 percentage points on average (95% paired CI [0.58, 0.97]).

(2) Shared-component energy showed little per-example correlation with evidence eviction in either diagnostic. Query alignment alone is therefore insufficient to determine whether the shared evidence is semantically redundant.

We used Llama-3.1-8B at 20% KV retention and measured evidence eviction under matched raw-only scoring. Evidence is defined by gold supporting passages on HotpotQA and target-word occurrences on RULER CWE.

TaskExamplesMean shared-component energySpearman correlation with evidence eviction (95% CI)
HotpotQA20081.94%−0.028 [−0.163, 0.106]
RULER CWE20081.99%0.078 [−0.067, 0.224]

In the default HotpotQA setting, DisenKV changes aggregate supporting-passage KV-slot retention by −0.29 percentage points (95% paired CI [−0.68, 0.07]) relative to matched raw-only. Taking the minimum retention rate among each example's supporting passages and then averaging across examples yields a gain of +0.78 percentage points. These metrics distinguish aggregate retention from coverage of the least-retained supporting passage.

On CWE, target-word KV-slot retention falls from 25.89% with DefensiveKV to 24.37% with DisenKV. This result is consistent with useful shared demand in tasks that require repeated evidence. DisenKV keeps the original queries available for coverage protection and one-sided correction to accommodate such demand. The diagnostics show a change in how evidence is retained, while the relationship between shared energy and individual eviction errors remains unresolved.

AuthorsOpenReview ↗

Response to BRkb (2/5) — End-to-End Shared-Component Intervention

Q2. Does intervening on the shared component affect actual KV eviction and generation?

(1) Controlled interventions establish an effect on retention scores. We extended the intervention to actual KV eviction and answer generation on 200 HotpotQA examples.

(2) Both methods have lower mean F1 point estimates at the higher shared-component strength. Both 95% paired confidence intervals span zero.

In Appendix E.6, we increased the shared-component multiplier γ\gamma from 00 to 22 while holding query norms fixed. The retention-risk gap between high- and low-support tokens increased from 0.50 to 3.26 under raw-query scoring and decreased from 0.73 to 0.19 under residual scoring. Residualization reduced the response slope on all 16 LongBench tasks.

For the end-to-end experiment, we used Llama-3.1-8B at 20% KV retention and increased γ\gamma from 0.50.5 to 1.51.5 only in the primary risk-scoring branch while holding query norms fixed. We then performed KV eviction and generated answers. At γ=1\gamma=1, both methods reproduced their corresponding default retained indices and answers on five validation examples.

MethodSupporting-passage KV-slot retentionAnswer F1F1 change (95% paired CI)
Matched raw-only20.09% → 20.15%55.75 → 51.10−4.65 [−9.73, 0.38]
DisenKV19.84% → 19.85%57.42 → 54.66−2.76 [−6.40, 0.76]

Across examples, the Spearman correlations between changes in supporting-passage retention and F1 were −0.075 for matched raw-only (95% CI [−0.220, 0.065]) and −0.057 for DisenKV (95% CI [−0.179, 0.066]). This experiment measures evidence retention and answer F1 after eviction. Intermediate attention-output changes remain unquantified.

The controlled scoring result establishes that residualization changes sensitivity to shared demand. The end-to-end results leave open whether the methods differ in their F1 response and how much of that response can be attributed to aggregate evidence retention.

AuthorsOpenReview ↗

Response to BRkb (3/5) — LOO, Mean-Centering, PCA, and Random Projection

Q3. How does LOO compare with mean-centering, PCA, and random projection?

(1) Single-direction LOO achieved the highest macro-average in this comparison. It scored 48.08, compared with 47.70 for mean-centering, 46.31 for rank-1 PCA, 46.22 for rank-2 PCA, and 47.01 for random projection.

(2) The leave-one-out direction is estimated separately for each query. Its residual supplies the primary risk score, and the original query supplies coverage protection and one-sided correction.

For each query qiq_i, LOO averages and normalizes the other W−1W-1 observation queries and removes the projection of qiq_i onto the resulting direction. This prevents the query being scored from contributing to its own direction estimate. Estimating the direction requires a mean and normalization.

We fixed Llama-3.1-8B, 20% KV retention, and the rest of the eviction pipeline. Only the query transformation in the primary risk branch changed. Mean-centering subtracts the window mean. The PCA controls fit a centered one- or two-dimensional subspace using the other W−1W-1 queries. The random-direction control uses one fixed seed.

TransformationNrtvQAQasperMF-enHotpot2WikiMQAMusiqueGovRepQMSumMultiNewsTRECTriviaQASAMSumPCountPR-enLccRB-PAvg.
Single-direction LOO29.8641.5751.2056.9741.5028.7731.5723.9024.4668.0091.9443.587.7599.5069.4459.2348.08
PCA rank 127.6937.7248.2850.9938.4428.3728.7523.2923.9766.0092.0844.035.1199.5068.2458.4346.31
PCA rank 228.2136.2446.1752.0535.6329.1928.4323.0824.3466.5091.9243.987.98100.0068.1157.7646.22
Mean-centering30.4643.4249.7656.4141.8228.8128.1324.3824.1169.0091.6643.445.8799.5068.2958.1947.70
Random-direction projection29.4040.2749.6652.4840.9928.7828.5623.1424.8366.0092.0843.597.1999.5067.9257.8347.01

LOO's macro-average advantages over mean-centering, rank-1 PCA, rank-2 PCA, and random projection are +0.38, +1.77, +1.86, and +1.07 points, respectively. These are point estimates for the matched configuration. The comparison supports using a single leave-one-out direction in this setting, with the random-direction result scoped to the tested seed.

Reviewer_vkKfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that observation queries used to score KV entries share a strong common direction (6868–8484% of query energy aligns with a leave-one-out direction), which repeatedly boosts the same tokens ("common-demand bias"). DisenKV removes each query's projection onto the mean direction of the other queries and uses these residual queries as the main eviction-risk signal, while raw queries provide coverage protection and a one-sided late correction. The method is training-free. On 1616 LongBench tasks and four models (3.83.8B–2424B) at 2020% retention, it improves over DefensiveKV by 1.081.08–1.401.40 points, and Layer-DisenKV improves over Layer-DefensiveKV under the same allocator and budget (47.8247.82 to 49.0749.07 on average). The paper also reports RULER results, ablations, observation-window sensitivity, efficiency measurements and mechanism analyses on Llama-3.1-8B.

Critical strengths

S1: The evaluation is broad and well controlled. Gains over the strongest baseline hold on all four models at 2020% retention and under a matched layer-wise comparison with the same allocator and budget, and on Llama-3.1-8B they also hold at 1010% retention and across three window sizes. The ablations separate direction from norm (norm-matched raw queries score 46.9446.94 vs. 48.0848.08) and show the benefit of leave-one-out estimation (47.6747.67 vs. 48.0848.08).

S2: The method is simple, training-free and cheap: it adds 1.81.8–3.53.5% prefill latency over DefensiveKV, and the same residual cue improves all five tested baselines. The paper also reports its own failure case (CWE) with a clear diagnostic.

Critical weaknesses

W1: The gains are modest and lack statistical support. Average improvements are about 11–1.41.4 points, no confidence intervals or significance tests are reported for task scores, and Layer-DisenKV improves Mistral on only 99 of 1616 tasks. At 4040% retention the two methods are nearly tied (49.3249.32 vs. 49.2149.21). On RULER, most of the 1.111.11-point gain comes from NIAH-MK2 (86.8086.80 to 99.5099.50); the net change over the other 1212 tasks is small, and CWE drops.

W2: The link between the proposed mechanism and accuracy is not established. The analyses in Appendix E show that residual risk depends less on raw-query support, but this is partly expected, since the residual removes the shared component and support is counted from raw attention. The paper itself notes that these analyses measure scores rather than accuracy, and evidence on answer relevance is limited to three case studies. The only quantitative relevance diagnostic (RULER CWE and FWE) shows lower gold-evidence survival on CWE and only a small increase on FWE.

Questions for authors

Q1: Can you report confidence intervals or paired significance tests (e.g., a bootstrap over examples) for the main LongBench comparisons?

Q2: How were the default hyperparameters (β\beta, kk, α\alpha, λraw\lambda_{\text{raw}} and the coverage target) chosen? Were they tuned on the LongBench evaluation data, and how sensitive are the results to λraw\lambda_{\text{raw}} and α\alpha?

Q3: The matched raw-only control (47.0047.00) already outperforms DefensiveKV (46.6846.68), so part of the gain comes from pipeline differences rather than residualization. Apart from residualization, how does DisenKV's pipeline differ from DefensiveKV's, and do the raw-only and norm-matched controls show a similar gap on another model?

Q4: Could you measure gold-evidence survival, as in Appendix E.8, on LongBench QA tasks? This would show whether residualization retains more answer-relevant tokens, rather than only reducing support dominance.

5 comments
AuthorsOpenReview ↗

Response to vkKf (1/4) — LongBench Uncertainty and Task-Level Gains

Q1. Can you quantify uncertainty for the main LongBench comparison?

DisenKV improves the LongBench macro-average by 1.40 points over DefensiveKV at 20% KV retention on Llama-3.1-8B. The 95% paired bootstrap confidence interval is [0.30, 1.67].

We used 3,750 paired examples. For each of 10,000 bootstrap replicates, we resampled examples with replacement within each task, preserved the pairing between methods, and averaged the 16 task-level differences with equal weight.

ComparisonDisenKVDefensiveKVDifference95% paired bootstrap CI
Main comparison48.0846.68+1.40[0.30, 1.67]

Per-task scores at 20% KV retention

MethodNrtvQAQasperMF-enHotpot2WikiMQAMusiqueGovRepQMSumMultiNewsTRECTriviaQASAMSumPCountPR-enLccRB-PAvg.
DefensiveKV29.9740.4646.2352.2038.4028.0629.9623.8924.1168.0091.5843.178.28100.0067.1755.4046.68
DisenKV29.8641.5751.2056.9741.5028.7731.5723.9024.4668.0091.9443.587.7599.5069.4459.2348.08
Difference−0.11+1.11+4.97+4.77+3.10+0.71+1.61+0.01+0.350.00+0.36+0.41−0.53−0.50+2.27+3.83+1.40

The gain varies across tasks, models, and budgets. In the main Llama comparison, DisenKV improves 12 of 16 tasks, ties one, and scores lower on three. On Mistral, Layer-DisenKV improves 9 of 16 tasks relative to Layer-DefensiveKV. At 40% retention on Llama, the standard variants are nearly tied (49.32 vs. 49.21). We also agree that the 1.11-point RULER gain is concentrated in NIAH-MK2 (86.80 to 99.50), while CWE declines. The bootstrap interval quantifies the uncertainty of the main LongBench macro-average gain under the reported setting.

AuthorsOpenReview ↗

Response to vkKf (2/4) — Default Parameters and Sensitivity

Q2. How were the defaults chosen, and how sensitive are the results to λraw\lambda_{\mathrm{raw}} and α\alpha?

(1) We did not use LongBench evaluation scores to select or tune the defaults.

We retained DefensiveKV's pooling kernel size k=5k=5 and raw-attention coverage target of 0.900.90. For historical positions, the coverage threshold is 0.90−bwin0.90-b_{\mathrm{win}}, where bwinb_{\mathrm{win}} is the attention mass on the recent observation window. The number of protected slots is capped by the cache budget. With a window of W=32W=32, α=0.95\alpha=0.95 aggregates the two highest-scoring observations. The noise-floor coefficient β=0.10\beta=0.10 subtracts 10% of each query's mean historical attention from its historical attention scores and clips at zero. After aligning the two risk scales, λraw=0.10\lambda_{\mathrm{raw}}=0.10 adds 10% of the positive gap between raw-query risk and residual-query risk.

(2) The three tested alternatives for λraw\lambda_{\mathrm{raw}} changed the four-task LongBench average by less than one point. Increasing the number of observations aggregated in the upper tail had a larger effect in this diagnostic.

We varied one parameter at a time on the same 200 examples, with 50 each from HotpotQA, MuSiQue, PCount, and PR-en. The last column gives the difference from the default and its 95% paired bootstrap confidence interval.

SettingHotpotQAMuSiQuePCountPR-enFour-task avg.Difference from default (95% CI)
Default: λraw=0.10\lambda_{\mathrm{raw}}=0.10, α=0.95\alpha=0.9550.3321.513.77100.0043.90
λraw=0\lambda_{\mathrm{raw}}=050.3322.815.77100.0044.73+0.83 [−0.38, 2.50]
λraw=0.05\lambda_{\mathrm{raw}}=0.0553.6721.673.77100.0044.78+0.88 [−0.22, 2.31]
λraw=0.20\lambda_{\mathrm{raw}}=0.2049.9323.655.10100.0044.67+0.77 [−0.33, 2.20]
α=0.975\alpha=0.975, m=1m=151.9321.613.10100.0044.16+0.26 [−0.78, 1.53]
α=0.88\alpha=0.88, m=4m=454.4024.567.17100.0046.53+2.63 [0.72, 4.92]

All three LongBench intervals for changes in λraw\lambda_{\mathrm{raw}} span zero. Aggregating four observations instead of two improves the diagnostic average by 2.63 points. On RULER CWE, increasing λraw\lambda_{\mathrm{raw}} from 0.100.10 to 0.200.20 reduces the score from 20.13 to 19.48 (−0.65 points). The sensitivity results show that the preferred setting can depend on the task.

AuthorsOpenReview ↗

Response to vkKf (3/4) — Pipeline Differences and Cross-Model Controls

Q3. How much of the gain comes from pipeline differences, and do the matched controls generalize?

(1) Pipeline differences account for part of the observed gain. Within the matched pipeline, full DisenKV scores 1.08 points above raw-only on Llama-3.1-8B and 0.62 points above raw-only on Phi-4-mini.

DefensiveKV takes maxima over the observation window and over query heads within a group after smoothing. DisenKV applies a soft noise floor before smoothing, averages the upper tail over the observation window, and aggregates within a head group using the geometric mean of the maximum and mean. Both methods use the hard mean floor, value and output magnitude weighting, and raw-attention coverage protection. Full DisenKV also adds a one-sided correction from raw-query risk. The matched raw-only control feeds raw queries to both branches, so the correction is zero.

(2) The advantage persists after matching query norms. The norm-matched raw control preserves the original query directions while matching the residual-query norms. Full DisenKV scores 1.14 points higher on Llama and 1.09 points higher on Phi.

Llama-3.1-8B at 20% KV retention

MethodLongBench macro-averageDifference from DisenKV
DisenKV48.08
DefensiveKV46.68−1.40
Matched raw-only47.00−1.08
Norm-matched raw46.94−1.14

Phi-4-mini at 20% KV retention

MethodNrtvQAQasperMF-enHotpot2WikiMQAMusiqueGovRepQMSumMultiNewsTRECTriviaQASAMSumPCountPR-enLccRB-PAvg.
DefensiveKV23.3334.9244.9850.9435.9826.8928.9221.6124.0063.5082.6344.062.5093.0063.2255.5743.50
DisenKV22.9638.7847.4250.6836.7729.3831.5922.4324.4366.5082.5745.353.0092.5063.6155.2744.58
Matched raw-only22.1434.4744.3448.9536.4824.3028.7021.7424.0067.5082.7644.161.5093.5062.8066.0043.96
Norm-matched raw22.4638.6945.3346.1233.5228.1130.2821.8923.9066.0083.4745.741.5093.0062.6653.1843.49
Full mean22.8238.3846.5251.2135.8027.8031.3521.9724.4067.0082.6945.182.0092.5063.6053.6144.18
No late correction22.7437.7047.0450.7835.6828.4231.6122.3224.3167.5083.6845.052.0092.5063.2453.6344.26
Residual-only22.9039.1047.7051.0036.2028.9030.9022.2024.2068.0081.0045.802.1092.0057.8047.6043.59

The matched raw-only control exceeds DefensiveKV by 0.32 points on Llama and 0.46 points on Phi. The additional gains over both matched controls support the use of query-specific residuals for primary risk scoring together with raw queries for coverage protection and correction.

Reviewer_AzVLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DisenKV, a training-free method for KV cache eviction. The approach employs a "Leave-One-Out" (LOO) projection to disentangle shared query directions from query-specific residuals. It aggregates original attention and residual attention separately, prioritizing an eviction risk score based on the residuals, while utilizing the original attention for one-sided correction. The authors evaluate four series of models on benchmarks such as LongBench and RULER, demonstrating that their pruning method outperforms other baselines.

Critical strengths
  1. Practical and easy to deploy. DisenKV is a training-free method and can be integrated with existing KV cache eviction methods. It can also be applied to KV cache pruning for open-source models.
  2. Broad experiments. The paper evaluates DisenKV across four model families of different scales and multiple benchmarks, including LongBench and RULER. The results demonstrate overall improvements over existing baselines, providing empirical support for the method’s effectiveness.
  3. Comprehensive ablation studies and component analysis. The paper evaluates the contributions of specific design elements—such as LOO projection, query-direction residuals, and original attention compensation—through ablation studies. It also examines performance across various observation window sizes, providing evidence to understand the role of each component and the method's stability.
Critical weaknesses
  1. Insufficient observation evidence. Geometric alignment among queries does not directly establish evidence redundancy. Further analysis of attention distributions and actual output behavior is needed to verify whether shared directions induce biased importance scores. Moreover, the current experiments do not sufficiently establish a causal link between suppressing shared directions and improved task performance.
  2. Inadequate retention rate settings. The experiments primarily compare different models at 20% KV retention, with limited testing at other retention rates. The method’s robustness and generalizability across pruning budgets remain to be verified.
  3. Limited justification and validation of method design. DisenKV employs a single LOO mean direction for projection, but provides limited justification for this design choice and its applicability conditions. While existing ablation studies confirm the effectiveness of LOO strategy, the paper fails to provide a systematic comparison with alternative approaches—such as PCA, multidimensional subspace projection, or robust direction estimation. Consequently, the rationale for selecting the current design and its applicability to different query distributions remain insufficiently clarified.
Questions for authors
  1. How do you verify that query alignment reflects evidence redundancy rather than useful shared information? And is there further evidence to establish a causal link between this phenomenon and performance on downstream tasks?
  2. Can you evaluate more KV retention rates to verify the method’s robustness across pruning budgets?
  3. Why is a single LOO mean direction preferred over alternatives such as PCA or multidimensional subspace projection? Can you provide comparative evidence to justify this design choice?
3 comments
AuthorsOpenReview ↗

Response to AzVL (1/3) — Mechanism and Downstream Accuracy

Thank you for raising these questions about the mechanism, performance across retention budgets, and the choice of projection. We address each question below.

Q1. Does query alignment imply evidence redundancy, and does the shared component affect downstream accuracy?

(1) The shared component affects retention scores under a controlled intervention. We extended the intervention to actual KV eviction and answer generation, where the confidence intervals for mean F1 changes span zero.

In Appendix E.6, we increased the shared-component multiplier γ\gamma from 00 to 22 while holding query norms fixed. The retention-risk gap between high- and low-support tokens increased from 0.50 to 3.26 under raw-query scoring and decreased from 0.73 to 0.19 under residual scoring. Residualization reduced the response slope on all 16 LongBench tasks.

We then evaluated actual KV eviction and generation on 200 HotpotQA examples with Llama-3.1-8B at 20% retention. We increased γ\gamma from 0.50.5 to 1.51.5 in the primary risk-scoring branch while holding query norms fixed.

MethodSupporting-passage KV-slot retentionAnswer F1Δ\DeltaF1 (95% paired CI)
Matched raw-only20.09% → 20.15%55.75 → 51.10−4.65 [−9.73, 0.38]
DisenKV19.84% → 19.85%57.42 → 54.66−2.76 [−6.40, 0.76]

Both F1 intervals span zero. Across examples, the Spearman correlations between changes in supporting-passage retention and changes in F1 were −0.075-0.075 for matched raw-only and −0.057-0.057 for DisenKV, with both 95% intervals spanning zero. The smaller F1 drop for DisenKV is a point estimate. The experiment leaves open how much the two methods differ in their F1 response and how much aggregate evidence retention explains that response.

(2) Query alignment alone cannot establish semantic redundancy. With Llama-3.1-8B at 20% retention, we measured the per-example relationship between shared-component energy and evidence eviction under matched raw-only scoring. Spearman ρ\rho was −0.028-0.028 on HotpotQA (95% CI [−0.163,0.106][-0.163, 0.106]) and 0.0780.078 on RULER CWE (95% CI [−0.067,0.224][-0.067, 0.224]), using 200 examples per task. Shared demand can also be useful. On CWE (Appendix E.8), target-word KV-slot retention decreased from 25.89% to 24.37% and the score decreased from 22.94 to 20.13 when moving from DefensiveKV to DisenKV.

Together, these results support the claim that residualization changes how retention scores respond to shared demand. The contribution of specific retained evidence to downstream accuracy remains open.

AuthorsOpenReview ↗

Response to AzVL (2/3) — KV Retention Budgets

Q2. How does DisenKV perform across KV retention budgets?

(1) Standard DisenKV scores higher than DefensiveKV at all four tested budgets, with its largest gain at 10% retention (+2.34 points). (2) Layer-DisenKV improves over Layer-DefensiveKV at 10%, 20%, and 40% retention and scores 0.22 points lower at 60%.

The table reports macro-averages over all 16 LongBench tasks with Llama-3.1-8B. Each layer-adaptive pair uses the same allocation rule and total budget.

Method10%20%40%60%
DefensiveKV41.5246.6849.2149.43
DisenKV43.8648.0849.3249.78
Layer-DefensiveKV42.7648.0149.4649.52
Layer-DisenKV45.6749.1850.2449.30

Full per-task scores at 60% retention

MethodNrtvQAQasperMF-enHotpot2WikiMQAMusiqueGovRepQMSumMultiNewsTRECTriviaQASAMSumPCountPR-enLccRB-PAvg.
Full Cache29.5544.6855.8257.5948.8932.6134.4025.5126.8373.0092.3643.277.3899.5063.4452.3649.20
StreamingLLM25.2139.9233.5349.7239.9822.6231.5123.1225.9169.5092.2742.753.0899.5066.2761.7545.42
SnapKV28.9240.3548.0056.7948.6030.1232.5424.2025.5864.5091.6444.538.8599.0065.0353.8147.65
CAKE29.9941.8753.0955.3942.8332.1732.1224.8725.6166.5092.5043.377.9699.5064.1152.6247.78
AdaKV30.1043.6151.2056.3749.7030.1832.3724.3825.5466.5091.4843.878.0299.5063.9154.7348.22
CriticalKV30.3143.5452.8257.3049.0931.7833.4825.1826.0072.5091.8043.957.4799.5064.0554.9248.98
DefensiveKV30.8843.2055.1755.8550.1731.8434.7925.2926.8473.0092.1443.289.0599.5063.8856.0249.43
DisenKV30.3745.0155.6457.7149.1533.6334.5825.3727.1673.5091.4743.238.9199.5064.4956.7149.78
Layer-DefensiveKV29.9544.1156.7857.1147.4732.5834.8125.1126.8072.0091.8343.1411.1099.5064.2155.8149.52
Layer-DisenKV30.5444.1954.4357.3447.4031.1837.1325.0027.5372.0091.7642.756.7099.5065.3755.9749.30

Per-task differences relative to the corresponding DefensiveKV baseline

VariantNrtvQAQasperMF-enHotpot2WikiMQAMusiqueGovRepQMSumMultiNewsTRECTriviaQASAMSumPCountPR-enLccRB-PAvg.
Standard 10%+0.81+4.97+4.68+1.46+8.12+5.05+6.15−0.02+0.78+1.50−0.66−1.21−0.74+1.00+0.50+5.04+2.34
Layer-adaptive 10%+1.93+7.23+4.99+2.35+7.55+1.66+3.07+0.94+0.79+11.50−0.51−0.86−0.34+1.50+2.26+2.46+2.91
Standard 20%−0.11+1.11+4.97+4.77+3.10+0.71+1.61+0.01+0.350.00+0.36+0.41−0.53−0.50+2.27+3.83+1.40
Layer-adaptive 20%+0.69+0.46+1.96+1.70−1.11+1.96+2.50+0.18+1.52+2.20+0.23+1.67+0.480.00+2.26+1.93+1.17
Standard 40%+0.39−0.61−1.05−1.34+0.75+0.99−0.11+0.30−1.07−1.50−0.31−0.35−1.13−0.50+2.86+4.48+0.11
Layer-adaptive 40%+0.60+2.04+0.66+1.46−0.38+1.28+2.04+0.18+0.18−0.50−0.08+0.05+0.840.00+2.41+1.63+0.78
Standard 60%−0.51+1.81+0.47+1.86−1.02+1.79−0.21+0.08+0.32+0.50−0.67−0.05−0.140.00+0.61+0.69+0.35
Layer-adaptive 60%+0.59+0.08−2.35+0.23−0.07−1.40+2.32−0.11+0.730.00−0.07−0.39−4.400.00+1.16+0.16−0.22

Both variants gain most at the tightest tested budget. At higher retention rates, the gains are smaller and depend on the allocation scheme.

AuthorsOpenReview ↗

Response to AzVL (3/3) — LOO and PCA Comparisons

Q3. Why use a single LOO direction, and how does it compare with PCA?

(1) Single-direction LOO achieved the highest macro-average in this comparison, scoring 48.08 versus 46.31 for rank-1 PCA and 46.22 for rank-2 PCA.

(2) The leave-one-out estimate provides query-specific residuals for primary risk scoring while preserving the original queries for coverage protection and one-sided correction.

For each query qiq_i, LOO estimates a shared direction by averaging and normalizing the other W−1W-1 observation queries. It then removes the projection of qiq_i onto that direction. Excluding qiq_i from the estimate prevents it from influencing its own projection direction. The residual supplies the primary risk score, while the original query remains available to preserve useful shared demand.

We held Llama-3.1-8B, 20% KV retention, and the rest of the eviction pipeline fixed. We varied only the query transformation in the primary risk-scoring branch. The PCA controls fit a centered one- or two-dimensional subspace using the same W−1W-1 other queries. We also evaluated mean-centering and random-direction projection with one fixed seed.

TransformationNrtvQAQasperMF-enHotpot2WikiMQAMusiqueGovRepQMSumMultiNewsTRECTriviaQASAMSumPCountPR-enLccRB-PAvg.
Single-direction LOO29.8641.5751.2056.9741.5028.7731.5723.9024.4668.0091.9443.587.7599.5069.4459.2348.08
PCA rank 127.6937.7248.2850.9938.4428.3728.7523.2923.9766.0092.0844.035.1199.5068.2458.4346.31
PCA rank 228.2136.2446.1752.0535.6329.1928.4323.0824.3466.5091.9243.987.98100.0068.1157.7646.22
Mean-centering30.4643.4249.7656.4141.8228.8128.1324.3824.1169.0091.6643.445.8799.5068.2958.1947.70
Random-direction projection29.4040.2749.6652.4840.9928.7828.5623.1424.8366.0092.0843.597.1999.5067.9257.8347.01

Increasing the PCA rank from one to two did not improve the macro-average in this setting. LOO scored 0.38 points above mean-centering and 1.07 points above the single-seed random control. These point estimates support using one leave-one-out direction in the residual branch for the tested configuration. The direction can be computed from a mean and normalization, without fitting a PCA subspace.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.