acceptodds
Under review as a conference paper at ICLR 2027

What Transfers Across Models in KV Cache Eviction? A Controlled Measurement Study

Abstract

Deploying a KV cache eviction method on an untuned model is a leap of faith. Whether published accuracy holds on a new model and a new task is unknown. We separate eviction into independent design choices and measure each on five models in one physical-eviction harness. For scoring, accuracy follows a computable property of the workload rather than of the model. Viewing eviction as estimation yields a parameter-free dilution regularity. Windowed scoring attenuates the task-conditional signal by the fraction of task rows in the window. It holds on all five models under controlled interventions in both directions and on real tasks spanning question answering, summarization, and long-document understanding. For allocation, per-layer fragility follows no cross-model pattern, so borrowed shapes must be measured rather than trusted. A starvation probe maps fragility from a few calibration prompts and tests borrowed shapes against a uniform split. Derived only from controlled measurements, this knowledge predicts where the complete methods CAKE and ChunkKV fail on a new model and prescribes a repair, lifting needle-in-a-haystack retrieval from 43% to 65% with 128 tokens per layer at 32K context. What transfers across models is not a method but knowledge of what transfers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache eviction, long-context inference, large language models, measurement study, attention mechanisms, inference efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_bd82OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares KV-cache scoring rules and cache-budget allocation across layers on different models.

Critical strengths

The paper examines KV-cache eviction across five models and includes experiments on synthetic retrieval and real-world tasks, providing empirical observations across different model and workload settings.

Critical weaknesses
  1. Poor presentation. The abstract is vague and the paper is hard to follow. Cumulative scoring, layer-wise allocation, and the configuration in Table 3 need clearer explanations and annotations.

  2. Limited insight and theoretical analysis. The method offers little new insight and lacks substantive theoretical analysis beyond a simple formulation. The results vary across models without providing clear practical guidance.

  3. Insufficient evaluation. The experiments lack a comparison with OBCache [1] and ablations on the calibration data used by the starvation probe.

[1] OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference

Questions for authors
  1. Please clarify cumulative scoring and layer-wise allocation, and identify the proposed configuration in Table 3.

  2. How do the choice and amount of calibration data affect the starvation-probe results?

Reviewer_YBnhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies why KV cache eviction methods fail to transfer across models. The authors derives a parameter-free method to predict and repair the failures.

Critical strengths

None

Critical weaknesses

The formatting of the manuscript deviates substantially from the required template, and the writing is difficult to follow.

In addition, the experimental results lack sufficient detail. For instance, only an overall score is reported for LongBench v2, with no breakdown. Similarly, the NIAH experiments are presented without a description of the setup and no appendix is provided to supply these details.

The paper positions its contribution as transferable knowledge rather than a new method, but the actionable outcomes largely reduce to hyperparameter choices.

Questions for authors

none

Reviewer_XDDfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether a KV cache eviction method for long -context LLM inference can transfer across different models. It finds that existing KV cache eviction methods fail on new models or unknown tasks. The authors propose a simple starvation probe method, which recovers some performance on five models.

Critical strengths
  • The problem of KV cache eviction for long-context LLM inference is important.
Critical weaknesses
  • The author changed the spacing between paragraphs, which violates the ICLR 2027 formatting instruction.
  • There are no experimental setup or implementation details, making the results difficult to reproduce.
  • The experiments are mainly on 32K setting, which is too short for long-context LLM inference. Papers over the past two years mostly conduct experiments using a setting of at least 64K context length.
  • Terrible writing: The paper is difficult to follow, doesn't look like it was written by human.
Questions for authors

Did the authors really check the LLM writen text?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.