acceptodds
Under review as a conference paper at ICLR 2027

Evaluating the accuracy of KV cache reuse techniques

Abstract

Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show that current evaluations of KV cache reuse techniques rely on measurements that fail to faithfully capture the loss of accuracy attributable to reuse, often artificially inflating the reported effectiveness. We also show that existing datasets do not exhibit the reuse dynamics needed to thoroughly evaluate such techniques. To address these issues, we propose an evaluation methodology that measures this accuracy loss without ambiguity and we introduce Boxoffice, a tool that programmatically generates evaluation datasets that exercise challenging KV cache reuse patterns.

Primary area:
datasets and benchmarks
Keywords:
KV-cache reuse, LLM inference, retrieval-augmented generation, benchmarking, synthetic benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WsdLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the evaluation of position-independent KV cache reuse. This paper argues that standard aggregate F1 on benchmarks (e.g., LongBench) is confounded by queries that full prefill already fails on, queries answerable from parametric memory, and low-information queries. This paper define a meaningful subset that better isolates accuracy loss caused by reuse and introduce Boxoffice, a programmatic workload generator that creates controlled cross-query chunk reuse and warmup histories. A key concept is cache staleness, where a chunk's cached contextual role from an earlier query conflicts with its role in the current query. Experiments across aligned, all-W, all-N, and flipped configuration show that cache history can drastically affect reuse accuracy.

Critical strengths

The paper identifies important limitations in existing benchmarks for evaluating KV cache reuse methods. In particular, it shows that commonly used aggregate metrics can be affected by queries that do not faithfully reflect the accuracy degradation caused by cache reuse, while existing datasets may also lack realistic cross-query chunk reuse patterns required to properly evaluate warm-cache strategies. Then, the paper introduces Boxoffice, a controlled evaluation framework designed to address these limitations. Boxoffice programmatically generates workloads with configurable cross-query reuse patterns and cache histories, enabling a more systematic evaluation of KV cache reuse methods, including challenging cases such as cache sstaleness.

Critical weaknesses
  1. The paper demonstrates the limitations of existing practices primarily on MuSiQue, HotpotQA, and 2WikiMQA, which are indeed widely used in prior KV-cache reuse studies. However, it remains unclear to what extent the identified issues generalize beyond these three multi-hop QA datasets. In particular, properties such as semantic clustering and limited cross-query chunk reuse are inherently dataset-dependent. A broader analysis covering additional commonly used tasks or datasets, or a clearer discussion of the scope of the claims, would strengthen the generality of the conclusions.
  2. The cache staleness scenarios constructed in Boxoffice appear relatively extreme and may not fully reflect realistic deployment settings. While these controlled cases are useful for stress testing, it remains unclear how frequently such strong mismatches between cached context and current context occur in real-world KV cache reuse workloads.
  3. Boxoffice assigns equal weight to the four configurations, with each accounting for 25% of the evaluation set, but this distribution may not reflect real-world workloads. Since the prevalence of aligned, all-W, all-N, and flipped cases is likely to be highly non-uniform in practice, the authors should report the performance of each configuration separately and clarify that the averaged result depends on this artificial weighting.
Questions for authors

Boxoffice is presented primarily as a programmatic workload generator, and its generated evaluation set can depend on the selected filter models. Have the authors considered defining and releasing a fixed standardized Boxoffice benchmark suite, in addition to the generator, to facilitate consistent and directly comparable evaluation across future KV cache reuse methods and models?

Reviewer_3UMFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that standard QA evaluations hide errors from KV cache reuse and poorly test caches built under earlier prompts. It proposes filtering existing benchmarks and introduces Boxoffice, a synthetic benchmark showing that reuse accuracy can depend strongly on the context in which a chunk was originally cached.

Critical strengths
  • With recomputation disabled, semantically similar passages outside the current prompt offer no consistent advantage over random passages, while actual context passages help (§4.2). This challenges the use of semantic similarity to choose useful cache-warming context.
Critical weaknesses
  • Table 1 labels gains on questions that full prefill gets wrong as accuracy "inflation". Removing those gains necessarily lowers the reported ratio, it does not establish that the gains were spurious or the original accuracy score was wrong.

  • The CacheCraft variant uses prefix-overlap selection and a fixed recomputation budget, replacing the original method’s contextualization scores and adaptive recomputation. These changes affect exactly the context-mismatch problem being tested, so its failures cannot establish that the original method fails.

Questions for authors

How much of the reported 42% inflation comes from fully correct answers that full prefill missed?

Reviewer_jHZJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that current evaluations of KV cache reuse techniques rely on measurements that fail to faithfully capture the loss of accuracy attributable to reuse, often artificially inflating the reported effectiveness. The paper also shows that existing datasets do not exhibit the reuse dynamics needed to thoroughly evaluate such techniques. To address these limitations, the paper proposes an evaluation methodology that measures this accuracy loss without ambiguity. Besides, the paper introduces Boxoffice, a tool that programmatically generates evaluation datasets that exercise challenging KV cache reuse patterns.

Critical strengths

(1) The paper is related to an emerging and practical topic. The paper is well motivated. The limitations of existing works are clearly detailed.

(2) The writing is polished and easy to follow. The illustration is intuitive. The evaluations are insightful.

Critical weaknesses

It seems that there are several benchmarks related to this topic, while these papers are not cited and discussed in the paper. It would be great to discuss how the developed benchmark distinguishes from other benchmark datasets. Some quantitive comparisons or evaluations on multiple benchmark datasets in Section 5 might also be helpful.

Li, Yucheng, et al. "Scbench: A kv cache-centric analysis of long-context methods." International Conference on Learning Representations. Vol. 2025. 2025.

Ikeda, Taiga, Daisuke Miyashita, and Jun Deguchi. "KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse." arXiv preprint arXiv:2610.13060 (2026).

Shi, Xi, and Qian Lou. "KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints." arXiv preprint arXiv:2609.10266 (2026).

Questions for authors

What are the differences between the developed benchmark and some other benchmark datasets? It seems that there are several benchmarks related to this topic, while these papers are not cited and discussed. It would be great to discuss how the developed benchmark distinguishes from other benchmark datasets. Some quantitive comparisons or evaluations on multiple benchmark datasets in Section 5 might also be helpful.

Li, Yucheng, et al. "Scbench: A kv cache-centric analysis of long-context methods." International Conference on Learning Representations. Vol. 2025. 2025.

Ikeda, Taiga, Daisuke Miyashita, and Jun Deguchi. "KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse." arXiv preprint arXiv:2610.13060 (2026).

Shi, Xi, and Qian Lou. "KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints." arXiv preprint arXiv:2609.10266 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.