acceptodds
Under review as a conference paper at ICLR 2027

KVBench: An Extensible Benchmark to Evaluate Approximate KV Cache Reuse

Abstract

Approximate KV-cache reuse accelerates large language model inference by reusing cached states beyond identical request prefixes. However, these states depend on their original context, so transferring them can alter generation quality, and the resulting quality–latency trade-off may vary across models, tasks, and request structures. We present KVBench, an extensible benchmark that separates tasks, models, reuse methods, and request workflows to support controlled evaluation of these effects. KVBench measures task quality and online time-to-first-token relative to full-prefill references across RAG-style context reuse, multi-agent communication, and agentic skill reuse, covering five models with full- or hybrid-attention architectures. Our evaluation reveals substantial task–model variation in quality retention and recovery through recomputation. Controlled studies further show that instruction placement can strongly affect approximate reuse even when full-prefill quality remains stable, and that reusable-context length affects the latency benefit. We also study reuse of independently prepared skill caches on SkillsBench tasks. HYPIC TRR retains 83.0% of the reward improvement provided by skills, while reducing first-call time-to-first-token to roughly 20–30% of full prefill on executions with large reusable-context fractions. These findings establish cache preparation and request structure as important evaluation dimensions and provide a common framework for studying the benefits and limitations of approximate KV-cache reuse.

Primary area:
datasets and benchmarks
Keywords:
KV Reuse, Benchmark, inference acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YCjCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces KVBench, a modular and extensible benchmark for evaluating approximate KV-cache reuse. It decouples tasks, models, reuse methods, and request workflows, providing a unified protocol for measuring task quality and time-to-first-token (TTFT) relative to full-prefill baselines. Covering RAG-style context reuse, multi-agent communication, and agentic skill reuse, KVBench enables systematic evaluation of the trade-off between task quality and latency. Through extensive experiments, the authors show that quality degradation and latency benefits are highly dependent on the task, model configuration, request structure, and reusable-context length. They further observe that aggregate cross-document dependency metrics have limited predictive power for quality degradation in the evaluated settings, while skill reuse offers promising latency reductions but does not fully preserve the gains from incorporating skills. Overall, the work highlights the need for controlled, workload-aware evaluation and provides a common framework for assessing the benefits and limitations of approximate KV-cache reuse.

Critical strengths
  1. Beyond covering different tasks, models, and reuse methods, KVBench explicitly incorporates cache preparation, request construction, and state sharing into its evaluation, measuring both quality and latency against paired full-prefill baselines. This design helps control experimental conditions that can otherwise be overlooked and provides a consistent, extensible foundation for evaluating future methods.

  2. The authors show that quality recovery varies across tasks and models, request structure and reusable-context length affect reuse performance, and aggregate dependency metrics have limited predictive power in the evaluated settings. These findings demonstrate why results from a single task or fixed prompt layout cannot establish general effectiveness. Beyond providing an evaluation tool, the paper offers empirical insights that can directly inform experimental design and deployment validation, strengthening its research contribution.

Critical weaknesses
  1. Standard tasks are evaluated on only 64 examples by default, and the skill-reuse experiments use a single agent rollout per task and configuration. Without sufficient repeated runs and uncertainty estimates, it is difficult to assess whether quality differences and method rankings are robust. The emphasis on TTFT provides an incomplete picture of efficiency, especially for long-horizon reasoning and multi-step agent tasks. Approximate reuse may change subsequent reasoning and tool-use trajectories. This could increase generation length or the number of execution steps, potentially offsetting the initial latency savings. Without measurements of end-to-end task runtime, total generated tokens, and throughput, the evidence remains insufficient to establish the practical efficiency benefits comprehensively.

  2. The paper documents task- and model-dependent behavior, sensitivity to instruction placement, and the limited predictive power of aggregate dependency metrics. However, it does not sufficiently explain which contextual interactions drive quality degradation or why the same repair strategy behaves differently across tasks and models. In particular, after establishing the weak explanatory power of existing dependency metrics, the analysis does not identify and validate alternative explanatory factors or predictors. Although these findings motivate broader evaluation coverage, they provide limited guidance for designing reuse policies, improving repair mechanisms, or selecting methods for a given workload, constraining the contribution of the empirical analysis to future research.

Questions for authors
  1. What motivated the use of a batch size of one and the emphasis on TTFT? Practical deployments often involve batched or concurrent requests, where throughput can be a key performance metric. Please clarify the rationale for this evaluation design and the extent to which the reported benefits are expected to hold in these settings.

  2. Can the authors provide explanations for the observed patterns, particularly the sensitivity to instruction placement and the variation in quality recovery across tasks and models? Can the insights from these observations help guide the design of approximate KV-cache reuse methods?

1 comment

I appreciate the authors' detailed rebuttal and the additional experimental results. The expanded evaluations with 200 samples, repeated SkillsBench rollouts, and end-to-end execution measurements meaningfully address several of my concerns regarding experimental robustness and practical efficiency. I also find the justification for using batch size one and TTFT as the primary efficiency metric reasonable for a controlled benchmark.

However, some concerns remain. The repeated agent experiments are limited to a selected subset of 20 relatively successful tasks, and no uncertainty estimates are provided. More importantly, while the additional analysis strengthens the evidence that aggregate cross-document dependency metrics have limited predictive power, the proposed explanations for quality degradation and instruction-placement sensitivity remain largely hypothetical, without direct experimental validation.

It is a tough decision, but I believe that the rebuttal does not sufficiently justify an upgrade of my score from 2 to 3.

Reviewer_zVd3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an evaluation framework called KVBench, which systematically evaluates the performance of approximate KV-cache reuse in long-context tasks. It investigates how reusing previously computed KV caches under different context arrangements, with cache adaptation or partial recomputation, affects downstream task performance and inference efficiency across different models.

Critical strengths
  1. The research question has practical value. Current agentic tasks often require extremely long context inputs, making it important to reduce the cost of repeated prefill computation. This paper provides a systematic empirical evaluation of KV-cache reuse in this setting.

  2. The experiments cover a wide range of models, tasks, and reuse methods. The paper also clearly describes the specific conditions under which the results are obtained, providing useful evidence for future applications.

Critical weaknesses
  1. The paper lacks sufficient discussion of how differences in model size and architecture affect the evaluation results. In particular, the difference in TTFT reduction between dense and MoE models is not well explained. Reporting more absolute latency measurements might help clarify this difference.

  2. The paper could include more experiments that are closely related to real-world applications, such as repository-level coding tasks. These tasks often involve frequent modifications to existing code files, rather than simply changing the order of context segments.

Questions for authors
  1. In the experiments with Qwen3.8-27B and Qwen3.5-35B-A3B, the dense model shows a significantly larger TTFT reduction than the MoE model. Is there any further explanation for this difference? Could it be related to differences in the baseline inference efficiency of the two models? Although these models have relatively similar total parameter counts, more discussion of the architectural differences between dense and MoE models might be helpful.

  2. Could the authors provide more discussion of how model size affects KV-cache reuse within the same model family? For example, is the degree of quality recovery through partial recomputation related to model size? If so, what might explain this relationship?

  3. The paper has already discussed full-attention and hybrid-attention architectures, including dense and MoE models. More model architectures could be considered if the computational budget allows, such as the Gemma 4 series, to further investigate whether the observed trends generalize across different architectures.

  4. The paper has already included coding tasks, but repository-level coding tasks in real-world agentic settings may involve frequent modifications to the same code files. In this case, the content corresponding to a previously computed KV cache would change, rather than simply appearing in a different context position. Have the authors considered evaluating KV-cache reuse under such scenarios? It would be interesting to see how different reuse methods perform when the reusable content itself is modified.

Reviewer_emDMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces KVBench, a benchmark for evaluating approximate KV-cache reuse across tasks, models, methods, and request workflows. It measures output quality and time to first token, and extends the evaluation to agentic skill reuse. The results highlight how reuse performance depends on the task, model, and organization of the input.

Critical strengths

1.I find the evaluation problem well motivated. Cache preparation and request structure can affect both quality and latency, and treating them as explicit evaluation dimensions helps clarify when approximate reuse is effective.

2.The evaluation has broad coverage, including multiple models and methods, full- and hybrid-attention architectures, and workloads spanning RAG, multi-agent communication, and skill reuse.

Critical weaknesses

My main concerns are: 1.Each SkillsBench task is evaluated with only one rollout per configuration. Given the randomness of agent execution, I would like to see some repeated runs for the main configurations to assess the uncertainty around the reported 83% skill-gain retention and the differences between methods. The authors acknowledge this limitation, but the stability of these results remains unclear.

2.The skill-reuse experiments measure first-call TTFT, as the paper clearly states. However, it remains unclear how much this reduction matters over a complete agent execution. End-to-end timing on representative tasks, together with an analysis of how cache preparation costs are amortized across repeated uses, would help establish the practical benefit.

3.CacheBlend is evaluated across several recomputation ratios, while several other full-attention methods are mainly represented by a single configuration. A few additional settings for the main tunable baselines, or a clearer justification of the configuration choices, would help distinguish differences between methods from differences due to the selected configurations.

Questions for authors

Please see Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.