- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces KVBench, a modular and extensible benchmark for evaluating approximate KV-cache reuse. It decouples tasks, models, reuse methods, and request workflows, providing a unified protocol for measuring task quality and time-to-first-token (TTFT) relative to full-prefill baselines. Covering RAG-style context reuse, multi-agent communication, and agentic skill reuse, KVBench enables systematic evaluation of the trade-off between task quality and latency. Through extensive experiments, the authors show that quality degradation and latency benefits are highly dependent on the task, model configuration, request structure, and reusable-context length. They further observe that aggregate cross-document dependency metrics have limited predictive power for quality degradation in the evaluated settings, while skill reuse offers promising latency reductions but does not fully preserve the gains from incorporating skills. Overall, the work highlights the need for controlled, workload-aware evaluation and provides a common framework for assessing the benefits and limitations of approximate KV-cache reuse.
Critical strengths
-
Beyond covering different tasks, models, and reuse methods, KVBench explicitly incorporates cache preparation, request construction, and state sharing into its evaluation, measuring both quality and latency against paired full-prefill baselines. This design helps control experimental conditions that can otherwise be overlooked and provides a consistent, extensible foundation for evaluating future methods.
-
The authors show that quality recovery varies across tasks and models, request structure and reusable-context length affect reuse performance, and aggregate dependency metrics have limited predictive power in the evaluated settings. These findings demonstrate why results from a single task or fixed prompt layout cannot establish general effectiveness. Beyond providing an evaluation tool, the paper offers empirical insights that can directly inform experimental design and deployment validation, strengthening its research contribution.
Critical weaknesses
-
Standard tasks are evaluated on only 64 examples by default, and the skill-reuse experiments use a single agent rollout per task and configuration. Without sufficient repeated runs and uncertainty estimates, it is difficult to assess whether quality differences and method rankings are robust. The emphasis on TTFT provides an incomplete picture of efficiency, especially for long-horizon reasoning and multi-step agent tasks. Approximate reuse may change subsequent reasoning and tool-use trajectories. This could increase generation length or the number of execution steps, potentially offsetting the initial latency savings. Without measurements of end-to-end task runtime, total generated tokens, and throughput, the evidence remains insufficient to establish the practical efficiency benefits comprehensively.
-
The paper documents task- and model-dependent behavior, sensitivity to instruction placement, and the limited predictive power of aggregate dependency metrics. However, it does not sufficiently explain which contextual interactions drive quality degradation or why the same repair strategy behaves differently across tasks and models. In particular, after establishing the weak explanatory power of existing dependency metrics, the analysis does not identify and validate alternative explanatory factors or predictors. Although these findings motivate broader evaluation coverage, they provide limited guidance for designing reuse policies, improving repair mechanisms, or selecting methods for a given workload, constraining the contribution of the empirical analysis to future research.
Questions for authors
-
What motivated the use of a batch size of one and the emphasis on TTFT? Practical deployments often involve batched or concurrent requests, where throughput can be a key performance metric. Please clarify the rationale for this evaluation design and the extent to which the reported benefits are expected to hold in these settings.
-
Can the authors provide explanations for the observed patterns, particularly the sensitivity to instruction placement and the variation in quality recovery across tasks and models? Can the insights from these observations help guide the design of approximate KV-cache reuse methods?
I appreciate the authors' detailed rebuttal and the additional experimental results. The expanded evaluations with 200 samples, repeated SkillsBench rollouts, and end-to-end execution measurements meaningfully address several of my concerns regarding experimental robustness and practical efficiency. I also find the justification for using batch size one and TTFT as the primary efficiency metric reasonable for a controlled benchmark.
However, some concerns remain. The repeated agent experiments are limited to a selected subset of 20 relatively successful tasks, and no uncertainty estimates are provided. More importantly, while the additional analysis strengthens the evidence that aggregate cross-document dependency metrics have limited predictive power, the proposed explanations for quality degradation and instruction-placement sensitivity remain largely hypothetical, without direct experimental validation.
It is a tough decision, but I believe that the rebuttal does not sufficiently justify an upgrade of my score from 2 to 3.