- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits the evaluation of position-independent KV cache reuse. This paper argues that standard aggregate F1 on benchmarks (e.g., LongBench) is confounded by queries that full prefill already fails on, queries answerable from parametric memory, and low-information queries. This paper define a meaningful subset that better isolates accuracy loss caused by reuse and introduce Boxoffice, a programmatic workload generator that creates controlled cross-query chunk reuse and warmup histories. A key concept is cache staleness, where a chunk's cached contextual role from an earlier query conflicts with its role in the current query. Experiments across aligned, all-W, all-N, and flipped configuration show that cache history can drastically affect reuse accuracy.
Critical strengths
The paper identifies important limitations in existing benchmarks for evaluating KV cache reuse methods. In particular, it shows that commonly used aggregate metrics can be affected by queries that do not faithfully reflect the accuracy degradation caused by cache reuse, while existing datasets may also lack realistic cross-query chunk reuse patterns required to properly evaluate warm-cache strategies. Then, the paper introduces Boxoffice, a controlled evaluation framework designed to address these limitations. Boxoffice programmatically generates workloads with configurable cross-query reuse patterns and cache histories, enabling a more systematic evaluation of KV cache reuse methods, including challenging cases such as cache sstaleness.
Critical weaknesses
- The paper demonstrates the limitations of existing practices primarily on MuSiQue, HotpotQA, and 2WikiMQA, which are indeed widely used in prior KV-cache reuse studies. However, it remains unclear to what extent the identified issues generalize beyond these three multi-hop QA datasets. In particular, properties such as semantic clustering and limited cross-query chunk reuse are inherently dataset-dependent. A broader analysis covering additional commonly used tasks or datasets, or a clearer discussion of the scope of the claims, would strengthen the generality of the conclusions.
- The cache staleness scenarios constructed in Boxoffice appear relatively extreme and may not fully reflect realistic deployment settings. While these controlled cases are useful for stress testing, it remains unclear how frequently such strong mismatches between cached context and current context occur in real-world KV cache reuse workloads.
- Boxoffice assigns equal weight to the four configurations, with each accounting for 25% of the evaluation set, but this distribution may not reflect real-world workloads. Since the prevalence of aligned, all-W, all-N, and flipped cases is likely to be highly non-uniform in practice, the authors should report the performance of each configuration separately and clarify that the averaged result depends on this artificial weighting.
Questions for authors
Boxoffice is presented primarily as a programmatic workload generator, and its generated evaluation set can depend on the selected filter models. Have the authors considered defining and releasing a fixed standardized Boxoffice benchmark suite, in addition to the generator, to facilitate consistent and directly comparable evaluation across future KV cache reuse methods and models?