- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates accuracy degradation from cross-context KV cache reuse, mainly focusing on KVCOMM. It proposes Budgeted Cache Repair (BCR) to obtain attention queries from a short draft, use them to rank cached positions, and selectively recompute a fixed budget using individual rows, fixed chunks, or grown spans.
Critical strengths
The paper shows that KVCOMM can appear lossless in short evaluations while losing accuracy over longer runs, exposing a blind spot in evaluating online cache reuse.
Critical weaknesses
-
The central insight is not established. Table 8’s row-to-chunk decline is budget-matched, but the call-level endpoint is not (50% and 100% budgets), and is combined with a separate allocation experiment. More importantly, the curve’s prediction fails at the task level as single-row selection (BCR-single, CacheBlend) does not beat 64-token chunks in any cell of Table 14.
-
The paper does not demonstrate the proposed method’s quality/cost advantage. Accuracy uses draft-scored repair and the advertised 2.85× speedup uses random repair and excludes the draft. And, algorithm1 does not make selected rows dense-exact: they still attend to erroneous un-repaired entries, which can contaminate their later layer representations.
Questions for authors
- Does Table 8 run Algorithm 1, or replace selected entries with entries copied from a dense reference? These measure different interventions.
- What accuracy and TTFT does the complete draft-scored implementation achieve, including scoring, rollback, and repair?
- Does the generated-token query improve selection over using only the final prompt-token query? Without this comparison, the contribution of generation remains unidentified.
12 comments
Q1 — Does Table 8 run Algorithm 1?
Q1 — Does Table 8 run Algorithm 1?
The reviewer is correct. Table 8 restores selected entries using their dense reference values rather than executing Algorithm 1. It therefore represents an oracle upper bound, not the actual output of our method.
We have clarified this distinction in the revised caption and added Table R10 to measure the residual error produced by Algorithm 1 directly.
Table R10. The residual at three budgets. GSM8K, five agents, pinned roles, , , , – repair calls per condition. Panel (a) varies the budget at -token chunks; panel (b) holds the budget and varies the ranking rule, with the rule in place of ρ and the decision unit in place of k. Panel (b) is tail-masked — its conditions cannot select the last positions, which is the only way both rules choose from the identical pool, since InfoFlowKV's needs a query block there — so the two panels are different regimes and the bold marks the best value within each panel, not across them. The italic row repeats (a)'s operating point unmasked. The three tail-masked floors agree to in keys and in values, which makes those residuals comparable. Both error columns share one denominator — the repaired rows' own pre-repair deviation — so floor and residual are directly comparable. already exact is the share of selected rows matching dense at every layer to ⁻² before the repair.
| ρ | k | residual keys ↓ | residual values ↓ | floor keys ↓ | floor values ↓ | already exact ↓ | accuracy ↑ | vs. Dense |
|---|---|---|---|---|---|---|---|---|
| KVComm (no repair) | — | — | — | — | — | — | % | , |
| (a) the budget, at -token chunks | ||||||||
| 2 | % | % | , | |||||
| 2 | 0.0103 | 0.0153 | 63.2% | 87.3% | , | |||
| 2 | 0.379 | 0.435 | % | % | , | |||
| (b) the ranking rule, at , tail-masked | ||||||||
| BCR (unmasked, from (a)) | chunk | % | % | , | ||||
| BCR | chunk | 0.356 | 0.405 | % | 81.3% | , | ||
| InfoFlowKV | chunk | % | % | , | ||||
| InfoFlowKV | token | 46.7% | 81.3% | , | ||||
| Dense (ceiling ref.) | — | — | — | — | — | — | % | — |
Q2 — Does the method improve latency?
Q2 — Latency
As discussed in our response to JnoT, we corrected the latency evaluation and now report the complete implementation cost in Table R3. We also distinguish the configurations used for the previously reported speedup figures.
Table R3. Complete-implementation latency. pinned GSM8K problems, five agents, .
| accuracy ↑ | TTFT, cached call ↓ | vs. KVComm ↓ | dense-prefill control | |
|---|---|---|---|---|
| KVComm (baseline) | % | 109.4 ms | — | ms |
| BCR, | 84.3% | ms | ms | ms |
| BCR, | % | ms | +82.8 ms | ms |
Dear Reviewer VwvB
We appreciate the reviewer's detailed comments on the experimental methodology and presentation. We agree with the concerns regarding the oracle-based evaluation, budget matching, and the interpretation of the granularity results.