- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when a KV-cache retention budget set for a short context can be reused for longer contexts. With a fixed ranking, the minimum budget for retaining an entire evidence set equals the number of evidence positions plus the number of competitors. Competitors are non-evidence positions ranked ahead of the lowest-ranked evidence position. In controlled retrieval experiments, adding filler text had little effect on the budget. Adding distractors at a fixed context length substantially increased the required budget. The authors used short-context experiments to predict budgets for longer contexts. In 22 of 25 testable settings, the predictions were within a factor of two of the measured budgets.
In the eight-value retrieval experiment, the model retained all annotated values but answered only 9 of 20 prompts correctly. At the same budget, retaining the full key–value records and the question raised the number of correct answers to 20 out of 20. This result suggests that correct answers also depend on linking the values to their keys and the question.
Critical strengths
-
The controlled experiments separate the effects of distractor count and context length. At 32K length, distractors raise Qwen3.5-9B’s retention budget from about 24 to 5,277 positions, while added filler has little effect.
-
The evaluation links evidence retention, answer accuracy, and physical cache savings. Even with all annotated values retained, only 9/20 ordered answers are correct (Section 5.4). Dense and hybrid models achieve approximately 97–102× and 11× storage reductions while matching full-cache answer accuracy on tested prompts.
Critical weaknesses
-
The budgeting guidance has limited validation beyond controlled retrieval with 15–20 paired prompts per condition. Rank audits require a specified evidence proxy, leaving unclear how budget reuse can be assessed in natural tasks where evidence is difficult to annotate.
-
The LongBench analysis applies the same 0.8 validity threshold to retrieval accuracy and token F1 without sufficient justification. All three cohorts fall below this threshold, with full-cache F1 scores of 0.298–0.335, so no budget-crossing estimates are reported. This limits the evidence about budget requirements for natural QA.
Questions for authors
-
For natural tasks without directly annotatable evidence, which of the described calibration or audit steps remain applicable, and what conditions for budget reuse can they test?
-
Can the authors justify applying the 0.8 threshold to both retrieval accuracy and LongBench token F1, and use the existing curves to assess how the threshold affects budget estimation?
3 comments
Response to your last concern
Thank you for engaging with the response, and for confirming that the 50/80/90% analyses address the threshold question. You raise three remaining points about E1: the post-hoc filter, the sample sizes, and the two datasets. We take them in turn. The first answer also corrects an imprecise statement in our earlier post.
1. The result does not depend on the post-hoc filter. In our earlier post we wrote that the registered analysis set was "censored at the lowest rung." That is accurate only for Qwen on 2WikiMQA, where SnapKV already reaches half of full accuracy at 32 retained positions in every condition. The other three registered sets have well-defined crossings. The registered set is specified in the frozen design: every example the full cache answers correctly at the shortest length, with no closed-book filter, so no selection was made after seeing results.
| Registered set, SnapKV-w32 | n | Length change | (positions) | Ratio [95% CI] | A fraction predicts |
|---|---|---|---|---|---|
| Llama-3.1-8B, 2WikiMQA | 27 | 8K → 32K | 42 → 42 | 1.00 [0.85, 1.18] | 4.0 |
| Llama-3.1-8B, HotpotQA | 28 | 16K → 32K | 44 → 44 | 0.99 [0.87, 1.13] | 2.0 |
| Qwen3.5-9B, HotpotQA | 32 | 16K → 32K | 32 → 33 | 1.03 [0.99, 1.17] | 2.0 |
These are the pre-registered estimates. Each interval sits near 1 and far from the fraction prediction, including a 4× length change on 2WikiMQA. The same sets also answer the competitor question: adding 48 Wikipedia passages gives 0.96 [0.77, 1.17], 1.00 [0.91, 1.12], and 0.97 [0.90, 1.06]. The post-hoc filter was never needed to obtain the result. It only made the Qwen 2WikiMQA cell measurable, and it gives the same answer there.
The same pattern holds with no selection of any kind: all 108 candidate examples per model, scored by token F1. On Llama, the SnapKV half-F1 budget is 46, 47, and 47 positions at 8K, 16K, and 32K on 2WikiMQA, and 45 → 48 on HotpotQA (16K → 32K). This secondary analysis has no bootstrap intervals, so we report it only as a consistency check.
2. The sample sizes are sufficient for the question being tested. The test is whether the budget scales with length, as a fraction does, or stays flat, as a count does. That is a contrast between a ratio of 1 and a ratio of 2–4, and the contrast is far larger than the uncertainty at these sample sizes. Every interval above excludes the fraction prediction. Even the upper bounds (1.13–1.18) are at least 1.7× below what a fraction predicts. The examples are also paired: each prompt is measured at every length, so prompt difficulty cancels out of the ratio. A larger n would narrow the intervals but could not move them from about 1 to 2–4. Across both models and both tasks, the registered sets contain 122 examples.
3. On the number of natural datasets. We agree that two LongBench tasks are a starting point, not the full range of natural workloads. Three other natural-format results bear on the same question:
- InfiniteBench passkey and 10-digit retrieval show count transfer, with NLL crossings of 23–104 positions on four models: Qwen3.5-9B, Qwen3.5-4B, Qwen2.5-7B-1M, and Phi-4.
- Single-span audits, frozen before the runs on five unseen InfiniteBench cells, never under-provisioned in 20 of 20 scorer–cell pairs (App. G).
- RULER, BABILong, and LongBench (Sec. 5.5) are where the paper predicts that a literal-retrieval count should not transfer, and it does not. That is the boundary the protocol is meant to detect.
For the revision, we will extend E1 to MuSiQue, the third LongBench multi-hop task, under the same frozen protocol. We will also state in the abstract that the natural-task validation covers multi-document QA on two model families.
Summary. Your remaining concern was that the natural-task evidence rests on a post-hoc filter and small samples. The pre-registered sets, with no filter, give the same conclusion, with tighter intervals than the filtered analysis in all three cells that are not censored. They exclude the fraction prediction in every case and cover 122 paired examples. We hope this resolves the first concern as well. If it does, we would be grateful if you would consider updating your rating.
MuSiQue result, and a test with no example selection
In our previous reply we committed to extending the natural-task test to MuSiQue, the third LongBench multi-hop task. That run is complete, and we report it here together with a second test that answers your sample-size point directly. Both designs, their predictions, and their inputs were hashed before the first eviction run (frozen 2026-11-17 19:28 UTC). The revised PDF now includes both.
1. MuSiQue under the same protocol (all three registered predictions held). We used all 100 LongBench MuSiQue examples on Qwen3.5-9B and Llama-3.1-8B. Only 11 originals fit in 16K, so the filler test runs from 32K to 64K, and 48 Wikipedia competitor passages are inserted at 64K. The analysis set is the registered one, with no post-hoc filter: every example the full cache answers at 32K.
| SnapKV-w32, registered set | n | Filler 32K → 64K, ratio [95% CI] | +48 passages at 64K, ratio [95% CI] | 90% count fixed at 32K: share of full at 64K |
|---|---|---|---|---|
| Qwen3.5-9B | 36 | 1.09 [0.93, 1.23] (32 → 35) | 0.92 [0.81, 1.11] | 1.00 |
| Llama-3.1-8B | 27 | 0.98 [0.74, 1.45] (57 → 56) | 1.03 [0.73, 1.41] | 0.94 |
- The count transfers on a third dataset. Doubling the length leaves the budget flat in both models, and both intervals exclude the 2.0 a fraction predicts. MuSiQue therefore behaves like 2WikiMQA and HotpotQA.
- The direct reuse test passes. The count chosen at 32K, applied unchanged at 64K, kept at least its target minus 0.10 in all 12 registered cells (two scorers, two models, targets of 50%, 80%, and 90%).
- Same-topic competitors again do not raise the budget, which replicates the E1 finding on new data.
- Two caveats, stated plainly. TOVA reaches half accuracy at the lowest tested budget in every condition, so its ratio carries no information. Llama's own full-cache accuracy also drops at 64K (from 1.00 to 0.67 on this set), so every share is relative to the full cache at the same length.
2. A test with no example selection at all (your sample-size point). We ran all 100 2WikiMQA and all 100 HotpotQA examples at their natural length, with nothing inserted and nothing filtered. For each model and task, we calibrated the count on the shortest third of examples (median 4.3K–10.0K tokens) and applied it unchanged to the longest third (median 10.6K–17.0K). This tests the claim on different, naturally longer examples, not on padded copies of the same ones.
- Registered prediction 1 held in 16 of 16 cells. At the 90% target, the count kept 0.93–1.00 of full-cache accuracy for SnapKV and 0.80–1.00 for TOVA. SnapKV's 90% budget did not grow with length: 169 → 91, 111 → 51, 676 → 104, and 62 → 62 positions.
- Registered prediction 2 failed in 9 of 16 cells, and we report it. A fraction calibrated the same way used 1.3–13.7× more positions. In 9 of 16 cells, that extra memory raised accuracy by more than 0.05 above a target the count already met.
What this means for your three points.
- Post-hoc filter: every number above uses registered sets, or no selection at all.
- Sample size: the natural-text evidence now covers 3 tasks, 2 model families, and 63 more registered MuSiQue examples, plus 400 unselected natural-length examples (100 per model and task).
- Two datasets: there are now three, and the result holds on all three.
The abstract now states this scope: count transfer on natural multi-hop QA, measured on three LongBench tasks and two model families.
We hope this closes the remaining questions. If it does, we would be grateful if you would reconsider your rating.
Thank you for the detailed response. E1 provides useful initial evidence. However, it covers only two datasets, with 34 and 23 context-dependent examples for Qwen and Llama, respectively. The post-hoc filtering also makes the results less conclusive. I therefore consider my first concern only partially addressed.
The additional analyses at 50%, 80%, and 90% of full-cache F1 adequately address my second concern. However, the limited natural-task validation remains a concern, so I will maintain my original rating.