- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether KV Cache can serve as an updatable long-term memory. The central challenge is that when new knowledge replaces outdated information, directly deleting or modifying the KV states associated with the old knowledge may also damage useful historical information. Through a series of controlled interventions, the authors compare strategies such as Whole-source Masking, Value-only Masking, and KV Recomputation, and analyze their effects across multiple models and tasks.
Critical strengths
- Clear Motivation: This paper studies the problems of using KV Cache as an updatable memory.
- Comprehensive Experiments: The experiments cover a wide range of models, tasks, and KV Cache updating strategies, with detailed controlled experiments.
- Interesting Findings: The paper reveals several interesting phenomena, such as KV Cache updates potentially damaging historical information and subsequent states retaining outdated information.
Critical weaknesses
-
Unclear Necessity of KV Cache Editing: Although Appendix D.7–D.9 evaluates the cost of KV Cache updates, the paper lacks a sufficient comparison with Text Editing + Re-prefill under optimized deployment settings. For example, the official vLLM benchmark [1] reports a median TTFT of 763.6 ms on a single H100 with Llama-3.1-8B, approximately 20K input tokens, and a concurrency of 8.
-
Fragmented Empirical Analysis: The paper includes many experiments, but the connections between them are not sufficiently clear. Although individual findings are analyzed, they do not form systematic scientific insights. For example:
- Figure 2: Value-only Masking achieves perfect accuracy on the Quantity Task.
- Table 1: The same strategy fails to consistently handle both current and historical questions on MQuAKE.
- Figure 3: Source Masking produces opposite effects on Qwen and Gemma.
The authors provide some local explanations but do not sufficiently identify general patterns across tasks and models.
-
Limited Analysis Beyond Observed Phenomena: Some conclusions remain at the level of empirical observations. For example, in Section 6.1, Learned KV Readout achieves an AUC of 0.862, but Detector-selected Masking does not clearly outperform Random Masking on natural text. Although the authors distinguish Recognition and Intervention in controlled experiments, they lack a systematic analysis of how these two types of errors contribute to the results on natural text.
[1] The State of FP8 KV-Cache and Attention Quantization in vLLM
Questions for authors
See Weaknesses.
4 comments
Response to Reviewer oB6m (2/2)
W3. "Limited Analysis Beyond Observed Phenomena"
Thank you for this point. Appendix B.12–B.14 relates detector decisions to generated answers on natural text, but does not combine the two error types or count the cost of false alarms; we do both here.
On natural text, even a perfect detector would fix only 9 of the 116 changed answers, and recognition errors keep every detector at a net gain of 3 or less. Intervention errors appear when a question still needs the old record.
The 0.862 AUC is for a learned KV readout on synthetic record pairs. Natural-text masks are chosen by text detectors; Qwen3.8 correctly classifies all 300 synthetic test cases (Section 6.1).
A detector can err in two ways: it can miss a real change, or it can flag an unchanged record and hide information that is still correct. The 116 natural-text questions (OAKS) all contain a real change, so they reveal only misses. Appendix B.11 pairs each question with two controls in which nothing changed (the later passage repeats the earlier one or comes from another book). We have now generated Qwen3.8-27B answers for all 348 cases, so false alarms can break answers.
Masking every real change would fix 9 answers and break none, so a perfect detector gains 9. Masking every control would break 52 correct answers. Seven detectors, each reading the two passages alone or with the question, give 14 selectors; each masks the earlier passage when it flags a change, and Qwen3.8-27B answers every question:
| Detector, input | Changes flagged (of 116) | Answers fixed (of 9) | False alarms (of 232) | Answers broken | Net change |
|---|---|---|---|---|---|
| Qwen3-4B, passages | 35 | 2 | 11 | 3 | −1 |
| Qwen3-4B, + question | 71 | 3 | 17 | 6 | −3 |
| Qwen3-8B, passages | 59 | 5 | 28 | 6 | 0 |
| Qwen3-8B, + question | 78 | 3 | 8 | 0 | +3 |
| Qwen3-32B, passages | 35 | 2 | 12 | 0 | +2 |
| Qwen3-32B, + question | 74 | 3 | 8 | 0 | +3 |
| Qwen3.8-27B, passages | 17 | 0 | 0 | 0 | 0 |
| Qwen3.8-27B, + question | 37 | 1 | 0 | 0 | +1 |
| Gemma 4 26B-A4B, passages | 16 | 1 | 1 | 0 | +1 |
| Gemma 4 26B-A4B, + question | 55 | 3 | 0 | 0 | +3 |
| Gemma 4 31B, passages | 40 | 2 | 1 | 0 | +2 |
| Gemma 4 31B, + question | 75 | 3 | 0 | 0 | +3 |
| Llama-3.1-70B, passages | 29 | 1 | 3 | 1 | 0 |
| Llama-3.1-70B, + question | 71 | 3 | 2 | 1 | +2 |
Qwen3-8B on passages alone also fixes one control, so its net change is 0.
-
Recognition errors. No selector gains more than +3 of the possible +9:
- Strict detectors such as Qwen3.8 raise no false alarms but flag only 17–37 of the 116 changes.
- Looser ones such as Qwen3-4B and Qwen3-8B flag 35–78, but their false alarms break up to 6 correct answers.
-
Intervention errors. The natural-text questions ask about the state after the change, and no correctly flagged change broke an answer. Historical questions still need the old record: with true replacement labels, whole-record masking lowers historical accuracy from 100% (300/300) to 93.3% (280) for Qwen3.8 and 82.3% (247) for Gemma 4 31B (Table 37, Appendix B.13). In Wikipedia revision histories, masking only the known changed passages lowers complete answers from 68.3% to 66.2% (Qwen3.8) and from 54.9% to 52.1% (Gemma 4 26B-A4B), a small but significant drop on questions that mostly ask about earlier versions (Section 5.2).
We have added this breakdown and the new answers to Section 6.1 and Appendix B.14 (Table R2).
All changes in the revised PDF are marked in blue. We hope these results address your concerns, and we would welcome further discussion.
Best regards,
The Authors
Thank you for the thoughtful and detailed rebuttal. I appreciate the authors' additional experiments and their candid discussion of the limitations. The new analyses address several of my concerns.
I also see potential long-term value in KV Cache editing, especially for future models that autonomously manage and update their own contexts. However, its practical advantages over optimized re-prefilling remain limited at this stage.
Overall, the rebuttal has improved my assessment of the paper, but I will maintain my original score of 2 (I would lean toward a score of 2.5 if that option were available).
Dear Reviewer oB6m,
Thank you for reading our rebuttal carefully and for recognizing our new experiments and analyses.
We completely agree that optimized re-prefilling is widely used in practice; frontier agent harnesses, including Codex and Claude Code, rely on context compaction and re-prefilling every day. And yet, KV-cache management still gives us plenty of headaches in everyday use :)
We only want to clarify in good faith that our goal is not to propose a KV-cache editing algorithm to replace re-prefilling, but to show through experiments where the growing practice of KV-cache reuse and in-place editing works and where it falls short.
As our results show, in-place editing and compaction each have their own strengths and blind spots. The two approaches do not have to compete, and can be used together: depending on the question, a system can reuse cached prefixes, mask only replaced values, and re-prefill when later states need to be updated.
We hope these findings help the community design future context-management systems with a clearer sense of what each operation keeps and what it loses.
Thank you again for your time and constructive feedback throughout the review process.
Best regards,
The Authors
Response to Reviewer oB6m (1/2)
Dear Reviewer oB6m,
Thank you for your review, and for finding the motivation clear, the experiments comprehensive and the findings interesting.
W1. "Unclear Necessity of KV Cache Editing"
Prompt Cache, CacheBlend, EPIC and KVLink serve requests from stored KV caches; editing a cached history in place is a natural way to update memory without re-reading it.
We share your question about whether cache editing pays off. In our experiments it is hard to make work well, and deleting the old text and re-prefilling (Recompute in Tables 1 and 3) faces the same problem: both must decide what to keep.
Since Appendices D.7 and D.9 do not use an optimized kernel, we added a FlashAttention 3 test: Llama-3.1-8B-Instruct on one H200, one quantity fact at 10%, 50% or 90% of a 20K-token history, then its current and historical questions, each asked 16 times (3-run medians; ranges over positions and re-prefill variants):
Optimized re-prefill can still lose many answers. Deleting the old record loses the unit here too. A faster kernel changes how fast the cache is rebuilt, not what it is rebuilt from, so the main-text losses carry over:
Cost depends on where the fact sits and how the edit changes generation. In your vLLM setting (Llama-3.1-8B-Instruct, ~20K tokens, 8 concurrent requests, prefix caching), TTFT after a one-word edit is 0.55 s when the fact is at 90% of the history, 1.9 s at 50% and 2.9 s at 10%, since only the prefix before the edit is reused. In the table above, masking the whole record takes ~30 ms, yet over 32 requests it is slowest (111–112 s) because its historical answers hit the token limit; Appendix D.9 shows the same for both Gemma models.
We have added both measurements to Appendix D.9 (Tables R4–R5) and Section 6.2.
W2. "Fragmented Empirical Analysis"
Thank you for this suggestion; we agree that the links between the experiments were not clear enough, and we have made them explicit at the end of Section 2 and in Section 8.
What connects the experiments is the mismatch in our title: hiding an outdated record's tokens is not the same as removing the outdated fact. An edit is exact about which positions it hides, but not about which information it removes, because a fact is not confined to its own positions. Your three examples show this:
The general patterns are:
No edit we tested works across all tasks and models, so we do not claim a general update rule. Our results instead caution against treating a KV cache as a memory that can be updated by masking or deleting an old fact's positions.