acceptodds
Under review as a conference paper at ICLR 2027

KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing

Abstract

Post-hoc context erasing over the KV cache is challenging because a local edit has a global consequence: once a span has been processed, its influence propagates into the cached states of all subsequent tokens. This issue arises naturally in long-context LLM applications, where stale, incorrect, or harmful context may be identified only after prefill. Exact erasing must then recompute all tokens after the deleted span, making its computational cost depend on suffix length rather than erased-span length. We introduce KVEraser, a learned KV-cache editing method for efficient localized context erasing. KVEraser replaces the KV states of the erased interval with learned steering states while reusing the remaining cache unchanged. To learn a transferable erasing mechanism, we use a two-stage pipeline: generic span-neighbor pre-training followed by task-specific fine-tuning. Experiments show that KVEraser nearly matches full recomputation in post-erasure performance on in-domain tasks across 1K–32K contexts, while its latency increases by only 29.6% compared with a 17.6 increase for full recomputation. KVEraser also generalizes to unseen long-document QA with harmful factual distractors and tool-selection with malicious skill-file injections, achieving the best performance among approximate baselines with a 2.9–10.7 speedup over full recomputation. Notably, full-parameter eraser training is unnecessary: a rank-16 LoRA eraser, which trains only approximately 0.6% as many parameters as the generator, performs comparably to or better than its full-parameter counterpart.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, KV cache

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5UBDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies post-hoc context erasing: after a long context has been prefilled into the KV cache, a contiguous span is found to be invalid (a stale tool result, a misleading passage, an injected instruction). The goal is to make future decoding behave as if that span had never been present. Exact erasure needs re-prefilling the entire suffix, so its cost depends on suffix length rather than span length. KVEraser keeps the prefix and suffix caches unchanged and replaces only the KV states at the erased positions with learned "steering" states. These states come from a rank-16 LoRA adapter on the frozen generator backbone. The adapter processes the erased span, conditioned on the prefix cache. It is trained with a teacher-forced NLL objective on continuations that the frozen model generates from the clean, edited prompt.

Critical strengths

None

Critical weaknesses
  1. I think the experiment setup is problematic. In section 3, the work clearly acknowledged two other families: kv cache reuse and kv cache eviction, which imho, might still work under the setting of the paper, yet they are never compared in the experiment setting. I understand the authors might claim they have justified with words in section 3, I am not convinced by this argument, so please refer to Questions for my requests.
  2. . It's also unclear whether the method removes contamination carried by the suffix. The motivation is that a local edit has a global consequence: suffix states were computed while attending to the span. But the steering block is just new KV entries at positions m..n. It can change what future tokens attend to, but it cannot change the contaminated suffix states themselves, at any layer. So it may only suppress direct attention to the span, not the influence that has already been absorbed into suffix representations.
  3. Erasure is behavioral, not informational. The authors acknowledge that the reused suffix states still reflect the erased span, but the paper never tests whether the erased content can still be extracted, e.g., by asking about it directly or with adversarial probes. This matters because several motivating examples (retracted preferences, malicious injections) are safety- or privacy-adjacent. Either a leakage evaluation should be added, or the claims should be explicitly scoped away from these uses.
Questions for authors
  1. The authors claimed that " In these cases, a short problematic span is identified only after prefill, when the long context has already been written into the KV cache and can influence future decoding. Such context can lead to incorrect answers, failed actions, or unsafe behavior. The desired inference-time operation is therefore context erasing..." I'm a reviewer with background in kv cache eviction, so why kv cache eviction couldn't work in this case? I understand the authors might claim the residual are still kept in neighbor KV, but I don't think so. Second, KV Cache Eviction does not require prefill recomputation at all. Can the authors provide concrete empirical experiment and evidence to convince me?
  2. Following on that, I believe KV Cache Eviction should be considered as baselines, so can the authors provide kv cache eviction baselines in the paper's experiment as well?
  3. The authors claim in several places that "Exact erasing would recompute the suffix cache under the edited prompt, but is computationally costly for a long suffix." The introduction also motivates the method with long-running agents, tool-augmented assistants, and multi-turn conversational memory. However, I don't see concrete benchmarks matching this setting in the experiments. Examples would include multi-turn conversation (e.g., LongMemEval, or the selective-forgetting tasks in Hu et al. (2026), which the paper already cites) and agentic benchmarks with long tool-use trajectories (e.g., τ-bench, SWE-bench, or WebArena-style tasks).
  4. The central claim, "behave as if the span never appeared," is evaluated only through recovery of one target answer. Preservation is not tested at evaluation time. Every evaluation measures whether the single query the erasure targets is answered correctly. There is no test of whether the steered cache keeps the model's behavior on other queries over the same context. Section 4.1 justifies query-agnostic conditioning by arguing that one steered cache can serve many future queries. That is a strong motivation, but it is never tested. I would like to see multi-query probes of a single steered cache.
6 comments

Thank you for the detailed review.

Weakness 1, Questions 1 and 2. KV-cache eviction and reuse methods may still work in this setting. The paper lacks associated empirical comparisons, particularly KV-cache eviction methods.

Delete-and-shift is already an eviction baseline. The most direct eviction for context erasing is to evict exactly the KV entries of the span to erase. This is precisely the delete-and-shift baseline in the paper (Sec. 5.3, Fig. 2–3). It fails on NIAH (EM 0.05 at 32K) and scores 0.2–0.6 EM lower than KVEraser on QA and BFCL.

New eviction experiments. This concern motivates testing a stronger hypothesis: the span's residual influence is concentrated in a subset of suffix states that eviction heuristics can identify and discard. To test it, we always evict the span's own KV entries and then apply StreamingLLM [1] and SnapKV [2] to evict an additional 15% or 30% of the suffix KV. The exact match results are as follows.

MethodNIAH (over all context sizes)2WikiMultiHopQAMuSiQueIIRCBFCL
Delete-and-shift (evict span only)0.0150.5780.5670.6970.28
StreamingLLM 15%0.0150.50.450.6240.48
StreamingLLM 30%0.020.3660.3870.5340.57
SnapKV 15%0.01330.5730.5530.7020.3
SnapKV 30%0.01170.5640.5570.6970.3
KVEraser0.9980.9040.850.910.86

Across NIAH and all three QA benchmarks, additional suffix eviction provides no meaningful improvement over delete-and-shift and often degrades performance. NIAH exact match remains near zero, while QA performance generally decreases, particularly with more aggressive StreamingLLM eviction. The only exception is a marginal improvement from SnapKV on IIRC (0.702 vs. 0.697). On BFCL, StreamingLLM improves exact match from 0.28 to 0.48 and 0.57 with 15% and 30% suffix eviction, respectively, but still falls substantially short of KVEraser (0.86). Overall, these results show that additional suffix eviction does not reliably recover the behavior of the cleanly edited context.

This outcome is expected. Eviction heuristics assess the importance of KV states under the original, contaminated context. Consequently, suffix states that have strongly absorbed information from the erased span may be considered important and retained. Moreover, eviction can only remove KV states, without correcting the contamination already encoded in the retained states. Fundamentally, eviction aims to preserve generation quality under the original context, whereas context erasing aims to change model behavior to reflect the removal of the target span.

KV cache reuse methods. The reviewer also mentions chunk-reuse methods. Our local-suffix-repair baseline (Sec. 5.3) represents an adaptation of this family, using EPIC-style [3] selective recomputation of suffix windows at the two locations motivated by prior findings on position-dependent attention. Our empirical results show that KVEraser consistently outperforms this baseline (Sec. 5.4 - 5.5).

[1] Xiao et al. Efficient Streaming Language Models with Attention Sinks.

[2] Li et al. SnapKV: LLM Knows What You are Looking for Before Generation.

[3] Hu et al. EPIC: Efficient Position-Independent Caching for Serving Large Language Models.

Weakness 2. It's also unclear whether the method removes contamination carried by the suffix. The motivation is that a local edit has a global consequence: suffix states were computed while attending to the span. But the steering block is just new KV entries at positions m..n. It can change what future tokens attend to, but it cannot change the contaminated suffix states themselves, at any layer. So it may only suppress direct attention to the span, not the influence that has already been absorbed into suffix representations.

KVEraser indeed does not directly modify the contaminated suffix KV states. Our goal is to achieve functional erasure rather than reconstruct the clean suffix cache. Specifically, the steering KV is trained to compensate for the residual influence of the erased span carried by the suffix during attention aggregation in future decoding. This goes beyond simply suppressing direct attention to the erased span or contaminated suffix states.

To test whether attention suppression alone can achieve similar effects, we conduct an additional experiment building on the delete-and-shift baseline, which removes the erased span's KV entries and adjusts the positional encodings of the suffix states. Specifically, we further scale the attention weights from future tokens to the first 15% of suffix tokens to 75%, 50%, and 25% of their original values. We target these tokens because they are closest to the erased span and may carry stronger contamination. The results are shown below.

MethodNIAH (over all context sizes)2WikiMultiHopQAMuSiQueIIRCBFCL
Delete-and-shift0.0150.5780.5670.6970.28
75% attention weights0.0170.5470.590.7130.34
50% attention weights0.0270.4740.5070.7020.45
25% attention weights0.0250.3810.400.5340.48
KVEraser0.9980.9040.850.910.86

Attenuating attention to the nearby contaminated suffix states provides limited or inconsistent improvements over delete-and-shift, and all variants remain substantially below KVEraser across all benchmarks. These results show that suppressing attention to the erased span and nearby suffix states is insufficient to achieve functional erasure. KVEraser's substantially stronger performance is consistent with its learned steering KV compensating for the residual influence carried by the suffix, rather than merely suppressing attention to contaminated states.

Weakness 3. Erasure is behavioral, not informational. The authors acknowledge that the reused suffix states still reflect the erased span, but the paper never tests whether the erased content can still be extracted, e.g., by asking about it directly or with adversarial probes. This matters because several motivating examples (retracted preferences, malicious injections) are safety- or privacy-adjacent. Either a leakage evaluation should be added, or the claims should be explicitly scoped away from these uses.

Thank you for raising this concern. We add a leakage evaluation as follows.

Experiment Setup. We evaluate 100 QA contexts. For each, GPT-6 Astra (extra high reasoning) generates two information-extraction queries targeting the erased span, including a single-hop and a multi-hop query.

To minimize contamination from parametric knowledge, we keep a context only if the frozen generator:

  • answers the queries incorrectly without context;
  • answers the queries correctly with the original context and erased span alone but incorrectly with the cleanly edited context, so a correct answer after erasing is counted as leakage;

Example queries from one context:

  • Single-hop: What is Anne Celeste Heche's full birth date?
  • Multi-hop: In which daytime soap did the actress from Donnie Brasco and Volcano play a dual role during 1987–91?

Results. Using GPT-6 Astra (extra high reasoning) to judge correctness, we find that KVEraser leaks erased information on only 1.0% of queries (2/200). While KVEraser does not aim to completely eliminate residual information from the KV cache, it seeks to suppress the influence of erased content on downstream generation. Our results demonstrate that KVEraser empirically achieves this functional erasure objective.

Question 4.

  1. Preservation. The central claim, "behave as if the span never appeared," is evaluated only through recovery of one target answer. Preservation is not tested at evaluation time. Every evaluation measures whether the single query the erasure targets is answered correctly. There is no test of whether the steered cache keeps the model's behavior on other queries over the same context.
  2. Multi-Query Probes. Section 4.1 justifies query-agnostic conditioning by arguing that one steered cache can serve many future queries. That is a strong motivation, but it is never tested. I would like to see multi-query probes of a single steered cache.

We add a preservation evaluation as follows.

Experiment Setup. We evaluate 100 QA contexts. For each, GPT-6 Astra (extra high reasoning) generates two preservation queries targeting retained information, including a single-hop and a multi-hop query.

To minimize contamination from parametric knowledge, we keep a context only if the frozen generator:

  • answers the queries incorrectly without context;
  • answers the queries correctly under both original and edited contexts (by removing the erased span in text).

Example queries from one context:

  • Single-hop: For what job did Columbia Pictures hire Fred F. Sears in 1946?
  • Multi-hop: Which 1952 serial brought together the director and producer of The Night the World Exploded?

Results. Using GPT-6 Astra (extra high reasoning) to judge correctness, we find that KVEraser correctly answers 99.5% of preservation queries (199/200). This demonstrates that KVEraser largely preserves access to retained information under our single-hop and multi-hop probes.

Multi-Query Probes. For the leakage and preservation evaluations above, we jointly filter the 100 QA contexts using both sets of criteria. For each context, we construct the steered cache only once and independently evaluate all four queries against it, without query-specific reconstruction. As shown above, KVEraser suppresses erased information on 198/200 extraction queries while correctly answering 199/200 preservation queries. These results demonstrate that a single query-agnostic steered cache can support multiple downstream queries.

Question 3. The authors claim in several places that "Exact erasing would recompute the suffix cache under the edited prompt, but is computationally costly for a long suffix." The introduction also motivates the method with long-running agents, tool-augmented assistants, and multi-turn conversational memory. However, I don't see concrete benchmarks matching this setting in the experiments. Examples would include multi-turn conversation (e.g., LongMemEval, or the selective-forgetting tasks in Hu et al. (2026), which the paper already cites) and agentic benchmarks with long tool-use trajectories (e.g., τ-bench, SWE-bench, or WebArena-style tasks).

Thank you for raising this important point.

Our existing BFCL evaluation already covers multi-turn interaction contexts. Specifically, we use the multi-turn long-context subset of BFCL v4, comprising 100 examples from 40 conversations, with a median context length of 10.9K tokens and a maximum of 31.1K tokens (Appendix I). Similar to LongMemEval, our evaluation conditions the model on a multi-turn interaction history and assesses its response to a subsequent single request. Specifically, we test whether KVEraser can recover the correct next-tool selection after erasing malicious skill-file instructions. We will clarify the multi-turn nature of this evaluation in the main text.

To isolate the effectiveness of the erasing operator, we need cases where removing the target span changes the generator's response from incorrect to correct. Our QA and tool-selection evaluations follow this protocol (Sec. 5.2), selecting cases where the frozen generator fails with the target span present but succeeds after clean deletion.

End-to-end agent benchmarks introduce additional challenges. In such cases, the models can have high run-to-run variability on individual tasks. Moreover, failures on these benchmarks can arise from planning, tool use, memory retrieval, or reasoning errors, making it difficult to attribute a failure to a particular context span. Even with artificially injected distractors, identifying reliable cases remains challenging: a single span must cause the eventual task failure, while its clean deletion must restore success.

We acknowledge that end-to-end agent evaluations remain important directions for future work. We will clarify this limitation and distinguish these motivating applications from the settings directly evaluated in our paper.

Reviewer_8CTaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KVEraser for post-hoc context erasing in an LLM's KV cache. Instead of recomputing the suffix after removing a span, a LoRA-based eraser uses the prefix cache and the erased span to generate replacement KV states. The remaining cache and its original positions are preserved. Training combines span-neighbor retrieval pretraining with task-specific fine-tuning to approximate the model's behavior on the edited context. Experiments on retrieval, long-document QA, and tool selection show higher accuracy than the tested approximate baselines and lower latency than full recomputation.

Critical strengths
  1. The paper clearly motivates the practical need for post-hoc context erasing. Its LoRA-based eraser replaces only the target span’s KV states while reusing the suffix cache. The eraser is trained to approximate the model’s behavior on the edited context without recomputing the full suffix. This is a useful design when a short span needs to be erased from a long context.

  2. KVEraser shows good results across retrieval, QA, and tool selection. It achieves near-perfect retrieval accuracy from 1K to 32K (Figure 2) and higher exact match than the tested approximate baselines on three QA datasets and BFCL (Figure 3). Full recomputation remains more accurate on QA and BFCL, but takes 2.9–4.4× and 10.7× as long, respectively. This is a useful trade-off when recomputing the context is expensive.

Critical weaknesses
  1. The main-text evaluation does not isolate the contribution of learned KV steering from position-preserving span masking. Although delete-and-shift re-rotates suffix keys, it also compacts their positions, while KVEraser preserves them. Without a mask-only control that retains the original positions, it remains unclear whether a simpler, training-free edit could achieve comparable performance. This uncertainty limits my confidence in the additional benefit provided by the learned eraser.
Questions for authors
  1. Could you compare KVEraser with a mask-only baseline that blocks attention to the erased span while keeping the remaining KV states and suffix/query positions unchanged? How much improvement does the learned eraser provide over this baseline? A random KV replacement with a similar scale to the original KV states would also be useful to check whether the gains depend on the learned replacement.

  2. Stage-1 training includes preservation of non-erased information. Do you also have evaluation results showing that the same edited cache can still answer multiple questions about retained prefix and suffix facts?

3 comments

Thank you for the supportive review!

Q1. Could you compare KVEraser with a mask-only baseline that blocks attention to the erased span while keeping the remaining KV states and suffix/query positions unchanged? How much improvement does the learned eraser provide over this baseline? A random KV replacement with a similar scale to the original KV states would also be useful to check whether the gains depend on the learned replacement.

Thank you for the insightful question. We compared against two additional baselines:

Mask-only: We mask attention to the erased span while keeping the remaining KV states and token positions unchanged. Random KV: For each layer and KV head, we independently sample standard Gaussian noise for K and V within the erased span, then rescale each to match the root mean square (RMS) of the corresponding original states, computed over tokens and feature dimensions. The exact match results are as follows.

MethodNIAH (over all context sizes)2WikiMultiHopQAMuSiQueIIRCBFCL
Mask-only0.0170.5780.5530.7020.27
Random KV0.010000
Delete-and-shift0.0150.5780.5670.6970.28
KVEraser0.9980.9040.850.910.86

Mask-only performs comparably to delete-and-shift, while random KV replacement performs substantially worse. KVEraser consistently outperforms both baselines across all benchmarks, demonstrating the substantial benefits of the learned eraser over masking or random replacement alone.

Q2. Stage-1 training includes preservation of non-erased information. Do you also have evaluation results showing that the same edited cache can still answer multiple questions about retained prefix and suffix facts?

We construct 100 samples, each concatenating three unrelated 128-256-token chunks from distinct Wikipedia articles: a prefix, a target chunk to erase, and a suffix. For each, GPT-6 Astra (extra high reasoning) generates two queries about the prefix and two about the suffix, mixing single-hop and multi-hop questions.

To minimize contamination from parametric knowledge, we include a sample only if the frozen generator:

  • Answers all four queries correctly with both the original context and the cleanly edited context (i.e., after removing the target chunk);
  • Fails to answer the queries without the context.

We also use GPT-6 Astra with extra high reasoning to judge the responses.

Below are example queries from one QA sample:

  • Prefix 1: Where did Ling and her friends obtain the alcohol they gave to Fei-hung during the fight?
  • Prefix 2: What poisonous substance made the husband of the woman who visited Kei-ying ill?
  • Suffix 1: How much wind power was estimated to be needed for the College of the Marshall Islands to become energy self-sufficient?
  • Suffix 2: In which capital city was the processing plant associated with the man who described coconut-oil fuel experiments in September 2007?

For each sample, we construct the edited KV cache once and independently evaluate all four queries using the same cache. KVEraser achieves 99.75% (399/400) accuracy on the queries. These results demonstrate that a single KVEraser-edited cache can reliably answer multiple independent queries about retained prefix and suffix information.

Thank you for running these additional experiments. The mask-only comparison addresses my main concern: KVEraser performs much better even when the baseline preserves the original token positions. The results on multiple questions using the same edited cache also help address my question about preserving the remaining information.

The preservation test uses short, unrelated passages, so I would mention this limitation alongside the result. Please include these experiments and their setup in the revised paper.

My main concern has been addressed, and I will keep my rating of 3 (Weak acceptance).

We are glad that we've addressed your concerns. We will include these experiments and their setup in the revised paper. Thank you!

Reviewer_Mf3uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces KVEraser, a learned KV cache editing method for efficient post-hoc context erasing in LLMs. Using a LoRA-based eraser, it removes unwanted content by replacing erased spans with learned steering states while preserving the remainder. The authors propose a two-stage training pipeline: generic span-neighbor pre-training on Wikipedia chunks, followed by task-specific fine-tuning on controlled tasks like needle-in-a-haystack. On Qwen3-8B and Llama-3.1-8B across 1K–32K tokens, KVEraser matches full recomputation accuracy while reducing latency growth.

Critical strengths

S1. KVEraser replaces only the marked interval and shares the frozen generator backbone via a rank-16 LoRA adapter—a practical, economical approach to post-hoc cache editing.

S2. Empirical benefits extend beyond synthetic benchmarks.

Critical weaknesses

W1. The paper motivates its query-agnostic design by reuse across requests, but does not report multiple independent queries against the same edited cache. In NIAH, the earlier needle is always erased and the later one retained, leaving a last-occurrence shortcut unruled out despite randomized positions.

Questions for authors

Q1. Does the NIAH result persist when the erased needle is randomly chosen to be the earlier or later occurrence?

3 comments

Thank you for the positive feedback!

W1. The paper motivates its query-agnostic design by reuse across requests, but does not report multiple independent queries against the same edited cache.

We ran a multi-query evaluation on 100 QA contexts. For each, GPT-6 Astra (extra high reasoning) generates two queries targeting the erased span and two targeting retained information (single- and multi-hop).

To minimize contamination from parametric knowledge, we keep a context only if the frozen generator

  • answers all four incorrectly without context;
  • answers both erasure queries correctly with the original context and the erased span alone but incorrectly with the cleanly edited context, so a correct answer after erasing is counted as leakage;
  • answers both preservation queries correctly under both original and edited contexts.

Example queries from one context:

  • Erasure 1: What is Anne Celeste Heche's full birth date?
  • Erasure 2 (multi-hop): In which daytime soap did the actress from Donnie Brasco and Volcano play a dual role during 1987–91?
  • Preservation 1: For what job did Columbia Pictures hire Fred F. Sears in 1946?
  • Preservation 2 (multi-hop): Which 1952 serial brought together the director and producer of The Night the World Exploded?

The steered cache is constructed once per context and all four queries are evaluated against it. GPT-6 Astra judges correctness. KVEraser leaks erased information on 1.0% of erasure queries (2/200) and answers 99.5% of preservation queries (199/200). A single query-agnostic steered cache thus serves multiple downstream queries.

Q1. Does the NIAH result persist when the erased needle is randomly chosen to be the earlier or later occurrence?

Thank you for the insightful question. We re-evaluated the original KVEraser checkpoint under a new setting where the erased needle is randomly chosen to be the earlier or later occurrence. The exact match results are as follows.

1K2K4K8K16K32K
110.99110.99

These results demonstrate that KVEraser can effectively erase either occurrence without relying on a fixed last-occurrence shortcut.

Thanks for the additional experiments. I'll keep the score.

Thank you for reading our response!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.