acceptodds
Under review as a conference paper at ICLR 2027

Safety in Numbers: The KV Cache Was Never That Fragile to Repeated Pruning

Abstract

Long-running agents accumulate stale context from failed retrievals and superseded reasoning, occupying GPU memory and distorting attention. Removing it at the text layer forces a re-encode; at the KV-cache layer it is nearly free: excise rows, re-rotate survivor positions, continue. Whether that composite cache behaves like an honest re-prefill of the surviving tokens is untested, and production practice stays conservative. We test it directly, holding surviving content byte-identical and varying only operation count (Q1), window position (Q2), and eviction timing (Q3); relevance is oracle-given and the mechanism a reused RoPE rotation, so the operation itself is the only remaining variable. The caution inverts: repeating the cheap operation makes the result more faithful, not less. Over 8 rounds at 92.9% pruning, composite-KV quality reaches 91.3% of a clean re-prefill and improves with round count, with no drift over 26+ real-trajectory rounds (Q1). Re-indexing costs only 0.8% and matters only when survivor positions exceed the model's trained window, confirmed by a pre-registered model swap (Q2). Timing is the genuine open variable (Q3): correctness slips only a little as junk lingers, a small, task-dependent cost that timely pruning keeps low. The worry that repeated pruning compounds error is unsupported; it is timing, not repetition, that costs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, KV cache, agent context

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_euAaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether repeated KV-cache pruning is reliable in long-running LLM agents. The authors ask three questions. Do pruning errors accumulate across rounds? When is RoPE position re-indexing necessary? How does eviction timing affect output quality? The authors compare direct KV-cache editing with full re-prefilling. They use LongBench-style tasks, BrowseComp-Plus, and AppWorld trajectories. Task performance shows no steady decline across pruning rounds. After eight rounds, performance is 91.3% of the full re-prefill baseline. RoPE re-indexing is especially helpful for token positions beyond the model’s training context window. Delaying eviction leads to a small drop in accuracy. The size of the drop varies by task.

Critical strengths

1.The paper investigates repeated KV-cache pruning by separately examining pruning frequency, position re-indexing, and eviction timing. The comparison with full re-prefilling while keeping the surviving content identical provides a relatively controlled setting for evaluating the effects of cache editing. 2.The experiments on LongBench-style tasks and BrowseComp-Plus show that repeated pruning does not lead to progressively worse task performance under the tested conditions. The comparison between Llama-128K and Qwen-40K also provides useful evidence that re-indexing is particularly beneficial when token positions exceed the trained context window.

Critical weaknesses

1.The experiments rely on prerecorded trajectories and oracle relevance judgments, leaving the effects of pruning on intermediate decisions and tool calls unexamined. It remains unclear whether the reported stability extends to closed-loop agents, where errors may propagate across interactions. 2.The paper does not report end-to-end latency or throughput measurements, despite computational efficiency being a key motivation. The overhead of repeated cache editing and its impact on prefix-cache reuse also remain unclear.

Questions for authors
  1. Can the authors verify that the observed pruning stability extends to closed-loop agent execution, where modified KV states may influence subsequent reasoning and tool calls?
  2. What are the end-to-end latency and throughput implications of repeated pruning, including RoPE re-indexing overhead and potential prefix-cache invalidation?
  3. How sensitive are the results to imperfect relevance judgments? Since the experiments assume oracle-given relevance, would occasional removal of useful chunks significantly affect the conclusions about pruning stability?
4 comments
AuthorsOpenReview ↗

Thank you for your thoughtful review and helpful questions.

  1. Closed-loop execution. We chose fixed replay to isolate and quantify the effect of KV editing on output quality, and closed-loop evaluation is a next step. Holding the trajectory and retained tokens fixed lets us compare final answers under matched inputs. The same design can assess an intermediate output by ending the replay at that step. In closed loop, an early change in reasoning or tool use can alter subsequent inputs and affect task success (C2 of “Additional measurements and shared clarifications”).

  2. Latency and throughput. Editing a resident cache lowers completion latency by 24–67% and raises effective output rate to 1.3–3.0× that of recomputing the same retained tokens from scratch. C1 of “Additional measurements and shared clarifications” shows that replacing fresh prefill with deletion and RoPE re-indexing accounts for most of the gain. The new benchmark measures one final edit. C1 explains the per-round reference for the original +0.8% re-indexing overhead. Our recomputation baseline starts from an empty cache.

    Prefix-cache invalidation. Our method speeds up continuation by directly reusing retained KV within a session, the dominant pattern of prefix reuse in the cited agent traces. [Fang et al., §2.2] found that 99.96% of reused prefix blocks in their AgentBank-style traces were reused within the same session. Even when pruning changes the exact prefix, a session that carries forward the edited cache can reuse the retained KV without re-prefilling those tokens. Tiered storage supports retaining KV beyond GPU memory [Chen et al., §2.1]; saving a separate pre-edit copy could therefore keep the original prefix available for resuming or forking. Request-level cache hit rates and the net serving cost of repeated edits remain outside this evaluation.

  3. Imperfect relevance judgments. We fix the retained tokens to separate the effect of KV editing from the quality of the retention decision. The agent runtime decides which content is stale. Existing APIs can clear older thinking blocks or replace stale tool results with placeholders. Our experiments do not vary mistaken eviction, so its effect on pruning stability remains unmeasured (C2). [Claude Context Editing]

I appreciate the clarification and the new latency results. However, my main concerns remain. The stability of repeated pruning has not been validated in closed-loop agent execution, and the effects of imperfect relevance judgments remain untested. I recognize the value of the controlled analysis, and I will maintain my current rating.

Thank you for the clarification. However, my main concerns remain unresolved.

Q1: The authors acknowledge that pruning frequency and dwell time are confounded. Therefore, improved performance with more pruning rounds does not establish that repeated KV-cache editing does not accumulate errors, which is a central claim of the paper.

Online evaluation: Fixed-trajectory replay does not capture how pruning affects subsequent agent decisions and error propagation. The lack of closed-loop evaluation remains a limitation.

Thank you for the follow-up.

Q1. The practical insight remains: proactive pruning with in-session KV reuse can improve both quality and efficiency. We agree that improved performance with more pruning rounds does not establish the absence of accumulated editing errors. We have revised the claim to: “Under the tested fixed-replay schedules, earlier and more frequent pruning improves final-answer quality in Q1 and on BrowseComp-Plus, with identical final retained tokens.”

The quality benefit may come from removing stale content earlier; our experiments do not isolate the contribution of editing count. Earlier eviction also reduces the active KV memory footprint sooner, while our resident-cache measurements demonstrate the latency savings from avoiding fresh prefill.

Online evaluation. Fixed-trajectory replay is a deliberate experimental-design choice for controlled comparisons of KV-pruning schedules. Holding the trajectory and final retained tokens fixed lets us compare pruning schedules and clean recomputation under controlled conditions. This controls trajectory variation, although it does not separate pruning frequency from dwell time. Closed-loop evaluation addresses the complementary question of how pruning-induced changes to intermediate decisions propagate through subsequent interactions.

As follow-up work, we have already implemented a closed-loop runtime for in-session KV pruning and reuse, with preliminary results showing comparable task success and closely aligned execution trajectories. On 200 BFCL multi-turn tasks with Qwen3-8B, StructKV achieved a 44% success rate, compared with 45% for prefix caching, with 14 task-level wins and 16 losses (exact two-sided McNemar’s test, p = 0.856). Pruning occurred in 197 of the 200 tasks. We observed no statistically significant difference in success rates and no systematic trajectory divergence in this evaluation. These initial findings complement our controlled replay results; broader closed-loop validation remains ongoing.

Reviewer_WS2zOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper empirically studies the repeated KV-cache pruning in long-running LLM agents via three research questions: operation count, window position, and eviction timing. The authors investigate these questions in Longbench-style experiments and BrowseComp-Plus replication, and AppWorld trajectories, using full re-prefill of the identical surviving tokens as the reference baseline.

The main findings from the paper: 1. Repetition does not compound error 2. re-indexing provides a reasonable benefit when original token positions exceed the model's trained context window and 3. eviction timing is important and earlier is somehow better.

Critical strengths
  • The problem of KV cache pruning itself is significant, and the motivation is good.
  • The exp of Q3 provides some interesting insights, which demonstrate that the eviction timing is important.
Critical weaknesses
  • The presentation is a significant weakness of the paper. Verbose, repetitive, poor notation (e.g., no_reindex, no_prune).
  • If I understand correctly, in Q1, the experiment does not properly isolate the number of pruning operations and dwell time.
  • Oracle-given relevance itself is very tricky and can't be practical in a real use case. We can't know if we select the wrong information during real generation.
  • The experiments are based on agent trajectories, while in a real LLM response, it doesn't actually measure how the agent's subsequent trajectory will change after pruning.
Questions for authors
  • Can the authors improve the presentation?
  • Can authors also investigate the latency and throughput?
  • Can the authors do online generation instead of reply-based generation?
1 comment
AuthorsOpenReview ↗

Response to Reviewer WS2z

Thank you for your careful review and constructive suggestions.

  1. Presentation. We will consolidate repeated descriptions and use the four labels in C3 of “Additional measurements and shared clarifications” consistently across the text, tables, and figures.

  2. Latency and throughput. With the full cache resident, pruning and re-indexing reduces completion latency by 24–67% versus recomputing the same retained tokens. Avoided re-prefill accounts for most of the gain. C1 reports effective output rates and separates preparation from decoding.

  3. Online generation. We use fixed replay to quantify the effect of cache editing on the final answer under matched inputs. An intermediate output can be evaluated in the same way by stopping the replay at that step. Online generation lets an early action change the rest of the trajectory, so closed-loop evaluation is the next step for measuring how such differences affect task success (C2).

Additional points

The paper’s Q1: Does repeated pruning hurt quality? We agree that Q1 cannot isolate the effect of pruning count from dwell time. On a fixed trajectory, specifying each block's dwell time fixes its eviction point and thus the number of rounds with an eviction. Conversely, in our schedules, pruning in more rounds shortens the dwell time of discarded content. The eight-round schedule starts evicting after the second original document, while the one-round schedule evicts after the ninth.

Earlier, more frequent pruning yielded a net gain in Q1 and on BrowseComp-Plus in Q3. With identical final retained tokens, Q1 containment rose from 0.296 with one pruning round to 0.344 with eight. On BrowseComp-Plus, pruning each round improved containment over end-only pruning by 3.0 percentage points on Llama and 2.4 on Qwen. The AppWorld differences were statistically inconclusive. These results support proactive pruning: even with identical final retained content, earlier and more frequent removal can yield better answers. We have revised the abstract, introduction, and conclusion to make this finding explicit, and clarified the coupling between frequency and dwell time in the experimental discussion and limitations.

Oracle relevance. Fixed retention lets us compare editing and recomputation on the same retained tokens. A runtime policy can evict useful content. The effect of mistaken eviction remains unmeasured (C2).

Reviewer_DxPSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Long-running agents accumulate stale context (failed retrievals, abandoned reasoning) that can be pruned. Rather than re-prefilling the tokens that follow an evicted chunk, a cheap alternative is to excise the chunk's KV, re-rotate the surviving KV to close the positional gap, and continue. However, the surviving KV was computed under the influence of the evicted tokens, and it was previously untested how this composite cache compares to a clean re-prefill of the same surviving tokens. This paper studies that question with oracle-given relevance, on synthetic LongBench-style data and replayed BrowseComp-Plus and AppWorld trajectories, asking: (1) does repeated pruning compound error, (2) when is re-indexing positions necessary, and (3) how does dwell time (how long junk lingers before eviction) affect quality. It finds that repeated pruning does not degrade quality (91.3% of re-prefill after 8 rounds on synthetic data; statistically tied on real trajectories), that re-indexing matters only when positions would exceed the trained window, and that delayed eviction carries a small (2–3 point) accuracy cost whose size the authors argue is task-dependent. These results provide useful guidance for implementing KV-level eviction.

Critical strengths
  1. Important, understudied problem. Reusing the existing cache after eviction, instead of re-prefilling, is a key optimization for long-running agents. Yet how such edits affect downstream quality relative to a clean re-prefill has received little study. The paper studies this question systematically, with a controlled design that keeps the surviving tokens byte-identical across conditions so that only the operation's count, position handling and timing vary.

  2. Useful finding on re-indexing. If survivor positions stay within the trained context window, the retained KV can be left at its original positions: re-indexing gives no measurable gain on Llama-3.1-8B (128K window), but is decisive on Qwen3-8B (40K window). The pre-registered model swap makes this convincing, and it means serving systems can often skip rewriting cached KV.

Critical weaknesses
  1. Limited, dated model coverage. Most experiments use only Llama-3.1-8B and Qwen3-8B, both 8B models from 2024–early 2025 with full attention in every layer. Many recent models use hybrid or sparse attention (sliding-window/global, linear-attention or state-space layers, learned sparse attention). There, junk may be folded irreversibly into a recurrent state, may age out of a window on its own, or may never be attended at all, so the conclusions on repetition, re-indexing and dwell time may not transfer.

  2. Simplified, open-loop setting. All experiments replay fixed, pre-recorded trajectories (BrowseComp-Plus trajectories were generated by a different agent, GPT-5), and the evaluated model generates only once at the end. In a real agent, the model acts at every ReAct step, so a contaminated cache could change its tool calls and reasoning, and those divergences could compound over many iterations. This closed-loop effect, the main risk in the deployment setting the paper targets, is not measured; the paper acknowledges this (Section 5), but it limits how far the conclusions can be applied. Relevance is also oracle-given, so the interaction with an imperfect selector, whose mistakes are irreversible under deletion, is untested.

Questions for authors
  1. Where is the question placed in the BrowseComp-Plus replays? Section 4.4 puts the question at a median position of ~55K, outside Qwen3-8B's window. But in BrowseComp-Plus the question is normally the first message, at position ~0. Is it restated at the end before generation? If not, please clarify what exactly falls outside the window.

  2. What do the replayed BrowseComp-Plus trajectories contain? Do they include GPT-5's intermediate reasoning or final answer? The 8B models' scores (~0.63–0.79 containment) are close to GPT-5's reported 70% accuracy with the same retriever, far above the ~10% reported for open models such as Qwen3-32B acting as search agents. This suggests the models may be copying the answer from the trajectory, which would make the task easier and less sensitive to junk than in a real agent.

  3. Was Qwen3's thinking mode on or off? Qwen3-8B supports both thinking and non-thinking modes, and the paper does not say which was used.

2 comments
AuthorsOpenReview ↗

Response to Reviewer DxPS

Thank you for your careful reading and detailed feedback.

  1. Question location and re-indexing. The question appears in the opening prompt and is repeated before generation. The ~55K median is the position of the final restatement without re-indexing. On the same 669 questions, re-indexing improves containment by 3.1 percentage points on Qwen and 0.4 on Llama, with the latter difference statistically inconclusive. We have clarified the position and limited the window-pressure conclusion to the tested setting.

  2. GPT-5 content and answer cues. We exclude GPT-5's final answer but retain its reasoning summaries and search queries alongside the retrieved documents. In the 830-question constructed set, the reference answer matches a string in the reasoning summary for 309 questions and in a search query for 397, with possible overlap. The compared conditions share these cues, which may make final answering easier and reduce sensitivity to pruning. We will disclose this limitation and interpret the results as containment under supplied replay. The cited search-agent scores also reflect the difficulty of choosing searches and retrieving evidence.

  3. Qwen3 thinking configuration. Fixed replay uses a custom prompt ending in ### Answer followed by greedy decoding, with no separate thinking trigger or Qwen chat-template switch. Thus, this setup does not use the standard chat-template thinking/non-thinking switch. We have clarified this configuration in the manuscript.

Additional points

Architecture coverage. Our main real-trajectory comparisons use Llama-3.1-8B and Qwen3-8B. The synthetic Q1 experiments also include Yi-1.5-9B, and Appendix A.8 includes a Qwen3-30B-A3B check with sparse feed-forward layers and full attention. This does not test sparse attention. Sliding-window, sparse-attention, and recurrent architectures can handle stale content differently; we have clarified the coverage and limited the claims to the tested architectures.

Closed-loop behavior and selection errors. Fixed replay and retention decisions let us measure the effect of cache editing on final-answer quality. A change in an agent's next action can alter the rest of the trajectory, so measuring its effect on task success calls for closed-loop evaluation. Robustness to mistaken eviction also requires a separate evaluation (C2 of “Additional measurements and shared clarifications”).

Thank you for the clarifications. I choose to keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional measurements and shared clarifications

We have revised the relevant descriptions in the manuscript to clarify the interpretation of the pruning schedules, the scope of the window-pressure findings, and the model configurations and architecture coverage. These changes are marked in red in the revised manuscript.

C1. Latency and throughput

When the full KV cache is already on the GPU, pruning and re-indexing cuts time to complete generation by 24–67% and raises the effective output rate to 1.3–3.0× that of fresh recomputation on the same retained tokens. The gain comes mainly from avoiding re-prefill. Decode rates are nearly identical for the same retained content, and the shorter cache decodes 4–22% faster than the full cache.

We timed three BrowseComp-Plus replays with Llama-3.1-8B-Instruct on one NVIDIA A800, averaging two runs per trace and condition with 64 output tokens per run. Total latency runs from a resident full cache to generation completion, and the effective output rate is 64 divided by this time.

Full → retained tokensTotal latency (s), prune / recompute / fullEffective output rate (tokens/s), prune / recompute / full
5,748 → 4,2941.238 / 1.630 / 1.27751.7 / 39.3 / 50.1
15,780 → 7,0261.340 / 1.995 / 1.62647.8 / 32.1 / 39.4
27,701 → 21,6461.901 / 5.675 / 2.06733.7 / 11.3 / 31.0
Full → retained tokensPreparation (ms), prune / recomputeDecode rate (tokens/s), prune / recompute / full
5,748 → 4,29416.6 / 405.054.5 / 54.4 / 52.4
15,780 → 7,02624.7 / 678.850.9 / 50.9 / 41.7
27,701 → 21,64659.1 / 3,821.337.2 / 37.2 / 33.5

Preparation is deletion plus RoPE re-indexing, or prefill of retained tokens from an empty cache. Full context has no preparation step. Total latency also includes question prefill and decoding.

The original +0.8% is the extra wall time from RoPE re-indexing relative to the per-round wall time of the checkpoint-based Q1 workflow without that step.

C2. Fixed replay and retention

We use fixed replay to quantify the effect of cache editing on final-answer quality in a controlled comparison with the same trajectory and retained tokens. The same comparison can be applied at an intermediate step by stopping the replay at that step and treating its next output as the endpoint. Our final-answer results provide a reference for such comparisons. In closed-loop execution, a changed action can immediately alter later tool calls and observations. Measuring how these differences propagate to task success is a direction for future work.

We fix the retained tokens to compare cache editing with clean recomputation for the same retention decision. A runtime policy can evict useful content. The experiments do not vary selection errors, so robustness to mistaken eviction remains open. Runtime policies can, for example, clear stale tool results or evict parts of the interaction history. Claude Context Editing · Semenov and Dorofeev, 2026

C3. Experimental conditions

We will define the four conditions once and use these labels throughout the text, tables, and figures.

ConditionContent at generationCache constructionPositions
Full contextFull inputPrefill full inputOriginal
Recompute retained tokensSpecified retained tokensFresh prefillContiguous
Prune, keep positionsSame retained tokensEdit existing cacheOriginal survivor positions
Prune and reindexSame retained tokensEdit existing cacheRe-indexed to contiguous positions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.