- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether repeated KV-cache pruning is reliable in long-running LLM agents. The authors ask three questions. Do pruning errors accumulate across rounds? When is RoPE position re-indexing necessary? How does eviction timing affect output quality? The authors compare direct KV-cache editing with full re-prefilling. They use LongBench-style tasks, BrowseComp-Plus, and AppWorld trajectories. Task performance shows no steady decline across pruning rounds. After eight rounds, performance is 91.3% of the full re-prefill baseline. RoPE re-indexing is especially helpful for token positions beyond the model’s training context window. Delaying eviction leads to a small drop in accuracy. The size of the drop varies by task.
Critical strengths
1.The paper investigates repeated KV-cache pruning by separately examining pruning frequency, position re-indexing, and eviction timing. The comparison with full re-prefilling while keeping the surviving content identical provides a relatively controlled setting for evaluating the effects of cache editing. 2.The experiments on LongBench-style tasks and BrowseComp-Plus show that repeated pruning does not lead to progressively worse task performance under the tested conditions. The comparison between Llama-128K and Qwen-40K also provides useful evidence that re-indexing is particularly beneficial when token positions exceed the trained context window.
Critical weaknesses
1.The experiments rely on prerecorded trajectories and oracle relevance judgments, leaving the effects of pruning on intermediate decisions and tool calls unexamined. It remains unclear whether the reported stability extends to closed-loop agents, where errors may propagate across interactions. 2.The paper does not report end-to-end latency or throughput measurements, despite computational efficiency being a key motivation. The overhead of repeated cache editing and its impact on prefix-cache reuse also remain unclear.
Questions for authors
- Can the authors verify that the observed pruning stability extends to closed-loop agent execution, where modified KV states may influence subsequent reasoning and tool calls?
- What are the end-to-end latency and throughput implications of repeated pruning, including RoPE re-indexing overhead and potential prefix-cache invalidation?
- How sensitive are the results to imperfect relevance judgments? Since the experiments assume oracle-given relevance, would occasional removal of useful chunks significantly affect the conclusions about pruning stability?
4 comments
I appreciate the clarification and the new latency results. However, my main concerns remain. The stability of repeated pruning has not been validated in closed-loop agent execution, and the effects of imperfect relevance judgments remain untested. I recognize the value of the controlled analysis, and I will maintain my current rating.
Thank you for the clarification. However, my main concerns remain unresolved.
Q1: The authors acknowledge that pruning frequency and dwell time are confounded. Therefore, improved performance with more pruning rounds does not establish that repeated KV-cache editing does not accumulate errors, which is a central claim of the paper.
Online evaluation: Fixed-trajectory replay does not capture how pruning affects subsequent agent decisions and error propagation. The lack of closed-loop evaluation remains a limitation.
Thank you for the follow-up.
Q1. The practical insight remains: proactive pruning with in-session KV reuse can improve both quality and efficiency. We agree that improved performance with more pruning rounds does not establish the absence of accumulated editing errors. We have revised the claim to: “Under the tested fixed-replay schedules, earlier and more frequent pruning improves final-answer quality in Q1 and on BrowseComp-Plus, with identical final retained tokens.”
The quality benefit may come from removing stale content earlier; our experiments do not isolate the contribution of editing count. Earlier eviction also reduces the active KV memory footprint sooner, while our resident-cache measurements demonstrate the latency savings from avoiding fresh prefill.
Online evaluation. Fixed-trajectory replay is a deliberate experimental-design choice for controlled comparisons of KV-pruning schedules. Holding the trajectory and final retained tokens fixed lets us compare pruning schedules and clean recomputation under controlled conditions. This controls trajectory variation, although it does not separate pruning frequency from dwell time. Closed-loop evaluation addresses the complementary question of how pruning-induced changes to intermediate decisions propagate through subsequent interactions.
As follow-up work, we have already implemented a closed-loop runtime for in-session KV pruning and reuse, with preliminary results showing comparable task success and closely aligned execution trajectories. On 200 BFCL multi-turn tasks with Qwen3-8B, StructKV achieved a 44% success rate, compared with 45% for prefix caching, with 14 task-level wins and 16 losses (exact two-sided McNemar’s test, p = 0.856). Pruning occurred in 197 of the 200 tasks. We observed no statistically significant difference in success rates and no systematic trajectory divergence in this evaluation. These initial findings complement our controlled replay results; broader closed-loop validation remains ongoing.
Thank you for your thoughtful review and helpful questions.
Closed-loop execution. We chose fixed replay to isolate and quantify the effect of KV editing on output quality, and closed-loop evaluation is a next step. Holding the trajectory and retained tokens fixed lets us compare final answers under matched inputs. The same design can assess an intermediate output by ending the replay at that step. In closed loop, an early change in reasoning or tool use can alter subsequent inputs and affect task success (C2 of “Additional measurements and shared clarifications”).
Latency and throughput. Editing a resident cache lowers completion latency by 24–67% and raises effective output rate to 1.3–3.0× that of recomputing the same retained tokens from scratch. C1 of “Additional measurements and shared clarifications” shows that replacing fresh prefill with deletion and RoPE re-indexing accounts for most of the gain. The new benchmark measures one final edit. C1 explains the per-round reference for the original +0.8% re-indexing overhead. Our recomputation baseline starts from an empty cache.
Prefix-cache invalidation. Our method speeds up continuation by directly reusing retained KV within a session, the dominant pattern of prefix reuse in the cited agent traces. [Fang et al., §2.2] found that 99.96% of reused prefix blocks in their AgentBank-style traces were reused within the same session. Even when pruning changes the exact prefix, a session that carries forward the edited cache can reuse the retained KV without re-prefilling those tokens. Tiered storage supports retaining KV beyond GPU memory [Chen et al., §2.1]; saving a separate pre-edit copy could therefore keep the original prefix available for resuming or forking. Request-level cache hit rates and the net serving cost of repeated edits remain outside this evaluation.
Imperfect relevance judgments. We fix the retained tokens to separate the effect of KV editing from the quality of the retention decision. The agent runtime decides which content is stale. Existing APIs can clear older thinking blocks or replace stale tool results with placeholders. Our experiments do not vary mistaken eviction, so its effect on pruning stability remains unmeasured (C2). [Claude Context Editing]