- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces cache engineering, which lets an agent or its harness directly control its active KV cache instead of editing the text history. The method defines two cache operations: drop, which removes selected KV entries from future attention without recomputing later states, and repos, which re-rotates retained keys to close position gaps. Since the resulting cache states depend on the operation history rather than the token prefix alone, the paper proposes AKV, an SGLang-based serving system with a stateless interface, a radix tree extended with drop and reposition nodes, drop-aware eviction, and segment-batch attention for restoring evicted states in one prefill pass. Experiments on three models and four agent benchmarks show that a rolling policy retaining the latest 12 tool responses improves average pass rate and reduces cost by 27.7% compared with summarization-based baselines, with up to 2.67× output throughput in serving.
Critical strengths
- The paper identifies a meaningful systems problem introduced by agent-controlled KV cache. Once drop and repos are allowed, the token prefix no longer determines the cache state (Figure 5), and the paper addresses this with a concrete serving design: drop and reposition nodes in the radix tree, history-aware matching across positional versions, and the separation of logical drop from physical eviction. This design is useful for agent-controlled KV methods in general, beyond the evaluated policy.
Critical weaknesses
-
(Quality) The accuracy gain is not clearly attributed to cache engineering, since the context policy and the caching mechanism are changed together.
a. The baselines combine a summarization policy with text re-prefill, while AKV combines a rolling observation-masking policy () with direct KV drop. Both factors therefore differ between the compared settings.
b. Text-level masking has been reported to perform comparably to or better than summarization for agent context management (Lindenbauer et al.), and the paper attributes AKV's advantage on long trajectories to information loss from summarization (Figure 7), which relates to the baseline policy rather than the mechanism.
c. The control that would directly support the main thesis, that KV computed while the dropped context was visible retains useful information, is the same rolling-masking policy with text re-prefill. This setting is currently reported for cost (1.85×) but not for accuracy.
d. In Table 3, drop without repos is below Standard on BCP, and repos accounts for most of the gain. Since repositioned keys no longer match the positions where their hidden states were computed, some analysis of why these states help would be valuable.
-
(Significance / Quality) The history-dependent cache semantics may make restoration cost scale with the logical history rather than the active context, which could limit the long-horizon setting the paper targets.
a. Because each KV state depends on what was visible when it was computed, restoring an evicted state seems to require recomputing earlier tokens including dropped ones (as in the Figure 10 example), so a cold restore may involve a prefill over the full logical sequence.
b. For the 500+ round trajectories that motivate the paper, this cost may approach or exceed the context window that drop and repos aim to stay within, and the stateless interface also resends the full history and operation log on each request.
c. Each method replays trajectories collected from its own policy, so the serving workloads differ across methods.
d. AKV at high concurrency is measured until the first 30 completed tasks, while Standard uses a fixed one-hour window with unfinished requests, which may affect the latency and throughput comparison.
Questions for authors
-
(ref Weakness 1) Could the authors report accuracy and cost for the same rolling-masking policy () with text re-prefill under the same harness?
-
(ref Weakness 1) What explains the 7-point improvement from repos over drop alone on BCP, and does the gain remain when the position budget is not binding?
-
(ref Weakness 2) How often are cold restorations triggered in the serving runs, what is their cost relative to active-context prefill, and how does AKV behave when the logical sequence exceeds the context length?
-
(ref Weakness 2) Would it be possible to repeat the serving comparison on a common replayed workload with identical measurement windows?
-
(Presentation, minor) The headline accuracy gain appears as 1.7 points (abstract), 3.5% (contributions), and 1.3–2.1% (Section 5.2). The RSI result is described as "expanding the cost reduction to 17.8%," which is smaller than 27.7% and uses a different baseline. Section 5.2 refers to "all four models" while Table 1 shows three. Minor typos: "Both the baselines deploy" and the missing space in "directly.AKV."
4 comments
W2.a-b, Q3.1: The cost and frequency of AKV restoration
The history-dependent cache semantics may make restoration cost scale with the logical history rather than the active context, which could limit the long-horizon setting the paper targets.
How often are cold restorations triggered in the serving runs, what is their cost relative to active-context prefill
We measure restoration frequency, length and cost on real agentic workloads from MLGym [1].
| Method | (0,32K) | [32K,64K) | [64K,128K) | [128K,194K) | [194K,+∞) | Request with restore (ratio, %) | Average restoration length (tokens/request) |
|---|---|---|---|---|---|---|---|
| Standard | 10.24% | 7.95% | 10.06% | 5.00% | 0.06% | 33.31% | 22,443.40 |
| AKV | 2.85% | 0.71% | 0.71% | 0.12% | 0.00% | 4.40% | 1,307.44 |
| Method | Overall cost (cents) | Restore cost (cents) | Restore cost ratio |
|---|---|---|---|
| Standard | 103.7841 | 55.8841 | 53.85% |
| AKV | 53.4181 | 3.2987 | 6.18% |
AKV reduces the total restoration cost to only 5.91% of Standard's, since it uses less cache for each task. It reduces the fraction of requests requiring restoration by 28.91%, and reduces average restoration length by 94.2%. Detailed setups and results are in Appendix A.4.8.
For extremely long trajectories without available historical caches, the harness can choose to compact the history into a new trajectory containing only the active context, similar as Claude Code [2].
[1] Nathani et al., 2025. MLGym: A new framework and benchmark for advancing ai research agents. arXiv:2502.14499. https://arxiv.org/abs/2502.14499
[2] An update on recent Claude Code quality reports. https://www.anthropic.com/research/april-23-postmortem
Q3.2: AKV behavior when trajectory length exceeds context length
How does AKV behave when the logical sequence exceeds the context length?
AKV is designed to support trajectories longer than the model's context window.
During decoding, AKV triggers repositioning before token positions reach the context limit. This compacts positional indices and allows generation to continue with a bounded active context (Section 5.1, L400).
During restoration, AKV use segment-batched attention to keep position encoding and active KV cache length within context length, resulting in the same cache states as in decoding (Section 4.2.2, L350).
W2.c-d, Q4: Speed comparison using same trajectory and measurement window
Each method replays trajectories collected from its own policy, so the serving workloads differ across methods.
Would it be possible to repeat the serving comparison on a common replayed workload with identical measurement windows?
We add a system efficiency comparison using the same trajectories and measurement window. Detailed settings are in Appendix A.4.6.
| System | Request latency mean(s) | Request latency P95(s) | TTFT mean(s) | TTFT P95(s) | TPOT mean(ms) | TPOT P95(ms) | Input throughput(token/s) | Output throughput(token/s) |
|---|---|---|---|---|---|---|---|---|
| Standard | 36.03 | 336.10 | 29.26 | 333.33 | 26.80 | 33.84 | 13535.33 | 94.86 |
| AKV | 9.24 | 30.05 | 3.11 | 10.35 | 24.46 | 32.08 | 51179.72 | 360.39 |
AKV achieves 3.8× overall throughput, consistent with 3.7× reported in the original paper (Table 2). The original experiments use trajectories collected under each method's own context-management policy to reflect real-world workloads.
Q5: Minor presentation mistakes
The headline accuracy gain appears as 1.7 points (abstract), 3.5% (contributions), and 1.3–2.1% (Section 5.2). The RSI result is described as "expanding the cost reduction to 17.8%," which is smaller than 27.7% and uses a different baseline. Section 5.2 refers to "all four models" while Table 1 shows three. Minor typos: "Both the baselines deploy" and the missing space in "directly.AKV."
We thank the reviewer for the suggestions and have corrected them in the revised paper.
Thank you for the response and the new experiments. The matched comparison suggests a somewhat different conclusion from the original claim, as text drop at K=12 reaches higher accuracy than cache drop, and the difference at K=1 is within the reported variance. The accuracy gain therefore seems to come mainly from the dropping policy, and I would suggest presenting the results as a cost–accuracy trade-off.
The restoration concern is partially addressed, as restoration is infrequent in practice but can still scale with the logical length in the worst case. The reposition analysis, the serving comparison on a common workload, and the presentation fixes resolve my other concerns.
The new results strengthen the systems contribution, and I appreciate the efforts of the authors. I will raise my score, assuming that the claims on accuracy, including those in the abstract, are revised as a cost–accuracy trade-off.
Thank you for the careful assessment and updated score. We will revise the accuracy claims throughout the paper to reflect the cost–accuracy trade-off, focusing on the new Pareto frontier that AKV achieves.
We sincerely thank Reviewer VYgd for the constructive suggestions. We appreciate your recognition of our practical motivation, system design, and usefulness. We address your concerns below.
W1.a-c, Q1: Direct comparison with text drop
We add text drop baselines, using the same rolling policy with retained rounds K = 1, 4, 12. Detailed results are updated in Figure 3.
The results support accuracy gains from both retained cache information and dropping old observations:
Cache drop retains more information than text drop, especially when retaining fewer rounds, like K=1.
Dropping, either with cache or text, provides accuracy gain, especially when retaining more rounds, like K=12.
Cache drop (AKV) advances the cost–accuracy Pareto frontier compared with summarization and text drop. AKV at K=12 achieves 1.5% higher accuracy than text drop at K=4, while using 14.8% lower cost.
We note that text drop achieves higher accuracy at K=12, but with higher cost and a larger standard deviation. Higher cost and variance presumably help explain why summarization is more commonly used than text drop.
AKV provides the formulation and infrastructure for the open-source community to explore better cache policies. We view rolling drop as our first demonstration in this direction.
W1.d, Q2: Why reposition improves accuracy
We add an analysis on why reposition improves accuracy, finding that reposition helps most on hard tasks requiring exploration near or beyond the model's context window.
On hard tasks with trajectories exceeding 96K tokens, reposition increases accuracy from 10.3% to 30.8%. It compacts the position indices of retained tokens, allowing continued exploration within the maximum position encoding. Without reposition, the model often reaches the context limit or answers prematurely before completing its exploration. One possible explanation for premature answers is length penalties during RL, such as DAPO’s overlong punishment [1].
On easy tasks that can be completed well within the model's context window, repositioning has little effect.
[1] Yu, Qiying, et al. "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." NeurIPS'25.