acceptodds
Under review as a conference paper at ICLR 2027

Cache Engineering for Agents

Abstract

Large Language Model (LLM) agents are rapidly evolving to handle complex tasks, navigating long sequences of observations and actions. However, as agent trajectories grow longer, Key-Value (KV) cache storage and I/O become dominant costs. Long KV caches can also degrade model performance, especially when an LLM's context window limit is exceeded. Prior work uses context engineering to reduce cache costs and support agent trajectories longer than the LLM’s context window. They typically trim or summarize information irrelevant to the current round, thereby reducing the cache used in subsequent attention computations. However, modifying the prefix invalidates all subsequent cached states and triggers costly re-computation of the cache for the remaining context. Compact summarizations can also directly lose critical information, reducing accuracy in long-horizon tasks. To mitigate these limitations, we propose a new cache engineering scheme. It lets agents proactively drop selected KV cache while directly reusing retained ones, without relying on context engineering as a proxy. The key challenge is the history dependence of cache states: the same token sequence can yield different caches under different cache-operation histories, making token-prefix matching insufficient to determine whether a cache can be reused. Thus, we propose AKV, an efficient serving engine to support cache engineering. It provides a simple, stateless interface compatible with existing API-based serving, and hides the complexity of stateful cache management inside. We further extend the radix tree and attention masking to maximize cache reuse, enable timely eviction, and efficiently restore missing cache states. Experiments show that AKV enables a simple rolling-drop cache engineering strategy that effectively improves benchmark scores by 1.7 points while reducing the average cost by 27.7% across various LLMs and agent benchmarks. It further enables recursive self-improvement of agents' cache-engineering strategy, expanding the cost reduction to 17.8%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Agent, KV cache, efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VYgdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces cache engineering, which lets an agent or its harness directly control its active KV cache instead of editing the text history. The method defines two cache operations: drop, which removes selected KV entries from future attention without recomputing later states, and repos, which re-rotates retained keys to close position gaps. Since the resulting cache states depend on the operation history rather than the token prefix alone, the paper proposes AKV, an SGLang-based serving system with a stateless interface, a radix tree extended with drop and reposition nodes, drop-aware eviction, and segment-batch attention for restoring evicted states in one prefill pass. Experiments on three models and four agent benchmarks show that a rolling policy retaining the latest 12 tool responses improves average pass rate and reduces cost by 27.7% compared with summarization-based baselines, with up to 2.67× output throughput in serving.

Critical strengths
  1. The paper identifies a meaningful systems problem introduced by agent-controlled KV cache. Once drop and repos are allowed, the token prefix no longer determines the cache state (Figure 5), and the paper addresses this with a concrete serving design: drop and reposition nodes in the radix tree, history-aware matching across positional versions, and the separation of logical drop from physical eviction. This design is useful for agent-controlled KV methods in general, beyond the evaluated policy.
Critical weaknesses
  1. (Quality) The accuracy gain is not clearly attributed to cache engineering, since the context policy and the caching mechanism are changed together.

    a. The baselines combine a summarization policy with text re-prefill, while AKV combines a rolling observation-masking policy (K=12K=12) with direct KV drop. Both factors therefore differ between the compared settings.

    b. Text-level masking has been reported to perform comparably to or better than summarization for agent context management (Lindenbauer et al.), and the paper attributes AKV's advantage on long trajectories to information loss from summarization (Figure 7), which relates to the baseline policy rather than the mechanism.

    c. The control that would directly support the main thesis, that KV computed while the dropped context was visible retains useful information, is the same rolling-masking policy with text re-prefill. This setting is currently reported for cost (1.85×) but not for accuracy.

    d. In Table 3, drop without repos is below Standard on BCP, and repos accounts for most of the gain. Since repositioned keys no longer match the positions where their hidden states were computed, some analysis of why these states help would be valuable.

  2. (Significance / Quality) The history-dependent cache semantics may make restoration cost scale with the logical history rather than the active context, which could limit the long-horizon setting the paper targets.

    a. Because each KV state depends on what was visible when it was computed, restoring an evicted state seems to require recomputing earlier tokens including dropped ones (as in the Figure 10 example), so a cold restore may involve a prefill over the full logical sequence.

    b. For the 500+ round trajectories that motivate the paper, this cost may approach or exceed the context window that drop and repos aim to stay within, and the stateless interface also resends the full history and operation log on each request.

    c. Each method replays trajectories collected from its own policy, so the serving workloads differ across methods.

    d. AKV at high concurrency is measured until the first 30 completed tasks, while Standard uses a fixed one-hour window with unfinished requests, which may affect the latency and throughput comparison.

Questions for authors
  1. (ref Weakness 1) Could the authors report accuracy and cost for the same rolling-masking policy (K=12K=12) with text re-prefill under the same harness?

  2. (ref Weakness 1) What explains the 7-point improvement from repos over drop alone on BCP, and does the gain remain when the position budget is not binding?

  3. (ref Weakness 2) How often are cold restorations triggered in the serving runs, what is their cost relative to active-context prefill, and how does AKV behave when the logical sequence exceeds the context length?

  4. (ref Weakness 2) Would it be possible to repeat the serving comparison on a common replayed workload with identical measurement windows?

  5. (Presentation, minor) The headline accuracy gain appears as 1.7 points (abstract), 3.5% (contributions), and 1.3–2.1% (Section 5.2). The RSI result is described as "expanding the cost reduction to 17.8%," which is smaller than 27.7% and uses a different baseline. Section 5.2 refers to "all four models" while Table 1 shows three. Minor typos: "Both the baselines deploy" and the missing space in "directly.AKV."

4 comments

We sincerely thank Reviewer VYgd for the constructive suggestions. We appreciate your recognition of our practical motivation, system design, and usefulness. We address your concerns below.

W1.a-c, Q1: Direct comparison with text drop

The accuracy gain is not clearly attributed to cache engineering, since the context policy and the caching mechanism are changed together. The control that would directly support ... that KV computed while the dropped context was visible retains useful information, is the same rolling-masking policy with text re-prefill.

Could the authors report accuracy and cost for the same rolling-masking policy with text re-prefill under the same harness?

We add text drop baselines, using the same rolling policy with retained rounds K = 1, 4, 12. Detailed results are updated in Figure 3.

Cost levelMethodSettingAcc (mean ± std)Cost ($/task)
ReferenceStandard96k55.00 ± 1.410.131
LowText dropK=157.75 ± 4.110.077
Cache drop (AKV)K=159.25 ± 4.790.062
HighText dropK=461.75 ± 2.220.128
Cache drop (AKV)K=1263.25 ± 1.500.109
ExtremeText dropK=1267.25 ± 5.740.201

The results support accuracy gains from both retained cache information and dropping old observations:

  1. Cache drop retains more information than text drop, especially when retaining fewer rounds, like K=1.

  2. Dropping, either with cache or text, provides accuracy gain, especially when retaining more rounds, like K=12.

Cache drop (AKV) advances the cost–accuracy Pareto frontier compared with summarization and text drop. AKV at K=12 achieves 1.5% higher accuracy than text drop at K=4, while using 14.8% lower cost.

We note that text drop achieves higher accuracy at K=12, but with higher cost and a larger standard deviation. Higher cost and variance presumably help explain why summarization is more commonly used than text drop.

AKV provides the formulation and infrastructure for the open-source community to explore better cache policies. We view rolling drop as our first demonstration in this direction.

W1.d, Q2: Why reposition improves accuracy

d. In Table 3, drop without repos is below Standard on BCP, and repos accounts for most of the gain. Since repositioned keys no longer match the positions where their hidden states were computed, some analysis of why these states help would be valuable.

We add an analysis on why reposition improves accuracy, finding that reposition helps most on hard tasks requiring exploration near or beyond the model's context window.

Task category (trajectory length)Repos thresholdw.o repos (%)w. repos (%)
Easy (<96K)48K75.078.1
Easy (<96K)96K75.075.0
Hard (>96K)96K10.330.8

On hard tasks with trajectories exceeding 96K tokens, reposition increases accuracy from 10.3% to 30.8%. It compacts the position indices of retained tokens, allowing continued exploration within the maximum position encoding. Without reposition, the model often reaches the context limit or answers prematurely before completing its exploration. One possible explanation for premature answers is length penalties during RL, such as DAPO’s overlong punishment [1].

On easy tasks that can be completed well within the model's context window, repositioning has little effect.

[1] Yu, Qiying, et al. "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." NeurIPS'25.

W2.a-b, Q3.1: The cost and frequency of AKV restoration

The history-dependent cache semantics may make restoration cost scale with the logical history rather than the active context, which could limit the long-horizon setting the paper targets.

How often are cold restorations triggered in the serving runs, what is their cost relative to active-context prefill

We measure restoration frequency, length and cost on real agentic workloads from MLGym [1].

Method(0,32K)[32K,64K)[64K,128K)[128K,194K)[194K,+∞)Request with restore (ratio, %)Average restoration length (tokens/request)
Standard10.24%7.95%10.06%5.00%0.06%33.31%22,443.40
AKV2.85%0.71%0.71%0.12%0.00%4.40%1,307.44
MethodOverall cost (cents)Restore cost (cents)Restore cost ratio
Standard103.784155.884153.85%
AKV53.41813.29876.18%

AKV reduces the total restoration cost to only 5.91% of Standard's, since it uses less cache for each task. It reduces the fraction of requests requiring restoration by 28.91%, and reduces average restoration length by 94.2%. Detailed setups and results are in Appendix A.4.8.

For extremely long trajectories without available historical caches, the harness can choose to compact the history into a new trajectory containing only the active context, similar as Claude Code [2].

[1] Nathani et al., 2025. MLGym: A new framework and benchmark for advancing ai research agents. arXiv:2502.14499. https://arxiv.org/abs/2502.14499

[2] An update on recent Claude Code quality reports. https://www.anthropic.com/research/april-23-postmortem

Q3.2: AKV behavior when trajectory length exceeds context length

How does AKV behave when the logical sequence exceeds the context length?

AKV is designed to support trajectories longer than the model's context window.

During decoding, AKV triggers repositioning before token positions reach the context limit. This compacts positional indices and allows generation to continue with a bounded active context (Section 5.1, L400).

During restoration, AKV use segment-batched attention to keep position encoding and active KV cache length within context length, resulting in the same cache states as in decoding (Section 4.2.2, L350).

W2.c-d, Q4: Speed comparison using same trajectory and measurement window

Each method replays trajectories collected from its own policy, so the serving workloads differ across methods.

Would it be possible to repeat the serving comparison on a common replayed workload with identical measurement windows?

We add a system efficiency comparison using the same trajectories and measurement window. Detailed settings are in Appendix A.4.6.

SystemRequest latency mean(s)Request latency P95(s)TTFT mean(s)TTFT P95(s)TPOT mean(ms)TPOT P95(ms)Input throughput(token/s)Output throughput(token/s)
Standard36.03336.1029.26333.3326.8033.8413535.3394.86
AKV9.2430.053.1110.3524.4632.0851179.72360.39

AKV achieves 3.8× overall throughput, consistent with 3.7× reported in the original paper (Table 2). The original experiments use trajectories collected under each method's own context-management policy to reflect real-world workloads.

Q5: Minor presentation mistakes

The headline accuracy gain appears as 1.7 points (abstract), 3.5% (contributions), and 1.3–2.1% (Section 5.2). The RSI result is described as "expanding the cost reduction to 17.8%," which is smaller than 27.7% and uses a different baseline. Section 5.2 refers to "all four models" while Table 1 shows three. Minor typos: "Both the baselines deploy" and the missing space in "directly.AKV."

We thank the reviewer for the suggestions and have corrected them in the revised paper.

Thank you for the response and the new experiments. The matched comparison suggests a somewhat different conclusion from the original claim, as text drop at K=12 reaches higher accuracy than cache drop, and the difference at K=1 is within the reported variance. The accuracy gain therefore seems to come mainly from the dropping policy, and I would suggest presenting the results as a cost–accuracy trade-off.

The restoration concern is partially addressed, as restoration is infrequent in practice but can still scale with the logical length in the worst case. The reposition analysis, the serving comparison on a common workload, and the presentation fixes resolve my other concerns.

The new results strengthen the systems contribution, and I appreciate the efforts of the authors. I will raise my score, assuming that the claims on accuracy, including those in the abstract, are revised as a cost–accuracy trade-off.

Thank you for the careful assessment and updated score. We will revise the accuracy claims throughout the paper to reflect the cost–accuracy trade-off, focusing on the new Pareto frontier that AKV achieves.

Reviewer_FnX6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes cache engineering. the agent harness drops selected KV entries and re-rotates the remaining keys instead of editing the text history, so the cache is reused without re-prefill. because the same retained tokens can carry different KV states depending on when drops happened, the authors build AKV on SGLang with a stateless cache API, a radix tree with drop and reposition nodes, and one-pass restoration. Keeping the latest 12 tool responses, AKV gains 1.3-2.1 points on average over a summarize-at-96K baseline on four agent benchmarks and three models, at 27.7% lower API-equivalent cost.

Critical strengths
  1. The authors identify a real systems problem.
  2. As shown, AKV is not the most expensive method on average.
Critical weaknesses
  1. The accuracy gain is small. There are some inconsistency as well, GPT-OSS-120B Standard on BCP is 57.5, 56.0 and 52.1 in Tables 1, 3 and 4, is it on different unnamed question sets? The paper claims retained KV "can also carry information from the dropped history", yet never reports the accuracy of text-level observation masking with the same K=12. Fig. 3 shows even K=1 beating Standard, so the gain may simply come from dropping old observations.
  2. The efficiency headlines come from a cost proxy and a memory-cliff regime. The 27.7% is an API-price calculation with repositioned tokens counted as cache hits.
Questions for authors
  1. How many questions and runs are behind each cell in Tables 1, 3 and 4, and why do the BCP Standard numbers differ?
  2. How does AKV compare with a matched-budget KV eviction baseline (SnapKV/H2O-style, or SideQuest/Leyline) on BCP?
  3. Can you rerun Table 2 with identical measurement windows and a random task sample instead of the longest 80?
2 comments

We sincerely thank Reviewer FnX6 for the constructive suggestions. We appreciate your recognition of our practical motivation and cost reduction. We address your concerns below.

W1.1: The accuracy gain is small

AKV achieves higher accuracy at substantially lower cost. Compared with Standard, AKV reduces average cost by 27.7% and improves average task accuracy by 1.7 points.

W1.2, Q1: Inconsistent results for Table 1, 3, 4

There are some inconsistency as well, GPT-OSS-120B Standard on BCP is 57.5, 56.0 and 52.1 in Tables 1, 3 and 4, is it on different unnamed question sets? How many questions and runs are behind each cell in Tables 1, 3 and 4, and why do the BCP Standard numbers differ?

Tables 1, 3, and 4 in the original paper used different question sets. Our main table (Table 1) used the full BCP benchmark, while drop strategy ablations (Table 3 and 4) each used 100 randomly sampled BCP tasks. We apologize for any confusion.

We update Tables 3 and 4 using the full benchmarks, matching the question sets in Table 1:

StrategyReposBCPSWEAverage
ScoreCostScoreCostScoreCost
Standard✗57.513.140.020.048.7516.55
Tool drop✗57.66.644.010.050.808.30
Reasoning drop✓55.88.636.214.646.0011.60
Both drop✓57.87.632.48.645.108.10
Tool drop✓61.911.443.29.752.5510.55

In the updated Table 3, tool drop with repos (AKV) still achives the highest average score, consistent with the original results.

ModelMetricMethod
StandardText Eng.CAT
GPT-OSS-120BScore57.552.759.0
Cost13.112.610.1
MiniMax-M2.7Score58.962.364.2
Cost5.25.35.3

In the updated Table 4, CAT retains the highest BCP score for both models, with lower per-task cost for GPT-OSS-120B and comparable cost for MiniMax-M2.7.

Both updated tables support the original conclusions.

W1.3: Direct comparison with text drop

The paper claims retained KV "can also carry information from the dropped history", yet never reports the accuracy of text-level observation masking with the same K=12. Fig. 3 shows even K=1 beating Standard, so the gain may simply come from dropping old observations.

We add text drop baselines, using the same rolling policy with retained rounds K = 1, 4, 12. Detailed results are updated in Figure 3.

Cost levelMethodSettingAcc (mean ± std)Cost ($/task)
ReferenceStandard96k55.00 ± 1.410.131
LowText dropK=157.75 ± 4.110.077
Cache drop (AKV)K=159.25 ± 4.790.062
HighText dropK=461.75 ± 2.220.128
Cache drop (AKV)K=1263.25 ± 1.500.109
ExtremeText dropK=1267.25 ± 5.740.201

The results support accuracy gains from both retained cache information and dropping old observations:

  1. Cache drop retains more information than text drop, especially when retaining fewer rounds, like K=1.

  2. Dropping, either with cache or text, provides accuracy gain, especially when retaining more rounds, like K=12.

Cache drop (AKV) advances the cost–accuracy Pareto frontier compared with summarization and text drop. AKV at K=12 achieves 1.5% higher accuracy than text drop at K=4, while using 14.8% lower cost.

We note that text drop achieves higher accuracy at K=12, but with higher cost and a larger standard deviation. Higher cost and variance presumably help explain why summarization is more commonly used than text drop.

AKV provides the formulation and infrastructure for the open-source community to explore better cache policies. We view rolling drop as our first demonstration in this direction.

W2.1: Efficiency measures cost as a proxy

The efficiency headlines come from a cost proxy

We use an API cost proxy because it does not depend on a specific hardware configuration or deployment setting, such as concurrency.

In addition to the cost proxy, we report AKV's wall-clock improvements in Figure 9(a) and Table 2 to demonstrate its efficiency on actual hardware.

W2.2: Efficiency measured at memory-cliff regime

The efficiency headlines come from a memory-cliff regime.

Our cost calculation is independent of memory pressure. Our wall-clock comparison (Figure 9) uses each method's throughput-maximizing concurrency, with neither method chosen from the memory-cliff regime. Moreover, AKV reaches the memory cliff at a higher concurrency than Standard, which is another efficiency advantage of AKV.

W2.3: Charging reposition at cache-hit price

27.7% is an API-price calculation with repositioned tokens counted as cache hits.

We clarify that we charge reposition as an extra operation, on top of cache hits and misses. Reposition price is the same as cache-hit, despite requiring far fewer FLOPs.

SymbolDefinition
LLNumber of Transformer layers
NNNumber of cache-hit tokens
MMNumber of newly extended tokens
BBBatch size
Hq,HkvH_q, H_{kv}Number of query and KV heads
dk,dvd_k, d_vKey and Value dimensions
droped_{\mathrm{rope}}RoPE dimension
bbBytes per KV element
OperationFLOPsMemory access (bytes)
Cache hit2BLMNHq(dk+dv)2BLMNH_q(d_k+d_v)BLNHkv(dk+dv)bBLNH_{kv}(d_k+d_v)b
Reposition*3BLNHkvdrope3BLNH_{kv}d_{\mathrm{rope}}BLNHkv(dk+dv)bBLNH_{kv}(d_k+d_v)b

* Assuming all cache-hit tokens are repositioned.

For MiniMax-M2.7 with typical sequence lengths of N=36KN=36K and M=2KM=2K, reposition incurs only 0.003% of the cache-hit FLOPs, with the same amount of memory access. The overhead calculation is detailed in Appendix A.3.1.

Q2: Add more baselines

How does AKV compare with a matched-budget KV eviction baseline (SnapKV/H2O-style, or SideQuest/Leyline) on BCP?

We add SnapKV as a baseline and compare accuracy at matched cache lengths. Detailed settings are provided in Appendix A.4.9.

BenchmarkStandardText EngineeringSnapKVAKV
BCP56.050.052.062.0
GAIA-Text67.065.050.070.0
xbench-DS38.034.036.039.0

AKV outperforms SnapKV across all three benchmarks at matched cache lengths.. As reported by SnapKV, H2O and SideQuest, they typically reduce task accuracy to save cost. In contrast, AKV simultaneously improves accuracy and saves cost by utilizing cache engineering. For concurrent works Leyline, no open-source implementation is currently avalible for faithful comparison.

Q3: Evaluate Table 2 with identical windows and random task samples

Can you rerun Table 2 with identical measurement windows and a random task sample instead of the longest 80?

We re-evaluate Table 2 with identical measurement windows and random task samples at a concurrency of 12.

MethodInput throughput(token/s)Output throughput(token/s)Latency mean/P95(s)TTFT mean/P95(s)TPOT mean/P95(ms)
Standard32247.01311.0611.83 / 39.023.27 / 9.5627.20 / 31.67
Text Engineering21318.64340.4212.58 / 42.734.12 / 16.6824.01 / 26.02
AKV35633.58408.268.17 / 28.531.95 / 6.0921.50 / 24.75

AKV improves overall throughput by 1.10$\times$ and 1.66$\times$ over Standard and Text Engineering, respectively, consistent with the original conclusion.

Reviewer_DQw4OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces cache engineering, a framework that enables LLM agents to directly control their active KV cache through explicit append, drop, and reposition operations, without modifying the underlying logical token sequence. To support these operations, the authors develop AKV, a serving system that exposes a stateless interface while internally managing history-dependent KV cache reuse, eviction, and restoration. Its key technical components include an extended radix tree that tracks cache-operation histories and segment-batch attention that reconstructs missing cache states within a single prefill pass. Experiments across multiple models and four agent benchmarks demonstrate improved task performance and reduced inference costs, with an average cost reduction of 27.7% relative to standard context engineering. The paper additionally explores agent-driven cache management and recursive policy optimization, suggesting opportunities for more adaptive and efficient long-horizon agent execution.

Critical strengths

1. A meaningful conceptual contribution that bridges agent-level context management and system-level KV cache optimization. The paper identifies an important limitation of conventional context engineering: modifying historical tokens invalidates downstream KV cache states and introduces expensive recomputation. Rather than treating KV cache management as an implementation detail hidden from agents, the proposed framework makes cache operations explicitly programmable while preserving the logical conversation history. This distinction is consequential because it introduces a new optimization interface through which agent policies can jointly influence computational efficiency and task performance. The framework has the potential to support more sophisticated cache-management strategies beyond the rolling-drop heuristic evaluated in the paper.

2. A technically well-motivated serving architecture with promising empirical improvements. The paper recognizes that KV cache states depend not only on token sequences but also on the histories of cache operations, making conventional prefix-based reuse insufficient. The extended radix tree and segment-batch attention provide concrete mechanisms to address this challenge while maintaining efficient cache reuse and restoration. The evaluation is reasonably broad, covering multiple model families and both research-oriented and software-engineering agent benchmarks. In addition to improvements in task success rates and inference costs, the serving experiments report up to 2.67× higher output throughput and 7.51× lower P95 request latency relative to the standard baseline. These results suggest that the proposed abstraction can translate into meaningful practical improvements rather than remaining purely conceptual.

Critical weaknesses

1. Insufficient differentiation from existing programmable KV-cache management systems. The proposed cache-engineering abstraction overlaps substantially with existing work, particularly Leyline (Ma et al., 2026a), which already supports agent-directed KV-cache editing and downstream cache reuse with positional corrections. While AKV introduces history-aware radix-tree matching and segment-batch restoration, the paper does not sufficiently clarify which capabilities fundamentally distinguish its framework from prior approaches.

The supplementary source code provides useful evidence of the implementation, but does not resolve this conceptual concern. In particular, the paper lacks a concrete comparison demonstrating scenarios where existing cache-editing systems are insufficient and AKV's additional mechanisms are necessary. This makes it difficult to assess the novelty and incremental practical significance of the proposed framework.

2. Insufficient isolation and validation of the proposed serving mechanisms. Although AKV reports substantial improvements in throughput and latency, the experiments do not sufficiently isolate the contributions of its key technical components. In particular, it remains unclear how much of the improvement comes from reducing the active attention length through rolling-drop, versus the extended radix tree and segment-batch restoration. While the paper provides a decode-time breakdown (Figure 12), it does not directly quantify the benefits of history-aware cache reuse or single-pass restoration over simpler alternatives.

Furthermore, Appendix A.1.4 indicates that serving experiments replay trajectories collected under different context-management policies, potentially introducing differences in request lengths, generation budgets, and workload characteristics. The high-concurrency measurements also use different stopping conditions. These factors complicate the interpretation of the reported performance gains. Although the supplementary code improves implementation transparency, it does not provide additional controlled measurements addressing these concerns.

References

Ma, B., Eitzinger, J., & Koestler, H. (2026). Leyline: KV Cache Directives for Agentic Inference. arXiv:2606.01065. https://arxiv.org/abs/2606.01065

Questions for authors
  1. Leyline (Ma et al., 2026a) already provides agent-directed KV-cache editing and downstream cache reuse. Could the authors clarify which capabilities of AKV cannot be supported by Leyline or similar existing systems? In particular, could the authors provide a concrete cache-operation trajectory where AKV's history-aware representation is necessary and explain how an existing approach would handle, or fail to handle, the same scenario?

  2. Could the authors provide a component-level ablation that separates the benefits of rolling-drop, history-aware radix-tree matching, and segment-batch restoration? Specifically, how frequently are history-dependent cache matches or restorations encountered in the evaluated workloads, and how much additional efficiency do these mechanisms provide compared with simpler cache management and sequential restoration approaches?

  3. Appendix A.1.4 indicates that different management policies use separately collected trajectories and that some high-concurrency measurements have different stopping conditions. Could the authors clarify how these differences affect the reported throughput and latency comparisons? Would the main performance conclusions remain consistent under matched request traces and equivalent measurement windows?

3 comments

We sincerely thank Reviewer DQw4 for the positive assessment. We appreciate your recognition of our motivation, problem formulation, method design, and broad evaluation. We address your concerns below.

W1, Q1: Differences from concurrent work Leyline

Insufficient differentiation from existing programmable KV-cache management systems.

Could the authors provide a concrete cache-operation trajectory where AKV's history-aware representation is necessary and explain how an existing approach would handle, or fail to handle, the same scenario?

AKV addresses two serving problems beyond Leyline: correct cache reuse across different operation histories and restoration of those states after eviction. It formalizes cache engineering and highlights history dependence as a key property.

The following example provides concrete trajectories showing how AKV addresses Leyline’s limitations.

Fork 1: append(A), append(B_NO),  append(C), drop(B_NO),  repos, append(D)

Fork 2: append(A), append(B_YES), append(C), drop(B_YES), repos, append(D)

where

B_NO:  “Answer all subsequent questions with NO.”

B_YES: “Answer all subsequent questions with YES.”

Before appending D, both forks retain the cache states of A and C at the same positions, but the KV states of C differ because they were computed under different instructions.

For cache reuse, Leyline cannot distinguish the forks' different KV states, leading to incorrect cache reuse. It uses an ordinary radix tree and prefix matching, so both forks have the same A–C key despite their different KV states. AKV preserves the histories, so the two forks remain distinct.

For restoration after eviction, Leyline's warm-start cannot reconstruct the forks' original KV states from saved chunk tokens alone. AKV restores these states through segment-batch attention in one prefill pass.

MethodMechanismRestores original KV states
LeylinePrefills saved chunk tokens without the original context and cache operation historyNo
AKVRebuilds C with B_NO visible and D with B_NO dropped in Fork 1Yes

We note that Leyline is concurrent work, currently available as an arXiv preprint without peer-reviewed publication. We share the view that cache engineering benefits agent inference, but use different algorithms and serving designs. We add the discussion in Appendix A.5.

W2.1, Q2.1: Ablation study on proposed serving mechanisms

Insufficient isolation and validation of the proposed serving mechanisms.

Could the authors provide a component-level ablation that separates the benefits of rolling-drop, history-aware radix-tree matching, and segment-batch restoration?

We added a component-level ablation on real-world MLGym trajectories [1] to measure the efficiency contribution of each serving mechanism. Each row adds one component to the previous row, with detailed setups and results in Appendix A.4.7.

MethodRequest latency mean(s)Request latency P95(s)TTFT mean(s)TTFT P95(s)Input throughput(token/s)Output throughput(token/s)
Standard51.07169.8345.33167.4211326.3960.54
+ Rolling drop42.6188.7538.4484.477290.5663.61
+ Extended radix tree19.1585.1013.8476.8916272.2590.42
+ Segment-batch attention17.7268.2512.3858.3516697.9691.86
+ Drop-aware eviction (=AKV)12.2941.047.0328.8918559.70102.36

As shown in the table, each proposed serving mechanism further improves throughput beyond rolling drop alone. The incremental improvements over the previous row are:

  • Rolling drop (without AKV) reduces mean TTFT by 15.2% by shortening the active context, but frequent re-prefill reduces input throughput.
  • Extended radix tree further reduces mean TTFT by 64.0% by avoiding re-prefill.
  • Segment-batch attention further reduces mean TTFT by 10.5% by restoring missing states in one prefill pass.
  • Drop-aware eviction further reduces mean TTFT by 43.2% by prioritizing dropped KV states for eviction, freeing space for reusable states.

[1] Nathani et al., 2025. MLGym: A new framework and benchmark for advancing ai research agents. arXiv:2502.14499. https://arxiv.org/abs/2502.14499

Q2.2: Frequency of cache match and restoration

How frequently are history-dependent cache matches or restorations encountered in the evaluated workloads?

We measured the cache hit rate, restoration frequency, and average restoration length on the same MLGym workload [1].

MethodAverage cache hit rateRequests requiring restorationAverage restoration length (tokens/request)
Standard63.97%33.31%22,443.40
AKV89.24%4.40%1,307.44

Compared with Standard, AKV improves cache hit rate by 25.27% and reduces restoration frequency by 28.91%. Among AKV's cache hits, 83.35% are only possible with history-dependent cache matching.

[1] Nathani et al., 2025. MLGym: A new framework and benchmark for advancing ai research agents. arXiv:2502.14499. https://arxiv.org/abs/2502.14499

W2.2, Q3: Speed comparison using same trajectory and measurement window

Appendix A.1.4 indicates that serving experiments replay trajectories collected under different context-management policies ...

Would the main performance conclusions remain consistent under matched request traces and equivalent measurement windows?

We add a system efficiency comparison using the same trajectories and measurement window. Detailed settings are in Appendix A.4.6.

SystemRequest latency mean(s)Request latency P95(s)TTFT mean(s)TTFT P95(s)TPOT mean(ms)TPOT P95(ms)Input throughput(token/s)Output throughput(token/s)
Standard36.03336.1029.26333.3326.8033.8413535.3394.86
AKV9.2430.053.1110.3524.4632.0851179.72360.39

AKV achieves 3.8× overall throughput, consistent with 3.7× reported in the original paper (Table 2). The original experiments use trajectories collected under each method's own context-management policy to reflect real-world workloads.

Dear Reviewer,

Thank you again for your feedback. We have added differentiation from concurrent works, isolation of each proposed mechanism, operation frequency analysis, and speed comparisons under the same trajectory. As the discussion period is closing soon, we would like to ask whether you have any remaining questions or concerns. We would be happy to clarify further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Key Points

We thank the reviewers and Area Chair for their time and constructive feedback. We appreciate the recognition of our motivation, problem formulation, serving system design, broad evaluation, and practical efficiency gains. We summarize the main recognitions and rebuttal updates below.

Key recognitions

  1. Practical motivation (DQw4, FnX6, VYgd). The reviewers recognize costly re-prefill as a key problem in context engineering, acknowledging that cache engineering addresses it.

  2. Cache-engineering formulation (DQw4, VYgd). . The reviewers recognize cache engineering as a new optimization scheme for agents to directly manage KV cache, beyond text-based engineering.

  3. Serving system design (DQw4, VYgd). The reviewers acknowledge the benefits of AKV serving system for correct and efficient cache reuse, eviction and restoration.

  4. Broad evaluation and efficiency gains (DQw4). The reviewer recognizes the broad evaluation across model families and agent benchmarks, with real-world gains in accuracy, inference cost, throughput, and latency.

Key rebuttal updates

  1. Additional ablations on system components.

    Address DQw4 (W2.1/Q2.1).

    Ablation study shows that each system component improves efficiency beyond rolling drop alone, where extended radix tree, segment-batch attention, and drop-aware eviction further contribute to 64.0%, 10.5%, and 43.2% TTFT reductions, respectively. (Appendix A.4.7, Table 10.)

  2. Additional comparison with KV eviction baseline.

    Address FnX6 (Q2).

    Compared with SnapKV, AKV improves task accuracy by 11% across all three benchmarks at matched cache lengths. (Appendix A.4.9, Table 12.)

  3. Additional comparison with text drop.

    Address FnX6 (W1.3) and VYgd (W1.a–c/Q1).

    Compared with text drop, AKV reduces cost by 14.8% while improving task accuracy by 1.5%, advancing the Pareto frontier. (Section 3.2, Figure 3.)

  4. Additional analysis on reposition benefits.

    Address VYgd (W1.d/Q2).

    We find that reposition benefits most on hard tasks with over 96K original trajectory lengths, increasing accuracy from 10.3% to 30.8%. It helps to steer LLMs to explore beyond the context window limit. (Appendix A.4.10, Table 13.)

  5. Additional analysis on restoration cost.

    Address DQw4 (Q2.2) and VYgd (W2.a–b/Q3).

    Compared with Standard, AKV reduces the total restoration cost to only 5.91%, the fraction of requests requiring restoration by 28.91%, and average restoration length by 94.2%. (Appendix A.4.8, Table 11.)

  6. Evaluate ablations on the full evaluation sets.

    Address FnX6 (W1.2/Q1).

    We update Tables 3 and 4 using the full evluation sets, matching main Table 1. Both updated tables support the original ablation conclusions. (Section 5.4, Tables 3–4.)

  7. Evaluate speed under extra settings.

    Address DQw4 (W2.2/Q3), FnX6 (Q3), and VYgd (W2.c-d/Q4).

    Under the random sampling setting, AKV improves end-to-end latency by 1.45$\times$ and 1.54$\times$ over Standard and Text Engineering. Under same trajectory setting, AKV achieves 3.9× end-to-end speedup. Both results are consistent with the original paper. (Appendix A.4.5, Table 8; Appendix A.4.6, Table 9.)

  8. Cache-aware fine-tuning with AKV.

    Address FnX6's follow-up.

    AKV enables LLM to learn to preserve information from dropped observations in the retained KV states. After fine-tuning, AKV improves accuracy from 46.75% to 52% while reducing inference cost to 40–70% of text-drop at a higher accuracy.

  9. Differences from concurrent work Leyline.

    Address DQw4 (W1/Q1).

    AKV uniquely addresses correct cache reuse across different operation histories, and the restoration of those states after eviction. It also implements an efficient system for real-world serving.

  10. Cost and presentation clarification.

Address FnX6 (W2.3) and VYgd (Q5).

We clarify how we charge reposition operations and update presentation details. (Appendix A.3.1.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.