acceptodds
Under review as a conference paper at ICLR 2027

AgentKV: Predictive Admission Prevents KV-Cache Collapse in Agentic LLM Serving

Abstract

Long-horizon language-model agents place unique demands on LLM serving systems because they repeatedly return from tool calls with increasingly long contexts. At high concurrency, these growing contexts can evict one another’s reusable KV cache, forcing repeated prefill, slowing completion, and keeping even more contexts resident. We call this positive-feedback failure KV-cache collapse. We show that the onset of collapse can be predicted from the relationship between available KV capacity and the average context demand of each instance. We introduce AGENTKV, a predictive admission controller that forecasts shorthorizon context growth, maintains a lifecycle-safe working set under an adaptive high-bandwidth memory (HBM) budget, and replans from live cache telemetry. Across ten Qwen3-8B and Qwen3.8-27B campaigns on four RTX PRO 6000 GPUs, the predicted capacity boundary closely matches the observed transition from stable prefix reuse to cache collapse. Beyond this boundary, AGENTKV outperforms vLLM, ThunderAgent, InferCept, Continuum-TTL, and cache-aware vLLM-router in every measured cell, reaching up to 3.7× the throughput of stock vLLM. At 240 concurrent instances, AGENTKV retains roughly 75% prefix reuse for Qwen3-8B workloads while stock vLLM falls to at most 0.4%, and removing predictive admission leaves only 27–77% of full-system throughput. These results show that predictive working-set control can prevent a high-load cache-collapse regime that allocation, routing, and persistence alone do not fully address.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
llm, infra

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vJ1VOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents AgentKV, a predictive admission control system designed to prevent KV-cache collapse in high-concurrency agentic LLM serving. AgentKV predicts future context growth and dynamically controls the set of active agents to keep their KV caches within GPU memory capacity. It also incorporates cache-aware feedback to adjust admission limits and selective CPU KV persistence to reduce recomputation. Experiments across multiple models and agent workloads show up to 3.7× throughput improvement over vLLM, with significantly higher prefix-cache hit rates and lower TTFT under high concurrency.

Critical strengths
  1. The paper evaluates AgentKV across a wide range of experimental settings, including multiple models, workloads, concurrency levels, and GPU platforms, covering recent GPU architectures such as NVIDIA B200.
Critical weaknesses
  1. Unclear contributions and benefits over existing work, with most proposed techniques showing limited effectiveness. The core contribution of AgentKV appears to be extending admission control (already proposed in ThunderAgent and CONCUR) with context growth prediction and cache-aware feedback. However, the paper does not clearly demonstrate the benefits of these additional mechanisms. In fact, the experimental results, particularly the ablation study, suggest that most of the proposed techniques provide little to no benefit: (a) batch-adaptive decoding is disabled in the evaluation; (b) reducing the prediction horizon from H=4 to H=1 has negligible impact on performance; and (c) disabling persistence even improves throughput. These observations raise concerns about the actual sources of AgentKV's performance gains. I suspect that much of the improvement may instead come from short-first scheduling, which is a relatively minor optimization that comes at the cost of worse tail latency and fairness.
  2. Artificially constructed evaluation workloads with questionable representativeness. The evaluation relies on manually selected workloads with rather unusual characteristics. In particular, each agent performs only four LLM calls, while the context is artificially extended to 24–25k tokens. Such settings may not accurately reflect real-world agent workloads. Given that one of the paper's primary contributions is integrating context growth prediction into admission control, evaluating agents with only four LLM calls is particularly concerning, as such short and controlled execution traces may make context growth easier to predict. More fundamentally, I am skeptical of the assumption that context growth can be reliably predicted in realistic agent workloads, especially coding agents, where tool outputs can vary significantly in length and occasionally produce extremely long, unpredictable prefill sequences.
Questions for authors
  1. What are the actual benefits of context growth prediction over existing admission control mechanisms?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vJ1V (1/3)

We thank the reviewer for highlighting the need for realistic agent trajectories, targeted component ablations, and a direct assessment of fairness. We added all three to the paper, and addressing it below. :)

We provided some new results that show AgentKV matches the strongest evaluated concurrency controller when compute binds and improves throughput when memory binds, with balanced waiting across context lengths.

W1. Contributions and benefits over existing admission control.

These experiments pinpoint the primary drivers behind AgentKV's gains: budgeting the resident working set, including tool-waiting contexts, and regulating admission near the engine's service peak. Meanwhile, FIFO admission with bounded bypass and longest-idle-first pausing govern which requests wait. Specifically, AgentKV operates with a service threshold of 48 requests, a value determined through a fixed-window sweep, and the common response details how these parameters were selected and held constant across evaluations.

First, we compared with stock vLLM, ThunderAgent in the same proxy, and our reimplementation of CONCUR's published control law on vLLM. The latter comparison evaluates the control law, not the authors' complete system. These runs use no CPU KV tier.

SettingvLLMThunderAgentCONCUR lawAgentKV
Recorded trajectories, 0.5 s tools, 288 instances30.277.598.498.0
Recorded trajectories, 30 s tools, 288 instances30.133.936.044.7
Sustained load, 384 slots, 30 s tools22.629.034.659.0
Original Terminal-Bench replay21.531.369.270.2

With short tool calls, output peaks at roughly 40–55 requests in the engine even without eviction. Operating near this peak, AgentKV stays within 0.8k of CONCUR's law across all six short-tool trajectory settings, which we regard as comparable performance. With long tool calls, however, resident idle KV becomes critical: whereas ThunderAgent discounts waiting contexts over time and the implemented CONCUR law relies on running-request KV usage and prefix-hit feedback, AgentKV explicitly budgets these resident idle contexts. Consequently, in the 288-instance batch, AgentKV reaches 44.7k compared to 36.0k for CONCUR's law, a comparison that includes window-opening time, with corresponding mid-run rates of 61.1k and 55.6k. Under sustained load, AgentKV maintains 59.0k, while vLLM drops to 22.6k alongside a 10% prefix-hit rate.

AuthorsOpenReview ↗

Response to Reviewer vJ1V (2/3)

The targeted ablations address the individual mechanisms raised in the review:

  • Service bound and feedback. Removing the service bound reduces throughput from 98.0k to 79.5k with 0.5 s tools and from 70.2k to 63.8k on Terminal-Bench. Removing the eviction-feedback rule used in the trajectory experiments reduces 30 s throughput from 44.7k to 42.9k in the measured run. This rule uses the fraction of expected-cached prefix tokens that missed; the absolute-threshold rule used with the load-sweep settings was triggered on 98% of short-tool control ticks, explaining its limited ablation effect.
  • Short-first scheduling and fairness. Replacing shortest-first with FIFO under the load-sweep settings gives 85.1k versus 84.9k, and we see that there is no measurable throughput advantage from shortest-first in that test. AgentKV's mean worst request waits for short/long-context quartiles are 77/75 s with 30 s tools, and the maximum falls from 676 s under the load-sweep settings to 180 s. In the sustained 30 s test, none of the 493 AgentKV instances launched during the first ten minutes waits over ten minutes. The maximum is 329 s, compared with 208 of 512 instances exceeding ten minutes under the load-sweep settings. These are measured cohort results, not a universal latency bound. Comparing both configurations without forecasts, AgentKV retains fixed-batch throughput and costs approximately 2% under sustained memory pressure (61.7k versus 63.0k), while removing the observed extreme waits.
  • Prediction horizon. The original Terminal-Bench replay begins with approximately 24k tokens and adds only about 70 tokens per call. These relatively small increments help explain why performance remains largely unchanged between H=1H = 1 and H=4H = 4. To more directly evaluate the effect of prediction horizon, we conduct additional experiments using 16-call trajectories, with results presented below.
  • Persistence. The 256 GiB tier holds 1.86M tokens, below the GPU cache's 2.36M, with the same recency order. In the transfer-counter experiment, persist-all writes 1.39 TB and restores zero bytes, explaining the limited benefit in this configuration. A 768 GiB tier retains the whole working set and lets vLLM reach 76.3k under sustained 30 s load, compared with 22.6k without the tier and 59.0k for AgentKV without it. This uses substantially more host memory (about 1,150 GiB) and establishes the scope of the memory-admission advantage. The service bound remains beneficial with that tier: at 0.5 s tools, AgentKV reaches 96.8k versus 80.8k for vLLM.
  • Batch-adaptive decoding. This is an optional integration point in Figure 1 and is not used in the reported experiments; none of the gains is attributed to it.

W2. Workload representativeness and irregular coding-agent growth.

This added workload replays the first 16 model calls from recorded SWE-agent and OpenHands runs, using their actual tool outputs without any artificial padding. Throughout these trajectories, contexts expand from roughly 1.6k to 17.9k tokens. From an initial pool of 2,476 runs containing at least 16 calls, we specifically select the 1,178 runs that reach 12k tokens by call 16 to target the long-context regime that heavily strains the cache.

Tool execution times span four configurations: fixed 0.5-second delays, a log-normal distribution fitted to published SWE-bench tool times (mean 0.93 s), a 30-second exponential distribution, and a heavy-tailed mixture. Across all experiments, every system receives identical replayed prompts and tool latencies, while sustained-load evaluations continuously refill client slots over a 20-minute run and report steady-state throughput across minutes 12–20.

A separate analysis of 4,090 trajectories directly addresses the workload irregularity raised in the review: tool outputs account for 80–85% of context growth, single-step increments reach up to 23k tokens, and lag-1 autocorrelations sit at just 0.14 for SWE-agent and −0.04 for OpenHands, making recent increments weak predictors of individual-run growth. However, aggregating across instances substantially improves forecast accuracy: across 100 instances, an estimator that shrinks exponential moving average (EMA) growth toward the population mean achieves a 8–10% 95th-percentile error four calls ahead, compared to 30–31% when relying on current occupancy alone. We explicitly distinguish this statistical estimation gain from end-to-end throughput improvements, which we isolate in a separate ablation.

AuthorsOpenReview ↗

Response to Reviewer vJ1V (3/3)

Q. What are the actual benefits of context-growth prediction over existing admission control?

We measured this directly, and the answer supports the reviewer: prediction is not what helps, and the revision says so (Section 6). In the earlier configuration the forecast gives +6% with short tool calls, nothing from being per-instance (86.3k with a population-mean forecast against 84.9k), and −18% to −24% with long ones. In AgentKV, with the forecast, without it and with an oracle: 44.7k, 46.2k and 45.8k at 30 s tools. With and without it: 45.8k and 45.9k, 43.4k and 43.8k on two further draws; 79.3k and 79.4k, 59.0k and 61.7k under the two sustained loads; 46.7k and 52.3k at 576 instances.

The benefits come from three other elements, each measured separately:

elementmeasured effect
a bound on the requests in the enginewithout it, 79.5k instead of 98.0k at 0.5 s tools, and 63.8k instead of 70.2k on Terminal-Bench
budgeting the resident working set, idle contexts included, without over-reserving43–46k at 30 s tools, against 32.7–36.5k for the earlier configuration (43.0–44.7k once its forecast is off)
FIFO with bounded bypass, and pausing the longest-idle instanceremoves starvation and the bias against long contexts; costs no throughput in fixed batches and 2% under sustained memory-bound load (whole controllers, no forecast)

With its forecast switched off, the earlier configuration is as fast as AgentKV where memory binds, or faster (63.0k against 59.0k under sustained load), but 153 of its first 536 instances then wait more than 10 minutes.

We would be glad to provide any further detail.

Reviewer_tsUbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper studies KV-cache eviction, repeated prefill, and performance degradation caused by growing contexts in highly concurrent agent serving.
  2. AgentKV estimates future demand using an EMA of per-instance context increments, adjusts its memory budget using cache feedback, and controls instance admission, pausing, and resumption.
  3. Experiments cover two GPU platforms and four models, reporting throughput, cache reuse, latency, and component ablations.
Critical strengths
  1. The system delivers clear performance gains. Across three serving replays on PRO 6000, total token throughput exceeds the strongest evaluated baseline by 17–42% at 240 concurrent instances.
  2. Multiple models and platforms provide evidence across configurations. Cache-reuse and latency measurements also expose performance tradeoffs.
  3. Disabling admission leaves only 27–77% of full-system throughput in three ablations, supporting the importance of active working-set control in these settings.
Critical weaknesses
  1. Predictive reservation and shortest-projected-context prioritization may leave long-context or overestimated instances waiting indefinitely. This directly affects the throughput–waiting-time tradeoff in the target setting and requires evaluation.
  2. The paper does not analyze the distribution or temporal variation of per-instance context increments, leaving the supporting workload patterns and applicability boundaries unclear.
  3. Past-Future Scheduler [1] and MAPS [2] already use predicted demand for capacity-aware scheduling. The paper does not discuss these works, while its system-level comparisons and whole-module ablations do not sufficiently isolate the added value of per-instance growth prediction.

[1] Gong et al. Past-Future Scheduler for LLM Serving under SLA Guarantees. ASPLOS 2025.

[2] Zhang et al. MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving. ICML 2026.

Questions for authors
  1. How does the system ensure progress when overestimation repeatedly blocks admission of an instance whose actual demand fits the available capacity?
  2. What measured context-growth patterns support extrapolating future demand from recent increments?
  3. What targeted experimental evidence supports the paper’s incremental contribution beyond predictive capacity-scheduling methods such as Past-Future Scheduler and MAPS mentioned in the weaknesses section?
6 comments
AuthorsOpenReview ↗

Response to Reviewer tsUb (1/2)

We thank the reviewer for the careful review and the three specific questions. :)

We added targeted experiments on waiting time, context growth, and the incremental value of forecasting.

W1 / Q1. Waiting time, overestimation, and progress.

The measurements show substantially more balanced waiting across context lengths. In fixed batches of 288 recorded trajectories, the shortest and longest context quartiles have nearly equal completion times under AgentKV:

System, tool timeThroughputCompletion time, short / long (s)Mean worst request wait, short / long (s)Maximum request wait (s)
Load-sweep settings, 0.5 s84.9326 / 46167 / 241424
AgentKV, 0.5 s98.0266 / 268172 / 174413
Load-sweep settings, 30 s32.7481 / 86611 / 383676
AgentKV, 30 s44.7582 / 59377 / 75180
CONCUR law, 30 s36.0723 / 731256 / 256509

We measure waiting time from the client’s request to the first output token, including admission delay.

Effect of admission and pausing order. Changing only the admission order from shortest-first to FIFO under the load-sweep settings leaves throughput essentially unchanged: 85.1k versus 84.9k. Separate ablations within AgentKV show how each scheduling choice affects waiting with 30 s tools:

  • Replacing FIFO with shortest-first resumption increases the longest request wait from 180 s to 371 s.
  • Pausing the largest contexts instead of the longest-idle instances increases long contexts’ mean worst request wait from 75 s to 141 s.

Fairness under sustained load. We keep client slots full for 20 minutes and measure the longest request wait for each instance launched during the first ten minutes. None of these AgentKV instances waits more than ten minutes for a request:

Tool-time settingAgentKV: instances waiting over 10 minLoad-sweep settings: instances waiting over 10 minAgentKV: longest wait
Heavy-tailed gaps0/51883/503159 s
30 s tools0/493208/512329 s

With forecasts disabled in both configurations, AgentKV reaches 61.7k versus 63.0k for the load-sweep settings under sustained 30 s load, a difference of about 2% (59.0k with AgentKV's forecast on). Fairer service spreads waiting more evenly across requests. Late in the runs, AgentKV’s 95th-percentile request wait is 38 s with heavy-tailed gaps and 168 s with 30 s tools. We report these waiting-time tradeoffs alongside throughput.

How requests make progress. Pending requests are ordered by arrival. A request that does not yet fit may be bypassed for up to 60 s; after that, later requests cannot overtake it. This protects the queue position of long-context or conservatively estimated requests. Eventual admission still requires that its memory reservation fit the budget and that admitted work release capacity. The 60 s limit therefore bounds bypassing, not total waiting time.

Sensitivity to overestimation. We stress-tested AgentKV by multiplying every growth estimate by 16. Throughput falls by 1% in the compute-bound setting. In the memory-bound setting, 4,607 of 4,608 calls complete within 45 minutes; throughput is 16.7k versus 44.7k, a 63% decrease, and the longest wait is 1,415 s. This test shows the throughput and waiting-time cost of severe overreservation. The current-context ablation below evaluates operation without growth forecasts; it is a separate configuration, not an automatic fallback.

AuthorsOpenReview ↗

Response to Reviewer tsUb (2/2)

W2 / Q2. Which measured growth patterns support extrapolation?

The useful regularity is in aggregate demand. Appendix C analyzes 2,902 SWE-agent and 1,188 OpenHands trajectories using the served model’s tokenizer:

ScaffoldIncrement per call: mean / median / p99 / max (tokens)Within-run CV, medianLag-1 autocorrelation, medianGrowth from tool output
SWE-agent535 / 235 / 2,167 / 23,0070.840.1480%
OpenHands939 / 385 / 8,493 / 18,8611.52−0.0485%

These heavy-tailed, weakly correlated increments limit extrapolation of individual runs from recent history. Aggregation provides a more stable admission signal:

EstimateDemand four calls ahead: p95 error, 10 instancesp95 error, 100 instances
Current occupancy40%30–31%
Per-instance EMA22–28%11–13%
EMA shrunk toward population mean18–20%8–10%
Population mean16%5–10%

While forecasting improves aggregate-demand accuracy in this analysis, population statistics alone already capture much of that gain. Its benefits also diminish in less favorable settings, such as small working sets or cohorts of similarly aged instances. Crucially, this separates the question of forecast accuracy from whether more accurate forecasts actually translate into higher serving throughput.

W3 / Q3. Relation to Past-Future Scheduler and MAPS; incremental contribution.

Thank you for these references! Both are now discussed in Section 2. They predict single-request output demand, while our accounting spans agent contexts that remain resident between calls, including tool-waiting periods. This distinction matters for the measured workloads, where tool outputs contribute 80–85% of context growth.

Although we did not run the complete Past-Future or MAPS systems. Instead, we isolated the demand estimate within the same controller. This controlled ablation evaluates history-based estimation alongside per-instance EMA, current occupancy, and an oracle, and it is not a reproduction or substitute for a system-level comparison with those papers.

Demand estimateLoad-sweep settings, 0.5 s toolsLoad-sweep settings, 30 s toolsTrajectory settings, 30 s tools
Current occupancy79.843.046.2
Per-instance EMA84.932.7–
Population mean / history-based estimate86.3–44.7
Oracle, true context four calls ahead91.836.145.8

Under the load-sweep settings, forecasting helps limit concurrency with short tools, but overreserves memory with long tools, and population history performs at least as well as the per-instance estimate in the short-tool comparison. With the trajectory settings, forecasting provides no measurable incremental throughput benefit in these tests, including the oracle comparison. In the most memory-constrained tests it reduces throughput by 4–11%: 59.0k versus 61.7k under sustained load and 46.7k versus 52.3k at 576 instances.

The supported contribution is therefore the controller’s accounting and admission rules. The service bound addresses compute-limited operation; resident-context accounting addresses memory pressure during tool waits; FIFO and longest-idle-first pausing improve the distribution of waiting. AgentKV matches the strongest evaluated concurrency law in compute-bound replays and exceeds it in memory-bound replays. Section 6 separates these measured system benefits from the forecast’s contribution, and Appendix C provides the fairness and growth analyses.

We appreciate the questions, which helped make both the evidence and the scope of the claims more precise :)

Thank you for the detailed rebuttal and additional experiments. The authors have substantially addressed my original concerns.

To confirm my understanding, the evidence-supported incremental contribution of AgentKV now lies primarily in lifecycle-aware working-set accounting and admission control for persistent agent contexts, complemented by service-bound scheduling and cache-feedback control, rather than per-instance context-growth forecasting itself.

Is this an accurate characterization of the paper's core technical contribution? If not, could the authors clarify what they consider the central methodological novelty?

This clarification will help me finalize my assessment of the paper's novelty and significance.

Reviewer_WWn5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focused on KV Cache management for agentic LLMs, identifying the "positive-feedback failure KV-cache collapse": growing contexts can evict one another’s reusable KV cache, forcing repeated prefill, slowing completion, and keeping even more contexts resident. To address this, they proposed AgentKV, a predictive admission controller that forecasts shorthorizon context growth, maintains a lifecycle-safe working set under an adaptive high-bandwidth memory (HBM) budget, and replans from live cache telemetry. According to the experinmental results presented in this paper, AgentKV give a consistent improvement over multiple models and benchmarks.

Critical strengths
  1. This paper identified a positive-feedback failure in which growing agent contexts cause prefix eviction, repeated prefill, slower completion, longer context residency, and further eviction.

  2. They formulated agentic serving as a dynamic working-set control problem and proposed AgentKV.

  3. Solid experinment on multiple models and benchmarks, demonstrating the effectiveness of AgentKV.

  4. The presentation is smooth and clear.

Critical weaknesses
  1. The growing contexts causing prefix eviction and repeated prefill is listed as a contribution. However this observation was mentioned by existing works.

  2. Some existing works like ThunderAgent, CONCUR focus on similar issues, the author should explicitly discuss the technical distinctions among AgentKV and existing works.

  3. In the ablation study, figure (a) shows the main performance contribution comes from "admission", but other modules seems no significant effect. Figure (b) shows that different persistence cap policies are nearly tied. Why is there no difference in performance?

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer WWn5 (1/2)

We sincerely thank the reviewer for the constructive review and encouraging assessment! :)

We added new experiments, and we want to clarify the technical contribution and explain the component ablations: AgentKV combines explicit budgeting of resident KV with a bound on engine service, matching the strongest evaluated concurrency controller when compute binds and improving throughput when memory binds. Below we address the three questions directly.

W1. How does the contribution go beyond the observation of eviction and repeated prefill?

We explicitly credits ThunderAgent’s KV-cache thrashing and CONCUR’s middle-phase thrashing (Section 1). Our contribution is to characterize when this failure occurs and which resources admission must control. The capacity boundary, C_HBM / average context demand, predicts the observed transition from stable prefix reuse to collapse. The new trajectory replays additionally distinguish this memory boundary from the engine’s service limit.

For example, a fixed-window sweep on recorded trajectories with 0.5 s tools and 288 instances gives:

Requests in the engine1054173
Throughput72.499.285.5
Time per output token (ms)1458246

Even though no eviction occurs anywhere in this sweep, throughput still peaks near 50 requests, well before reaching the approximately 240-instance KV-capacity boundary. When introducing 30-second tool calls, however, idle contexts accumulate and shift the bottleneck, making memory the binding resource instead. Together, these measurements demonstrate the need to control both resident KV and active requests within the engine, as monitoring KV occupancy alone cannot capture both performance limits.

W2. What distinguishes AgentKV from ThunderAgent and CONCUR?

Sections 2 and 5.3 compare the mechanisms and their measured effects. The key distinction is how each controller accounts for contexts while agents wait for tools:

ControllerAdmission signal and treatment of idle (tool-waiting) contexts
ThunderAgentPauses and resumes programs against a token capacity, progressively discounting tool-waiting contexts by 2^(−t).
CONCUR lawAdjusts an active-agent window from running-request KV usage and cache-hit rate; these signals do not directly measure resident idle KV.
AgentKVExplicitly budgets the resident working set, including idle contexts, and separately bounds requests in the engine.

For CONCUR, we re-implemented the published control law in the same proxy on vLLM, using a 5 s control period and checking 1 s as well. This is a comparison with that law, not an end-to-end evaluation of the authors’ system.

When systems are memory-bound, AgentKV consistently outperforms competing frameworks across varying instance scales. At 288 instances, AgentKV achieves 44.7k, surpassing the CONCUR law (36.0k), ThunderAgent (33.9k), and vLLM (30.1k). ThunderAgent’s lower performance stems from its discounting mechanism, which undercounts waiting contexts and forces the recomputation of 31–38% of reusable prefix tokens. Meanwhile, the CONCUR law opens its window too slowly at 288 instances and over-admits under heavier loads; adjusting it to a 1-second period admits instances sooner but overshoots capacity, yielding a comparable 36.4k. This gap widens at higher scales: at 576 instances, AgentKV reaches 46.7k compared to the CONCUR law’s 37.6k, and under a sustained 30-second load, it maintains 59.0k against the CONCUR law’s 34.6k.

Conversely, when systems are compute-bound, the CONCUR law operates near its service peak, and the two approaches achieve practical parity. AgentKV’s engine-service bound keeps its throughput within 0.8k of the CONCUR law across all six short-tool settings on recorded trajectories. While the full evaluation table—which includes heavy-tailed tool gaps—shows slight steady-state differences in these compute-bound scenarios, we treat these results as parity rather than claiming them as performance gains.

AuthorsOpenReview ↗

Response to Reviewer WWn5 (2/2)

W3. Which components matter, and why were persistence policies nearly tied?

The new one-at-a-time ablations separate throughput contributions from effects on waiting time:

ComponentResult with / without it
Engine-service bound, 0.5 s tools98.0k / 79.5k
Engine-service bound, Terminal-Bench replay70.2k / 63.8k
Eviction feedback, 30 s tools44.7k / 42.9k (one run)
FIFO with bounded bypass, versus shortest-first resumptionWorst wait of 180 s / 371 s at 30 s tools; throughput 44.7k / 43.9k

Thus, admission is the main source of throughput improvement, with the service bound supplying a substantial gain when compute binds. FIFO resumption and longest-idle-first pausing primarily improve waiting-time behavior: among instances launched in the first ten minutes of sustained 30 s load, AgentKV’s longest observed wait is 329 s, whereas 208 of 512 instances under the load-sweep settings wait more than ten minutes. These are measured outcomes for those arrival cohorts.

The forecast ablation also clarifies the original horizon result. The original replays grow little after their initial long context, about 70 tokens per call on Terminal-Bench, leaving little opportunity for H = 4 to improve on H = 1. On the new replays, per-instance forecasting provides no measurable incremental throughput benefit to AgentKV with the trajectory settings: at 30 s tools, it reaches 44.7k with the forecast, 46.2k without it, and 45.8k with an oracle. Section 6 distinguishes these component results from the overall admission gains.

For persistence, transfer counters explain the tied policies. At 147 kB per KV token, the original 256 GiB CPU tier holds 1.86M tokens, less than the GPU cache’s 2.36M, with the same recency order. In the measured all-persist run it writes 1.39 TB but reads back 0 bytes, providing little additional reusable coverage.

Adding a 768 GiB CPU tier, large enough to hold the entire working set, fundamentally changes these results. With this tier available, vLLM reaches 55.4k at 288 instances and climbs to 76.3k under a sustained 30-second load, surpassing AgentKV’s 59.0k baseline without a tier. This establishes a key scope condition: because a sufficiently large CPU tier eliminates the underlying memory bottleneck, AgentKV’s memory-admission gains specifically apply to engines operating without one. However, AgentKV's engine-service bound remains advantageous even when the tier is present. When running 0.5-second tools alongside the same 768 GiB tier, AgentKV achieves 96.8k compared to vLLM’s 80.8k, a distinction detailed further in Section 6 and Appendix F.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Quick Summary of Paper Revisions, New Experiments, and Rebuttal

We sincerely thank all three reviewers for their constructive reviews. We ran new experiments on the points raised and report the results, including those that narrow our claims.

What is new

  • A real workload. Each instance replays the first 16 model calls of a recorded SWE-agent or OpenHands run, with its real tool outputs, unpadded (1.6k tokens at the first call, 17.9k at the last; 288 instances end at 2.2 times the KV capacity). The pool is the 1,178 of 2,476 runs with 16 calls whose context reaches 12k tokens. Tool time is 0.5 s as in the paper, exponential (30 s mean), log-normal fitted to SWE-bench tool times, or heavy-tailed.
  • Baselines: vLLM; in the same proxy, ThunderAgent and CONCUR's published control law (Eq. 1 with its constants; our re-implementation, not the authors' system). The paper's other baselines were not rerun.
  • One-at-a-time isolations of the forecast, the admission order, the feedback and the CPU tier, and sustained-load runs (client slots full for 20 minutes).
  • AgentKV as revised (the controller of the revised PDF; the submitted configuration, used in the paper's load sweeps, is called the earlier configuration or the load-sweep settings). It bounds the requests in the engine (at 48, read from a fixed-window sweep), admits on net demand (forecast growth minus expected completions), pauses the longest-idle instances, and serves FIFO with bounded bypass. It was developed on two of the 0.5 s draws, one 30 s draw and the Terminal-Bench replay (and in simulation, incl. 576 instances and heavy-tailed gaps), then frozen before it ran any other cell on the real engine.

Results (thousand tokens/s; engine without a CPU tier, as in the paper's main sweeps; ranges cover the draws run; sustained rows: minutes 12–20)

settingvLLMThunderAgentCONCUR lawAgentKV
0.5 s tools, 288 instances (≤4 draws)29.3–30.275.8–77.596.8–98.497.6–98.0
30 s tools, 288 instances (≤3 draws)28.9–30.132.0–33.935.0–36.043.4–45.8
30 s tools, 576 instances––37.646.7
sustained, 384 slots, 30 s tools22.629.034.659.0
sustained, 288 slots, heavy-tailed gaps71.073.577.079.3
the paper's Terminal-Bench replay21.531.369.270.2

Findings

  1. The collapse occurs on real agent runs, and admission prevents it. Under sustained load with 30 s tools, vLLM settles at a 10% prefix-hit rate and 22.6k; AgentKV holds 59.0k.
  2. There are two regimes, and AgentKV matches or leads all three baselines in both. With short tool calls the engine is compute-bound: output peaks at 40–55 requests in the engine, CONCUR's law settles near that point, and AgentKV matches it; the earlier configuration is 11–14% below on recorded runs. With long tool calls memory binds; mechanisms that do not budget idle KV recompute prefixes (vLLM, ThunderAgent), open too slowly or over-admit (CONCUR's law), and AgentKV leads them.
  3. The gains come from what is budgeted, not from prediction. Without its forecast AgentKV loses at most 0.3% in any setting, and the forecast costs it 4–11% where memory binds hardest.
  4. Fairness costs little throughput. Shortest-first ordering buys no throughput (85.1k with FIFO against 84.9k), and the earlier configuration starves long contexts: under sustained 30 s load 208 of its first 512 instances wait more than 10 minutes. AgentKV is FIFO and nearly size-neutral; its longest wait in that run is 329 s. Against the earlier configuration, both without forecast, this costs nothing in fixed batches and 2% under that load (61.7k against 63.0k).
  5. Persistence is explained, and it limits our claim. The paper's 256 GiB tier holds fewer tokens than the GPU cache, so its three policies could only tie. A tier that holds the whole working set (768 GiB) needs no memory admission: vLLM alone then reaches 76.3k under the sustained 30 s load, above every controller in that row. The paper's claim is therefore scoped to engines without such a tier (App. F); the bound on engine service remains valuable with one (at 0.5 s tools, 96.8k against 80.8k for vLLM with the tier).

Revised PDF (changes in blue): Section 4 adds the service bound and FIFO order, 5.3 trajectory replays and sustained load, a new Section 6 ablations (forecast, order, tier), Appendix C the details, and Section 2 ThunderAgent, CONCUR, Past-Future and MAPS.

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close soon, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.