- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents AgentKV, a predictive admission control system designed to prevent KV-cache collapse in high-concurrency agentic LLM serving. AgentKV predicts future context growth and dynamically controls the set of active agents to keep their KV caches within GPU memory capacity. It also incorporates cache-aware feedback to adjust admission limits and selective CPU KV persistence to reduce recomputation. Experiments across multiple models and agent workloads show up to 3.7× throughput improvement over vLLM, with significantly higher prefix-cache hit rates and lower TTFT under high concurrency.
Critical strengths
- The paper evaluates AgentKV across a wide range of experimental settings, including multiple models, workloads, concurrency levels, and GPU platforms, covering recent GPU architectures such as NVIDIA B200.
Critical weaknesses
- Unclear contributions and benefits over existing work, with most proposed techniques showing limited effectiveness. The core contribution of AgentKV appears to be extending admission control (already proposed in ThunderAgent and CONCUR) with context growth prediction and cache-aware feedback. However, the paper does not clearly demonstrate the benefits of these additional mechanisms. In fact, the experimental results, particularly the ablation study, suggest that most of the proposed techniques provide little to no benefit: (a) batch-adaptive decoding is disabled in the evaluation; (b) reducing the prediction horizon from H=4 to H=1 has negligible impact on performance; and (c) disabling persistence even improves throughput. These observations raise concerns about the actual sources of AgentKV's performance gains. I suspect that much of the improvement may instead come from short-first scheduling, which is a relatively minor optimization that comes at the cost of worse tail latency and fairness.
- Artificially constructed evaluation workloads with questionable representativeness. The evaluation relies on manually selected workloads with rather unusual characteristics. In particular, each agent performs only four LLM calls, while the context is artificially extended to 24–25k tokens. Such settings may not accurately reflect real-world agent workloads. Given that one of the paper's primary contributions is integrating context growth prediction into admission control, evaluating agents with only four LLM calls is particularly concerning, as such short and controlled execution traces may make context growth easier to predict. More fundamentally, I am skeptical of the assumption that context growth can be reliably predicted in realistic agent workloads, especially coding agents, where tool outputs can vary significantly in length and occasionally produce extremely long, unpredictable prefill sequences.
Questions for authors
- What are the actual benefits of context growth prediction over existing admission control mechanisms?
5 comments
Response to Reviewer vJ1V (2/3)
The targeted ablations address the individual mechanisms raised in the review:
- Service bound and feedback. Removing the service bound reduces throughput from 98.0k to 79.5k with 0.5 s tools and from 70.2k to 63.8k on Terminal-Bench. Removing the eviction-feedback rule used in the trajectory experiments reduces 30 s throughput from 44.7k to 42.9k in the measured run. This rule uses the fraction of expected-cached prefix tokens that missed; the absolute-threshold rule used with the load-sweep settings was triggered on 98% of short-tool control ticks, explaining its limited ablation effect.
- Short-first scheduling and fairness. Replacing shortest-first with FIFO under the load-sweep settings gives 85.1k versus 84.9k, and we see that there is no measurable throughput advantage from shortest-first in that test. AgentKV's mean worst request waits for short/long-context quartiles are 77/75 s with 30 s tools, and the maximum falls from 676 s under the load-sweep settings to 180 s. In the sustained 30 s test, none of the 493 AgentKV instances launched during the first ten minutes waits over ten minutes. The maximum is 329 s, compared with 208 of 512 instances exceeding ten minutes under the load-sweep settings. These are measured cohort results, not a universal latency bound. Comparing both configurations without forecasts, AgentKV retains fixed-batch throughput and costs approximately 2% under sustained memory pressure (61.7k versus 63.0k), while removing the observed extreme waits.
- Prediction horizon. The original Terminal-Bench replay begins with approximately 24k tokens and adds only about 70 tokens per call. These relatively small increments help explain why performance remains largely unchanged between and . To more directly evaluate the effect of prediction horizon, we conduct additional experiments using 16-call trajectories, with results presented below.
- Persistence. The 256 GiB tier holds 1.86M tokens, below the GPU cache's 2.36M, with the same recency order. In the transfer-counter experiment, persist-all writes 1.39 TB and restores zero bytes, explaining the limited benefit in this configuration. A 768 GiB tier retains the whole working set and lets vLLM reach 76.3k under sustained 30 s load, compared with 22.6k without the tier and 59.0k for AgentKV without it. This uses substantially more host memory (about 1,150 GiB) and establishes the scope of the memory-admission advantage. The service bound remains beneficial with that tier: at 0.5 s tools, AgentKV reaches 96.8k versus 80.8k for vLLM.
- Batch-adaptive decoding. This is an optional integration point in Figure 1 and is not used in the reported experiments; none of the gains is attributed to it.
W2. Workload representativeness and irregular coding-agent growth.
This added workload replays the first 16 model calls from recorded SWE-agent and OpenHands runs, using their actual tool outputs without any artificial padding. Throughout these trajectories, contexts expand from roughly 1.6k to 17.9k tokens. From an initial pool of 2,476 runs containing at least 16 calls, we specifically select the 1,178 runs that reach 12k tokens by call 16 to target the long-context regime that heavily strains the cache.
Tool execution times span four configurations: fixed 0.5-second delays, a log-normal distribution fitted to published SWE-bench tool times (mean 0.93 s), a 30-second exponential distribution, and a heavy-tailed mixture. Across all experiments, every system receives identical replayed prompts and tool latencies, while sustained-load evaluations continuously refill client slots over a 20-minute run and report steady-state throughput across minutes 12–20.
A separate analysis of 4,090 trajectories directly addresses the workload irregularity raised in the review: tool outputs account for 80–85% of context growth, single-step increments reach up to 23k tokens, and lag-1 autocorrelations sit at just 0.14 for SWE-agent and −0.04 for OpenHands, making recent increments weak predictors of individual-run growth. However, aggregating across instances substantially improves forecast accuracy: across 100 instances, an estimator that shrinks exponential moving average (EMA) growth toward the population mean achieves a 8–10% 95th-percentile error four calls ahead, compared to 30–31% when relying on current occupancy alone. We explicitly distinguish this statistical estimation gain from end-to-end throughput improvements, which we isolate in a separate ablation.
Response to Reviewer vJ1V (3/3)
Q. What are the actual benefits of context-growth prediction over existing admission control?
We measured this directly, and the answer supports the reviewer: prediction is not what helps, and the revision says so (Section 6). In the earlier configuration the forecast gives +6% with short tool calls, nothing from being per-instance (86.3k with a population-mean forecast against 84.9k), and −18% to −24% with long ones. In AgentKV, with the forecast, without it and with an oracle: 44.7k, 46.2k and 45.8k at 30 s tools. With and without it: 45.8k and 45.9k, 43.4k and 43.8k on two further draws; 79.3k and 79.4k, 59.0k and 61.7k under the two sustained loads; 46.7k and 52.3k at 576 instances.
The benefits come from three other elements, each measured separately:
| element | measured effect |
|---|---|
| a bound on the requests in the engine | without it, 79.5k instead of 98.0k at 0.5 s tools, and 63.8k instead of 70.2k on Terminal-Bench |
| budgeting the resident working set, idle contexts included, without over-reserving | 43–46k at 30 s tools, against 32.7–36.5k for the earlier configuration (43.0–44.7k once its forecast is off) |
| FIFO with bounded bypass, and pausing the longest-idle instance | removes starvation and the bias against long contexts; costs no throughput in fixed batches and 2% under sustained memory-bound load (whole controllers, no forecast) |
With its forecast switched off, the earlier configuration is as fast as AgentKV where memory binds, or faster (63.0k against 59.0k under sustained load), but 153 of its first 536 instances then wait more than 10 minutes.
We would be glad to provide any further detail.
Response to Reviewer vJ1V (1/3)
We thank the reviewer for highlighting the need for realistic agent trajectories, targeted component ablations, and a direct assessment of fairness. We added all three to the paper, and addressing it below. :)
We provided some new results that show AgentKV matches the strongest evaluated concurrency controller when compute binds and improves throughput when memory binds, with balanced waiting across context lengths.
W1. Contributions and benefits over existing admission control.
These experiments pinpoint the primary drivers behind AgentKV's gains: budgeting the resident working set, including tool-waiting contexts, and regulating admission near the engine's service peak. Meanwhile, FIFO admission with bounded bypass and longest-idle-first pausing govern which requests wait. Specifically, AgentKV operates with a service threshold of 48 requests, a value determined through a fixed-window sweep, and the common response details how these parameters were selected and held constant across evaluations.
First, we compared with stock vLLM, ThunderAgent in the same proxy, and our reimplementation of CONCUR's published control law on vLLM. The latter comparison evaluates the control law, not the authors' complete system. These runs use no CPU KV tier.
With short tool calls, output peaks at roughly 40–55 requests in the engine even without eviction. Operating near this peak, AgentKV stays within 0.8k of CONCUR's law across all six short-tool trajectory settings, which we regard as comparable performance. With long tool calls, however, resident idle KV becomes critical: whereas ThunderAgent discounts waiting contexts over time and the implemented CONCUR law relies on running-request KV usage and prefix-hit feedback, AgentKV explicitly budgets these resident idle contexts. Consequently, in the 288-instance batch, AgentKV reaches 44.7k compared to 36.0k for CONCUR's law, a comparison that includes window-opening time, with corresponding mid-run rates of 61.1k and 55.6k. Under sustained load, AgentKV maintains 59.0k, while vLLM drops to 22.6k alongside a 10% prefix-hit rate.