acceptodds
Withdrawn from ICLR 2027

EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents resend their whole conversation on every turn, and most of it was already processed on the previous turn. Serving systems avoid recomputing it by caching its key–value (KV) state and, when GPU memory runs out, by offloading that state to host memory. For agents, offloading gives inconsistent results: on the same coding-agent workload it speeds up one deployment, slows down another, and changes nothing on a third, even where loading a token back is several times cheaper than recomputing it. The reason is that cached state must survive until it is used again. While one agent waits for its tool, the server processes the contexts of all other agents, so an agent's prefix is reused only if the host tier holds the reusable context of the whole agent pool, which we call the reuse working set. A smaller tier keeps writing state that is evicted before anyone reads it. We present EfficientAgent, which sizes and manages the host tier by this working set. A stack-distance model estimates the working set from agent histories to size the host tier; its predictions, made before the experiments, located the capacity at which offloading starts to pay. When the tier is too small, a runtime policy stops writing large refills of evicted context and keeps extending prefixes that are still cached; when the tier is large enough, it writes everything. On SWE-bench Verified coding agents, a host tier sized to the estimated working set cuts recomputed prompt tokens by 93% and end-to-end time by 39%. With a small fixed tier, the policy cuts recomputation by 35%; with a large tier, it avoids the 4.3-fold increase caused by always filtering writes. Across three GPU types and two models, offloading pays off when the GPU has little compute per byte of host bandwidth and the host tier holds the working set. Code is available at https://anonymous.4open.science/r/efficientagent_release-6927.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, KV cache, LLM serving, KV cache offloading, prefix caching, working set, cache admission, agentic inference

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_ASgbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the cost and efficiency of KV cache offloading for LLM inference engines serving concurrent agentic requests and identified that it is important to look at the cost of restoring cached prefix versus recomputing and the chance a cached token survives until its next reuse. The authors then introduce a stack-distance-based model for estimating the reuse working set and a capacity-conditioned admission policy that bypasses large cache refills under memory pressure. They have also conducted experiments on SWE-bench Verified and demonstrate that sufficient host memory substantially reduces prefill recomputation and adaptive admission improves performance under constrained host-memory budgets.

Critical strengths
  1. The topic of this paper is relevant as concurrent agentic serving system is getting increasingly important while KV re-use is highly relevant to the deployment cost and runtime efficiency.
  2. The authors have explained clearly the seemingly inconsistent offloading performance. The experiments across RTX 3090, H20, and H800 demonstrate that faster host restoration does not necessarily translate into better end-to-end performance.
Critical weaknesses
  1. I find that the paper is not easy to read as the language is not formal, complete and smoothly transitioned in many places, perhaps due to AI drafting. The author needs to put in efforts to make the paper more logically smooth and academically formal.
  2. The novelty of this paper is limited. Despite sufficient analysis of the existing KV cache offloading, the resulting algorithm is a simple heuristic rather than a fundamentally new cache-management mechanism.
  3. The experimental evaluation primarily compares against standard LMCache offloading and a fixed write-admission variant. However, several relevant alternatives are not evaluated including Marconi and Continuum.
  4. The authors have only conducted experiments on the SWE-Bench Verified. However, other agentic workloads such as browser-use and deep research might exhibit substantially different characteristics. The user needs to demonstrate the generality of the proposed adaptive admission method.
Questions for authors
  1. How can we assess whether the proposed algorithm is good enough? How does it compare against future-reuse-aware oracle that knows subsequent KV references? What fraction of the theoretically achievable improvement does the proposed policy capture?
Reviewer_bjnHOpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EfficientAgent studies when host-memory KV-cache offloading helps concurrent coding agents. It separates the cost of restoring a cached prefix from the probability that the prefix survives until reuse. A stack-distance model estimates useful consecutive-prefix coverage after subtracting GPU-resident state, and a working-set estimate guides host-tier sizing. The runtime then suppresses large refill writes only when the estimated working set exceeds host capacity and the tier is full and evicting. Dependency-preserving replay of SWE-bench Verified trajectories holds token work fixed while allowing serving policies to change task interleaving. On H20, increasing host capacity from 5 to 20 GiB per rank reduces computed prefill by 93.1% and makespan by 38.7%; at a fixed 5 GiB, conditioned admission reduces computed prefill by about 35% and makespan by 10.4%.

Critical strengths
  1. The study gives a useful explanation for inconsistent offloading results. Cheap restoration is insufficient if concurrent agents evict each other’s reusable prefixes before the next turn. The analysis goes beyond counting cache hits by retaining the exact-prefix constraint and subtracting state already present on the GPU. The capacity sweep and active-pool experiment support this account, and the reversal of fixed write admission above and below the working set is particularly informative. This is a meaningful application of classical working-set ideas to an increasingly important workload.

  2. The replay methodology makes the central comparisons substantially more convincing. The experiments retain all recorded prompt and output tokens and preserve each task’s dependencies and inter-call delays, while allowing queueing and interleaving to change. This avoids attributing differences in generated trajectories to a cache policy. Complete token, transfer, preemption, and makespan counters are provided, and predictions for additional capacities were recorded before those runs. The policy also reproduces unrestricted offload when its filter does not engage, which is a useful implementation control.

Critical weaknesses
  1. The incremental value of the proposed admission policy is not established against stronger alternatives. Its comparisons are unrestricted offload and an always-on version of its own refill filter. These show why the filter should depend on capacity, but not whether the working-set estimator is preferable to a simpler feedback-based controller or an existing adaptive admission policy. The paper cites several relevant admission and agent-aware cache systems without an empirical comparison at this layer. This matters because the largest headline improvement comes from increasing host memory; the new runtime policy’s fixed-capacity gain is the smaller 10.4% makespan reduction at 5 GiB.

  2. The predictive and adaptive claims need validation beyond the source workload’s interleavings. The detailed sizing and admission results use one H20 coding-agent trace. Appendix C constructs prediction ranges from that workload’s 5 and 40 GiB reference runs, which is useful for interpolation across capacity but does not demonstrate transfer to new agent histories or tool-delay distributions. Errors also grow when concurrency changes: at eight active tasks and 4.5 GiB, predicted restored volume is 22.25–29.36M tokens, versus 13.22M measured (Table 5). The runtime additionally uses fixed 60-second activity and 256-request prompt windows, while the sensitivity analysis mainly reclassifies existing decision streams. This leaves uncertainty about how well the controller responds when the workload changes rather than replaying the same task mix.

Questions for authors
  1. Policy comparison: Can you compare conditioned admission with a feedback-only controller and at least one adaptive admission baseline under identical host/GPU budgets? In particular, how much does the working-set estimate add beyond occupancy and eviction feedback?
  2. Prediction and adaptation: Can you test the sizing model and controller on a held-out trace or a controlled change in tool delays, context-length skew, or active-pool size? Please explain the restored-volume overprediction in Table 5 and report whether estimation errors produce harmful admission decisions.
  3. Deployment guidance: Can you separate the effect of hardware compute/transfer cost from cache residency in the cross-GPU results, for example with matched working-set pressure and measured effective prefill/restore costs? The deployment-level observations are useful, but the current GPU configurations differ in TP degree and memory allocation, so peak FLOP/host-byte ratio alone does not establish a generally reliable offloading rule.
Reviewer_NcDQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when KV cache offloading benefits concurrent LLM agents. It distinguishes the cost savings of restoring cached KV states from their likelihood of remaining cached until reuse, arguing that offloading effectiveness depends on both hardware characteristics and the agent pool’s reuse working set. The proposed system, EfficientAgent, uses a stack-distance model to estimate host-cache capacity requirements and a capacity-conditioned admission policy that suppresses large cache refills under memory pressure while retaining incremental prefix extensions. The authors evaluate the approach using dependency-preserving replay of SWE-bench Verified agent trajectories, complemented by experiments across three GPU types and two models. On the main H20 workload, increasing host capacity from 5 to 20 GiB per GPU reduces recomputed prefill tokens by 93.1% and makespan by 38.7%. At a fixed 5 GiB capacity, the admission policy reduces recomputation by approximately 35% and makespan by 10.4%, while avoiding the performance degradation caused by unconditional write filtering at larger capacities.

Critical strengths
  1. The paper separates the value of recovering cached KV states from whether those states survive until reuse. This distinction provides a useful explanation for why inexpensive host restoration alone does not guarantee end-to-end speedups.

  2. The replay fixes prompts and generated tokens while preserving dependencies between model calls and tool execution. This enables comparisons on identical computational work while allowing serving policies to affect queueing and task interleaving.

  3. The experiments demonstrate that unconditional write filtering helps under tight capacity but becomes harmful when sufficient cache is available. The proposed capacity-conditioned policy captures the benefit in the constrained regime and avoids this degradation at larger capacities.

Critical weaknesses
  1. Limited evidence for the admission policy’s advantage over simpler alternatives. The main comparisons are against default offloading and unconditional write filtering. They do not isolate the contribution of the working-set estimate relative to an occupancy/eviction-only controller or establish an advantage over concurrency tuning.

  2. The scope of the predictive validation is narrower than the deployment claims suggest. The capacity predictions use reference runs from the same workload, and the working-set scale uses its full-trace mean prompt length. This supports capacity prediction after profiling, but does not establish generalization to unseen workloads or prediction from limited observations. Errors are also substantial in some configurations: at eight active tasks and 4.5 GiB, predicted restoration is 22.25–29.36M tokens versus 13.22M measured.

  3. The evaluated deployments differ in tensor parallelism, GPU cache budgets, and sometimes workload and timing scope. On H800, offloading also provides little improvement in total prefix-hit rate, making it difficult to attribute the slowdown specifically to the compute-to-bandwidth ratio.

  4. Proposition 1 assumes a fixed reference stream and rejects only insertions satisfying a future reuse-distance condition. The runtime policy approximates this condition, and only 63.6% of declined chunks satisfy it at 10 GiB.

  5. The central admission experiments focus on one coding-agent workload, one model, and H20 hardware; the broader hardware experiments primarily evaluate offloading rather than the proposed policy. Most policy configurations lack repeated runs, and threshold sensitivity is assessed on recorded decision streams rather than through closed-loop executions. Consequently, the stability of smaller gains and sensitivity to changing workload dynamics remain unclear.

Questions for authors

How much does the working-set estimate contribute beyond cache-pressure feedback? Could you compare the proposed policy against a controller using only occupancy and eviction signals, and against active-pool tuning under the same resource budget?

Can the capacity model generalize beyond the profiled workload? How accurately does it predict the capacity transition using only an initial profiling window or a separate task set? What explains the restoration discrepancy at eight active tasks and 4.5 GiB in Table 5?

What evidence isolates the hardware compute-to-bandwidth ratio as a cause of the observed performance differences? Could you report measured prefill and restoration costs alongside GPU/host cache coverage, accounting for differences in tensor parallelism and memory budgets?

What are the consequences of rejecting writes that do not satisfy Proposition 1’s sufficient condition? Can you quantify the useful reuse sacrificed by these decisions and examine whether protecting resident prefixes disproportionately increases latency for particular tasks?

How robust are the admission-policy gains across repeated runs and workload changes? Could you provide variability estimates for the main comparisons and closed-loop evaluations with different thresholds, tool delays, or context-length distributions?

Reviewer_mg4UOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why host-memory KV offloading sometimes fails to help concurrent LLM agents even when restoration is cheaper than recomputation. It attributes this to cache survival across interleaved agent turns, estimates the required capacity using reuse distances, and proposes a pressure-conditioned admission rule that avoids large refills when the host tier is undersized. On SWE-bench Verified replay, increasing host capacity from 5 to 20 GiB per rank reduces computed prefill by 93.1% and makespan by 38.7%. At a fixed 5 GiB capacity, the admission rule reduces makespan by 10.4% relative to unfiltered offloading.

Critical strengths
  1. The paper clearly separates the cost of restoring a prefix from whether that prefix survives until reuse. The capacity sweep and the reversal of unconditional filtering at large capacity support this explanation. This is a useful diagnosis of when host offloading wastes work.

  2. Fixing prompt and output tokens while preserving within-task dependencies makes the replay a useful way to compare serving policies. The prefill and transfer counters also help explain the observed changes in makespan, rather than leaving the result as an unexplained speedup.

Critical weaknesses
  1. The practical importance of the capacity constraint is not established. The main experiment reserves 7.86 GiB of GPU KV memory per rank and obtains the policy gain with a 5 GiB host tier. Simply increasing host capacity substantially reduces the problem; once capacity is sufficient, the conditioned filter does not intervene. These experiments show that the rule helps an undersized cache, but do not explain when a realistic deployment must remain in that regime. A real agent trace does not establish that the memory allocation is representative. The paper needs a concrete host-memory budget, including competing uses and the cost of provisioning more capacity, to show why selective bypass is preferable to a sufficiently provisioned tier. A larger SSD tier has different access costs from DRAM, but its exclusion also needs justification rather than treating the small host tier as the only available design.

  2. Refill size is a weak proxy for the value of caching a request. A cold, large refill may be reused soon and avoid expensive recomputation, while a small resident-prefix extension may never be needed again. The runtime rule does not explicitly account for next-use timing, remaining reuse, or the compute saved relative to write/restore costs. Proposition 1 assumes a condition on future reuse distance that the rule does not enforce. The policy also favors tasks whose prefixes are already resident: a task that loses its prefix may repeatedly require a large refill and be denied admission again. This is a risk of persistent cache exclusion, not evidence of request starvation. Aggregate makespan and prefill reductions do not resolve it. Without a reuse/cost-aware comparison and per-task latency results, it is unclear whether the gain reflects better cache allocation or shifting recomputation onto a subset of tasks.

Questions for authors
  1. Deployment constraint. In the H20 setup, what concrete server-memory budget makes 5 GiB per rank necessary, rather than the 20 GiB setting that largely removes recomputation? Please report total host DRAM and its competing allocations, explain the GPU KV reservation, and state why a larger lower storage tier is excluded. I am asking whether the evaluated pressure is a realistic constraint, not assuming DRAM or SSD access is free.

  2. Admission value. At 5 and 10 GiB, do requests rejected by the filter differ from admitted requests in their time to next use and subsequent recomputation cost? Please compare the rule with a matched-budget admission baseline that estimates next use from past inter-turn timing and accounts for compute saved versus transfer cost. This would test whether the eight-chunk cutoff identifies low-value writes or primarily selects already-resident tasks.

  3. Per-task effects. For the same replay tasks at 5 and 10 GiB, please report the distribution of task-completion-time ratios against unfiltered offloading, including the worst slowdowns. For tasks that lose their host prefix, how many consecutive later requests have their writes rejected, and how many regain substantial host-prefix coverage before completion? These measurements would reveal whether the aggregate gain comes with persistent cache exclusion for some tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.