acceptodds
Under review as a conference paper at ICLR 2027

Harness-Agnostic Predictive KV-Cache Prefetch Engine for LLM Agents

Abstract

Long-running LLM agents alternate between model inference and tool execution, creating idle periods during which cached prefixes may be evicted. Subsequent requests can also encounter uncached tool outputs or cold-started subagents. Because inference servers do not observe tool execution, returned results, or subagent creation before the next request arrives, they cannot use these gaps to prepare the KV cache. We introduce a harness-agnostic predictive KV-cache prefetch engine that uses agent-runtime lifecycle events to move KV-cache preparation off the critical path. During tool execution, the engine issues low-priority warmups for the known prefix; after completion, it warms the anticipated next-request prefix with the actual tool result. It also prepares initial prefixes for newly spawned subagents. A contextual Thompson sampling policy selects whether and how to prefetch based on cache state, prefix length, and predicted tool duration, balancing cache reuse against warmup cost and foreground interference. Across Claude Code, Hermes, and Codex on GAIA and SWE-bench Lite, the engine reduces aggregate time to first token by 67.6%, 19.6%, and 68.9%, respectively. Subagent warming reduces median first-call TTFT by 68.6%, from 564 ms to 177 ms. At eight concurrent sessions, warmups add 0.5% prompt-token overhead. These results demonstrate that harness-visible execution gaps can be used to prepare anticipated KV state and reduce cold-prefill latency in long-running agents.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Agents, KV-Cache Optimization, LLM Inference, Predictive Prefetching, Inference Serving Systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ymsGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a harness-agnostic KV-cache prefetch engine that reduces LLM agent latency by warming cached prefixes during tool execution. It handles cache eviction, newly returned tool outputs, and subagent cold starts using background inference requests. A Thompson Sampling policy decides when prefetching is beneficial to avoid unnecessary overhead. Experiments across Claude Code, Hermes, and Codex show 19.6–68.9% reductions in aggregate time to first token, with only 0.5% additional prompt-token overhead at eight concurrent sessions. The main limitation is that prefetching consumes inference capacity and may interfere with foreground requests under high load.

Critical strengths
  1. Good direction on agentic inference.
Critical weaknesses
  1. Terrible use of AI technology in paper writing. The graphs are AI generated and screenshot into the paper.
  2. No comparison with previous methods over the past 2 years...
Questions for authors
  1. How is this novel and not done by previous people? YOu have no comparison with prior work!! This is outrageous
Reviewer_rds5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper implements harness-triggered KV-cache warming using lifecycle events from tool execution and subagent creation. It warms stable prefixes during tool calls, augments prefixes with actual tool results after completion through PASTE, and reuses captured initial prefixes for subsequent subagent spawns. Admission heuristics and a contextual Thompson-sampling policy select warmups, with Dynamo/vLLM routing and telemetry support. Content-preserving replay of Claude Code, Hermes, and Codex traces reports aggregate TTFT reductions of 67.6%, 19.6%, and 68.9%, respectively. A separate subagent experiment reports a median first-call TTFT decrease from 564 to 177 ms.

Critical strengths
  1. The harness is a useful observation point for distinguishing eviction misses, newly available tool-result tokens, and recurring subagent prefixes. The implementation supports both in-process intercepts and CLI hook/proxy integrations, making the architecture more concrete than a single-framework proposal.

  2. The replay evaluation preserves request contents and includes native prefix caching under cache pressure, rather than only a cache-busted comparison. The separate PASTE and worker-routing measurements help establish that correct prefix construction and locality can produce real KV hits. The paper also reports short-gap inactivity and some latency regressions instead of claiming uniform benefit.

Critical weaknesses
  1. The timing and interference measurements do not yet establish the end-to-end value of post-tool warming. PASTE is dispatched only after the actual tool result arrives and is said not to delay the next foreground request (Section 3.3), leaving its useful lead time unclear. Section 3.1 also suppresses warmups while a designated measurement request is in flight, and main replays pin warmups and foreground requests to one worker. Without event timelines and an inclusive latency boundary, lower request TTFT could partly reflect work moved before measurement or favorable replay scheduling. The separate prompt-token overhead experiment does not measure GPU service time or harm to concurrent NoiseRunner traffic.

  2. The main architectural and adaptive-policy claims lack direct controls. The paper explicitly does not compare with prior cache-management systems (Section 6), so the large gains over native caching do not establish an advantage over TTL retention or existing agent-aware management under the same constraints. The bandit improves normalized TTFT from 0.286 to 0.282 over a learned heuristic, without reported uncertainty. Its Markov action sends the same warmup payload as standard warming (Section 3.1), so the benefit of next-tool prediction is not demonstrated. These gaps leave the contribution of the predictive policy unclear relative to a simpler event-triggered warmup rule.

Questions for authors
  1. For PASTE misses without tool-result reuse, please show tool completion, warmup enqueue/start/end, foreground release/enqueue, and first-token timestamps. What lead time is available in live execution, and what latency is obtained when measured from tool completion?

  2. Which requests activate the measurement guard, and is it enabled in every main B/C comparison? Please report task JCT, foreground tail latency, concurrent-tenant latency, and actual GPU/prefill work under matched offered load.

  3. Under the same backend, affinity, and cache capacity, how does the engine compare with a retention baseline and a deterministic event-triggered warmup policy? Please separate admission controls, PASTE, bandit selection, and the duplicate-payload Markov action, with repeated-run uncertainty.

Reviewer_B3eaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds a KV-cache warmup engine to the agent harness. During tool calls it sends one-token, low-priority requests to keep the session prefix cached; when a tool returns, it warms the prefix plus the tool output (PASTE); and it pre-warms subagent prefixes. In trace replays on Llama-3.1-Nemotron-70B, with background load that forces eviction, total TTFT drops by 67.6%, 19.6% and 68.9% for Claude Code, Hermes and Codex compared with native prefix caching.

Critical strengths

The harness is the right place for this, since it knows about tool gaps and subagent spawns that the server can't see. The engine works with three real harnesses without changes to serving code, and the replay protocol is careful (cache flush, canary check, contaminated trials discarded; L791-792).

Critical weaknesses
  1. The gains depend on heavy load, and the cost to other traffic isn't measured. Condition-B stalls in Fig. 3 range from about 8 to 115 s, far longer than prefill should take at ~37 ms per 1,000 tokens, which suggests queueing. Keeping one session warm under that load must delay other work, and the paper admits there's no strict isolation, yet only the replayed session's TTFT is reported. In the 86-session Codex run, sessions with fast baselines see almost no benefit (median C/B 0.997 for 54 sessions vs 0.841 for the other 32).
  2. The 0.5% overhead comes from a separate experiment in which 8 sessions used 14,837 warmup tokens, about two re-warms of the 7,000-token prefix in Sec. 4.1. That experiment can't have involved much eviction rescue, so the overhead of the Table 1 runs is unknown. Re-warming an evicted prefix costs the same prefill it saves; it only moves it off the critical path.
  3. There are no baselines beyond native prefix caching, such as TTL retention, InferCept-style preservation or CPU offloading. Mode 1 and PASTE aren't ablated separately in the main runs, and PASTE is isolated only by KV-hit rate in one live session.
  4. The learned parts add little. The "Markov" action sends the same payload as a standard warmup, so next-tool prediction never changes what's warmed, and the bandit beats a heuristic by 0.282 vs 0.286 with no variance.
Questions for authors
  1. What are the background sessions' TTFT and total throughput with and without the engine, and how do the gains change at lower load?
  2. How many warmup tokens and how much GPU time did the Table 1 runs use?
  3. What is a designated measurement request? If it covers the reported requests, interference is excluded by construction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.