acceptodds
Under review as a conference paper at ICLR 2027

Q-Bridge: Agent-Directed KV Cache Lifecycle Management for Efficient LLM Serving

Abstract

Agentic workloads repeatedly revisit, branch from, and abandon contexts across model and tool calls. However, execution intent available to the agent is not necessarily reflected in the request history used by inference systems to manage key–value (KV) caches. This mismatch can leave obsolete contexts resident while evicting contexts needed by subsequent calls. We propose Q-Bridge, an agent-directed protocol for KV cache lifecycle management. Q-Bridge lets agents issue explicit retention, release, and revision commands for identified contexts, while an infrastructure controller admits and executes these commands under memory and transfer constraints. Versioned lifecycle state persists across requests, allowing execution events to update cache decisions without requiring repeated state exchange before every model invocation. When intent is uncertain, the controller selectively requests additional information based on its expected decision benefit and communication cost. We formulate the resulting coordination problem as minimizing serving latency over an agent execution episode subject to cache capacity and communication constraints. Our evaluation targets persistent-cache workloads with interleaved agent executions, measuring time to first token from request arrival, including protocol overhead, alongside cache reuse, recomputation, and task quality. Q-Bridge provides a framework for studying explicit agent control over cache lifetimes and its tradeoffs against inference-only policies and unconditional state sharing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, KV cache management, agent-infrastructure coordination, efficient inference, LLM serving

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mBqoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Q-Bridge proposes an Agent-to-Infrastructure interface that allows agent applications to explicitly control physical KV-cache retention in vLLM. By leveraging application-known future branch selections, Q-Bridge reduces redundant prefill computation and TTFT.

Critical strengths
  1. Leveraging application information to guide KV cache management is a reasonable and promising direction.
Critical weaknesses
  1. Limited novelty. The core idea of the paper is application-informed KV cache management, which has already been explored in prior works such as KVFlow and PBKV. The paper does not propose any new cache management policies or prediction methods. Instead, its main contribution appears to be providing interfaces for application-informed cache management, which is primarily an engineering effort and has already been implemented in systems such as Pie.
  2. Unrealistic assumptions in the evaluation. The evaluation heavily relies on knowing which execution branch will be taken in the future. Under this assumption, retaining the corresponding KV cache can naturally improve performance. However, I question whether this assumption is realistic, as real-world applications typically cannot determine future execution branches without a dedicated prediction mechanism. The paper neither proposes such a mechanism nor evaluates the impact of prediction uncertainty, making it unclear whether the reported benefits would translate to practical deployments.
Questions for authors

No

Reviewer_MSUxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Q-Bridge, which enables application-level cache retention with an independent cache controller to reduce local KV recomputation.

Critical strengths
  1. The idea of building an API-Infra KV retention mechanism is interesting.
  2. The TTFT results look promising.
Critical weaknesses
  1. The paper writing is very subpar, with incoherent sentences and out-of-context terms throughout the paper. While AI-assisted tools are allowed for paper writing, the readability of this paper is very low for human readers.

  2. Overall, this paper leans more toward a technical report instead of a scientific paper. Most importantly, this paper lacks a proper structure to guide the readers to understand the why and how behind the core idea: an API-to-backend interface that allows more flexible KV retention. Section 3 just briefly introduces the system design but lacks the proper reasoning and insights behind the design.

  3. Figure 1 is also not ideal for readers to understand the mechanism, with too many subtexts. A separate algorithm or workflow figure could significantly improve readability.

  4. The baseline methods for Q-Bridge are unclear to me. There are no citations of the selected baseline methods, which makes it hard to judge the novelty of this work.

  5. There are also many small typos throughout the paper. For instance, why is C=768 blocks in equation 1? What is TTL in line 124? What does two-GPU study mean here? Does it mean you are running in TP=2 or spawning two instances?

Questions for authors

Please see the weaknesses.

Reviewer_UbtbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Q-Bridge, a system that allows LLM agent applications to communicate cache-retention intent to the underlying inference infrastructure. The central motivation is that agent applications may possess information about future context reuse that is unavailable to conventional cache management policies. Q-Bridge introduces an application-to-infrastructure interface connecting a Pi Coding Agent extension, an independent validation sidecar, and a modified vLLM backend. Through explicit context identification, retention requests, physical cache reservations, and acknowledgement mechanisms, the system allows applications to influence KV-cache residency while preserving the original inference requests and application behavior. Experiments demonstrate reductions in final-request time-to-first-token and executed prefill computation under cache pressure. However, overall workflow latency does not improve, and a simpler one-bit retention control reproduces the same prefill benefits, suggesting that the demonstrated advantage arises primarily from application-informed retention decisions rather than the expressiveness of the proposed interface.

Critical strengths

The paper aims to address a systems challenge at the intersection of agent orchestration and inference infrastructure. Its central premise—that applications can expose useful future-reuse information to improve physical KV-cache management is compelling, particularly as agent workloads increasingly involve branching, verification, and repeated context access. The explicit separation between application-level intent, controller validation, and backend physical enforcement is a reasonable architectural design that could facilitate integration with existing serving systems. The implementation also includes practical considerations such as context identity, bounded reservations, acknowledgement handling, and release semantics. The inclusion of complete-workflow timing and equal-information controls provides useful evidence about the actual source and limitations of the measured performance improvements.

Critical weaknesses

The main weakness is the limited evidence that the proposed interface provides a substantial benefit beyond simpler application-guided cache hints. The one-bit control achieves equivalent prefill reductions, indicating that the richer Q-Bridge protocol has not demonstrated additional performance value in the evaluated setting. Improvements in final-request TTFT do not translate into end-to-end workflow acceleration; the reported mean workflow time is slightly higher, suggesting that communication, verification, and controller overhead may offset the local benefits. The experimental setting is also relatively narrow, relying primarily on a single small language model, older V100 hardware, and controlled agent workloads. This limits confidence in the system's scalability and applicability to contemporary multi-GPU serving deployments. The evaluation does not sufficiently establish the benefits of multiple competing retention leases, dynamic global cache allocation, or heterogeneous agent workloads. The reliance on application-provided retention intent introduces potential concerns regarding inaccurate hints, resource monopolization, and interference between independent agents or tenants.

Questions for authors
  1. Given that a one-bit application-guided control reproduces Q-Bridge's prefill improvements, what additional functionality or measurable benefit justifies the complexity of the proposed protocol?

  2. Can the authors explain why the substantial reductions in final-request TTFT do not translate into improved end-to-end workflow latency, and identify the dominant sources of overhead?

  3. How does Q-Bridge behave when multiple agents simultaneously request cache reservations that exceed the available physical KV-cache capacity?

  4. Can the authors evaluate the system using larger models, modern accelerators, and realistic concurrent agent workloads to establish its scalability and practical deployment value?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.