- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the cost and efficiency of KV cache offloading for LLM inference engines serving concurrent agentic requests and identified that it is important to look at the cost of restoring cached prefix versus recomputing and the chance a cached token survives until its next reuse. The authors then introduce a stack-distance-based model for estimating the reuse working set and a capacity-conditioned admission policy that bypasses large cache refills under memory pressure. They have also conducted experiments on SWE-bench Verified and demonstrate that sufficient host memory substantially reduces prefill recomputation and adaptive admission improves performance under constrained host-memory budgets.
Critical strengths
- The topic of this paper is relevant as concurrent agentic serving system is getting increasingly important while KV re-use is highly relevant to the deployment cost and runtime efficiency.
- The authors have explained clearly the seemingly inconsistent offloading performance. The experiments across RTX 3090, H20, and H800 demonstrate that faster host restoration does not necessarily translate into better end-to-end performance.
Critical weaknesses
- I find that the paper is not easy to read as the language is not formal, complete and smoothly transitioned in many places, perhaps due to AI drafting. The author needs to put in efforts to make the paper more logically smooth and academically formal.
- The novelty of this paper is limited. Despite sufficient analysis of the existing KV cache offloading, the resulting algorithm is a simple heuristic rather than a fundamentally new cache-management mechanism.
- The experimental evaluation primarily compares against standard LMCache offloading and a fixed write-admission variant. However, several relevant alternatives are not evaluated including Marconi and Continuum.
- The authors have only conducted experiments on the SWE-Bench Verified. However, other agentic workloads such as browser-use and deep research might exhibit substantially different characteristics. The user needs to demonstrate the generality of the proposed adaptive admission method.
Questions for authors
- How can we assess whether the proposed algorithm is good enough? How does it compare against future-reuse-aware oracle that knows subsequent KV references? What fraction of the theoretically achievable improvement does the proposed policy capture?