- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes KVFlow, a cross-instance KV cache management system for multi-instance LLM serving. It addresses the trade-off between KV cache reuse and load balancing that cannot be fully resolved by request scheduling alone. KVFlow introduces KV cache access pressure to estimate the future request load associated with cached blocks and proactively replicates hot KV blocks across instances to balance this pressure. The system is implemented on vLLM and LMCache, and experiments on two LLMs and two agentic workloads show that KVFlow reduces average TTFT by 12.5%-54.3% compared with vLLM, Dynamo, and LMetric.
Critical strengths
- The paper addresses an important and practical problem in multi-instance LLM serving. It identifies the fundamental tension between KV cache reuse and load balancing.
- The proposed system is simple and practical to implement. KVFlow uses a global KV cache index and lightweight pressure-based replication and eviction policies. Its integration with vLLM and LMCache requires no major changes to the inference pipeline.
- The experimental results demonstrate promising performance improvements. KVFlow consistently reduces TTFT across the evaluated models and workloads.
Critical weaknesses
- The access pressure metric does not adequately capture request-level cache affinity. It simply sums block-level access heat, ignoring that multiple blocks may be accessed by the same request. For example, two instances can have equal access pressure of 200, while one serves 100 requests accessing two consecutive blocks and the other serves 200 requests accessing different blocks. Thus, equal access pressure does not necessarily imply equal request load or computation cost, as suggested in Section 4.2. The paper should justify why block-level access pressure is an appropriate proxy for request-level load and clarify the assumptions underlying Equations (2) and (3).
- Lack of key baselines and ablations that validate the core algorithmic contributions. The experiments compare KVFlow against request scheduling methods, but none of the baselines performs proactive KV cache replication. A simple hot-block replication strategy based on access heat alone might achieve similar gains. The paper should include this baseline to demonstrate the benefit of access-pressure-aware replication, along with ablations of the replication and eviction policies.
- The evaluation does not sufficiently demonstrate robustness across diverse workloads. Both evaluated workloads have high prefix reuse rates (94% and 72%), which naturally favor proactive KV cache replication. Experiments with lower reuse rates, rapidly changing cache hotspots, and heterogeneous prefix lengths are missing.
- The overhead and scalability of KVFlow are insufficiently evaluated. Each cache adjustment takes 2.98 seconds on average, but the paper does not quantify replication traffic, memory overhead, or interference with online inference. Furthermore, experiments are limited to eight instances, leaving the scalability and overhead of centralized KV cache management unclear.
Questions for authors
- How is the access heat updated when a request reuses multiple consecutive KV blocks? How well does the estimated access pressure correlate with actual per-instance request load?
- How much TTFT improvement remains if replication candidates are selected based only on access heat, rather than the proposed replication benefit?
- How frequently do cache hotspots change in the evaluated workloads? Can KVFlow adapt effectively when hotspot lifetimes are shorter than the average cache adjustment time of 2.98 seconds?
- What are the total replication traffic and additional CPU memory overhead under the evaluated workloads? Does background replication interfere with online inference?
- How do the global KV cache index size and manager overhead scale with the number of instances and cached KV blocks?
7 comments
Response to Q2 - Q4
Question 2: How much TTFT improvement remains if replication candidates are selected based only on access heat, rather than the proposed replication benefit?
Response: We have added a Heat-Based Replication baseline that ranks blocks by access heat while preserving prefix dependencies. It replicates prefixes every 10 s from the instance with the highest total block heat to the one with the lowest. This interval approximately matches the mean interval between KVFlow adjustment starts during its active replication period. (Section 7.1, Page 7, Lines 365–367).
We evaluate this baseline using Qwen3-30B-A3B on SWE-bench with eight NVIDIA A100 GPUs connected by a 10 Gbps network. Both methods use the same Dynamo router, vLLM/LMCache backend, and CPU KV-cache allocation. Heat-Based Replication improves mean TTFT over Dynamo and LMetric, but KVFlow achieves a further 14.1% reduction at 6 req/s (Section 7.2, Page 8, Lines 401–415; Figure 6).
Heat-only ranking ignores existing replica counts and therefore the diminishing benefit of additional replicas. Its fixed schedule can also initiate replication before hotspots form or continue after the imbalance has been relieved. KVFlow instead uses a pressure-based trigger to identify when adjustment is needed and replica-aware selection to prioritize blocks with greater marginal pressure reduction.
We have also added component ablations to evaluate prefix grouping, heat adjustment, and tail-first eviction under the same model, workload, and GPU configuration (Section 7.3, Page 9, Lines 450–467; Figure 11):
- Without grouping: Mean TTFT increases from 2.13 s to 2.64 s. Partial prefix replication can leave the original instance more attractive to the router, maintaining load concentration.
- Without heat adjustment: Mean TTFT increases to 2.41 s, as outdated heat can bias subsequent replication decisions.
- Without tail-first eviction: Mean TTFT increases slightly to 2.16 s. Plain LRU can temporarily leave unusable suffix blocks, although subsequent evictions limit their performance impact.
These results show that effective KV-cache replication requires carefully designed replication triggers, replica-aware block selection, and heat updates.
Question 3: How frequently do cache hotspots change in the evaluated workloads? Can KVFlow adapt effectively when hotspot lifetimes are shorter than the average cache adjustment time of 2.98 seconds?
Response:
-
For Qwen3-30B-A3B on SWE-bench at 4 req/s using eight NVIDIA RTX PRO 6000 GPUs connected by a 10 Gbps network, the mean interval between adjustment starts is 8.04 s (Appendix A.8, Page 19, Lines 986–992). This provides an operational indication of how frequently changes in cache hotspots require adjustment.
-
Short-lived hotspots on individual blocks are generally less likely to accumulate sufficient access pressure to trigger adjustment. KVFlow triggers adjustment only when the pressure gap between the highest- and lowest-pressure instances exceeds . This gap reflects the aggregate pressure of cached blocks on each instance, rather than an isolated block’s activity. We select through coarse tuning on synthetic workloads to identify a threshold that reflects stable hotspots. This aggregate pressure-based trigger helps filter short-term fluctuations and retain replication opportunities that provide sustained reuse benefits.
Question 4: What are the total replication traffic and additional CPU memory overhead under the evaluated workloads? Does background replication interfere with online inference?
Response:
- For Qwen3-30B-A3B on SWE-bench at 4 req/s using eight NVIDIA RTX PRO 6000 GPUs, the total replication traffic per run is (Appendix A.8, Page 19, Lines 986–992):
| Bandwidth | Adjustments per run | Total replication traffic |
|---|---|---|
| 10 Gbps | 13 | 31,044 MB |
| 1 Gbps | 12 | 29,580 MB |
| 500 Mbps | 13 | 30,771 MB |
Replicated blocks occupy the same 80 GB per-instance CPU KV-cache allocation used by all methods. The global index adds 41.9 MB of metadata when all eight CPU caches are full, measured using Qwen3-30B-A3B on SWE-bench at 6 req/s (Section 7.4, Pages 9–10, Lines 483–510).
- Replication runs asynchronously on the CPU without using GPU compute resources or blocking online GPU inference. During copying, the source blocks remain available for requests to reuse. Inference does not wait for replication to complete.
Response to Q5
Question 5: How do the global KV cache index size and manager overhead scale with the number of instances and cached KV blocks?
Response: We have added large-scale simulations using the KV cache manager running on an Intel Xeon Platinum 8470Q CPU, driven by KV-cache events from simulated instances serving Qwen3-30B-A3B on SWE-bench. These events exercise the manager’s normal index maintenance, pressure computation, and planning code. We vary instance count, cached block count, and request rate. Index updates are processed using 22 worker threads (Appendix A.7, Pages 17–18, Lines 917–952; Figure 23).
The measurements show the following scaling behavior:
- Index memory grows approximately linearly with cached block count, including replica-location metadata.
- Index maintenance time grows approximately linearly with request rate when the average number of affected blocks per request is fixed.
- Access-pressure computation time grows approximately linearly with instance count at fixed per-instance cache capacity, and with cached block count at fixed instance count.
- Adjustment-plan generation time remains approximately constant as instance count increases because each plan concerns one source–target pair. Larger per-instance caches increase the number of candidate blocks and therefore planning cost.
For 1,024 instances with full 80 GB CPU caches per instance, the measured overheads are:
| Operation | Measured overhead |
|---|---|
| Global index memory | 5,340.9 MB |
| Access-pressure computation | 12.856 ms |
| Adjustment-plan generation | 0.91 ms |
| Index maintenance at 1,024 req/s | 398.3 ms to process events generated by 1 s of requests |
Detailed results appear in Appendix A.7 (Pages 17–19; Figure 23).
For the evaluated model and workload, high serving latency limits the practical request rate per instance to below 1 req/s. A manager deployed on a single CPU node can keep pace with index maintenance, access-pressure computation, and adjustment-plan generation for a 1,024-GPU cluster. These results demonstrate KVFlow’s low management overhead and scalability.
Response to Q1
We thank the reviewer for the suggestions on the pressure metric, replication policies, workload coverage, and system overhead. We have clarified the analysis, added a heat-based replication baseline and component ablations, and expanded the reporting of replication traffic, memory overhead, and manager scalability.
Before addressing the individual questions, we clarify the workload scope raised in Weakness 3. Multi-turn conversations and agentic applications repeatedly reuse prior context, creating substantial opportunities for prefix reuse. This setting also motivates cache-aware schedulers such as Dynamo and LMetric. KVFlow targets these workloads, where preserving cache reuse while distributing requests across instances is an important serving concern. When little reusable KV state is available, the benefits of cache-aware routing and cache redistribution diminish, making conventional load-based routing appropriate. Our study therefore focuses on KV-cache management in workloads with substantial prefix reuse.
Question 1: How is the access heat updated when a request reuses multiple consecutive KV blocks? How well does the estimated access pressure correlate with actual per-instance request load?
Response: When a request reuses m consecutive KV blocks, we increment the heat of each reused block by one:
Hk←Hk+1.
The request therefore contributes m block accesses, rather than being counted as m requests.
Our previous description of access pressure as cache affinity was imprecise. As your example illustrates, equal aggregate pressure can arise from different numbers of requests and does not imply equal request-level affinity or computation cost. We have updated the interpretation of Pi: a larger Pi indicates a larger share of system-wide block-access demand assigned to instance i under uniform replica sharing, providing a signal for redistributing cached blocks to mitigate cache-induced load concentration (Section 4.1, Page 4, Lines 181–183). We use pressure as a cache-placement signal, rather than a direct estimate of request count or computation load.
We have also revised Equations (2)–(3) to establish the benefit of pressure balance within this model. For fixed block-access demands Hk, Equation (2) shows that replication preserves total modeled demand:
∑i=1NPi=∑kHk.
Equation (3) then bounds the maximum instance pressure:
maxiPi≥N1∑i=1NPi=N1∑kHk.
Balancing Pi attains this lower bound and thus minimizes peak modeled block-access demand under uniform replica sharing (Section 4.2, Page 4, Lines 187–200). Replicating popular prefixes gives the router more instances on which requests can reuse those prefixes, helping reduce load concentration.