acceptodds
Under review as a conference paper at ICLR 2027

KVFlow: Cross-Instance KV Cache Flow for Efficient and Balanced LLM Serving

Abstract

Multi-instance large language model (LLM) serving must balance KV cache reuse and load balancing. Prior work addresses this trade-off through request scheduling. However, scheduling cannot eliminate the fundamental conflict because hot KV cache couples reuse benefits with instance load, preventing requests from simultaneously achieving low computation time through reuse and low queueing delay through load balancing. Our key insight is to manage KV cache directly by replicating hot KV blocks across instances, preventing hot cache from causing load imbalance. We first define KV cache access pressure to characterize how cache distribution affects future request load across instances and show that balancing access pressure decouples KV cache reuse benefits from instance load. Based on this analysis, we propose KVFlow, the first KV cache management system that balances access pressure across instances. Across multiple LLMs and real LLM serving workloads, KVFlow reduces average time-to-first-token (TTFT) by 12.5%–54.3% compared with state-of-the-art production systems, including vLLM, Dynamo, and LMetric.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving, KV Cache, Request Scheduling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_B2qAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KVFlow, a cross-instance KV cache management system for multi-instance LLM serving. It addresses the trade-off between KV cache reuse and load balancing that cannot be fully resolved by request scheduling alone. KVFlow introduces KV cache access pressure to estimate the future request load associated with cached blocks and proactively replicates hot KV blocks across instances to balance this pressure. The system is implemented on vLLM and LMCache, and experiments on two LLMs and two agentic workloads show that KVFlow reduces average TTFT by 12.5%-54.3% compared with vLLM, Dynamo, and LMetric.

Critical strengths
  • The paper addresses an important and practical problem in multi-instance LLM serving. It identifies the fundamental tension between KV cache reuse and load balancing.
  • The proposed system is simple and practical to implement. KVFlow uses a global KV cache index and lightweight pressure-based replication and eviction policies. Its integration with vLLM and LMCache requires no major changes to the inference pipeline.
  • The experimental results demonstrate promising performance improvements. KVFlow consistently reduces TTFT across the evaluated models and workloads.
Critical weaknesses
  • The access pressure metric does not adequately capture request-level cache affinity. It simply sums block-level access heat, ignoring that multiple blocks may be accessed by the same request. For example, two instances can have equal access pressure of 200, while one serves 100 requests accessing two consecutive blocks and the other serves 200 requests accessing different blocks. Thus, equal access pressure does not necessarily imply equal request load or computation cost, as suggested in Section 4.2. The paper should justify why block-level access pressure is an appropriate proxy for request-level load and clarify the assumptions underlying Equations (2) and (3).
  • Lack of key baselines and ablations that validate the core algorithmic contributions. The experiments compare KVFlow against request scheduling methods, but none of the baselines performs proactive KV cache replication. A simple hot-block replication strategy based on access heat alone might achieve similar gains. The paper should include this baseline to demonstrate the benefit of access-pressure-aware replication, along with ablations of the replication and eviction policies.
  • The evaluation does not sufficiently demonstrate robustness across diverse workloads. Both evaluated workloads have high prefix reuse rates (94% and 72%), which naturally favor proactive KV cache replication. Experiments with lower reuse rates, rapidly changing cache hotspots, and heterogeneous prefix lengths are missing.
  • The overhead and scalability of KVFlow are insufficiently evaluated. Each cache adjustment takes 2.98 seconds on average, but the paper does not quantify replication traffic, memory overhead, or interference with online inference. Furthermore, experiments are limited to eight instances, leaving the scalability and overhead of centralized KV cache management unclear.
Questions for authors
  1. How is the access heat updated when a request reuses multiple consecutive KV blocks? How well does the estimated access pressure correlate with actual per-instance request load?
  2. How much TTFT improvement remains if replication candidates are selected based only on access heat, rather than the proposed replication benefit?
  3. How frequently do cache hotspots change in the evaluated workloads? Can KVFlow adapt effectively when hotspot lifetimes are shorter than the average cache adjustment time of 2.98 seconds?
  4. What are the total replication traffic and additional CPU memory overhead under the evaluated workloads? Does background replication interfere with online inference?
  5. How do the global KV cache index size and manager overhead scale with the number of instances and cached KV blocks?
7 comments
AuthorsOpenReview ↗

Response to Q1

We thank the reviewer for the suggestions on the pressure metric, replication policies, workload coverage, and system overhead. We have clarified the analysis, added a heat-based replication baseline and component ablations, and expanded the reporting of replication traffic, memory overhead, and manager scalability.

Before addressing the individual questions, we clarify the workload scope raised in Weakness 3. Multi-turn conversations and agentic applications repeatedly reuse prior context, creating substantial opportunities for prefix reuse. This setting also motivates cache-aware schedulers such as Dynamo and LMetric. KVFlow targets these workloads, where preserving cache reuse while distributing requests across instances is an important serving concern. When little reusable KV state is available, the benefits of cache-aware routing and cache redistribution diminish, making conventional load-based routing appropriate. Our study therefore focuses on KV-cache management in workloads with substantial prefix reuse.

Question 1: How is the access heat updated when a request reuses multiple consecutive KV blocks? How well does the estimated access pressure correlate with actual per-instance request load?

Response: When a request reuses mm consecutive KV blocks, we increment the heat of each reused block by one:

Hk←Hk+1.H_k \leftarrow H_k+1.

The request therefore contributes mm block accesses, rather than being counted as mm requests.

Our previous description of access pressure as cache affinity was imprecise. As your example illustrates, equal aggregate pressure can arise from different numbers of requests and does not imply equal request-level affinity or computation cost. We have updated the interpretation of PiP_i: a larger PiP_i indicates a larger share of system-wide block-access demand assigned to instance ii under uniform replica sharing, providing a signal for redistributing cached blocks to mitigate cache-induced load concentration (Section 4.1, Page 4, Lines 181–183). We use pressure as a cache-placement signal, rather than a direct estimate of request count or computation load.

We have also revised Equations (2)–(3) to establish the benefit of pressure balance within this model. For fixed block-access demands Hk\\{H_k\\}, Equation (2) shows that replication preserves total modeled demand:

∑i=1NPi=∑kHk.\sum_{i=1}^{N}P_i=\sum_k H_k.

Equation (3) then bounds the maximum instance pressure:

max⁡iPi≥1N∑i=1NPi=1N∑kHk.\max_i P_i \geq \frac{1}{N}\sum_{i=1}^{N}P_i = \frac{1}{N}\sum_k H_k.

Balancing PiP_i attains this lower bound and thus minimizes peak modeled block-access demand under uniform replica sharing (Section 4.2, Page 4, Lines 187–200). Replicating popular prefixes gives the router more instances on which requests can reuse those prefixes, helping reduce load concentration.

AuthorsOpenReview ↗

Response to Q2 - Q4

Question 2: How much TTFT improvement remains if replication candidates are selected based only on access heat, rather than the proposed replication benefit?

Response: We have added a Heat-Based Replication baseline that ranks blocks by access heat while preserving prefix dependencies. It replicates prefixes every 10 s from the instance with the highest total block heat to the one with the lowest. This interval approximately matches the mean interval between KVFlow adjustment starts during its active replication period. (Section 7.1, Page 7, Lines 365–367).

We evaluate this baseline using Qwen3-30B-A3B on SWE-bench with eight NVIDIA A100 GPUs connected by a 10 Gbps network. Both methods use the same Dynamo router, vLLM/LMCache backend, and CPU KV-cache allocation. Heat-Based Replication improves mean TTFT over Dynamo and LMetric, but KVFlow achieves a further 14.1% reduction at 6 req/s (Section 7.2, Page 8, Lines 401–415; Figure 6).

Heat-only ranking ignores existing replica counts and therefore the diminishing benefit of additional replicas. Its fixed schedule can also initiate replication before hotspots form or continue after the imbalance has been relieved. KVFlow instead uses a pressure-based trigger to identify when adjustment is needed and replica-aware selection to prioritize blocks with greater marginal pressure reduction.

We have also added component ablations to evaluate prefix grouping, heat adjustment, and tail-first eviction under the same model, workload, and GPU configuration (Section 7.3, Page 9, Lines 450–467; Figure 11):

  • Without grouping: Mean TTFT increases from 2.13 s to 2.64 s. Partial prefix replication can leave the original instance more attractive to the router, maintaining load concentration.
  • Without heat adjustment: Mean TTFT increases to 2.41 s, as outdated heat can bias subsequent replication decisions.
  • Without tail-first eviction: Mean TTFT increases slightly to 2.16 s. Plain LRU can temporarily leave unusable suffix blocks, although subsequent evictions limit their performance impact.

These results show that effective KV-cache replication requires carefully designed replication triggers, replica-aware block selection, and heat updates.

Question 3: How frequently do cache hotspots change in the evaluated workloads? Can KVFlow adapt effectively when hotspot lifetimes are shorter than the average cache adjustment time of 2.98 seconds?

Response:

  1. For Qwen3-30B-A3B on SWE-bench at 4 req/s using eight NVIDIA RTX PRO 6000 GPUs connected by a 10 Gbps network, the mean interval between adjustment starts is 8.04 s (Appendix A.8, Page 19, Lines 986–992). This provides an operational indication of how frequently changes in cache hotspots require adjustment.

  2. Short-lived hotspots on individual blocks are generally less likely to accumulate sufficient access pressure to trigger adjustment. KVFlow triggers adjustment only when the pressure gap between the highest- and lowest-pressure instances exceeds βCgpu\beta C_{\mathrm{gpu}}. This gap reflects the aggregate pressure of cached blocks on each instance, rather than an isolated block’s activity. We select β\beta through coarse tuning on synthetic workloads to identify a threshold that reflects stable hotspots. This aggregate pressure-based trigger helps filter short-term fluctuations and retain replication opportunities that provide sustained reuse benefits.

Question 4: What are the total replication traffic and additional CPU memory overhead under the evaluated workloads? Does background replication interfere with online inference?

Response:

  1. For Qwen3-30B-A3B on SWE-bench at 4 req/s using eight NVIDIA RTX PRO 6000 GPUs, the total replication traffic per run is (Appendix A.8, Page 19, Lines 986–992):
BandwidthAdjustments per runTotal replication traffic
10 Gbps1331,044 MB
1 Gbps1229,580 MB
500 Mbps1330,771 MB

Replicated blocks occupy the same 80 GB per-instance CPU KV-cache allocation used by all methods. The global index adds 41.9 MB of metadata when all eight CPU caches are full, measured using Qwen3-30B-A3B on SWE-bench at 6 req/s (Section 7.4, Pages 9–10, Lines 483–510).

  1. Replication runs asynchronously on the CPU without using GPU compute resources or blocking online GPU inference. During copying, the source blocks remain available for requests to reuse. Inference does not wait for replication to complete.
AuthorsOpenReview ↗

Response to Q5

Question 5: How do the global KV cache index size and manager overhead scale with the number of instances and cached KV blocks?

Response: We have added large-scale simulations using the KV cache manager running on an Intel Xeon Platinum 8470Q CPU, driven by KV-cache events from simulated instances serving Qwen3-30B-A3B on SWE-bench. These events exercise the manager’s normal index maintenance, pressure computation, and planning code. We vary instance count, cached block count, and request rate. Index updates are processed using 22 worker threads (Appendix A.7, Pages 17–18, Lines 917–952; Figure 23).

The measurements show the following scaling behavior:

  • Index memory grows approximately linearly with cached block count, including replica-location metadata.
  • Index maintenance time grows approximately linearly with request rate when the average number of affected blocks per request is fixed.
  • Access-pressure computation time grows approximately linearly with instance count at fixed per-instance cache capacity, and with cached block count at fixed instance count.
  • Adjustment-plan generation time remains approximately constant as instance count increases because each plan concerns one source–target pair. Larger per-instance caches increase the number of candidate blocks and therefore planning cost.

For 1,024 instances with full 80 GB CPU caches per instance, the measured overheads are:

OperationMeasured overhead
Global index memory5,340.9 MB
Access-pressure computation12.856 ms
Adjustment-plan generation0.91 ms
Index maintenance at 1,024 req/s398.3 ms to process events generated by 1 s of requests

Detailed results appear in Appendix A.7 (Pages 17–19; Figure 23).

For the evaluated model and workload, high serving latency limits the practical request rate per instance to below 1 req/s. A manager deployed on a single CPU node can keep pace with index maintenance, access-pressure computation, and adjustment-plan generation for a 1,024-GPU cluster. These results demonstrate KVFlow’s low management overhead and scalability.

Reviewer_QU3jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents KVFlow, a cross-instance KV-cache management system for multi-instance LLM serving. It proactively replicates popular cached prefixes to reduce the conflict between cache reuse and load balancing. A cache access pressure metric, derived from access counts and replica counts, guides replication and eviction through a global prefix index. The implementation primarily manages CPU-resident caches using vLLM and LMCache. Experiments with two models and two agentic workloads on eight machines report lower TTFT than the evaluated schedulers while maintaining similar cache hit rates and TPOT.

Critical strengths
  1. The paper addresses an important serving problem through cache placement. Cache-aware routing can concentrate requests on instances holding popular prefixes. Proactively placing additional replicas creates opportunities to reuse those prefixes on less-loaded instances. The prefix-aware index and background replication mechanism turn this observation into a practical system rather than relying only on a different routing objective.

  2. The evaluation provides useful evidence of practical benefits. The comparisons use a common vLLM/LMCache backend, and the paper shows improvements when KVFlow is combined with either Dynamo or LMetric. Similar cache hit rates together with reduced TTFT support the intended benefit of reducing queueing while preserving reuse. The replication, planning-overhead, and network-sensitivity results also provide evidence about the cost of the approach under the evaluated conditions.

Critical weaknesses
  1. The contribution of pressure-aware replication is not sufficiently isolated. The scheduler integrations demonstrate the benefit of adding KVFlow, but do not establish how much of that benefit comes from its pressure formulation rather than proactive replication itself. A simpler popularity-based replication policy, preserving prefix dependencies and using matched cache and transfer resources, would be an informative comparison. This matters for assessing the distinctive contribution of the proposed placement rule.

  2. The decoupling claim requires additional assumptions. Section 4.2 infers equal expected computation times from equal aggregate access pressure. However, the pressure metric divides popularity by replica count, whereas a candidate request's cache benefit depends on which prefixes each instance holds. Equal pressure therefore need not imply equal expected cache affinity under a common request distribution. The probability space underlying Equations (2)–(3) needs clarification. This weakens the analytical justification, although it does not by itself invalidate the empirical system results.

Questions for authors
  1. Additional evidence: How does pressure-aware replication compare with a simple heat-based policy under matched CPU cache capacity and transfer budgets? Please report the resulting transfer volume as well as TTFT, so that the benefit of the placement rule can be separated from the amount of replication.

  2. Clarification of the analytical claim: Consider three homogeneous instances: A and C cache prefix X, while B caches an equally sized prefix Y. Let H_X=2n, H_Y=n, r_X=2, and r_Y=1. All three pressures equal n, but under a common candidate-request distribution with X:Y frequencies of 2:1, their hit probabilities are 2/3, 1/3, and 2/3. With identical hit and miss costs, their expected computation times differ. What assumptions exclude this case? If the expectation conditions on requests already assigned to each instance, how does that equality justify routing a new request?

  3. Resource-accounting clarification: Section 7.1 specifies 80 GB of CPU memory per machine and a common cache backend. Does each method use the same actual CPU KV-cache capacity? Where is the global manager deployed, and how are its memory and compute costs accounted for?

6 comments
AuthorsOpenReview ↗

Response to Q1

We thank the reviewer for the comments on isolating the contribution of pressure-aware replication, clarifying the analytical assumptions, and accounting for system resources. We have added a heat-based baseline and component ablations, revised the pressure analysis, and clarified cache capacity and manager overhead. We address each question below.

Question 1: Additional evidence: How does pressure-aware replication compare with a simple heat-based policy under matched CPU cache capacity and transfer budgets? Please report the resulting transfer volume as well as TTFT, so that the benefit of the placement rule can be separated from the amount of replication.

Response: We have added a Heat-Based Replication baseline that periodically replicates prefixes from the instance with the highest total block heat to the one with the lowest. It ranks blocks by heat and preserves prefix dependencies, as in KVFlow.

We evaluate this baseline using Qwen3-30B-A3B on SWE-bench with eight NVIDIA A100 GPUs connected by a 10 Gbps network. Both methods use the same Dynamo router, vLLM/LMCache backend, and CPU KV-cache allocation. At 6 req/s, KVFlow reduces mean TTFT by 14.1% compared with Heat-Based Replication (Section 7.2, Page 8, Lines 410–415; Figure 6).

Heat-Based Replication adjusts every 10 s and copies an average of 20 blocks per adjustment. During the reported 311 s run, KVFlow initiates 13 adjustments between 55 s and 180 s, averaging approximately 10 s between starts and 101 blocks per adjustment, or approximately 1,313 blocks in total (Section 7.2, Page 8, Lines 401–408).

The fixed schedule can initiate replication before hotspots form or continue after the imbalance has been relieved. Heat-only ranking also ignores existing replica counts. KVFlow instead triggers replication when the pressure gap exceeds the threshold and suppresses further adjustments once the gap falls below it. Its block selection accounts for the diminishing benefit of additional replicas.

We have also added component ablations to evaluate the effectiveness of prefix grouping, heat adjustment, and tail-first eviction under the same model, workload, and GPU configuration (Section 7.3, Page 9, Lines 450–467; Figure 11):

  • Without grouping: Blocks are selected individually while preserving prefix dependencies. Mean TTFT increases from 2.13 s to 2.64 s. Partial prefix replication can leave the original instance with a longer reusable prefix, continuing to attract requests.
  • Without heat adjustment: Block heat remains unchanged after eviction. Mean TTFT increases to 2.41 s, as outdated heat can bias subsequent replication decisions.
  • Without tail-first eviction: Plain LRU replaces tail-first eviction. Mean TTFT increases slightly to 2.16 s. Plain LRU can temporarily leave unusable suffix blocks, although subsequent evictions limit their performance impact.
AuthorsOpenReview ↗

Response to Q2

Question 2: Clarification of the analytical claim: Consider three homogeneous instances: A and C cache prefix X, while B caches an equally sized prefix Y. Let HX=2nH_X=2n, HY=nH_Y=n, rX=2r_X=2, and rY=1r_Y=1. All three pressures equal nn, but under a common candidate-request distribution with X:Y frequencies of 2:1, their hit probabilities are 2/3, 1/3, and 2/3. With identical hit and miss costs, their expected computation times differ. What assumptions exclude this case? If the expectation conditions on requests already assigned to each instance, how does that equality justify routing a new request?

Response: We thank the reviewer for the example, which highlights a case overlooked in our previous formulation. In such cases, equal aggregate pressure does not imply equal expected computation times under a common incoming-request distribution. We have revised Section 4.2 to state the more precise claim that balancing access pressure minimizes peak modeled block-access demand under uniform replica sharing.

For fixed block-access demands Hk\\{H_k\\}, the model assigns each replica an equal share Hk/rkH_k/r_k. The total pressure therefore satisfies Equation (2):

∑i=1NPi=∑k∑i∈RkHkrk=∑kHk.\sum_{i=1}^{N}P_i = \sum_k\sum_{i\in R_k}\frac{H_k}{r_k} = \sum_k H_k.

Replication redistributes this modeled demand without changing its total. Consequently, Equation (3) gives

max⁡iPi≥1N∑i=1NPi=1N∑kHk.\max_i P_i \geq \frac{1}{N}\sum_{i=1}^{N}P_i = \frac{1}{N}\sum_k H_k.

Pressure balance attains this lower bound. This establishes a placement objective for reducing peak modeled demand, rather than equality of cache benefits or computation times for incoming requests (Section 4.2, Page 4, Lines 187–200).

Replicating popular prefixes expands the set of instances that can serve requests with cache reuse, giving the router more opportunities to distribute requests without recomputing those prefixes. We now explicitly state that pressure balance does not guarantee identical cache benefits across instances or eliminate all reuse–load conflicts. Actual routing assigns each request as a whole and depends on its reusable prefix and current instance load (Page 4, Lines 201–209).

AuthorsOpenReview ↗

Response to Q3

Question 3: Resource-accounting clarification: Section 7.1 specifies 80 GB of CPU memory per machine and a common cache backend. Does each method use the same actual CPU KV-cache capacity? Where is the global manager deployed, and how are its memory and compute costs accounted for?

Response: Yes. All methods use the same actual CPU KV-cache capacity of 80 GB per instance. We have clarified the common cache allocation in Section 7.1 (Page 7, Lines 352–353).

The global manager runs on an Intel Xeon Platinum 8470Q CPU on one serving machine. We measure its computation time and index memory footprint separately. Using Qwen3-30B-A3B on SWE-bench at 6 req/s with eight NVIDIA RTX PRO 6000 GPUs, index maintenance, access-pressure computation, and adjustment-plan generation take 8.5 ms, 0.047 ms, and 0.86 ms, respectively. The index occupies 41.9 MB when all eight 80 GB CPU KV caches are full. This is additional metadata overhead. Manager computation runs asynchronously on the CPU without using GPU resources. It does not block online request processing and remains off the inference critical path. (Section 7.4, Pages 9–10, Lines 483–510).

We have also added large-scale measurements by running the manager on the same CPU and sending it KV-cache events from simulated instances serving Qwen3-30B-A3B on SWE-bench. These events exercise the manager’s normal index maintenance, pressure computation, and planning code. We vary instance count, cached block count, and request rate (Appendix A.7, Pages 17–18, Lines 917–945; Figure 23).

  • Index maintenance grows approximately linearly with request rate for a fixed average number of affected blocks per request. Using 22 worker threads, processing the events generated by 1 s of requests at 1,024 req/s takes 398.3 ms, allowing the manager to keep pace with the simulated event stream.
  • Access-pressure computation grows approximately linearly with instance count at fixed per-instance cache capacity, and with cached block count at fixed instance count. It takes 12.856 ms for 1,024 instances with full 80 GB caches, and remains below 27 ms with 160 GB per instance.
  • Adjustment-plan generation remains approximately constant as instance count increases because each plan concerns one source–target pair. It takes 0.91 ms at 1,024 instances with 80 GB caches. Larger per-instance caches increase planning cost, but it remains below 4 ms across the tested 40–160 GB configurations.
  • Index memory grows approximately linearly with cached block count. At 1,024 instances with full 80 GB caches, the index occupies 5,340.9 MB on the manager node.

For the evaluated model and workload, high serving latency limits the practical request rate per instance to below 1 req/s. A manager deployed on a single CPU node can keep pace with index maintenance, access-pressure computation, and adjustment-plan generation for a 1,024-GPU cluster. These results demonstrate KVFlow’s low management overhead and scalability. (Appendix A.7, Pages 17–19; Figure 23).

Reviewer_Xm74OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at data-parallel LLM serving where prefill and decode share an instance, and argues that cache-aware routers such as Dynamo and LMetric can only trade KV reuse against load balance, because a hot prefix cached on a few instances pulls load onto them. Instead of changing the router, KVFlow changes where the cache lives. It defines a per-instance access pressure P_i = Σ H_k / r_k (hits over replica count) and, when the max–min pressure gap exceeds βC_gpu, copies prefix-contiguous groups of blocks from the highest- to the lowest-pressure instance, ranked by the marginal pressure drop Δp_k. A tail-first LRU eviction that also lowers block heat completes the design. The prototype extends LMCache/vLLM and moves mostly CPU-resident blocks in the background. On 8 machines (one RTX PRO 6000 each, 10 Gbps) with Qwen3-30B-A3B and DeepSeek-V2-Lite on SWE-bench and GAIA agentic traces, KVFlow on top of Dynamo reduces mean TTFT at 6 req/s by 12.5–20.7% vs. Dynamo, 14.8–21.1% vs. LMetric and 25.8–54.3% vs. least-loaded vLLM, with TPOT unchanged. The paper also reports hit rate, prefill imbalance, replica counts, β sensitivity and a bandwidth sweep down to 500 Mbps.

Critical strengths
  1. The idea is simple and sits next to the router rather than replacing it. KVFlow runs in the background as an LMCache extension, improves both Dynamo and LMetric (−10.7% and −8.3% mean TTFT on top of LMetric at 6 req/s, Fig. 10), and holds up from 10 Gbps down to 500 Mbps (Fig. 15). Treating cache placement as a second knob next to routing is a useful framing for practitioners.

  2. The gains are consistent across all four model/trace settings (12.5–20.7% over the strongest baseline at 6 req/s), and the diagnostics line up with the claimed mechanism: the hit rate is essentially unchanged (0.624 vs. 0.631), prefill imbalance mostly drops, 96.6% of blocks stay at ≤3 replicas, and planning costs 0.86 ms.

Critical weaknesses
  1. The closest alternatives are missing. KVFlow is essentially proactive replication of hot prefixes, so the natural comparisons are on-demand cross-instance fetch (route by load, then pull the prefix from a peer or a shared LMCache backend) and a simple top-k hot-prefix copy heuristic.

  2. There is no component ablation (index without replication, group vs. per-block selection, tail-first vs. plain LRU, the heat update). This matters because KVFlow manages CPU-resident LMCache blocks: if the baseline routers do not see CPU-tier hits the way KVFlow's index does, part of the gain could come from visibility rather than replication. The DeepSeek-V2-Lite/SWE-bench result, where the cache-aware baselines are no better than least-loaded vLLM despite ~94% reuse (Fig. 8), makes me want to see this checked.

  3. β is tuned on the evaluation traces (6 for SWE-bench, 2 for GAIA, Fig. 11), and the sensitivity is larger than the text suggests. On GAIA, β = 1.5 or 3 gives 1.20/1.18 s, roughly where Dynamo sits at the same load (≈1.19 s in Fig. 7, assuming Fig. 11 is at 5 req/s, which the values suggest). The threshold is also scaled by GPU block capacity even though the managed cache lives on the CPU.

Questions for authors
  1. In the Dynamo/LMetric runs, does the router count CPU-resident LMCache hits the same way as when KVFlow is enabled? A "KVFlow index + router, replication off" variant would answer most of my concern.

  2. How does KVFlow compare with on-demand fetch from a peer instance or a shared LMCache backend, and with a simple hot-prefix copy heuristic, under the same routers?

  3. How many runs per point, how long is each run, and what is the variance? Could you report gains at all request rates and goodput under a fixed TTFT SLO?

  4. Which β was used in each experiment (including DeepSeek-V2-Lite), at what request rate was Fig. 11 measured, and was β chosen on the evaluation traces? Also, what is the total replication volume per run, and how long does one adjustment take at 1 Gbps and 500 Mbps?

6 comments
AuthorsOpenReview ↗

Response to Q1, Q2

We thank the reviewer for the comments and suggestions on baseline comparisons, component ablations, parameter sensitivity, and experimental reporting. We have added baselines and ablations, revised the parameter-selection discussion, and expanded the reporting of variability, SLO attainment, goodput, and replication overhead. We address each question below.

Questions 1: In the Dynamo/LMetric runs, does the router count CPU-resident LMCache hits the same way as when KVFlow is enabled? A "KVFlow index + router, replication off" variant would answer most of my concern.

Response: Yes. Dynamo and LMetric have the same visibility into CPU-resident LMCache blocks with and without KVFlow. KVFlow does not modify either scheduler or how it counts cache hits. Its global index supports cache management and does not provide additional cache visibility to the routers.

Cache-aware routing can incur higher tail latency than load-based routing because prioritizing cache reuse can concentrate requests on cached instances. For DeepSeek-V2-Lite on SWE-bench, the P99 TTFT gap between the cache-aware baselines and least-loaded vLLM is more pronounced than in the other evaluated configurations (Figure 19, Page 17). This suggests stronger cache-induced load imbalance, whose queueing overhead can offset the computation savings, explaining why Dynamo and LMetric provide little improvement in mean TTFT in this configuration.

We have also added component ablations using Qwen3-30B-A3B on SWE-bench with eight NVIDIA A100 GPUs (Section 7.3, Page 9, Lines 450–467; Figure 11):

  • Without grouping: Blocks are selected individually while preserving prefix dependencies. Mean TTFT increases from 2.13 s to 2.64 s because partial prefix replication can leave the original instance more attractive to the router.
  • Without heat adjustment: Block heat remains unchanged after eviction. Mean TTFT increases to 2.41 s because outdated heat can bias subsequent replication decisions.
  • Without tail-first eviction: Plain LRU replaces tail-first eviction. Mean TTFT increases slightly to 2.16 s. Plain LRU can temporarily leave unusable suffix blocks, although subsequent evictions limit their performance impact.

These experiments isolate the contributions of grouping, heat adjustment, and tail-first eviction.

Questions 2: How does KVFlow compare with on-demand fetch from a peer instance or a shared LMCache backend, and with a simple hot-prefix copy heuristic, under the same routers?

Response: We have added On-Demand Fetch and Heat-Based Replication baselines:

  • On-Demand Fetch routes requests to the least-loaded instance, which fetches locally missing KV blocks from peer instances when available.
  • Heat-Based Replication replicates prefixes every 10 s from the instance with the highest total block heat to the one with the lowest. It ranks blocks by heat and preserves prefix dependencies.

We evaluate these baselines using Qwen3-30B-A3B on SWE-bench with eight NVIDIA A100 GPUs connected by a 10 Gbps network. All methods use the same vLLM/LMCache backend and CPU KV-cache allocation. Heat-Based Replication uses the same Dynamo router as KVFlow, while On-Demand Fetch uses least-loaded routing followed by peer fetch.

At 6 req/s, KVFlow reduces mean TTFT by 33.1% over On-Demand Fetch and 14.1% over Heat-Based Replication (Section 7.2, Pages 7–8, Lines 376–415; Figure 6). On-Demand Fetch places KV transfers on the request critical path. Heat-Based Replication generally improves mean TTFT over Dynamo and LMetric but remains slower than KVFlow. Heat-Based Replication runs at fixed intervals, even before hotspots form or after replication has relieved the imbalance. It also ranks blocks by heat without accounting for existing replicas. In contrast, KVFlow triggers replication only when the pressure gap exceeds the threshold and stops issuing adjustments once the gap falls below it. Its block selection also accounts for the diminishing benefit of additional replicas. These results highlight the importance of both replication timing and replica-aware block selection.

These results, together with the component ablations in Section 7.3 (Page 9, Figure 11), show that proactive replication benefits depend on replication timing, prefix selection, and heat updates. KVFlow addresses these through a pressure-based trigger, replica-aware selection, and eviction-driven heat adjustment.

AuthorsOpenReview ↗

Response to Q3

Questions 3: How many runs per point, how long is each run, and what is the variance? Could you report gains at all request rates and goodput under a fixed TTFT SLO?

Response: Each data point is averaged over five runs. Each run replays 1,821 requests for SWE-bench or 1,563 requests for GAIA. At 3–6 req/s, the replay duration is approximately 304–607 s for SWE-bench and 261–521 s for GAIA, excluding the time needed to finish outstanding requests.

For Qwen3-30B-A3B on SWE-bench using eight NVIDIA A100 GPUs, the relative standard deviations are 2.2%–4.7% for KVFlow, 1.8%–2.3% for vLLM, 2.1%–4.2% for Dynamo, 2.3%–4.4% for LMetric, 1.9%–2.6% for On-Demand Fetch, and 3.2%–4.8% for Heat-Based Replication. The revised manuscript reports the aggregate range of 1.8%–4.8% in Section 7.2 (Page 8, Lines 413–415).

We report TTFT comparisons at all evaluated request rates—3, 4, 5, and 6 req/s—and have added SLO attainment and goodput results:

  • Qwen3-30B-A3B/SWE-bench: Section 7.2, Pages 7–8, Figures 6–8.
  • Qwen3-30B-A3B/GAIA: Appendix A.6, Page 17, Figures 17–18.
  • DeepSeek-V2-Lite/SWE-bench: Appendix A.6, Page 17, Figures 19–20.
  • DeepSeek-V2-Lite/GAIA: Appendix A.6, Page 18, Figures 21–22.

We use fixed TTFT SLOs of 5 s for SWE-bench and 3 s for GAIA, choosing a more relaxed threshold for SWE-bench to account for its longer average input length (25,996.33 versus 10,152.43 tokens). We define goodput as the token throughput contributed by requests meeting the corresponding SLO.

For Qwen3-30B-A3B on SWE-bench using eight NVIDIA RTX PRO 6000 GPUs at 6 req/s, KVFlow reduces mean TTFT by 45.0%, 12.5%, and 17.5% over vLLM, Dynamo, and LMetric, respectively. Its SLO attainment reaches 91.8%, exceeding these baselines by 34.4, 8.5, and 11.6 percentage points, while goodput increases by 88.0%, 9.8%, and 12.8%, respectively (Section 7.2, Page 8, Lines 416–424; Figures 7–8).

AuthorsOpenReview ↗

Response to Q4

Questions 4: Which β was used in each experiment (including DeepSeek-V2-Lite), at what request rate was Fig. 11 measured, and was β chosen on the evaluation traces? Also, what is the total replication volume per run, and how long does one adjustment take at 1 Gbps and 500 Mbps?

Response: Parameter settings and selection. We use β=6\beta=6 for SWE-bench and β=2\beta=2 for GAIA across both models, including DeepSeek-V2-Lite, and both GPU configurations. In the revised evaluation, we select these values using separate synthetic workloads matching the corresponding real workloads in prefix reuse rate and average input and output lengths. We then keep them fixed when evaluating the real traces.

The revised parameter ablation experiment, now Figure 10, uses Qwen3-30B-A3B with eight NVIDIA A100 GPUs at 6 req/s. The best tested settings are β=6\beta=6 for SWE-like and β=2\beta=2 for GAIA-like (Section 7.3, Page 9, Lines 442–449).

GPU-capacity normalization. Although replication manages CPU-resident blocks, CgpuC_{\mathrm{gpu}} represents the GPU KV-block capacity available for online request execution. We therefore use it to scale the tolerated pressure imbalance according to execution capacity, rather than CPU storage capacity. Consequently, the absolute threshold βCgpu\beta C_{\mathrm{gpu}} changes with GPU KV capacity even when β\beta remains fixed (Section 5.3, Page 5, Lines 257–262).

Sensitivity clarification. Our statement that performance is insensitive to β\beta refers to similar performance near the best tested settings. To avoid suggesting robustness across all β\beta values, we have narrowed the statement to “suitable β\beta values span a range and can be found with a few coarse trials” (Section 5.3, Page 5, Lines 264–266). In Figure 10, increasing β\beta from 6 to 7 on SWE-like increases mean TTFT by 2.5%; increasing it from 2 to 2.5 on GAIA-like increases mean TTFT by 0.7%. Appendix A.3 explains the conditions supporting coarse tuning (Page 16, Lines 816–826).

The GAIA results at β=1.5\beta=1.5 and β=3\beta=3 in the original Figure 11 are close to Dynamo because these thresholds reduce the effectiveness of proactive replication. A smaller β\beta can trigger unnecessary replication before stable hotspots emerge, while a larger β\beta can delay hotspot relief. Both can reduce the benefit over routing alone, highlighting the importance of replication timing.

Replication overhead. We have added the adjustment counts and replication overhead measurements in Appendix A.8 (Page 19, Lines 986–996). These experiments use Qwen3-30B-A3B on SWE-bench at 4 req/s with eight NVIDIA RTX PRO 6000 GPUs.

BandwidthAdjustments per runMean volume per adjustmentTotal volume per runMean adjustment time
10 Gbps132388 MB31,044 MB2.98 s
1 Gbps122465 MB29,580 MB28.42 s
500 Mbps132367 MB30,771 MB52.25 s

Lower bandwidth delays replica availability, but only a few early requests miss the benefits during this delay. Once adjustment completes, many subsequent requests benefit from repeated cache reuse, preserving most of KVFlow’s overall gains. This explains the similar TTFT across the tested bandwidths in Figure 15.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.