acceptodds
Under review as a conference paper at ICLR 2027

CacheFlow: Efficient LLM Serving via Automated 3D-Parallel KV Cache Restoration

Abstract

KV cache restoration is becoming a major bottleneck in long-context LLM serving, including multi-turn conversations, agentic workloads, and RAG. Existing advances largely treat restoration as a coarse per-request choice between recomputing cached states and loading them from external storage (e.g., CPU memory or remote machines), overlooking structured parallelism within the model and resource contention across batched requests. Increasingly, modern hybrid architectures complicate restoration as their interleaved attention and recurrent layers introduce heterogeneous compute costs and state footprints. We present CacheFlow, which rethinks restoration as a multi-dimensional parallel execution problem. To our knowledge, CacheFlow is the first framework to jointly exploit restoration parallelism across tokens, layers, and GPUs. CacheFlow unifies token- and layer-wise restoration as a staircase partition of the token–layer space, while lightweight boundary activations enable concurrent restoration across model shards. A batch-aware dynamic-programming planner jointly determines what to recompute and what to load across requests and layer blocks, balancing GPU computation against shared I/O resources. Across dense and hybrid models spanning single-digit to hundred-billion parameters, diverse workloads, and hardware conditions, CacheFlow reduces restoration latency by 2.24–3.00 on average over existing advances, directly improving end-to-end serving latency by 1.64.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache, LLM serving, inference optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fwW2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

CacheFlow restores offloaded LLM states by combining recomputation and loading across tokens, layers, and model shards. A staircase partition describes which prefix chunks each layer block recomputes, and a dynamic-programming planner balances compute and transfer across requests. Cached shard-boundary activations allow different GPUs to restore their local layers concurrently. The implementation builds on vLLM and LMCache and is evaluated on dense and hybrid models, reporting substantial TTFT improvements and 1.21–1.43× mean TTFT speedups on the 122B model under matched parallel configurations.

Critical strengths
  • The staircase partition is a concrete technical contribution. It captures token-wise and layer-wise restoration in one dependency-constrained formulation, including the recurrent-state loading required when a hybrid block is only partly recomputed.
  • The system is evaluated across several models and operating conditions, including a 122B hybrid MoE on eight H100 GPUs with matched parallel configurations. The low measured planning overhead and latency-prediction accuracy provide useful evidence that the planner can be used at runtime.
Critical weaknesses
  • The overhead of saving and restoring boundary activations is not clearly accounted for. These activations enable concurrent restoration across model shards, but the paper does not report their storage footprint, the cost of capturing and writing them, or how much is loaded during restoration. This matters particularly for hybrid models with relatively few full-attention layers, where the additional activations may be substantial relative to the KV cache. The measured latency improvements are promising; a breakdown of these costs would make the practical tradeoff much clearer.
  • Minor: Figure 2 is squeezed into a narrow text-wrapped area, making the labels small and the surrounding explanation harder to follow. Please give the figure more space, preferably as a separate full-width figure.
Questions for authors
  • Please report the storage footprint and capture/write overhead of boundary activations for the evaluated models. Are their transfer costs included in all reported restoration and end-to-end latencies, and how much prefix history can be retained under a fixed total storage budget?
  • As an additional ablation, how much does joint batch planning improve over independent per-request planning with the same execution engine, particularly when requests have different prefix lengths?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fwW2

We sincerely thank the reviewer for the feedback. We have uploaded a revised PDF with all changes marked in blue.

W1, Q1: What are the storage footprint, capture/write cost, and loaded bytes of boundary activations? Are their transfers included in the reported latencies, and how much prefix history fits under a fixed storage budget?

Answer: Storage footprint. At each pipeline-stage boundary, CacheFlow stores the hidden state and residual entering the next stage (2 ×\times hidden size ×\times 2 bytes per token). Unlike the KV and recurrent-state cache, which grows with every layer a GPU holds, this cost is fixed per boundary. Boundary activations are stored alongside the KV cache in each stage's host-memory pool. Production KV-caching systems already provision such capacity deliberately to trade storage for computation (Mooncake, FAST'25), so a 10–13% increase per boundary is a small cost for the restoration speedup it enables. For hybrid models, the base cache is the full-attention KV plus the recurrent-state checkpoints saved at each chunk boundary. Under a fixed total storage budget, the estimated prefix history retained is:

ModelBase cache/token: KV + recurrent checkpointsBoundary Activation /token/boundaryPrefix history retained (PP=2)Prefix history retained (PP=4)
Llama-3.1-8B128 KiB16 KiB89%73%
Qwen3.5-9B127 KiB16 KiB89%73%
Qwen3.5-35B-A3B80 KiB8 KiB91%77%

Capture cost. Boundary activations are written when a prefix is first computed. On Qwen3.5-9B with OSWorld 2.0 at PP=4, this takes 9–18 ms per request per stage (1.3–2.6% of mean TTFT), and it is included in the reported latencies.

Loaded bytes and latency accounting. During restoration, downstream stages retrieve boundary activations at 16 KiB per token for Llama-3.1-8B and Qwen3.5-9B, and 8 KiB for Qwen3.5-35B-A3B. Their PCIe transfer time is included in measured restoration TTFT. We retain these activations in local host memory, enabling retrieval over high-bandwidth PCIe rather than the slower, bandwidth-capped KV tier. Their storage counts toward the host-memory budget. We have added these results to Appendix B (Table 1).

W2: Figure 2 is too narrow; make it a full-width figure.

Answer: We thank the reviewer for this suggestion. We have revised the paper accordingly.

Q2: How much does joint batch planning improve over independent per-request planning with the same engine, especially when prefix lengths differ?

Answer: We compare joint DP and independent per-request DP on the same engine and identical batches, using Qwen3.5-9B on H100 GPUs. We test two workloads, one with mixed prefix lengths and one with uniform prefix lengths.

  • Mixed prefixes (1 GPU). 12 requests: four with short prefixes (528 cached + 7,920 new tokens each), four with medium prefixes (4,224 cached + 256 new), and four with long prefixes (16,896 cached + 256 new).
  • Uniform prefixes (4 pipeline stages on 4 GPUs). 64 requests arriving together, each with 1,056 cached + 32 new tokens.

Values are the speedup of joint DP over per-request DP in TTFT.

Workload10 Gb/s40 Gb/s80 Gb/s
Mixed prefixes, 12 requests, 1 GPU1.12×1.30×1.13×
Uniform prefixes, 64 requests, 4 stages1.22×1.21×1.08×

Joint DP is faster than per-request DP in every setting, by 1.08×–1.30×. The gain holds for both mixed and uniform prefix lengths, and is largest (1.30×) for mixed prefixes at 40 Gb/s.

Mixed prefixes. Per-request DP splits each request's cached prefix between network loading and GPU recomputation, so it keeps both busy only when that one request has enough work for each. Here, a short-prefix request is almost all compute work, so the network sits idle, and a medium- or long-prefix request is almost all loading work, so the compute sits idle. Joint DP plans the batch as a whole and loads the medium and long prefixes' KV while the GPU computes the short-prefix requests' new tokens. The gain peaks at 40 Gb/s, where the loading and the computing take about the same time and so overlap almost fully.

Uniform prefixes. Every request is almost all loading work, and all 64 share one link. Per-request DP plans each request as if it were alone, so all 64 make the same choice and queue on the link while the GPUs are mostly idle. Joint DP sees the whole batch and moves part of the prefix work from the link to the GPUs. The gain shrinks at 80 Gb/s, where the link is fast enough that queuing on it costs little.

Thanks for the rebuttal. My technical concerns are addressed and I have increased the soundness score.

AuthorsOpenReview ↗

Thank You for the Follow-up

Dear Reviewer fwW2,

Thank you for the follow-up and for raising the soundness score. We're glad our response addressed your concerns, and we will incorporate these results in the final version.

Best regards,

The Authors

Reviewer_sKR4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses KV cache restoration under the case when a request’s cached prefix KV is not resident on the GPU (offload, discarded, etc). When this happens, we need to either recompute or load from external storage, and the paper proposes a hybrid method of combining the two in a concurrent way along three dimensions. Token-wise, the authors use a similar idea to Cake, where early chunks are recomputed while later chunks are loaded (due to heavier computations and dependency). Layer-wise, early layers are recomputed while deeper layers are loaded for similar reasons. And finally device-wise, cached boundary activations at pipeline-shard boundaries let each shard restore its layers independently without waiting for upstream shards. The final result is a dynamic programming-based strategy of figuring out how to combine each of these by minimizing the final SLO. Different architectures are tested (dense, GDN, etc) and the implementation is in vLLM + LMCache. Across many benchmarks, the authors show that CacheFlow enables 2.24-3 times lower mean TTFT than Cake, LMCache, and vLLM. Also a great boost of 1.64x lower p95 TTLT in online serving.

Critical strengths
  1. It addresses a real and significant problem in current agentic serving scenarios where long and repeatedly reused contexts appear in the workloads.
  2. The staircase formulation is clean and would make the token and layer level restoration as special cases.
  3. The great observation that identical blocks make the assignment order irrelevant leads to a simple DP method (only takes < 1% of the TTFT as the overhead)
  4. Different architectures are handled explicitly. E.g. being able to partially recompute a block of GDN does not mean that loading its final recurrent state is not useful anymore.
  5. Broad evaluation. Arch wise, they have dense and hybrid attentions + MoE. Workload wise, they have two GPU generations, across different bandwidths, parallelism configs and online serving.
Critical weaknesses
  1. The paper argues that recomputation under PP is sequential across shards and introduces boundary activations to remove this dependency. But chunked recomputation under PP is naturally pipelined: when stage 2 works on chunk 1, stage 1 can already work on chunk 2 (vLLM). In this case, if I understand correctly, boundary activations only remove the pipeline fill and bubble. Theorem 5 is on single device plan, and Figure 11 on 1 or 2 GPUs, none of which ablates this.
  2. I feel like the novelty is narrower than claimed but it is a great paper. Token wise is from Cake. Splitting layers is the core of HCache, and boundary activations are a special case of HCache’s hidden state caching. The main novelty is the staircase DP, the batch extension, and the special treatment of GDN, and the value of those should be more well demonstrated.
Questions for authors
  1. At the same PP degree, what is the TTFT of token-chunked hybrid restoration with ordinary stage-to-stage pipelining but without boundary activations? How much of the multi-GPU gain in Figure 9 remains?
  2. What fraction of DP plans in the main experiments are non-trivial staircases, and what is CacheFlow’s gain over a per-request oracle best of token-wise and layer-wise restoration, on one GPU at equal bandwidth?
4 comments
AuthorsOpenReview ↗

Response to Reviewer sKR4

We sincerely thank the reviewer for the feedback. We have uploaded a revised PDF with all changes marked in blue.

W1, Q1: At the same PP degree, how much of the multi-GPU gain survives if stages just pipeline chunks the ordinary way?

Answer: Ordinary chunk pipelining overlaps recomputation across stages, but downstream stages still wait for upstream activations. Boundary activations remove this dependency and allow each shard to select its restoration plan independently. We ran a controlled experiment to quantify the benefit over ordinary pipelining.

We use Qwen3.5-9B on H100 GPUs with PP4/PP8, 10 Gbps bandwidth, 16 requests per engine batch, and fixed cached prefixes of either 528 or 4,224 tokens per request. Each request has a 256-token uncached suffix and generates one output token. We report the median TTFT.

PP degreeCached tokens/requestChunk pipeline TTFT (ms)CacheFlow with Boundary Activation TTFT (ms)Speedup
PP4528304.4203.51.50×
PP44,224627.7495.31.27×
PP8528262.5183.01.43×
PP84,224566.4391.91.45×

At the same PP degree, boundary activations provide 1.27–1.50× TTFT speedups over ordinary chunk pipelining across all four configurations. Boundary activations help in two ways:

(1) Removing the fill-bubble cost. With jj recomputed chunks, PP stages, and per-stage chunk time tct_c, pipelined recomputation finishes at (j+P−1)⋅tc(j+P-1)\cdot t_c, and stage kk (counting from 0) idles for its first k⋅tck\cdot t_c. With boundary activations every stage starts at once and finishes after about j⋅tcj\cdot t_c plus a one-time load of its boundary activations. Without them, the bubble adds a fixed delay to recomputation, and the planner can only compensate by recomputing less and loading more, which is slow when bandwidth is limited.

(2) Enabling shard-local plans in CacheFlow. A downstream stage can recompute a chunk even when the preceding stage loads that chunk’s KV rather than producing its output activation. Without boundary activations, such a plan needs extra upstream recomputation or a different global plan. In our multi-GPU runs, 88–91% of requests received plans of this kind.

W2, Q2: Novelty relative to Cake and HCache. Is the staircase DP worth it? How often are plans non-trivial staircases, and how much does the DP gain over the better of token-wise or layer-wise per request?

Answer: Token-wise splitting comes from Cake, but our use of activations differs from HCache in its cost tradeoff. HCache stores per-layer hidden states in place of KV and rebuilds the KV by projection. This saves transfer under multi-head attention, where a hidden state is half the size of its KV. Under grouped-query attention, which all three models we evaluate use, the hidden state is 2× the KV, so the saving disappears.

CacheFlow keeps KV as the stored state and adds one activation per shard boundary instead of one per layer. A boundary activation is the input from which a downstream shard recomputes its part of the prefix, so each GPU starts restoring immediately instead of waiting for the shard before it.

We ran a controlled ablation that keeps the execution engine and restoration mechanisms fixed and changes only the planning strategy. Requests in a batch share the GPU and link, so one request’s TTFT depends on the others’ strategies. We therefore take the oracle per batch. We evaluate Qwen3.5-9B on H100 GPU with PP=1, batch size 8, and 10 Gbps bandwidth. Values below are mean batch TTFT normalized to CacheFlow.

Planning strategyMean batch TTFT
Token-wise race1.188×
Layer-wise race1.119×
Best of token-wise/layer-wise1.114×
CacheFlow staircase DP1.000×

The staircase DP provides a 1.11× mean-TTFT speedup over best-of-two selection, beyond the benefits of the shared execution mechanisms. 49–69% of selected plans are neither token-wise nor layer-wise. These plans vary recomputation boundaries across layer blocks to better balance computation and transfer. Planning costs 0.019% of TTFT at the median, so the DP adds negligible overhead.

AuthorsOpenReview ↗

Kindly Requesting Feedback

Dear Reviewer sKR4,

Thank you again for your comments. We have posted responses addressing your concerns, along with additional analysis and experiments.

As the discussion period closes on Nov 18 (AoE), we would really appreciate it if you could let us know whether our response resolves your concerns, and take it into account in your final assessment. We are happy to answer any further questions.

Thank you so much.

Best regards,

The Authors

Thanks for the rebuttal. I'd love to keep my original positive score and good luck!

AuthorsOpenReview ↗

Thank You for the Follow-up

Dear Reviewer sKR4,

We sincerely appreciate the time and effort you spent reviewing our paper, and your positive assessment of our work. We will incorporate the new results into the final version.

Best regards,

The Authors

Reviewer_TB9pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CacheFlow, a system for accelerating KV cache restoration in LLM serving. The main idea is to jointly optimize recomputation and loading of KV states across tokens, layers, and devices, rather than treating restoration as a sequential process. The authors introduce a batch-aware dynamic programming (DP) planner which determines which states to load and which to recompute, considering computational dependencies and available resources. They then evaluate the approach across several models (some dense, some hybrid/MoE), workloads, and hardware configs, reporting 2.24–3.00x reductions in restoration latency over existing approaches.

I lean weak reject overall. I'm not really convinced about the practical motivation for this system. The proposed optimization is interesting, but the paper doesn't sufficiently establish when it would actually be useful compared to other ways of managing and retrieving KV caches (AND how often that case happens in the real-world given the serving constraints there is). I also have some concerns about the evaluation, particularly the online serving results (huge TTLT values) and whether the reported improvements translate to realistic deployments. Some important comparisons and ablations are also missing.

Critical strengths
  • I am not convinced about the use case of this system BUT in case the authors can convince me, the joint optimization across tokens/layers/devices is interesting. Accounting for different recomputation costs across layers for hybrid models with heterogeneous state sizes is also interesting.

  • The batch-aware planning formulation is a reasonable extension over existing recomputation/loading approaches. The authors also provide theoretical analysis of the optimization, which is useful.

  • There's a good level of experimental results across models, I/O bandwidths, and GPU configurations. Even though some of the experimental setups seem questionable at the minimum, such as the fact that the model choices are different in fig. 4 for the different benchmarks (why??).

Critical weaknesses
  • My main concern is the practical motivation. How often is KV cache restoration actually a major bottleneck in realistic serving systems (and this MUST be analyzed in comparison to the existing bottlenecks on compute, btw)?? I know it's a nice thing to optimize but is it actually a major bottleneck that warrants using compute resources for re-computations of KV caches given the current realistic seving constraints?

  • Related to my previous point, there's no discussion of specific SLO requirements for the system to make sense. I don't think demonstrating faster restoration alone sufficiently motivates the system (esp. compared to other sources of latency).

  • Also not convinced why spending additional GPU compute on recomputation is necessarily preferable to just loading the cached states. In heavily utilized systems, wouldn't those compute resources be better spent serving other requests? The paper doesn't adequately examine the resulting tradeoff with throughput or overall system efficiency.

  • The online serving experiments are rather confusing to me. In fig. 14, TTLT seems to reach around 1,000–2,000 seconds, despite requests arriving over a 300-second window. Why are these latencies so large? Are the systems operating under severe overload? (again, I'm not sure what these results tell us about serving performance under realistic conditions.)

  • I'm not particularly convinced by the evaluation of the DP planner itself. Figure 9 shows the contributions of different parallelism dimensions, but how much does the actual optimization improve over simpler scheduling heuristics using the same mechanisms?

  • The experimental baselines seem somewhat limited. Previous works such as KVPR, eLLM, and CacheGen consider related problems in recomputation and efficient KV cache transfer (and one could even argue a system like InfiniteLLM is fairly useful here). Can the authors provide a more detailed comparison or explanation of why these approaches aren't evaluated?

  • The contribution of the proposed DP planner is also not entirely clear from the experiments. Figure 9 evaluates different parallelism dimensions, but how much improvement does the actual optimization provide over simpler scheduling heuristics using the same parallelism? This seems particularly important given that the planner is one of the main technical contributions.

  • The evaluation seems somewhat limited in terms of realistic serving loads. The authors do include online serving experiments, although I don't think these fully address how the approach behaves when GPU resources are highly utilized (and with different arrival times).

  • The authors evaluate prediction accuracy, but how would the approach behave under fluctuating network bandwidth or changing resource contention? (this is a more general feedback point though, probably not an immediate issue with the paper)

  • Lastly, the authors do compare against Cake and show meaningful improvements, but a stronger ablation of the proposed scheduling algorithm would help clarify what actually accounts for these gains (how substantial is the algorithmic contribution over existing approaches such as Cake??).

Questions for authors
  • See my questions in the weaknesses section (and one in strengths section).

  • Could the authors provide more evidence that KV cache restoration is a sufficiently common and important bottleneck in real-world serving systems? Under which SPECIFIC conditions would CacheFlow provide the greatest benefit and would it warrant using scarce compute resources for somewhat faster (of course would be faster) restoration?

  • How does the additional recomputation affect serving throughput when GPUs are already heavily utilized? Have the authors considered evaluating goodput under different load levels?

  • Why would recomputation be preferable to distributing or prefetching cached states using additional storage/network resources? Can the authors compare these approaches under similar resource budgets AND heavily utilized compute?

  • How much of the improvement comes specifically from the proposed DP planner? Why specifically wouldn't simpler greedy or layer-aware scheduling strategies achieve similar results?

6 comments
AuthorsOpenReview ↗

Response to Reviewer TB9p (1/3)

We sincerely thank the reviewer for the feedback. We have uploaded a revised PDF with all changes marked in blue.

S1: Model choices are different in Fig. 4 for different benchmarks.

Answer: Due to the space limit, we paired each benchmark with a different model so that the main results cover three architectures whose cost of loading compared with recomputing differ: (1) Llama3.1-8B: dense model with full attention (2) Qwen3.5-9B: hybrid-attention (3) Qwen3.5-35B-A3B: hybrid MoE model. We have added results for all 3 models x 3 benchmarks in Fig. 18. CacheFlow outperforms the baselines in all 9 settings.

W1, W3, Q2: How often is KV restoration a major bottleneck relative to compute in realistic serving? In heavily utilized systems, why recompute KV states on the GPU rather than simply load them, and what is the tradeoff with throughput and overall system efficiency? Under which specific conditions does CacheFlow help most?

Answer: We thank the reviewer for these important questions. We kindly note that KV restoration is an important problem in production serving.

(1) Reused prefixes require restoration in production. In Mooncake's production traces, 40% of input tokens in the conversation workload and 59% in the tool-and-agent workload belong to reusable prefixes, and even a 3M-token local DRAM cache captures <50% of the achievable prefix hits (Mooncake, FAST'25). Any reusable, non-GPU-resident KV must be restored by loading or by recomputing. In Alibaba's deployment, Pythia's profiling of a multi-agent coding assistant shows that tool calls can take tens of seconds, while intervening traffic largely evicts cached prefixes before agents resume, suggesting substantial restoration demand (Pythia, arXiv:2604.25899).

(2) KV restoration takes a large share of execution time. Recomputation is a standard fallback in serving. vLLM recomputes KV by default when a preempted request resumes, and SGLang recomputes prefix KV that it cannot recover from cache. In InferCept’s mixed tool-use workload, rebuilding discarded KV accounted for 37–40% of model forwarding time (InferCept, ICML'24). Loading can also dominate prefill. In Strata’s SGLang CPU-offload experiment on LooGLE, KV transfers blocked 74% of prefill execution time (Strata, OSDI'26).

Spending additional GPU compute on recomputation can be preferable to loading when KV transfers limit batch progress or service needs a low latency, even in a heavily utilized system. In batches with long cached prefixes and short new suffixes, there may be too little fresh prefill to hide KV transfers. Selective recomputation uses GPU time that would otherwise wait on the transfer to reduce the bytes transferred. CacheFlow is 3.64× faster than load-only (see W6).

Asynchronous loading can preserve GPU cycles for other requests when their computation hides the transfers. However, under tight GPU memory, reserving space for incoming KV can reduce concurrency, while waiting for an active request to free space shortens the window for overlap. Also, if scheduling priorities change after a prefetch starts, it can consume bandwidth and GPU memory needed by the request now scheduled to run first. Limited overlap leaves more of the KV transfer on the request’s critical path. CacheFlow shortens that transfer by recomputing selected parts of the prefix and loading only the remaining KV.

CacheFlow’s planner compares selective recomputation with load-only, using profiled compute costs and the available transfer bandwidth, and automatically chooses load-only when recomputation would slow the batch. We measured the resulting tradeoff in the Fig. 14 setting against asynchronous load-only, which overlaps KV transfers with other requests’ computation: CacheFlow meets joint SLOs of TTFT ≤\leq 2 seconds and TPOT ≤\leq 50 ms/token for 2.0×, 1.9×, and 1.4× as many requests as asynchronous load-only at ×0.1 (moderate load), ×0.15 (high load), and ×0.2 (near saturation) Mooncake arrival scales. Full results are in our answer to W4, W8, Q3.

CacheFlow helps most when requests reuse long, offloaded prefixes and have short new suffixes, so KV transfers dominate prefill and there is little other computation to overlap with them. These cases are very popular in real deployments, such as in agentic systems. CacheFlow’s gains grow with the length of the cached prefix (Fig. 7) and persist from 10 to 80 Gbps (Fig. 12).

AuthorsOpenReview ↗

Response to Reviewer TB9p (2/3)

W2: There's no discussion of specific SLO requirements for the system to make sense. Faster restoration alone does not sufficiently motivate the system (esp. compared to other sources of latency).

Answer: CacheFlow targets the TTFT SLO, covering queueing, prefix restoration, and prefill. Restoration dominates this work: median prompt reuse is 88%, and KV transfers consume 74% of prefill time (Strata, OSDI’26). It also drives queueing by occupying GPU and I/O resources. Decode latency falls under the separate TPOT SLO (DistServe, OSDI’24), while tool execution occurs between requests, outside the inference server. Optimizing either is orthogonal and complementary to CacheFlow.

SLO attainment. We added joint TTFT and TPOT SLO attainment results in §4. We replay SWE-Bench on Llama-3.1-8B with Mooncake arrival timing scaled by ×0.1 (moderate load), ×0.15 (high load), and ×0.2 (near saturation). Following MLPerf Inference’s Llama-3.1-8B server scenario, we set TTFT ≤ 2 s, and we additionally set TPOT ≤ 50 ms/token.

Arrival scaleCacheFlowCakeLMCache async load-onlyLMCache sync load-onlyvLLM
×0.137.5%1.8%18.7%1.5%1.3%
×0.1534.6%1.5%18.5%2.1%1.4%
×0.221.7%1.5%15.4%1.5%1.4%

CacheFlow attains the SLO for 1.41–2.01× as many requests as asynchronous loading. Absolute attainment is low for all systems because SWE-Bench prefixes are long: at 10 Gbps, load-only or recompute-only restoration alone exceeds 2s for many requests.

SLO-aware planning. CacheFlow currently minimizes restoration time and does not take deadlines as input. Our DP can support them: adding each request's remaining time budget to the DP state lets the planner prioritize requests close to their deadline. Each request's budget and batch are decided by the request scheduler, so this requires co-design with the scheduler. We consider it as complementary and future work (§5).

W4, W8, Q3: “Why are Fig. 14’s TTLTs so large, and how do CacheFlow’s throughput and goodput change across load levels, especially under high utilization?”

Answer: Fig. 14 is an overload test: requests arrive at ~6 req/s, while the deployment completes only 0.8–1.2 req/s, so TTLT is mostly queueing. This further shows that CacheFlow is effective under heavy load. CacheFlow re-plans every batch as load grows and opportunistically uses available GPU capacity to recompute part of the prefix while the rest loads. It achieves 1.22 req/s, compared with 0.78–0.81 req/s for baselines, and drains the backlog faster.

Throughput and goodput across loads. We additionally evaluate Mooncake arrival scales ×0.1 (moderate load), ×0.15 (high load), and ×0.2 (near saturation) with Llama-3.1-8B on SWE-Bench. Goodput is rate of requests satisfying both TTFT ≤\leq 2 seconds and TPOT ≤\leq 50 ms/token. Values below are ordered ×0.1 / ×0.15 / ×0.2.

MethodThroughput (req/s)Goodput (req/s)
CacheFlow0.578 / 0.847 / 1.0720.224 / 0.310 / 0.259
Async load-only0.572 / 0.827 / 1.0500.112 / 0.166 / 0.184
Sync load-only0.562 / 0.728 / 0.7920.009 / 0.019 / 0.018
vLLM0.555 / 0.705 / 0.7610.008 / 0.013 / 0.017
Cake0.566 / 0.688 / 0.7480.011 / 0.013 / 0.018

CacheFlow achieves 1.41–2.01× higher goodput than asynchronous loading and also higher throughput across these three loads.

W5, W7, W10, Q5: How much of the gain over Cake comes from the DP planner itself, and would simpler greedy or layer-aware heuristics with the same parallelism do as well?

Answer:

We ran an ablation that keeps other settings fixed and changes only the scheduling strategy. We compare (i) Cake-style token-wise, (ii) layer-wise, (iii) oracle best-of-two, which, for each batch, takes the lower measured TTFT of (i) and (ii), and (iv) CacheFlow’s staircase DP. Values are mean batch TTFT on Qwen3.5-9B relative to CacheFlow:

Scheduling strategyMean batch TTFT
Token-wise race1.188×
Layer-wise race1.119×
Best of token-wise/layer-wise1.114×
CacheFlow1.000×

The staircase DP provides a 1.11× speedup over best-of-two selection. Token-wise and layer-wise strategies restrict the restoration partition to a single shape, whereas the DP selects different recomputation boundaries across layer blocks to better balance compute and transfer costs.

The table isolates the planner. The rest of the gain over Cake comes from device-wise restoration with boundary activations and batch-aware joint planning. Fig. 9 shows how these components accumulate.

AuthorsOpenReview ↗

Response to Reviewer TB9p (3/3)

W6: Why are KVPR, eLLM, CacheGen and InfiniteLLM not evaluated as baselines?

Answer: We ran KVPR, the closest work among these four, as a baseline. Since its released code is built on FlexGen, we port its split rule unchanged and implement it on vLLM and LMCache.

Llama-3.1-8B, SWE-Bench batches of 8, 10 Gbps, mean batch TTFT relative to CacheFlow:

MethodPP=1PP=4
CacheFlow1.00×1.00×
Load-only3.64×2.92×
KVPR (released rule)7.08×5.41×
KVPR (rule with GQA K/V size)3.64×2.91×

KVPR recomputes KV from transferred activations. These are smaller than KV under multi-head attention, but twice their size under grouped-query attention. Its released rule therefore doubles the bytes loaded, and with the actual GQA KV size its rule falls back to load-only.

The others solve different problems from restoring an exact prefix, and are complementary to CacheFlow; eLLM and InfiniteLLM also have no official code release:

  • CacheGen compresses KV lossily, trading accuracy for speed.
  • eLLM saves GPU memory during decoding by recomputing part of each running request’s KV at every step, which raises throughput. CacheFlow instead cuts TTFT by restoring a cached prefix once, before prefill.
  • InfiniteLLM keeps KV on GPUs longer, so restoration is needed less often, but it is still needed.

W9: How would CacheFlow behave under fluctuating bandwidth or changing resource contention?

Answer: CacheFlow re-plans every restoration batch, and a plan takes 0.4 ms at the median (3.6 ms at p99), while a restoration batch itself lasts 339 ms at the median (2.1 s at p99) across our sweep, so CacheFlow can respond to changes between batches. For resource contention, CacheFlow already adjusts online. We rescale the compute-cost model by the measured ratio of actual to predicted compute time, which absorbs GPU contention from concurrent decoding and data loading. Our online experiments already run under such contention, since batches and concurrent decoding change the GPU time available to each request, and CacheFlow improves p95 TTFT and TTLT consistently (Fig. 14).

For network bandwidth, CacheFlow can support online bandwidth adaptation by feeding a moving average of measured transfer bandwidth into the planner. Since bandwidth is an explicit planning input and CacheFlow re-plans every restoration batch, subsequent plans can adjust the balance between recomputation and loading as bandwidth changes.

Q4: Why recompute instead of distributing or prefetching cached states with additional storage or network resources? How do these approaches compare under a similar resource budget when compute is heavily utilized?

Answer: These approaches are complementary to CacheFlow. Adding bandwidth makes loading faster, and CacheFlow uses whatever bandwidth is available. From 10 to 80 Gbps it loads 1.4–3.3$\times$ more of the cached state and recomputes less, and its gains persist (Fig. 12). Prefetching needs two things restoration usually lacks. The first is knowing when a request will return, but tool-call durations vary widely (InferCept, ICML’24). The second is free GPU memory to hold the prefetched KV. GPU memory is the scarce resource whose shortage caused the eviction, so prefetching stalls once it is full, however fast the storage and network are. Under the same resources, CacheFlow achieves 2.0×, 1.9×, and 1.4× the goodput of asynchronous load-only, at moderate load, high load, and near saturation, while also sustaining higher throughput (see answer to W4, W8, Q3).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

We sincerely thank the AC and all three reviewers for their constructive feedback, which substantially improved our work.

Reviewer TB9p (Current Rating: 3, Confidence: 3): found the joint optimization across tokens, layers, and devices interesting and the batch-aware planner a reasonable extension, and raised concerns about practical motivation, behavior under heavy load, the planner's contribution, and baseline coverage. We added a planner ablation, comparisons against asynchronous loading, SLO attainment and goodput at three load levels, KVPR as a new baseline, and results for all model–benchmark pairs. The reviewer stated that the rebuttal substantially strengthened the evidence and raised the score from 2 to 3. We have also answered the reviewer's follow-up questions.

Reviewer sKR4 (Current Rating: 3, Confidence: 3): regarded the problem as real and significant, the staircase formulation as clean, and the evaluation as broad, and asked how much boundary activations add over ordinary pipelining and how the staircase DP compares with simpler strategies. We provided a controlled comparison at the same PP degree, a planner ablation against an oracle best of token-wise and layer-wise restoration, and a clarification of our novelty relative to Cake and HCache.

Reviewer fwW2 (Current Rating: 3, Confidence: 3): valued the staircase partition as a concrete technical contribution and the evaluation up to a 122B model on eight GPUs. After our accounting of boundary-activation storage, capture, and transfer costs and our ablation of joint versus per-request planning, the reviewer confirmed that the technical concerns were addressed and raised the soundness score.

We have responded to all concerns raised by the reviewers, with new experiments and clarifications provided in our responses. We are grateful to the AC and all three reviewers for their helpful feedback, and will revise our paper accordingly to incorporate these results.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.