- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
CacheFlow restores offloaded LLM states by combining recomputation and loading across tokens, layers, and model shards. A staircase partition describes which prefix chunks each layer block recomputes, and a dynamic-programming planner balances compute and transfer across requests. Cached shard-boundary activations allow different GPUs to restore their local layers concurrently. The implementation builds on vLLM and LMCache and is evaluated on dense and hybrid models, reporting substantial TTFT improvements and 1.21–1.43× mean TTFT speedups on the 122B model under matched parallel configurations.
Critical strengths
- The staircase partition is a concrete technical contribution. It captures token-wise and layer-wise restoration in one dependency-constrained formulation, including the recurrent-state loading required when a hybrid block is only partly recomputed.
- The system is evaluated across several models and operating conditions, including a 122B hybrid MoE on eight H100 GPUs with matched parallel configurations. The low measured planning overhead and latency-prediction accuracy provide useful evidence that the planner can be used at runtime.
Critical weaknesses
- The overhead of saving and restoring boundary activations is not clearly accounted for. These activations enable concurrent restoration across model shards, but the paper does not report their storage footprint, the cost of capturing and writing them, or how much is loaded during restoration. This matters particularly for hybrid models with relatively few full-attention layers, where the additional activations may be substantial relative to the KV cache. The measured latency improvements are promising; a breakdown of these costs would make the practical tradeoff much clearer.
- Minor: Figure 2 is squeezed into a narrow text-wrapped area, making the labels small and the surrounding explanation harder to follow. Please give the figure more space, preferably as a separate full-width figure.
Questions for authors
- Please report the storage footprint and capture/write overhead of boundary activations for the evaluated models. Are their transfer costs included in all reported restoration and end-to-end latencies, and how much prefix history can be retained under a fixed total storage budget?
- As an additional ablation, how much does joint batch planning improve over independent per-request planning with the same execution engine, particularly when requests have different prefix lengths?
3 comments
Thanks for the rebuttal. My technical concerns are addressed and I have increased the soundness score.
Thank You for the Follow-up
Dear Reviewer fwW2,
Thank you for the follow-up and for raising the soundness score. We're glad our response addressed your concerns, and we will incorporate these results in the final version.
Best regards,
The Authors
Response to Reviewer fwW2
We sincerely thank the reviewer for the feedback. We have uploaded a revised PDF with all changes marked in blue.
Answer: Storage footprint. At each pipeline-stage boundary, CacheFlow stores the hidden state and residual entering the next stage (2 × hidden size × 2 bytes per token). Unlike the KV and recurrent-state cache, which grows with every layer a GPU holds, this cost is fixed per boundary. Boundary activations are stored alongside the KV cache in each stage's host-memory pool. Production KV-caching systems already provision such capacity deliberately to trade storage for computation (Mooncake, FAST'25), so a 10–13% increase per boundary is a small cost for the restoration speedup it enables. For hybrid models, the base cache is the full-attention KV plus the recurrent-state checkpoints saved at each chunk boundary. Under a fixed total storage budget, the estimated prefix history retained is:
Capture cost. Boundary activations are written when a prefix is first computed. On Qwen3.5-9B with OSWorld 2.0 at PP=4, this takes 9–18 ms per request per stage (1.3–2.6% of mean TTFT), and it is included in the reported latencies.
Loaded bytes and latency accounting. During restoration, downstream stages retrieve boundary activations at 16 KiB per token for Llama-3.1-8B and Qwen3.5-9B, and 8 KiB for Qwen3.5-35B-A3B. Their PCIe transfer time is included in measured restoration TTFT. We retain these activations in local host memory, enabling retrieval over high-bandwidth PCIe rather than the slower, bandwidth-capped KV tier. Their storage counts toward the host-memory budget. We have added these results to Appendix B (Table 1).
Answer: We thank the reviewer for this suggestion. We have revised the paper accordingly.
Answer: We compare joint DP and independent per-request DP on the same engine and identical batches, using Qwen3.5-9B on H100 GPUs. We test two workloads, one with mixed prefix lengths and one with uniform prefix lengths.
Values are the speedup of joint DP over per-request DP in TTFT.
Joint DP is faster than per-request DP in every setting, by 1.08×–1.30×. The gain holds for both mixed and uniform prefix lengths, and is largest (1.30×) for mixed prefixes at 40 Gb/s.
Mixed prefixes. Per-request DP splits each request's cached prefix between network loading and GPU recomputation, so it keeps both busy only when that one request has enough work for each. Here, a short-prefix request is almost all compute work, so the network sits idle, and a medium- or long-prefix request is almost all loading work, so the compute sits idle. Joint DP plans the batch as a whole and loads the medium and long prefixes' KV while the GPU computes the short-prefix requests' new tokens. The gain peaks at 40 Gb/s, where the loading and the computing take about the same time and so overlap almost fully.
Uniform prefixes. Every request is almost all loading work, and all 64 share one link. Per-request DP plans each request as if it were alone, so all 64 make the same choice and queue on the link while the GPUs are mostly idle. Joint DP sees the whole batch and moves part of the prefix work from the link to the GPUs. The gain shrinks at 80 Gb/s, where the link is fast enough that queuing on it costs little.