acceptodds
Withdrawn from ICLR 2027

CacheFlow: Fast and Memory Efficient LLM Inference via In-Network Prefix Cache Pool

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic systems consist of multiple copies of the same agent, defined by a common prompt prefix, that are hosted on multiple physical machines. Each of the machines is required to process the same common prefixes separately, leading to redundant computation across and increased latency. We recognize that all of the physical machines hosting the agents are connected by a computer network, making the network the trivial choice when storing the shared data needed across all machines. Leveraging this idea, this paper presents CacheFlow, a design and an implementation of a prefix cache pool that stores KV cache on a network device and delivers it to the GPU via GPUDirect RDMA bypassing the host CPU and DRAM. Our evaluations show that CacheFlow improves the overall prefix cache efficiency compared to vanilla vLLM by reducing the average Time-to-First-Token (TTFT) by up to 82%, increasing throughput by 39%, reducing the host memory storage requirement by 50%. In addition, CacheFlow shows up to 3X improvement in I/O throughput compared to host DRAM based cache systems.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV Cache, Prefix Cache, In-Network compute, Network, SmartNIC, Infrastructure

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EwcEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CacheFlow keeps prefix KV caches in the on-board DRAM of a BlueField-3 DPU instead of GPU or host memory, implemented as a vLLM KV Connector v1 plug-in. The host scheduler hashes the prefix and looks it up on the DPU over the DOCA Comm Channel, and KV blocks move by DOCA DMA, which the paper describes as GPUDirect RDMA, with buffer pooling, batching and layer-wise asynchronous loading. The main testbed is a desktop-class host (Ryzen 9 5900, 32 GB DDR4, one RTX 4090) plus a BlueField-3, running three 3–4B models on vLLM's synthetic prefix-repetition benchmark (10 × 20K-token prefixes, 100 requests, QPS 1–10). The baselines are GPU-only prefix caching in vLLM and LMCache with a local-CPU backend. The abstract's −82% TTFT and +39% throughput are vLLM-relative numbers for Qwen3-4B at QPS 1. A second setup cables two DGX Spark nodes to one BlueField-3 and compares only with vLLM (−29% to −43% TTFT, +49% to +55% throughput). The abstract also claims a 50% reduction in host memory and up to 3× I/O throughput over host-DRAM caches.

Critical strengths
  1. It is a real, working prototype on real hardware, packaged as a drop-in vLLM connector with code, and it runs in both single-host and two-node configurations. Showing that DPU memory can act as a KV tier inside a production serving engine is useful on its own.

  2. Against LMCache, the only offloading baseline, CacheFlow has lower mean TTFT in all 30 (model, QPS) cells of Table 1 (roughly 4–73% lower) and about 12–16% higher mean throughput, and the trend mostly holds at P90/P99. If this survives the controls below, it is a credible engineering improvement.

Critical weaknesses
  1. The central claim, that the DPU is the right place for the prefix pool, is never isolated. In the single-node setup the DPU→GPU and pinned-host→GPU paths cross the same PCIe Gen4 link to the GPU, and CacheFlow uses 2,048-token blocks, batching, four copy streams and layer-wise loading while LMCache runs with 256-token chunks on a 32 GB host. The obvious control, the same connector backed by pinned host DRAM, is missing, so the gains could come from implementation choices.

  2. The data path is described inconsistently. The abstract says KV bypasses the host CPU and DRAM via GPUDirect RDMA, but §3.1 moves tensors "between the host DMA buffers and the DPU memory" and Table 7 mentions GPU staging buffers. The "on the query path" benefit argued in Fig. 2 is also not what is implemented, since lookups are issued by the host scheduler.

  3. Two headline claims have no supporting measurement. Nothing in the paper measures the "3× I/O throughput", and the "50% host memory" figure is actually a back-of-the-envelope GPU-memory estimate (12 GB of 24 GB) that conflicts with the 23 GB shown in Fig. 1(a); Table 3's KV sizes are also about 10× smaller than Eq. 7 gives. The −82%/+39% figures are against the weakest baseline; against LMCache in the same cell they are −43%/+8%.

  4. The baselines and the rigor are weak overall. The multi-node experiment compares only against vLLM with no shared cache, although the DPU sits in a separate server and is effectively a remote memory node, so LMCache's remote backend or an RDMA KV store such as Mooncake Store would be the fair comparison. Hit rates are never reported, even though the DPU pool (71,680 tokens) is smaller than the 200,000-token working set. Every number comes from a single run of 100 requests, CacheFlow's curves are non-monotonic in load, and some LMCache/Ministral entries look rounded.

Questions for authors
  1. On a cache hit, does KV go directly from DPU DRAM to GPU HBM, or through host buffers? How was GPUDirect RDMA enabled on an RTX 4090 (my understanding is that NVIDIA documents it for data-center and professional GPUs), and can you show that host memory is not on the path?

  2. Which experiment produced the 3× I/O number? Could you add DPU→GPU vs. pinned-host→GPU bandwidth microbenchmarks, and run the same connector backed by host DRAM?

  3. What are the prefix hit rates and eviction counts per system and QPS, and how does CacheFlow beat LMCache when its pool holds only about 36% of the unique prefix tokens?

  4. Is the 50% memory claim about host or GPU memory, and was it measured? Please reconcile 12 GB vs. 23 GB and Table 3 vs. Eq. 7, and state how many runs each cell is based on.

Reviewer_WhecOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents CacheFlow, a prefix-cache offloading system that stores reusable LLM key-value (KV) states in SmartNIC memory. It targets serving scenarios in which multiple inference instances process requests with shared prefixes, leading to redundant prefill computation and replicated cache storage. Implemented as a vLLM KV connector on NVIDIA BlueField-3, CacheFlow combines a DPU-resident cache pool with direct GPU data transfers, buffer pooling and batching, and layer-wise asynchronous loading to overlap cache retrieval with computation. The paper also develops an analytical model of the trade-off between retrieving cached states and recomputing them.

The evaluation compares CacheFlow with vLLM’s automatic prefix caching and LMCache’s local CPU offloading across three small language models using a synthetic prefix-repetition workload. A separate two-server experiment evaluates shared access to a centralized SmartNIC cache. The results generally show lower time-to-first-token and higher throughput than the tested baselines. The main contribution is the design and implementation of a SmartNIC-resident prefix-cache tier integrated with an existing LLM serving engine.

Critical strengths
  1. CacheFlow uses SmartNIC memory as an additional cache tier and integrates it through vLLM’s KV connector interface. This addresses a relevant serving problem: repeated prefixes consume GPU memory and incur redundant computation across inference instances. The integration with an established serving engine makes the design potentially useful without requiring changes to model weights or architecture.

  2. Across the three evaluated models, CacheFlow generally improves both TTFT and throughput over the tested vLLM and local LMCache configurations. From Table 1, its mean throughput across request rates is approximately 12–16% higher than LMCache’s, depending on the model. The additional two-server experiment provides initial evidence that the shared cache can benefit multiple inference instances.

Critical weaknesses
  1. The paper does not establish its contribution relative to the closest prior work. ShadowServe already uses SmartNICs for distributed prefix-cache fetching, with direct GPU DMA, asynchronous execution, pipelining, and preallocated buffers, but is neither cited nor compared. CacheFlow’s use of SmartNIC memory as a shared cache pool may distinguish it, yet this distinction is not evaluated against an existing distributed cache such as Mooncake Store. The multi-node experiment compares only with vLLM, while the LMCache comparison uses local CPU offloading. Consequently, the results do not establish whether the proposed placement offers an advantage over contemporary shared-cache systems.

  2. The evaluation is too narrow to establish the benefits of the proposed design or explain where its gains come from. The main experiments use only ten fixed 20K-token prefixes, no request-specific suffix, and 100 output tokens per request. This setting does not capture the evolving contexts, partial prefix matches, and variable reuse patterns of the agentic workloads motivating the paper. Moreover, there are no component ablations separating the effects of SmartNIC placement, transfer batching, buffer pooling, and asynchronous overlap. The comparison with LMCache also uses different transfer granularities, making performance attribution difficult. Without controlled ablations and experiments varying cache capacity, reuse, and bandwidth, it remains unclear when SmartNIC caching is preferable to a well-tuned host-memory or distributed cache.

Questions for authors
  1. How does CacheFlow differ from ShadowServe beyond retaining reusable KV states in SmartNIC memory? Which mechanisms are new, and what benefit does this placement provide over a shared RDMA cache such as Mooncake Store?

  2. Can you isolate the benefits of SmartNIC placement, batching, buffer pooling, and layer-wise overlap? In particular, does the advantage over LMCache persist when usable cache capacity and transfer granularity are matched?

  3. Does the advantage persist with nonzero request-specific suffixes, partial prefix reuse, and working sets exceeding DPU capacity? Evidence from a small controlled sweep would help determine whether the gains extend beyond the fixed-prefix workload.

Reviewer_rq5COpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CacheFlow, an inference system that stores a shared prefix KV cache in SmartNIC/DPU memory. Its core idea is to enable multiple inference instances to share the cache, thereby reducing redundant prefilling and cache duplicates, whilst minimising access overhead through direct data transfer, batch operations and layer-by-layer asynchronous loading. Implemented as a vLLM plugin, the system demonstrates improvements in first-token latency and throughput on duplicate-prefix workloads compared to native vLLM and LMCache, which utilises host memory.

Critical strengths
  1. The paper focuses on the computation of repeated prefixes and cache redundancy in multi-instance reasoning, addressing efficiency issues in shared-context scenarios.
  2. The system is implemented via a vLLM plugin and combines batch transmission with layer-by-layer asynchronous loading, offering considerable practical value for deployment.
Critical weaknesses
  1. The model approximates the total time taken for cached reads and the computation of the missed portions as the maximum of the two, which is equivalent to assuming full overlap; however, suffix attention relies on the prefix KV of the corresponding layer, and in reality there are layer-by-layer dependencies, synchronisation and pipeline initiation overheads. The model also ignores the cost of long-context attention and queuing effects.
  2. The multi-node experiments only covered two servers sharing a single DPU, and were limited to comparisons with the native vLLM; there were no direct comparisons with other shared-cache schemes,
  3. Tests were conducted using only a single RTX 4090 (24 GB) and three small 3–4B models; the scale and representativeness of the experiments were insufficient.
  4. The description of the overall contribution is not sufficiently clear. The paper fails to adequately distinguish between the innovations inherent to the SmartNIC cache architecture itself and the integration of existing technologies such as direct transmission, batch operations and asynchronous pipelining; it also lacks ablation experiments to clarify the contribution of each design element to performance gains.
  5. The paper is generally focused on engineering implementation, with insufficient depth in theoretical analysis. The existing model relies on simplifying assumptions, such as full overlap between transmission and computation, and fails to adequately capture layer-by-layer data dependencies, synchronisation overhead and multi-node resource contention; consequently, its explanatory power regarding the system’s applicability boundaries and performance bottlenecks is limited.
Questions for authors

The paper describes throughput as the ‘token generation rate’; however, given the workload and numerical values, it is necessary to clarify whether this includes input tokens, to avoid it being interpreted as the output generation rate. Regarding memory, the abstract states that host memory is saved, whilst the main text primarily discusses graphics memory and provides inconsistent estimates of prefix cache capacity. Moving the cache to the DPU does not necessarily equate to a reduction in total memory consumption; the actual memory usage of the GPU, host and DPU should be reported separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.