- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CacheFlow keeps prefix KV caches in the on-board DRAM of a BlueField-3 DPU instead of GPU or host memory, implemented as a vLLM KV Connector v1 plug-in. The host scheduler hashes the prefix and looks it up on the DPU over the DOCA Comm Channel, and KV blocks move by DOCA DMA, which the paper describes as GPUDirect RDMA, with buffer pooling, batching and layer-wise asynchronous loading. The main testbed is a desktop-class host (Ryzen 9 5900, 32 GB DDR4, one RTX 4090) plus a BlueField-3, running three 3–4B models on vLLM's synthetic prefix-repetition benchmark (10 × 20K-token prefixes, 100 requests, QPS 1–10). The baselines are GPU-only prefix caching in vLLM and LMCache with a local-CPU backend. The abstract's −82% TTFT and +39% throughput are vLLM-relative numbers for Qwen3-4B at QPS 1. A second setup cables two DGX Spark nodes to one BlueField-3 and compares only with vLLM (−29% to −43% TTFT, +49% to +55% throughput). The abstract also claims a 50% reduction in host memory and up to 3× I/O throughput over host-DRAM caches.
Critical strengths
-
It is a real, working prototype on real hardware, packaged as a drop-in vLLM connector with code, and it runs in both single-host and two-node configurations. Showing that DPU memory can act as a KV tier inside a production serving engine is useful on its own.
-
Against LMCache, the only offloading baseline, CacheFlow has lower mean TTFT in all 30 (model, QPS) cells of Table 1 (roughly 4–73% lower) and about 12–16% higher mean throughput, and the trend mostly holds at P90/P99. If this survives the controls below, it is a credible engineering improvement.
Critical weaknesses
-
The central claim, that the DPU is the right place for the prefix pool, is never isolated. In the single-node setup the DPU→GPU and pinned-host→GPU paths cross the same PCIe Gen4 link to the GPU, and CacheFlow uses 2,048-token blocks, batching, four copy streams and layer-wise loading while LMCache runs with 256-token chunks on a 32 GB host. The obvious control, the same connector backed by pinned host DRAM, is missing, so the gains could come from implementation choices.
-
The data path is described inconsistently. The abstract says KV bypasses the host CPU and DRAM via GPUDirect RDMA, but §3.1 moves tensors "between the host DMA buffers and the DPU memory" and Table 7 mentions GPU staging buffers. The "on the query path" benefit argued in Fig. 2 is also not what is implemented, since lookups are issued by the host scheduler.
-
Two headline claims have no supporting measurement. Nothing in the paper measures the "3× I/O throughput", and the "50% host memory" figure is actually a back-of-the-envelope GPU-memory estimate (12 GB of 24 GB) that conflicts with the 23 GB shown in Fig. 1(a); Table 3's KV sizes are also about 10× smaller than Eq. 7 gives. The −82%/+39% figures are against the weakest baseline; against LMCache in the same cell they are −43%/+8%.
-
The baselines and the rigor are weak overall. The multi-node experiment compares only against vLLM with no shared cache, although the DPU sits in a separate server and is effectively a remote memory node, so LMCache's remote backend or an RDMA KV store such as Mooncake Store would be the fair comparison. Hit rates are never reported, even though the DPU pool (71,680 tokens) is smaller than the 200,000-token working set. Every number comes from a single run of 100 requests, CacheFlow's curves are non-monotonic in load, and some LMCache/Ministral entries look rounded.
Questions for authors
-
On a cache hit, does KV go directly from DPU DRAM to GPU HBM, or through host buffers? How was GPUDirect RDMA enabled on an RTX 4090 (my understanding is that NVIDIA documents it for data-center and professional GPUs), and can you show that host memory is not on the path?
-
Which experiment produced the 3× I/O number? Could you add DPU→GPU vs. pinned-host→GPU bandwidth microbenchmarks, and run the same connector backed by host DRAM?
-
What are the prefix hit rates and eviction counts per system and QPS, and how does CacheFlow beat LMCache when its pool holds only about 36% of the unique prefix tokens?
-
Is the 50% memory claim about host or GPU memory, and was it measured? Please reconcile 12 GB vs. 23 GB and Table 3 vs. Eq. 7, and state how many runs each cell is based on.