- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a disaggregated KV-cache system where the memory pool that stores the KV-cache is connected to the GPUs through CXL. The paper showcases the bottlenecks in the RDMA-based existing systems and evaluates the performance gains of CXL-based system over RDMA. Unlike prior methods, this paper addresses and evaluates an important aspect of disaggregated KV-cache systems: how to handle the emerging sparse KV-cache accesses.
Critical strengths
-
The manuscript is very well-written, easy to follow; the prior work, contributions, and novel points are clear. I enjoyed reading this paper.
-
The idea of using CXL in case of sparse attention is sensible and a promising direction.
-
The evaluations are thorough and clearly show the performance benefits of the proposed method.
Critical weaknesses
None
Questions for authors
You mention in Section 3.1: "KV caches are typically stored using a page-first layout on the remote but require a layer-first layout for local processing. This layout transformation introduces additional latency overhead". Can't we just change the layout and store the KV-cache in layer-first layout in RDMA-based systems? What would be the downside of it?
In Fig. 10, why is the speedup in TTFT so much higher than TBT? I would actually expect the opposite because in prefill, many input tokens will attend different parts of the prefix, so it will get closer to the dense attention, in which the RDMA should be equally good (except if there is a bandwidth difference between RDMA baseline and CXL)
Thanks for your valuable review! We address your two questions below.
Question 1: Layer-first layout in RDMA pools
We agree with your point. Using a layer-first layout in an RDMA pool can save local processing time. In my opinion, the current page-first layout is an engineering choice, as efficiently building and maintaining a layer-first layout in an RDMA pool is much harder. When retrieving a cached prefix, a layer-first layout requires scattered transfers. It also makes it more complicated to adaptively split a request’s KV cache across different nodes. Managing KV in blocks is more natural for an RDMA pool and follows the current community ecosystem.
SAC uses a different interconnect: CXL provides memory load/store semantics, allowing the shared pool to be accessed much like local memory. This makes using the same layer-first layout as HBM the simplest software design on CXL. Besides, SAC uses a physically centralized memory pool, which avoids much of the complexity of maintaining a layer-first layout in a distributed RDMA pool.
Question 2: TTFT and TBT speedups
During the prefill process, a chunk of tokens is processed together, making DSA access closer to dense attention. Therefore, in Fig. 9 (Round-1), where prefill accounts for most of the execution time, SAC and RDMA have almost the same TTFT. However, in Fig. 10 (Round-2), all prefix KV entries are already stored in the pool. RDMA needs to transfer the complete KV cache from the disaggregated pool to local DRAM. At high concurrency, RDMA bandwidth becomes saturated, causing transfers to queue and leading to long transfer times. As a result, KV transfer accounts for most of the TTFT in the RDMA case. While SAC does not need to fetch the complete KV cache. It only fetches the top-k entries when each layer computes, substantially reducing TTFT compared with RDMA.
Besides, the reported TBT does not include request-queueing time. It only measures the interval between output tokens after a decoding request enters the inference batch. Meanwhile, the RDMA RNIC continuously transfers large amounts of KV data to local DRAM, consuming substantial PCIe bandwidth and contending with HiSparse’s DRAM-to-GPU transfers. This affects TBT. To summarize, SAC’s TBT speedup comes from a different cause than its TTFT speedup, so the two speedup ratios are not correlated.