acceptodds
Under review as a conference paper at ICLR 2027

SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL

Abstract

The scaling of LLMs toward long-context inference has shifted the primary serving system bottleneck from computation to memory capacity. Traditional solutions for dense attention models rely on RDMA-based disaggregated memory pools, which perform coarse-grained fetching of the entire prefix KV Cache from remote storage to local memory before decoding. However, this approach is fundamentally inefficient for emerging sparse attention models. While only a small fraction of KV entries are active during decoding, these systems still fetch the full KV cache locally, leading to severe transmission bottlenecks and local memory wastage. To address this, we propose SAC, the first efficient disaggregated KV cache system optimized for sparse attention models. By leveraging the low-latency, cache-line granularity load/store semantics of Compute Express Link (CXL), SAC fetches only the required top-k KV entries on demand during inference. Evaluations on DeepSeek-V3.2 using SGLang show that SAC achieves 2.1x higher throughput, 9.7x lower TTFT, and 1.8x lower TBT compared to RDMA-based baselines, establishing CXL-based disaggregation as the superior infrastructure for emerging sparse attention models.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Sparse Attention, KV Cache, CXL

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_H8FFOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a disaggregated KV-cache system where the memory pool that stores the KV-cache is connected to the GPUs through CXL. The paper showcases the bottlenecks in the RDMA-based existing systems and evaluates the performance gains of CXL-based system over RDMA. Unlike prior methods, this paper addresses and evaluates an important aspect of disaggregated KV-cache systems: how to handle the emerging sparse KV-cache accesses.

Critical strengths
  • The manuscript is very well-written, easy to follow; the prior work, contributions, and novel points are clear. I enjoyed reading this paper.

  • The idea of using CXL in case of sparse attention is sensible and a promising direction.

  • The evaluations are thorough and clearly show the performance benefits of the proposed method.

Critical weaknesses

None

Questions for authors

You mention in Section 3.1: "KV caches are typically stored using a page-first layout on the remote but require a layer-first layout for local processing. This layout transformation introduces additional latency overhead". Can't we just change the layout and store the KV-cache in layer-first layout in RDMA-based systems? What would be the downside of it?

In Fig. 10, why is the speedup in TTFT so much higher than TBT? I would actually expect the opposite because in prefill, many input tokens will attend different parts of the prefix, so it will get closer to the dense attention, in which the RDMA should be equally good (except if there is a bandwidth difference between RDMA baseline and CXL)

1 comment

Thanks for your valuable review! We address your two questions below.

Question 1: Layer-first layout in RDMA pools

We agree with your point. Using a layer-first layout in an RDMA pool can save local processing time. In my opinion, the current page-first layout is an engineering choice, as efficiently building and maintaining a layer-first layout in an RDMA pool is much harder. When retrieving a cached prefix, a layer-first layout requires scattered transfers. It also makes it more complicated to adaptively split a request’s KV cache across different nodes. Managing KV in blocks is more natural for an RDMA pool and follows the current community ecosystem.

SAC uses a different interconnect: CXL provides memory load/store semantics, allowing the shared pool to be accessed much like local memory. This makes using the same layer-first layout as HBM the simplest software design on CXL. Besides, SAC uses a physically centralized memory pool, which avoids much of the complexity of maintaining a layer-first layout in a distributed RDMA pool.

Question 2: TTFT and TBT speedups

During the prefill process, a chunk of tokens is processed together, making DSA access closer to dense attention. Therefore, in Fig. 9 (Round-1), where prefill accounts for most of the execution time, SAC and RDMA have almost the same TTFT. However, in Fig. 10 (Round-2), all prefix KV entries are already stored in the pool. RDMA needs to transfer the complete KV cache from the disaggregated pool to local DRAM. At high concurrency, RDMA bandwidth becomes saturated, causing transfers to queue and leading to long transfer times. As a result, KV transfer accounts for most of the TTFT in the RDMA case. While SAC does not need to fetch the complete KV cache. It only fetches the top-k entries when each layer computes, substantially reducing TTFT compared with RDMA.

Besides, the reported TBT does not include request-queueing time. It only measures the interval between output tokens after a decoding request enters the inference batch. Meanwhile, the RDMA RNIC continuously transfers large amounts of KV data to local DRAM, consuming substantial PCIe bandwidth and contending with HiSparse’s DRAM-to-GPU transfers. This affects TBT. To summarize, SAC’s TBT speedup comes from a different cause than its TTFT speedup, so the two speedup ratios are not correlated.

Reviewer_BsiWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAC, a CXL-based disaggregated KV-cache system tailored to sparse-attention LLMs (e.g., DeepSeek-V3.2). Unlike RDMA-based systems that prefetch entire prefixes, SAC leverages CXL’s low-latency, cache-line-granular load/store semantics to fetch only the per-layer top-k KV entries on demand, thereby reducing transmission overhead and avoiding local memory bloat. Integrated into SGLang/HiSparse, the paper reports large end-to-end gains over an RDMA baseline.

Critical strengths
  • The paper offers a novel research perspective. Recognizing that sparse-attention models access only a small subset of KV entries at each layer, it questions the necessity of full KV cache prefetching in conventional disaggregated systems and explores how to better align disaggregated KV cache management with the dynamic, fine-grained access patterns of sparse attention.
  • The evaluation is comprehensive, covering end-to-end performance, ablation studies, statistic significance, tail latency, and request-level throughput, providing strong empirical support for the effectiveness of SAC.
  • The paper considers three representative baselines: Local DRAM (a local-memory performance reference), GPU Only (a non-disaggregated configuration), and RDMA Memory Pool (a conventional disaggregated approach). These baselines provide complementary perspectives for evaluating SAC's performance advantages.
Critical weaknesses
  • The end-to-end hardware evaluation is limited to DeepSeek-V3.2, leaving the generalizability of SAC to other sparse-attention models insufficiently validated.
  • The top-k value is fixed at 2048 throughout the experiments, without a sensitivity analysis of SAC's performance under different sparsity levels. Evaluating throughput and latency under different top-k settings would provide a more comprehensive understanding of SAC's performance across varying sparsity levels.
  • The sensitivity analysis of device_buffer_size in Section 5.5 is limited to only two configurations (4K and 6K).
Questions for authors

None. I have no question.

3 comments

Thanks for your valuable review! We address the weaknesses you pointed out below.

Weakness 1: Evaluation on other models

We choose DeepSeek-V3.2 because it was a mainstream sparse-attention model when we conducted this research. DeepSeek-V3.2 first introduced the basic DSA architecture, which has influenced later models, including GLM-5.2 and GLM-5.3, as well as DeepSeek-V4 and V4.1 with CSA.

We are working on deploying SAC on these models, especially GLM-5.3, which is architecturally similar and widely used as an open-source model. This will be part of our subsequent work. In DSA models with IndexShare, such as GLM-5.3, four DSA layers share one top-k selection, enabling cross-layer prefetching. This provides greater tolerance for CXL bandwidth and latency, making these models well suited for SAC.

In Appendix E, we also evaluate SAC’s performance with GLM-5.2 under multi-P–D configurations through emulation. By combining the computation and CXL communication components, both measured on real hardware, we scale the evaluation to CXL-interconnected inference clusters. These results complement the main experiments.

Weakness 2: Different top-k settings

For a smaller top-k, SAC performs better under high concurrency because fewer KV entries need to be swapped in, reducing pressure on CXL bandwidth. For a larger top-k, CXL bandwidth becomes more saturated, increasing the performance gap relative to the DRAM baseline.

We have added experiments with top-k values of 1024, 2048, and 4096, using the same configuration as Round-2 in §5.1: 64K input tokens, 1K output tokens, and concurrency 64.

Output Throughput (tokens/s)Top-k = 1024Top-k = 2048Top-k = 4096
CXL / SAC1050.4937.9847.2
DRAM1123.71035.4960.0
CXL Throughput Reduction vs. DRAM6.5%9.4%11.8%
Mean TBT (ms/token)Top-k = 1024Top-k = 2048Top-k = 4096
CXL / SAC59.166.273.3
DRAM55.259.964.6
CXL TBT Increase vs. DRAM7.1%10.5%13.5%

Weakness 3: Choice of GPU buffer sizes

In our experiments, we deploy DeepSeek-V3.2 (671B) on 8 × H20 GPUs (96 GB each). We tried a larger buffer size, but HBM capacity is tight. We cannot support a buffer size of 8192 at our experimental batch size, so we only evaluate 4096 and 6144.

Besides, the GPU buffer size of 6144 follows the configuration in the SGLang HiSparse guide [1]. This buffer size determines how many hot KV entries are stored on the GPU. At each sparse-attention step, the I/O kernel swaps missing KV entries into the buffer. A larger buffer means fewer KV swap-ins each time, but also a higher scanning cost. In most scenarios, 6144 provides a good trade-off, and further increasing the buffer size brings little additional benefit.

Thanks for the authors' thoughtful response. My response is as follows.

  • Weakness 1: What I mean is that no end-to-end hardware evaluations have been conducted on models other than DeepSeek-V3.2.
  • Weakness 2: Resolved
  • Weakness3: Smaller device_buffer_size could be considered in the sensitivity analysis of device_buffer_size.

Based on the authors' thoughtful response, I will increase the Soundness rating.

Thanks for your reply and for increasing the Soundness rating!

  • Weakness 1 (End-to-end hardware evaluation on other models):

Thank you for the clarification. We acknowledge that our current end-to-end hardware evaluation focuses on DeepSeek-V3.2. We are actively working on extending end-to-end hardware evaluation to additional sparse-attention models as part of our future work.

  • Weakness 3 (Smaller device_buffer_size configurations):

Since the model uses top-k = 2048, setting device_buffer_size below 2048 cannot accommodate the top-k KV entries required for a single decode step. Between 2048 and 4096, reducing device_buffer_size leaves limited to no headroom on GPU HBM to retain hot KV entries across consecutive decode steps, substantially increasing the per-step cache miss rate and triggering more frequent evictions and CXL swap-ins, which degrades both throughput and TBT. We measure at a 64K context length in the experiment config of Section 5.5. Compared to device_buffer_size = 4096, reducing it to 3072 (1.5× top-k) decreases throughput by 11.8%, and further reducing it to 2048 (1× top-k) decreases throughput by 24.5%. We will add this analysis in the follow-up revised version of the paper.

Reviewer_BznHOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SAC, a CXL-based disaggregated KV cache system for large language models with sparse attention. Traditional approaches fetch the entire prefix KV cache before decoding, although sparse attention uses only a subset of the entries at each layer. SAC keeps the full KV cache in a CXL memory pool and builds on SGLang/HiSparse to fetch the required entries on demand, reducing data movement and local memory consumption. On DeepSeek-V3.2, the paper reports an average throughput improvement of 2.1× over the evaluated RDMA baseline, along with substantial reductions in TTFT and TBT.

Critical strengths

1)The system implementation is comprehensive, and the results are convincing. The authors analyze access latency, integrate the system into an inference framework, and conduct end-to-end evaluations on real hardware. Performance close to that of local DRAM suggests that CXL is a viable option for storing KV caches for sparse attention. 2)The evaluation covers a range of configurations. The paper examines context length, concurrency, output length, CXL device interleaving, and GPU buffer size, helping readers understand how these factors affect performance.

Critical weaknesses

1)The end-to-end hardware evaluation primarily focuses on DeepSeek-V3.2. Although the paper discusses applicability to other sparse attention models, KV cache sizes and access patterns may differ across models. 2)The most substantial improvements are observed in Round-2, where the prefix KV cache is already populated in the memory pool and prefill is bypassed. This setup helps isolate KV access overhead, but real-world serving typically involves a mixture of cache hits and misses. The overall benefit therefore also depends on the extent of prefix reuse.

Questions for authors

1)What is the main rationale for setting the default GPU buffer size to 6144? For practical deployments, do the authors have simple guidelines for choosing this parameter based on context length or concurrency? 2)Would the authors still recommend SAC for short-context or low-concurrency workloads? Does the system provide configuration options for selecting different KV cache storage approaches based on the workload?

1 comment

Thanks for your valuable review! We address your questions and the weaknesses you pointed out below.

Question 1: Choice of GPU buffer size

The GPU buffer size of 6144 in HiSparse follows the configuration in the SGLang HiSparse guide [1]. This buffer size determines how many hot KV entries are stored on the GPU. At each sparse-attention step, the I/O kernel swaps missing KV entries into the buffer. A larger buffer means fewer KV swap-ins each time, but also a higher scanning cost. In most scenarios, 6144 or 8192 provides a good trade-off, and further increasing the buffer size brings little additional benefit.

Besides, in our experiments, we deploy DeepSeek-V3.2 (671B) on 8 × H20 GPUs (96 GB each), where HBM capacity is tight. We cannot support a larger buffer size of 8192 at our experimental batch size, so we only evaluate buffer sizes up to 6144.

Question 2: Suitability for short contexts or low concurrency

We do not recommend using SAC or HiSparse for short-context or low-concurrency workloads. The motivation of HiSparse is to relieve GPU HBM limitations and support larger batches of decoding requests. The motivation of SAC is to relieve the RDMA bandwidth limitation in KV transfer cross nodes. Both target high-concurrency workloads.

In serving, the decode instance is memory-bound and seeks higher concurrency to reduce inference cost while meeting the TPOT SLO. In RL post-training rollouts, the demand for high throughput is even more direct. SAC is specifically designed for these high-concurrency scenarios. For low-concurrency workloads, directly storing the complete KV cache in the decode instance’s HBM without HiSparse or SAC is more suitable.

Weakness 1: Evaluation on other models

We choose DeepSeek-V3.2 because it was a mainstream sparse-attention model when we conducted this research. DeepSeek-V3.2 first introduced the basic DSA architecture, which has influenced later models, including GLM-5.2 and GLM-5.3, as well as DeepSeek-V4 and V4.1 with CSA.

We are working on deploying SAC on these models, especially GLM-5.3, which is architecturally similar and widely used as an open-source model. This will be part of our subsequent work. In DSA models with IndexShare, such as GLM-5.3, four DSA layers share one top-k selection, enabling cross-layer prefetching. This provides greater tolerance for CXL bandwidth and latency, making these models well suited for SAC.

In Appendix E, we also evaluate SAC’s performance with GLM-5.2 under multi-P–D configurations through emulation. By combining the computation and CXL communication components, both measured on real hardware, we scale the evaluation to CXL-interconnected inference clusters. These results complement the main experiments.

Weakness 2: Dependence on prefix reuse

In our experiments, we use a two-round setup with high KV reuse in the second round to put pressure on the decode instance. However, SAC’s benefit is not limited to workloads with high prefix hit rates. The core contribution of SAC is to address the KV transfer bottleneck and saves local memory when decode instances are under heavy load.

Even for workloads with low prefix hit rates, such as QA workloads, practical serving deployments adjust the ratio of prefill to decode instances (e.g., 8P–1D) to fully utilize both sides. In this case, the decode instance still faces substantial P–D KV handoff traffic, which can become one of the system bottlenecks.

Besides, agentic workloads now account for much of production inference traffic. These workloads involve multi-turn requests with high prefix reuse rates, often reaching 95%–99% [2]. Under high concurrency, decode instances can easily face heavy load, making SAC particularly useful.

[1] SGLang Team. “HiSparse: Hierarchical Sparse Attention.” SGLang Documentation. https://lmsysorg.mintlify.app/docs/advanced_features/hisparse_guide.

[2] Cam Quilici et al. “AgentX – InferenceXv3: Does CUDA Moat Hold up in Agentic Inferencing?” SemiAnalysis, August 24, 2026. https://newsletter.semianalysis.com/p/agentx-inferencexv3-does-cuda-moat.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.