- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RheoServe is a CPU–GPU collaborative sparse-attention serving system. It computes attention over host-resident KV and merges CPU and GPU partial results using LSE. It further maintains a dynamic GPU working set through adaptive block migration. The paper reports 1.4 and 1.57 times peak throughput over matched PCIe Retrieval for Qwen3-4B and Qwen3-8B.
Critical strengths
-
Clean evaluation of the core execution idea. RheoServe and the PCIe Retrieval baseline use the same sparse predictor and selected blocks, isolating CPU-side compute versus host-to-GPU KV retrieval.
-
Reasonable system design. Dynamic block migration keeps frequently accessed blocks on GPU, while CPU attention, GPU attention, and migration are overlapped. The latency breakdown also clearly explains when CPU execution becomes the bottleneck.
Critical weaknesses
-
Novelty over prior CPU–GPU collaborative attention systems is insufficiently validated. MagicPIG and HGCA already perform CPU–GPU collaborative attention. RheoServe argues that its dynamic GPU working set and cache management are different, but provides no direct end-to-end comparison against these closest systems.
-
The serving evaluation is relatively limited. The main experiments use RTX 3090/5090.
Questions for authors
-
Can the authors directly compare against MagicPIG or HGCA, or otherwise isolate the benefit of RheoServe’s dynamic KV residency policy over existing CPU–GPU attention?
-
Does the advantage persist on H200/B200 systems or multi-GPU serving, where GPU compute, PCIe, and host-memory bandwidth ratios differ substantially? Or, theoretically, determine whether server-grade GPUs can still be effective.
6 comments
Author Response (1/3) — W1 & Q1: Contribution Beyond CPU–GPU Attention
RheoServe argues that its dynamic GPU working set and cache management are different, but provides no direct end-to-end comparison against these closest systems.
We have two core contributions: the compute offload strategy and the hot/cold cache migration strategy, which together make RheoServe surpass vLLM. Making the two mechanisms work together efficiently is both important and difficult, and no existing work fully achieves this. We therefore believe the novelty is sufficient.
MagicPIG maps keys into LSH buckets for prefill tokens and fully offloads them to the CPU. All required tokens are then computed on the CPU without GPU caching; as shown in Figure 6 and Table 2 of the manuscript, this causes severe throughput degradation. Moreover, its newly generated KV cache stays on the GPU, where dense attention is performed over these tokens, which incurs a further efficiency drop and eventually leads to OOM.
To validate this, we compare MagicPIG and RheoServe under matched conditions: both use Qwen3-4B, batch size 12, an 8K initial context, and the same continuation of 6,072 output tokens per sequence. MagicPIG uses its native LSH configuration (K=10, L=150). Longer sequences cause MagicPIG to run out of memory, whereas RheoServe can sustain even longer contexts through dynamic block migration — precisely the contribution we claim.
Table Q1a. End-to-end decode throughput: MagicPIG versus RheoServe.
| Method | Average decode throughput (tokens/s) |
|---|---|
| MagicPIG | 102.27 |
| RheoServe | 276.68 |
| Speedup | 2.71× |
RheoServe achieves a 2.71× speedup over MagicPIG under these conditions.
HGCA does implement GPU caching, but its caching strategy is first-in-first-out (FIFO): newly generated cache entries directly evict old ones to the CPU, which does not consider the dynamic change of attention. For an aligned comparison, we run RheoServe with the dynamic block migration strategy replaced by FIFO:
Table Q1b. Dynamic migration versus FIFO residency policy.
| Context | Dynamic (tokens/s) | FIFO (tokens/s) | Speedup |
|---|---|---|---|
| 12K | 181.16 | 97.90 | 1.85× |
| 32K | 147.76 | 62.38 | 2.37× |
The significant throughput drop of FIFO relative to Dynamic demonstrates that our adaptive migration strategy is fundamentally different from and superior to the static policies used by HGCA and MagicPIG, confirming it as a core contribution.
Author Response (2/3) — Q1: A Direct Comparison of CPU Sparse Attention
Or otherwise isolate the benefit of RheoServe's dynamic KV residency policy over existing CPU–GPU attention?
Setting aside the benefit of dynamic KV residency, we compare CPU-side sparse attention performance. The table below reports RheoServe's sparse kernel speedup over the others:
Table Q1c. CPU sparse-attention kernel latency and speedup.
| CPU | CPU tokens/KV head | RheoServe (ms) | HGCA (ms) | MagicPIG (ms) | vs. HGCA | vs. MagicPIG |
|---|---|---|---|---|---|---|
| Intel Core i9 / 8 P-cores | 64 | 0.06 | 0.16 | 0.11 | 2.7× | 1.9× |
| Intel Core i9 / 8 P-cores | 1024 | 1.19 | 6.46 | 1.64 | 5.4× | 1.4× |
| Intel Xeon / 32 cores | 64 | 0.07 | 0.79 | 0.08 | 11.5× | 1.2× |
| Intel Xeon / 32 cores | 1024 | 0.28 | 1.53 | 1.12 | 5.5× | 4.0× |
Both RheoServe and MagicPIG use AVX2 on Intel Core i9 and AVX-512 on Intel Xeon for vectorized operations. Beyond MagicPIG, we employ memory prefetching and cache-aware GQA computation, exploiting both spatial and temporal locality to make effective use of CPU caches. HGCA dispatches multiple sparse attention kernels through PyTorch threads, which introduces additional scheduling and synchronization overhead compared with RheoServe's native CPU kernel.
Across all measured settings, our sparse kernel achieves a higher speedup than both systems. Combined with our dynamic cache and migration design, this allows RheoServe to surpass both works overall and genuinely compete with production serving frameworks such as vLLM.
Author Response
We thank the reviewer for the careful reading and for the two concrete questions, which helped us to strengthen the evaluation substantially. We reply in a separate comment below: (1/3) contribution beyond CPU–GPU attention, including an aligned residency-policy comparison; (2/3) a direct comparison of CPU sparse-attention kernels; (3/3) server-grade GPU and multi-GPU serving results on H20 and H100.
We hope these new comparisons clearly demonstrate the distinct contributions of RheoServe over prior CPU–GPU collaborative attention systems, and we would be grateful if the reviewer would reconsider raising the score accordingly.