acceptodds
Under review as a conference paper at ICLR 2027

RheoServe: CPU-GPU Collaborative Sparse Attention for Efficient LLM Decoding

Abstract

Sparse attention is a promising approach for efficient LLM decoding over large KV caches, yet managing these caches remains a bottleneck: eviction-based methods sacrifice accuracy, while retrieval-based offloading is constrained by limited PCIe bandwidth, thereby bottlenecking decoding throughput. To address this, we present RheoServe, a CPU–GPU collaborative sparse-attention framework that actively leverages host memory bandwidth via block-level cache management. RheoServe adopts a compute offload paradigm: rather than moving massive data to the GPU, it moves compute to the data, executing sparse attention directly on DDR-resident blocks and merging results via Log-Sum-Exp (LSE) reduction. Furthermore, RheoServe employs a unified cache mechanism that intelligently manages block placement, improving cache hit rate to compensate for the bandwidth gap. Extensive evaluations demonstrate that RheoServe maintains reasoning accuracy and achieves and the peak throughput of matched PCIe Retrieval for Qwen3-4B and Qwen3-8B, respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
CPU-GPU Collaboration, Sparse Attention, KV Cache Management

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Q3zWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RheoServe is a CPU–GPU collaborative sparse-attention serving system. It computes attention over host-resident KV and merges CPU and GPU partial results using LSE. It further maintains a dynamic GPU working set through adaptive block migration. The paper reports 1.4 and 1.57 times peak throughput over matched PCIe Retrieval for Qwen3-4B and Qwen3-8B.

Critical strengths
  1. Clean evaluation of the core execution idea. RheoServe and the PCIe Retrieval baseline use the same sparse predictor and selected blocks, isolating CPU-side compute versus host-to-GPU KV retrieval.

  2. Reasonable system design. Dynamic block migration keeps frequently accessed blocks on GPU, while CPU attention, GPU attention, and migration are overlapped. The latency breakdown also clearly explains when CPU execution becomes the bottleneck.

Critical weaknesses
  1. Novelty over prior CPU–GPU collaborative attention systems is insufficiently validated. MagicPIG and HGCA already perform CPU–GPU collaborative attention. RheoServe argues that its dynamic GPU working set and cache management are different, but provides no direct end-to-end comparison against these closest systems.

  2. The serving evaluation is relatively limited. The main experiments use RTX 3090/5090.

Questions for authors
  1. Can the authors directly compare against MagicPIG or HGCA, or otherwise isolate the benefit of RheoServe’s dynamic KV residency policy over existing CPU–GPU attention?

  2. Does the advantage persist on H200/B200 systems or multi-GPU serving, where GPU compute, PCIe, and host-memory bandwidth ratios differ substantially? Or, theoretically, determine whether server-grade GPUs can still be effective.

6 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the careful reading and for the two concrete questions, which helped us to strengthen the evaluation substantially. We reply in a separate comment below: (1/3) contribution beyond CPU–GPU attention, including an aligned residency-policy comparison; (2/3) a direct comparison of CPU sparse-attention kernels; (3/3) server-grade GPU and multi-GPU serving results on H20 and H100.

We hope these new comparisons clearly demonstrate the distinct contributions of RheoServe over prior CPU–GPU collaborative attention systems, and we would be grateful if the reviewer would reconsider raising the score accordingly.

AuthorsOpenReview ↗

Author Response (1/3) — W1 & Q1: Contribution Beyond CPU–GPU Attention

RheoServe argues that its dynamic GPU working set and cache management are different, but provides no direct end-to-end comparison against these closest systems.

We have two core contributions: the compute offload strategy and the hot/cold cache migration strategy, which together make RheoServe surpass vLLM. Making the two mechanisms work together efficiently is both important and difficult, and no existing work fully achieves this. We therefore believe the novelty is sufficient.

MagicPIG maps keys into LSH buckets for prefill tokens and fully offloads them to the CPU. All required tokens are then computed on the CPU without GPU caching; as shown in Figure 6 and Table 2 of the manuscript, this causes severe throughput degradation. Moreover, its newly generated KV cache stays on the GPU, where dense attention is performed over these tokens, which incurs a further efficiency drop and eventually leads to OOM.

To validate this, we compare MagicPIG and RheoServe under matched conditions: both use Qwen3-4B, batch size 12, an 8K initial context, and the same continuation of 6,072 output tokens per sequence. MagicPIG uses its native LSH configuration (K=10, L=150). Longer sequences cause MagicPIG to run out of memory, whereas RheoServe can sustain even longer contexts through dynamic block migration — precisely the contribution we claim.

Table Q1a. End-to-end decode throughput: MagicPIG versus RheoServe.

MethodAverage decode throughput (tokens/s)
MagicPIG102.27
RheoServe276.68
Speedup2.71×

RheoServe achieves a 2.71× speedup over MagicPIG under these conditions.


HGCA does implement GPU caching, but its caching strategy is first-in-first-out (FIFO): newly generated cache entries directly evict old ones to the CPU, which does not consider the dynamic change of attention. For an aligned comparison, we run RheoServe with the dynamic block migration strategy replaced by FIFO:

Table Q1b. Dynamic migration versus FIFO residency policy.

ContextDynamic (tokens/s)FIFO (tokens/s)Speedup
12K181.1697.901.85×
32K147.7662.382.37×

The significant throughput drop of FIFO relative to Dynamic demonstrates that our adaptive migration strategy is fundamentally different from and superior to the static policies used by HGCA and MagicPIG, confirming it as a core contribution.

AuthorsOpenReview ↗

Author Response (2/3) — Q1: A Direct Comparison of CPU Sparse Attention

Or otherwise isolate the benefit of RheoServe's dynamic KV residency policy over existing CPU–GPU attention?

Setting aside the benefit of dynamic KV residency, we compare CPU-side sparse attention performance. The table below reports RheoServe's sparse kernel speedup over the others:

Table Q1c. CPU sparse-attention kernel latency and speedup.

CPUCPU tokens/KV headRheoServe (ms)HGCA (ms)MagicPIG (ms)vs. HGCAvs. MagicPIG
Intel Core i9 / 8 P-cores640.060.160.112.7×1.9×
Intel Core i9 / 8 P-cores10241.196.461.645.4×1.4×
Intel Xeon / 32 cores640.070.790.0811.5×1.2×
Intel Xeon / 32 cores10240.281.531.125.5×4.0×

Both RheoServe and MagicPIG use AVX2 on Intel Core i9 and AVX-512 on Intel Xeon for vectorized operations. Beyond MagicPIG, we employ memory prefetching and cache-aware GQA computation, exploiting both spatial and temporal locality to make effective use of CPU caches. HGCA dispatches multiple sparse attention kernels through PyTorch threads, which introduces additional scheduling and synchronization overhead compared with RheoServe's native CPU kernel.

Across all measured settings, our sparse kernel achieves a higher speedup than both systems. Combined with our dynamic cache and migration design, this allows RheoServe to surpass both works overall and genuinely compete with production serving frameworks such as vLLM.

Reviewer_rvg6OpenReview ↗
Rating
4
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

RheoServe proposes a CPU-GPU collaborative KV cache offloading and sparse attention framework to address challenge that the HBM iof GPU is limited, and PCIe bandwidth hinders fast KV cache loading during long-context generation. This paper first introduce a Dual-Track Cache with a lightweigh indexer to efficiently identify important KV blocks through a coarse-grain KV selection. Second, RheoServe adopts a compute offloading strategy that allows the GPU to process hot KV blocks while the CPU directly computes attention over cold blocks stored in host memory without moving them back to HBM. The results of these two parts are merged through LSE. Additionally, an adaptive block migration mechanism dynamically adjusts KV block placement based on historical attention scores to improve GPU cache hit rates and overlap data movement with computation. Experimental results show that RheoServe maintains competitive reasoning accuracy while achieving up to 1.40× and 1.57× higher peak decoding throughput than PCIe retrieval for Qwen3-4B and Qwen3-8B, respectively.

Critical strengths
  1. The design of the proposed system is novel, and most available system optimizations are used in the proposed system. By conducting attention compute on both GPU and CPU, the system is able to alleviate the heavy PCIe communication that bounds all KV cache offloading systems. The scheduling system is complicated but effective, showing the effort payed by the authors on optimizing the whole system.

  2. The presentation of this paper is informative. The figures and tables are easy to read, and can fully show the efficiency and effectiveness of the proposed system. Experiments across multiple models, tasks, and systems make this paper very solid and sound.

  3. The comparison between the proposed system and vLLM/SGLang fully show the effectiveness of all the proposed method. By surpassing the SOTA inference engines, the proposed system is a out-of-box framework that can be very influential to the LLM inference/serving community.

Critical weaknesses
  1. The baselines seem to be a little old and under-optimized. There are better training-free KV cache offloading systems, e.g., ShadowKV [1], and Quest [2], that shall be compared with or discuss with.

Update on 11/16: this weakness is resolved.

  1. The number of CPU cores used in the experiments (16) is quite limited. SOTA clusters are often equipped with hundreds of CPU cores. Would increase the number of CPU cores accelerate the attention computation on CPU, and lead to a different scheduling strategy?

Update on 11/16: this weakness is also resolved. The additional results can fully show the scalability of the proposed method in terms of CPU cores.


[1] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

[2] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

Questions for authors

See above.

5 comments
AuthorsOpenReview ↗

Author Response

We sincerely thank the reviewer for the positive assessment and for recognizing our system design, presentation, and evaluation; this is a great encouragement to us. We also appreciate the two constructive suggestions on baselines and CPU core scaling, which inspired us to carry out additional comparisons and measurements presented in this response. We reply in two separate comments: (1/2) comparison with ShadowKV and Quest, covering quality and throughput under matched conditions; (2/2) CPU core scaling and its implications for scheduling. We hope these results further demonstrate the contribution and robustness of RheoServe.

AuthorsOpenReview ↗

Author Response (1/2) — W1: Comparison with ShadowKV and Quest

There are better training-free KV cache offloading systems, e.g., ShadowKV and Quest, that shall be compared with or discussed.

There are two points we would like to clarify. Although ShadowKV and Quest are both training-free sparse-attention prediction methods, they differ in how readily they can serve as a ready-to-use KV cache offloading system like RheoServe.

ShadowKV is dedicated to the prefill stage. It concatenates the key heads (e.g., 8 heads × 128 dim = 1024 dim), performs SVD factorization over the whole key cache, and reduces it to 160 dim. It then uses this reduced key cache to predict important blocks and reconstructs the keys from the compressed representation, while storing the full value cache on the CPU.

Although this method also employs offloading, it differs from the design adopted by recent mainstream models such as DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, which follow the same direction as RheoServe:

  1. Its sparse prediction requires SVD to factorize and restore, which is slow in compression and loses full information compared with storing the full key cache and retrieving on demand.

  2. Its compression direction is along the attention-head dimension, performed only once at the prefill stage; during decoding it keeps full attention over newly generated tokens. The recent mainstream choice of DeepSeek and Qwen is instead to pack a block of key embeddings into a single embedding of the same size (sequence-wise), which preserves sparse attention throughout generation. This is also the choice RheoServe makes ahead of them.

  3. Its key embedding is reduced to only 15.6% (160/1024) of the original size but stays entirely on the GPU and grows continuously with sequence length, eventually exhausting GPU memory. In contrast, RheoServe's Key Index requires just 3.1% (1/32) of the key cache to remain on the GPU as a compact index. All other KV blocks are safely offloaded to the CPU.

These structural issues make ShadowKV unsuitable for long-generation tasks and prevent an aligned accuracy comparison. Therefore, we directly align the experimental settings to evaluate throughput. Using the same configuration as Figure 5 (Qwen3-4B, batch size 12, 8K initial context), we report the average decode throughput over a common generation interval of 6K output tokens per sequence:

ShadowKV versus RheoServe: average decode throughput.

MethodAverage decode throughput (tokens/s)
ShadowKV265.78
RheoServe276.68

At this point, ShadowKV reaches its configured GPU buffer capacity, whereas RheoServe can continue generating. These results demonstrate RheoServe's higher throughput and ability to continue beyond ShadowKV's configured capacity.


For Quest, the compression direction is sequence-wise: it stores max and min vector representations of a key block and indexes the full cache, which makes it suitable for comparison. It is training-free, but the authors did not implement an offloading system, so the full cache still incurs OOM under limited GPU memory.

We list Qwen3-8B quality results below, using temperature 0 and a 1,024-token sparse-attention budget. LongBench reports the average across tasks, while AIME reports pass@1.

Table W1a. Quality comparison on Qwen3-8B.

MethodLongBenchAIME24 (%)AIME25 (%)
Full attention37.7266.6756.67
RheoServe36.9166.6746.67
Quest36.3746.6726.67

Under the same indexing and attention budget, RheoServe achieves higher accuracy than Quest while avoiding the OOM issues it encounters.

AuthorsOpenReview ↗

Author Response (2/2) — W2: CPU Core Usage and Scheduling Strategy

The number of CPU cores used in the experiments (16) is quite limited. SOTA clusters are often equipped with hundreds of CPU cores.

In our experiment, the rented RTX 5090 container provides only 16 vCPUs within a single memory node, directly determining our use of 16 CPU workers. Another reason is that the server CPUs we can evaluate usually bind no more than 16 cores to each memory node. Take the bare-metal server we use for our extended experiments as an example: it is equipped with two AMD EPYC 7773X processors, providing 128 physical cores across eight memory nodes, with each node binding 16 physical cores. Our choice of 16 workers therefore reflects both the container's CPU allocation and the underlying core–memory topology.

Would increasing the number of CPU cores accelerate the attention computation on CPU, and lead to a different scheduling strategy?

Since CPU sparse attention is bandwidth-bound, simply increasing the number of cores within a single memory node will not continue to yield speedup once peak memory bandwidth is reached. However, because servers typically have multiple memory nodes, increasing the number of CPU cores will accelerate computation as long as we distribute tasks to the cores bound to each respective node.

Importantly, this will not lead to a different scheduling strategy, as RheoServe was designed to maximize parallelism by distributing tasks across KV heads × batch elements. In our test, we simply divide the attention heads evenly across the available memory nodes and let each node process the batch for its assigned heads.

The following experiments in Table W2a show the scaling within a single memory node. The workload uses a batch size of 32, and each KV head processes 32 selected CPU blocks (32 tokens per block).

Table W2a. Scaling within one memory node.

Threads on one memory node124816
Attention kernel (ms)18.5999.2434.6922.6791.159
Speedup1.0×2.0×4.0×6.9×16.0×

We observe nearly linear scaling from 1 to 16 cores. We then distribute the tasks from one memory node to multiple nodes. As shown in Table W2b, increasing the number of CPU cores alongside the number of memory nodes continues to achieve substantial speedup.

Table W2b. Multi-memory node scaling.

Memory nodes1248
Total physical cores163264128
Attention kernel (ms)1.1590.6140.4060.240
Speedup1.0×1.9×2.9×4.8×

These results suggest that RheoServe can easily exploit larger CPU allocations to accommodate heavier CPU-side attention workloads caused by more GPU-residency misses. Furthermore, scaling across multiple nodes yields substantial speedups without requiring any changes to the scheduling strategy.

Reviewer_ozyPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Efficient LLM decoding is heavily bottlenecked by KV cache management. While sparse attention offers a promising avenue for efficiency, existing approaches face significant trade-offs: eviction-based methods often degrade model accuracy, and retrieval-based offloading is fundamentally constrained by limited PCIe bandwidth. This paper addresses the critical gap in utilizing host memory bandwidth for LLM serving, identifying that moving massive data to the GPU is less efficient than leveraging the CPU’s local memory hierarchy. The authors propose RheoServe, a novel CPU-GPU collaborative sparse attention framework that shifts the paradigm from data movement to compute offloading. Extensive evaluations on Qwen3-4B and Qwen3-8B models across RTX 3090 and RTX 5090 platforms demonstrate RheoServe’s effectiveness.

Critical strengths

1 The paper accurately identifies PCIe bandwidth as the primary bottleneck for retrieval-based KV cache offloading. Rather than relying on theoretical assumptions, the authors effectively use empirical profiling to demonstrate the stark contrast between PCIe and DDR bandwidth utilization. This data-driven motivation strongly justifies the shift towards a compute-offload paradigm, making the problem formulation both convincing and practically relevant to current hardware architectures.

2 The architectural decision to move computation to the data is highly elegant. The use of LSE reduction to mathematically guarantee the correctness of merged attention results from heterogeneous compute units is a significant theoretical contribution. Beyond the algorithmic design, the engineering implementation is impressive, encompassing a Block Controller for logical-to-physical mapping, collaborative execution paths, and a runtime scheduler with Zero-Reallocation Batching. This demonstrates a high level of system maturity.

3 The evaluation methodology is robust and spans diverse hardware and software configurations. Testing on both RTX 3090 and RTX 5090 effectively showcases the system’s adaptability to varying bandwidth profiles. Evaluating across Qwen3-4B and Qwen3-8B models with batch sizes ranging from 1 to 16 provides a holistic view of performance, covering both latency-sensitive (BS=1) and throughput-optimized regimes.

Critical weaknesses

1 RheoServe relies on the SeerAttention-R gating network to generate sparse attention masks, but the paper does not isolate the CPU overhead attributable solely to this gating mechanism. Since the gating computation and the actual sparse attention computation share CPU resources, the unmeasured overhead of the predictor directly impacts the net benefit of compute offloading. Without a breakdown of CPU latency (gating vs. attention), it is difficult to assess the true efficiency of the offload strategy.

2 The accuracy evaluation is restricted to reasoning-heavy benchmarks GPQA, MATH-500, AIME24/25. For an LLM serving system, generative quality in open-ended dialogue is equally critical. The absence of evaluations on benchmarks like MT-Bench, AlpacaEval, or LLM-as-a-judge metrics leaves a gap in understanding how sparse attention affects conversational fluency and instruction following. Relying solely on reasoning tasks may underestimate potential quality degradation in broader deployment scenarios.

3 The adaptive block migration strategy depends entirely on historical attention score ranking. The paper does not analyze the system’s behavior under non-stationary workloads, such as sudden shifts from reasoning to dialogue or adversarial input patterns that disrupt attention locality. Without a stability analysis or discussion of warmup strategies, it remains unclear whether the policy might suffer from oscillation or slow adaptation, potentially causing latency spikes in real-world serving.

Questions for authors

see weakness. If these are well adderessed, my rating will be raised.

6 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the patience in reviewing and the willingness to hear our response. We reply to each weakness in a separate comment below: (1/3) gating placement and latency breakdown, (2/3) conversational fluency and instruction following, and (3/3) stability analysis and warm-up strategy.

We hope these clarifications and additional results address the reviewer's concerns, and we would be grateful if the reviewer would reconsider raising the score.

AuthorsOpenReview ↗

Author Response (1/3) — W1: Gating Placement and CPU Latency Breakdown

Since the gating computation and the actual sparse attention computation share CPU resources, the unmeasured overhead of the predictor directly impacts the net benefit of compute offloading.

We would like to clarify our operation layout: the gating computation is on the GPU, and only sparse attention is on the CPU. As shown in Figure 2(a), gating computation, by which we mean Fast Indexing in this figure, is performed on the GPU. Here we compute block-level attention scores and perform top-K selection. A vectorized mask then quickly identifies which of the top-K selected blocks are not GPU-resident, and only a lightweight bitmask is sent to the CPU to tell which blocks are needed for collaborative attention. Gating therefore consumes no CPU cycles and does not contend with CPU-side attention.

This reflects our design philosophy: we keep the lightweight Key Index Cache (1/32 of the Key Cache) fully resident on the GPU, so it does not quickly inflate memory usage, and we exploit the GPU's speed to quickly locate the required blocks through the Index Cache.

Below we report the latency breakdown of the different parts of collaborative attention (BS=12, per-layer average). CPU and GPU attention can be overlapped:

Table W1. Per-component latency of collaborative attention (BS=12, per-layer average).

ComponentDeviceTime per layer
Key-Index scoringGPU0.0344 ms
Top-K selectionGPU0.0121 ms
D2H: query + visit planPCIe0.0094 ms
GPU attention over resident blocksGPU0.4584 ms
CPU attention over selected blocksCPU0.6082 ms
H2D: partial output + LSEPCIe0.0064 ms
LSE mergeGPU0.0020 ms

Thus we can see that the overhead of the predictor/gating mechanism (Key-Index scoring + Top-K selection) only occupies 2.63% of the collaborative-attention pipeline latency. And as CPU/GPU attention runs in parallel, most CPU time can be hidden while the GPU computes. We proved in Appendix D that although CPU attention has a slight overhead over GPU attention, it can still achieve peak throughput. We hope this helps to assess the true efficiency of the offload strategy.

AuthorsOpenReview ↗

Author Response (2/3) — W2: Conversational Fluency and Instruction Following

The absence of evaluations on benchmarks like MT-Bench, AlpacaEval, or LLM-as-a-judge metrics leaves a gap in understanding how sparse attention affects conversational fluency and instruction following.

We trained the sparse predictor on UltraChat, an open-domain multi-turn instruction dataset, precisely to preserve basic language capability and instruction following. The MATH/GPQA/AIME results in Fig. 4 are therefore already out-of-distribution tests for the selector.

Following the reviewer's suggestion, we directly evaluate on MT-Bench (multi-turn dialogue quality) and AlpacaEval (single-turn instruction following). Both require an LLM-as-a-judge metric, so we use the DeepSeek-V4.1-Flash-1017 API as the judge at temperature 0. For AlpacaEval, we report the preference rate relative to the archived GPT-4 reference answers. The table below shows Qwen3-4B with temperature set to 0.

Table W2. Conversational fluency and instruction following on Qwen3-4B (temperature 0).

Attention budgetMT-Bench score (0–10)AlpacaEval preference rate (%)
Full attention7.025029.3478
1,0247.087529.2857
2,0487.031329.2857

Scores stay close to full attention at every budget.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We thank the reviewers for recognizing the motivation behind RheoServe’s compute-offload design, CPU–GPU collaboration, and evaluation. Our additional experiments address the reviewers’ concerns and further demonstrate RheoServe’s efficiency, scalability, and stability.

Compute Offload Scalability and Server-Grade Multi-GPU Serving. In response to reviewer Q3zW’s question, we show that RheoServe outperforms ArkVale on H20 and H100 across different GPU, PCIe, and host-memory bandwidth ratios (Q3zW Table Q2a). With larger batches and expanded KV capacity, multi-GPU throughput scales superlinearly on H100 (Q3zW Table Q2b). Following reviewer rvg6’s suggestion, we demonstrate that scaling across more CPU cores and memory nodes accelerates attention, highlighting RheoServe’s scalability (rvg6 Tables W2a–W2b).

Benefits of Compute Offloading and Dynamic Cache Management. Following the suggestions of reviewers Q3zW and rvg6, we demonstrate that CPU migration extends context capacity beyond Quest’s and MagicPIG’s GPU cache limits (rvg6 W1; Q3zW W1 & Q1). The residency ablation shows higher throughput than HGCA’s FIFO policy, and our CPU kernel outperforms HGCA’s reproduction and MagicPIG (Q3zW Tables Q1b–Q1c). RheoServe preserves competitive accuracy, surpassing Quest on LongBench/AIME (rvg6 Table W1a), and exceeds MagicPIG/ShadowKV throughput within common supported generation intervals (Q3zW Table Q1a; rvg6 ShadowKV table). Together, compute offloading and dynamic cache management improve capacity and efficiency while preserving quality.

Mixed-Workload Stability and Fine-Grained Latency Analysis. In response to reviewer ozyP’s concerns, we evaluate a mix of long-prefill, long-decode, and combined long-prefill–long-decode workloads. We demonstrate higher throughput than vLLM even under round-robin scheduling, with controlled decode-latency variation even when round-robin scheduling deliberately suspends and resumes requests to stress-test cache migration (ozyP Tables W3a–W3b). Per-layer profiling shows GPU-resident indexing and distributed KV storage enable fast selection and overlapping CPU–GPU attention (ozyP Table W1). We further demonstrate that RheoServe preserves quality in dialogue as well as reasoning (ozyP Table W2; rvg6 Table W1a).

We thank the Area Chair and reviewers for their time and careful consideration of our work. We believe these experiments address the main concerns and further demonstrate RheoServe’s contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.