- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
On tightly coupled machines such as GH200, the GPU can read CPU memory directly over NVLink-C2C at comparable speed to the Grace CPU's own DRAM bandwidth. so leave the expert in CPU memory and let the GPU kernel execute it in place, with no copy and no CPU compute become feasible
the paper implements this with one mechanism and one policy. The mechanism is a heterogeneous tensor: all experts of a model form a single contiguous tensor allocated with mmap and exposed via torch::from_blob, with mbind binding each expert's address range to either the CPU or the GPU NUMA node. Swapping this tensor in for vLLM's expert parameters reuses vLLM's scheduler, KV cache, Triton fused-MoE kernels, and CUDA Graphs unchanged. The policy is deliberately minimal
- profile expert access counts on TruthfulQA, and per layer place the least-frequently-used experts in CPU memory for a target offload ratio
- No runtime migration, prefetching, prediction, or custom kernels
Critical strengths
- On GH200, GPU reads ~450 GB/s one-way C2C. it makes executing in place over the link feasible. With GB200 and Vera Rubin continuing the C2C design, the observation will age well
- the implementation is elegant, and the paper is upfront about its moving parts. One tensor, mmap + per-expert mbind + from_blob; no custom kernels, no migration daemon, no router prediction. Reusing vLLM's scheduler/KV/Triton kernels and retaining CUDA Graph compatibility
- the numbers are internally checkable, and the authors do the arithmetic calculation: measured 79.79 tok/s implies ≥289 GB/s sustained over C2C. This kind of self-consistent bound is rare and welcome
- Honest about where it loses. Prefill is not hidden: INM degrades with batch size, offload ratio, and input length
Critical weaknesses
-
the cross-system comparisons confound hardware, engine, and strategy — and the abstract claims them at face value
- ktransformers is measured on a different machine (x86 + H100 over PCIe, different CPU and DRAM) and a different engine (SGLang)
- INM outperforms existing offloading systems bundles three effects the reader cannot separate. the narrower result is more interesting: INM loses prefill, wins decode above 50% offload. The abstract and conclusion should claim that, not the broader version.
-
The algorithmic contribution is thin, and the paper knows it — the question is whether the venue should mind.
- The placement policy is explicitly "similar to that used in ktransformers": The delta is per-expert granularity + explicit NUMA binding (with the huge-page benefit). That is a worthwhile systems contribution, but the paper needs to argue it as such
-
No bound on how good static placement can be, and no stress test of profile mismatch. random placement is a weak comparator. Two cheap ablations would pin down the contribution: (i) an oracle placement profiled on the evaluation dataset itself; (ii) a mismatched profile, profile on ShareGPT, evaluate on SWE-bench
-
Evaluation scope is narrow for the generality of the claims: one GH200 node, one GPU, two Qwen models, fixed 128-token outputs, batch sizes 1/16/64, and no tail-latency or realistic-arrival-rate serving measurement — vLLM's continuous batching with variable lengths is the deployment the paper targets, and it is not what is measured
Questions for authors
For the comparisons with ktransformers and FreeToken, how much of the gap do you attribute to the placement strategy itself, as opposed to everything around it — for ktransformers, a different machine (PCIe vs. GH200) and a different serving engine (SGLang vs. vLLM); for FreeToken, its own serving system? The same-engine, same-hardware results in Sec. 3.1 seem to be the cleanest evidence — do you agree that is the comparison the headline claim should rest on?
How close is the TruthfulQA-profiled placement to an oracle placement profiled on the evaluation data itself, and how much does performance drop when the profiling and evaluation distributions differ (e.g., profile on ShareGPT, evaluate on SWE-bench)?
Did you compare against CUDA Unified Memory with cudaMemAdvise set to CPU-preferred, measured rather than argued from qualitative observations? Appendix A.1.3 gives qualitative reasons to prefer mbind (huge pages, no implicit migration), but on GH200 this is the most obvious alternative a practitioner would try first.
7 comments
Answer to the other questions
Q2. How close is the TruthfulQA-profiled placement to an oracle placement profiled on the evaluation data itself, and how much does performance drop when the profiling and evaluation distributions differ (e.g., profile on ShareGPT, evaluate on SWE-bench)?
A2. We added a breakdown of MoE execution time for FreeToken and our method in section 3.4 to explain the effectiveness of our method. For single-request workloads, our method achieves comparable latency to FreeToken on MMLU, ShareGPT, and Needle-in-a-Haystack. However, on SWE-bench, our method exhibits 19% slower latency due to the distribution shift in expert accesses. In contrast, for batched workloads, our method achieves more than 23% faster latency than FreeToken at batch size 8 and more than 28% faster latency at batch size 16. We attribute this advantage to the increased diversity of expert accesses across multiple requests as the batch size grows. This makes FreeToken less suitable for batched requests.
Q3. Did you compare against CUDA Unified Memory with cudaMemAdvise set to CPU-preferred, measured rather than argued from qualitative observations? Appendix A.1.3 gives qualitative reasons to prefer mbind (huge pages, no implicit migration), but on GH200 this is the most obvious alternative a practitioner would try first.
A3. In our preliminary experiments, we evaluated matrix multiplication with CUDA Unified Memory configured using CPU-preferred setting in addition to NUMA APIs. However, we observed unexpectedly high performance that could not be explained by accessing data residing in CPU memory over NVLink-C2C. This suggested that implicit page migration was occurring during execution, rather than the data remaining CPU-resident as intended. This access-counter-based migration is also reported in the paper titled "Harnessing integrated cpu-gpu system memory for hpc: a first look into grace hopper".
Follow-up on Q2: expert distribution shift
We have added the following to the Appendix of the revised PDF:
- Data distribution (Appendix A.2.1): We compare the expert access distributions of each evaluation dataset with the TruthfulQA profile, and evaluate SWE-bench, whose distribution differs most from the profile.
- Other model families (Appendix A.3): We evaluate Gemma 4 with Needle-in-a-Haystack and SWE-bench and obtain the similar results consistent with the main text.
In the single-request setting, the distribution shift does affect performance: on SWE-bench, the MoE-related kernel latency of our method is 19% higher than FreeToken (you can see in section 3.4, Figure 6). In the batch-request setting, our method keeps its advantage even on SWE-bench, since the expert accesses become more diverse across concurrent requests and FreeToken's LRU caching loses its locality, whereas static placement does not depend on per-request locality.
Thank you for reviewing our paper and there are updates for our paper
Thank you for your review. First, we would like to clarify the central claims of our work.
This work demonstrates that, for MoE serving on coupled CPU-GPU architectures such as the GH200, even a simple combination of two techniques can achieve substantial performance improvements on this new class of hardware:
Our central claim is that combining these two straightforward techniques is already effective for improving MoE serving performance on emerging coupled CPU-GPU architectures.
Furthermore, another important advantage of our approach is its ease of integration into existing, highly optimized inference frameworks. Specifically, on the GH200, both techniques can be implemented relatively easily in vLLM by replacing the torch.Tensor memory, without requiring the development of a specialized serving system from scratch, such as FreeToken with LRU cache.
These are the main points we aim to demonstrate in this work.
And, there are updates for our paper:
We have updated the experimental results to reflect the latest software versions, as there have been significant software updates since the paper was submitted. In particular, the previous version of FreeToken had a critical limitation for this work: it could not fully utilize the available NVLink-C2C bandwidth. The updated results incorporate the changes. All revisions have been reflected in the PDF from the Experiment section.
The main changes are summarized below:
MoE execution breakdown: We conducted additional experiments to analyze the breakdown of MoE-related execution time during decoding across multiple datasets and quantized models. We have added these results as a new Section 3.4. We also compared our method against FreeToken, our primary baseline. For single-request workloads, we observed an interesting result: when static placement works effectively, the MoE execution time of our method becomes comparable to that of FreeToken. This suggests that zero-copy execution on GH200 may consume a similar amount of memory bandwidth to the LRU cache management overhead in FreeToken. We also found that our method underperforms on some datasets in a single request. We have added further discussion of this limitation in Appendix 2.
Single-request performance: We found that FreeToken achieves nearly the same performance as our method for single requests, and even outperforms it on some datasets. This indicates that our method still has room for improvement in single-request scenarios. We have added a discussion of this point.
Batch-request performance: FreeToken and other method remains less suitable for batch requests. With multiple concurrent requests, the expert locality assumed by its LRU cache breaks down, increasing the data transfer overhead associated with expert caching.
Analysis of Distribution Shifts and Quantization: From appendix 2.1, we have added an analysis of the impact of MoE distribution shifts under single-request workloads, as well as an evaluation using quantized models.
And we answer the questions:
Q1: For the comparisons with ktransformers and FreeToken, how much of the gap do you attribute to the placement strategy itself, as opposed to everything around it — for ktransformers, a different machine (PCIe vs. GH200) and a different serving engine (SGLang vs. vLLM); for FreeToken, its own serving system? The same-engine, same-hardware results in Sec. 3.1 seem to be the cleanest evidence — do you agree that is the comparison the headline claim should rest on?
A1. We agree that the comparison in section 3.1 provides the cleanest evidence for isolating the effect of the placement strategy itself. In this comparison, both methods use the same serving framework and hardware, and the placement strategy is the primary difference, apart from prefetching. We therefore consider this comparison the most appropriate basis for our main claims regarding the placement strategy. Regarding ktransformers, as explained in the experiment section, we believe that the performance gap is influenced not only by its CPU-based execution but also substantially by the use of a PCIe-based machine rather than GH200. If feasible, we will additionally include experimental results for ktransformers running on GH200 to provide a fairer comparison under the same hardware conditions. From a hardware perspective, we expect ktransformers to underperform our method because it executes experts placed in CPU memory on the CPU, whose computational throughput (TFLOPS) is one to two orders of magnitude lower than that of the GPU. For FreeToken, we have added a more detailed comparison of MoE execution time breakdowns in section 3.4 to better isolate the differences between the two approaches.