acceptodds
Under review as a conference paper at ICLR 2027

Imbalance-aware NUMA Memory Placement: A Simple Approach To Mixture-of-Experts Serving on Tightly Coupled Architectures

Abstract

Serving large Mixture-of-Experts (MoE) models is constrained by limited GPU memory capacity, which motivates offloading experts to CPU memory. Existing offloading systems are designed for PCIe-connected machines, where the interconnect is much slower than host memory. They migrate experts to the GPU or execute CPU-resident experts on the CPU. However, on tightly coupled architectures, the GPU can directly access CPU memory through NVLink-C2C at a bandwidth comparable to that of host memory. In this work, we propose a heterogeneous tensor that uses NUMA APIs to place both CPU-resident and GPU-resident experts within a single tensor. This allows GPU kernels to directly execute CPU-resident experts without explicit expert migration. We further propose Imbalance-aware NUMA Memory Placement (INM), a simple static strategy that places each expert in CPU or GPU memory based on its precomputed access frequency. Since our approach only replaces the expert parameter tensor, we integrate it into vLLM and reuse its existing scheduler, KV cache management, and kernels. On GH200, our approach outperforms existing offloading systems in decode throughput for a single request at offload ratios of more than 50%. For batched requests, it achieves higher end-to-end throughput and lower latency than existing systems across all evaluated offload ratios. On the larger Qwen3.5-122B-A10B with 87.5% of experts offloaded, it achieves 1.27–1.46 higher decode throughput than previous research across diverse datasets. These results show that even a simple placement strategy enables efficient MoE serving on tightly coupled architectures.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Inference, Mixture-of-Expert, LLM Serving, Expert Offloading

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YQLQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

On tightly coupled machines such as GH200, the GPU can read CPU memory directly over NVLink-C2C at comparable speed to the Grace CPU's own DRAM bandwidth. so leave the expert in CPU memory and let the GPU kernel execute it in place, with no copy and no CPU compute become feasible

the paper implements this with one mechanism and one policy. The mechanism is a heterogeneous tensor: all experts of a model form a single contiguous tensor allocated with mmap and exposed via torch::from_blob, with mbind binding each expert's address range to either the CPU or the GPU NUMA node. Swapping this tensor in for vLLM's expert parameters reuses vLLM's scheduler, KV cache, Triton fused-MoE kernels, and CUDA Graphs unchanged. The policy is deliberately minimal

  • profile expert access counts on TruthfulQA, and per layer place the least-frequently-used experts in CPU memory for a target offload ratio
  • No runtime migration, prefetching, prediction, or custom kernels
Critical strengths
  • On GH200, GPU reads ~450 GB/s one-way C2C. it makes executing in place over the link feasible. With GB200 and Vera Rubin continuing the C2C design, the observation will age well
  • the implementation is elegant, and the paper is upfront about its moving parts. One tensor, mmap + per-expert mbind + from_blob; no custom kernels, no migration daemon, no router prediction. Reusing vLLM's scheduler/KV/Triton kernels and retaining CUDA Graph compatibility
  • the numbers are internally checkable, and the authors do the arithmetic calculation: measured 79.79 tok/s implies ≥289 GB/s sustained over C2C. This kind of self-consistent bound is rare and welcome
  • Honest about where it loses. Prefill is not hidden: INM degrades with batch size, offload ratio, and input length
Critical weaknesses
  • the cross-system comparisons confound hardware, engine, and strategy — and the abstract claims them at face value

    • ktransformers is measured on a different machine (x86 + H100 over PCIe, different CPU and DRAM) and a different engine (SGLang)
    • INM outperforms existing offloading systems bundles three effects the reader cannot separate. the narrower result is more interesting: INM loses prefill, wins decode above 50% offload. The abstract and conclusion should claim that, not the broader version.
  • The algorithmic contribution is thin, and the paper knows it — the question is whether the venue should mind.

    • The placement policy is explicitly "similar to that used in ktransformers": The delta is per-expert granularity + explicit NUMA binding (with the huge-page benefit). That is a worthwhile systems contribution, but the paper needs to argue it as such
  • No bound on how good static placement can be, and no stress test of profile mismatch. random placement is a weak comparator. Two cheap ablations would pin down the contribution: (i) an oracle placement profiled on the evaluation dataset itself; (ii) a mismatched profile, profile on ShareGPT, evaluate on SWE-bench

  • Evaluation scope is narrow for the generality of the claims: one GH200 node, one GPU, two Qwen models, fixed 128-token outputs, batch sizes 1/16/64, and no tail-latency or realistic-arrival-rate serving measurement — vLLM's continuous batching with variable lengths is the deployment the paper targets, and it is not what is measured

Questions for authors

For the comparisons with ktransformers and FreeToken, how much of the gap do you attribute to the placement strategy itself, as opposed to everything around it — for ktransformers, a different machine (PCIe vs. GH200) and a different serving engine (SGLang vs. vLLM); for FreeToken, its own serving system? The same-engine, same-hardware results in Sec. 3.1 seem to be the cleanest evidence — do you agree that is the comparison the headline claim should rest on?

How close is the TruthfulQA-profiled placement to an oracle placement profiled on the evaluation data itself, and how much does performance drop when the profiling and evaluation distributions differ (e.g., profile on ShareGPT, evaluate on SWE-bench)?

Did you compare against CUDA Unified Memory with cudaMemAdvise set to CPU-preferred, measured rather than argued from qualitative observations? Appendix A.1.3 gives qualitative reasons to prefer mbind (huge pages, no implicit migration), but on GH200 this is the most obvious alternative a practitioner would try first.

7 comments
AuthorsOpenReview ↗

Thank you for reviewing our paper and there are updates for our paper

Thank you for your review. First, we would like to clarify the central claims of our work.

This work demonstrates that, for MoE serving on coupled CPU-GPU architectures such as the GH200, even a simple combination of two techniques can achieve substantial performance improvements on this new class of hardware:

  1. Imbalance-aware NUMA Memory placement: Placing experts according to their access frequencies.
  2. Zero-copy execution: Executing directly on CPU-resident expert weights without explicitly transferring them to GPU memory.

Our central claim is that combining these two straightforward techniques is already effective for improving MoE serving performance on emerging coupled CPU-GPU architectures.

Furthermore, another important advantage of our approach is its ease of integration into existing, highly optimized inference frameworks. Specifically, on the GH200, both techniques can be implemented relatively easily in vLLM by replacing the torch.Tensor memory, without requiring the development of a specialized serving system from scratch, such as FreeToken with LRU cache.

These are the main points we aim to demonstrate in this work.

And, there are updates for our paper:

We have updated the experimental results to reflect the latest software versions, as there have been significant software updates since the paper was submitted. In particular, the previous version of FreeToken had a critical limitation for this work: it could not fully utilize the available NVLink-C2C bandwidth. The updated results incorporate the changes. All revisions have been reflected in the PDF from the Experiment section.

The main changes are summarized below:

  • MoE execution breakdown: We conducted additional experiments to analyze the breakdown of MoE-related execution time during decoding across multiple datasets and quantized models. We have added these results as a new Section 3.4. We also compared our method against FreeToken, our primary baseline. For single-request workloads, we observed an interesting result: when static placement works effectively, the MoE execution time of our method becomes comparable to that of FreeToken. This suggests that zero-copy execution on GH200 may consume a similar amount of memory bandwidth to the LRU cache management overhead in FreeToken. We also found that our method underperforms on some datasets in a single request. We have added further discussion of this limitation in Appendix 2.

  • Single-request performance: We found that FreeToken achieves nearly the same performance as our method for single requests, and even outperforms it on some datasets. This indicates that our method still has room for improvement in single-request scenarios. We have added a discussion of this point.

  • Batch-request performance: FreeToken and other method remains less suitable for batch requests. With multiple concurrent requests, the expert locality assumed by its LRU cache breaks down, increasing the data transfer overhead associated with expert caching.

  • Analysis of Distribution Shifts and Quantization: From appendix 2.1, we have added an analysis of the impact of MoE distribution shifts under single-request workloads, as well as an evaluation using quantized models.

And we answer the questions:

Q1: For the comparisons with ktransformers and FreeToken, how much of the gap do you attribute to the placement strategy itself, as opposed to everything around it — for ktransformers, a different machine (PCIe vs. GH200) and a different serving engine (SGLang vs. vLLM); for FreeToken, its own serving system? The same-engine, same-hardware results in Sec. 3.1 seem to be the cleanest evidence — do you agree that is the comparison the headline claim should rest on?

A1. We agree that the comparison in section 3.1 provides the cleanest evidence for isolating the effect of the placement strategy itself. In this comparison, both methods use the same serving framework and hardware, and the placement strategy is the primary difference, apart from prefetching. We therefore consider this comparison the most appropriate basis for our main claims regarding the placement strategy. Regarding ktransformers, as explained in the experiment section, we believe that the performance gap is influenced not only by its CPU-based execution but also substantially by the use of a PCIe-based machine rather than GH200. If feasible, we will additionally include experimental results for ktransformers running on GH200 to provide a fairer comparison under the same hardware conditions. From a hardware perspective, we expect ktransformers to underperform our method because it executes experts placed in CPU memory on the CPU, whose computational throughput (TFLOPS) is one to two orders of magnitude lower than that of the GPU. For FreeToken, we have added a more detailed comparison of MoE execution time breakdowns in section 3.4 to better isolate the differences between the two approaches.

AuthorsOpenReview ↗

Answer to the other questions

Q2. How close is the TruthfulQA-profiled placement to an oracle placement profiled on the evaluation data itself, and how much does performance drop when the profiling and evaluation distributions differ (e.g., profile on ShareGPT, evaluate on SWE-bench)?

A2. We added a breakdown of MoE execution time for FreeToken and our method in section 3.4 to explain the effectiveness of our method. For single-request workloads, our method achieves comparable latency to FreeToken on MMLU, ShareGPT, and Needle-in-a-Haystack. However, on SWE-bench, our method exhibits 19% slower latency due to the distribution shift in expert accesses. In contrast, for batched workloads, our method achieves more than 23% faster latency than FreeToken at batch size 8 and more than 28% faster latency at batch size 16. We attribute this advantage to the increased diversity of expert accesses across multiple requests as the batch size grows. This makes FreeToken less suitable for batched requests.

Q3. Did you compare against CUDA Unified Memory with cudaMemAdvise set to CPU-preferred, measured rather than argued from qualitative observations? Appendix A.1.3 gives qualitative reasons to prefer mbind (huge pages, no implicit migration), but on GH200 this is the most obvious alternative a practitioner would try first.

A3. In our preliminary experiments, we evaluated matrix multiplication with CUDA Unified Memory configured using CPU-preferred setting in addition to NUMA APIs. However, we observed unexpectedly high performance that could not be explained by accessing data residing in CPU memory over NVLink-C2C. This suggested that implicit page migration was occurring during execution, rather than the data remaining CPU-resident as intended. This access-counter-based migration is also reported in the paper titled "Harnessing integrated cpu-gpu system memory for hpc: a first look into grace hopper".

AuthorsOpenReview ↗

Follow-up on Q2: expert distribution shift

We have added the following to the Appendix of the revised PDF:

  • Data distribution (Appendix A.2.1): We compare the expert access distributions of each evaluation dataset with the TruthfulQA profile, and evaluate SWE-bench, whose distribution differs most from the profile.
  • Other model families (Appendix A.3): We evaluate Gemma 4 with Needle-in-a-Haystack and SWE-bench and obtain the similar results consistent with the main text.

In the single-request setting, the distribution shift does affect performance: on SWE-bench, the MoE-related kernel latency of our method is 19% higher than FreeToken (you can see in section 3.4, Figure 6). In the batch-request setting, our method keeps its advantage even on SWE-bench, since the expert accesses become more diverse across concurrent requests and FreeToken's LRU caching loses its locality, whereas static placement does not depend on per-request locality.

Reviewer_cGdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents INM, a static expert-placement scheme for MoE inference on tightly coupled CPU–GPU systems. It uses NUMA APIs to place experts within one tensor across CPU memory and GPU HBM, allowing existing GPU kernels to access CPU-resident weights directly. Experts are ranked by access frequencies profiled on TruthfulQA. Experiments on GH200 show gains over random placement and module-level UVA offloading, alongside competitive performance against existing serving systems. On Qwen3.5-122B-A10B with 87.5% offloading, INM reports 1.27–1.46× higher decode throughput than FreeToken.

Critical strengths
  1. Expert placement avoids explicit migration and reuses existing MoE kernels, scheduling, and KV-cache management.

  2. On the same GH200 and vLLM stack, INM outperforms random placement and module-level UVA offloading across the tested partial-offload settings.

  3. Results span two model sizes, several datasets, and single-request and batched execution, while identifying where prefetching remains preferable.

Critical weaknesses
  1. Direct GPU access to host memory and NUMA placement are established GH200 capabilities, and Section 2.3 explicitly adopts an existing frequency-based policy. The heterogeneous tensor is useful engineering, but the paper offers limited new insight into placement beyond applying these mechanisms to MoE serving.

  2. Batched GEMMs reuse expert weights across tokens, so memory traffic need not scale with routing counts. Algorithm 1 also assigns the same offload fraction to every layer, regardless of differences in access skew. Neither choice is justified against the actual latency saved per unit of HBM.

  3. Fixed placement favors decode but can lose to prefetching on long prefills, as the paper’s own results show. It lacks a cost model explaining when direct access is preferable to staging weights in HBM.

  4. Section 3.3 reports a 3.0–4.8× advantage over FreeToken even across settings that include zero offloading, and attributes much of it to vLLM batching. Those gains cannot establish the benefit of INM itself; the same-engine results support a narrower claim.

  5. Testing several datasets is useful, but the paper does not report how expert rankings change from TruthfulQA or how much performance is lost relative to workload-specific placement. This matters because INM may not adapt once serving begins.

Questions for authors
  1. What is the main technical advance beyond established NUMA memory access and frequency-based expert placement?

  2. Why use token counts and a uniform per-layer offload ratio rather than estimated memory traffic or latency savings?

  3. Can a cost model identify when direct access should give way to prefetching?

  4. How much of the FreeToken speedup remains after accounting for the zero-offload engine gap?

  5. How close is TruthfulQA-based placement to workload-specific placement on each evaluation dataset?

5 comments
AuthorsOpenReview ↗

Thank you for reviewing our paper and there are updates for our paper

Thank you for your review. First, we would like to clarify the central claims of our work.

This work demonstrates that, for MoE serving on coupled CPU-GPU architectures such as the GH200, even a simple combination of two techniques can achieve substantial performance improvements on this new class of hardware:

  1. Imbalance-aware NUMA Memory placement: Placing experts according to their access frequencies.
  2. Zero-copy execution: Executing directly on CPU-resident expert weights without explicitly transferring them to GPU memory.

Our central claim is that combining these two straightforward techniques is already effective for improving MoE serving performance on emerging coupled CPU-GPU architectures.

Furthermore, another important advantage of our approach is its ease of integration into existing, highly optimized inference frameworks. Specifically, on the GH200, both techniques can be implemented relatively easily in vLLM by replacing the torch.Tensor memory, without requiring the development of a specialized serving system from scratch, such as FreeToken with LRU cache.

These are the main points we aim to demonstrate in this work.

And, there are updates for our paper:

We have updated the experimental results to reflect the latest software versions, as there have been significant software updates since the paper was submitted. In particular, the previous version of FreeToken had a critical limitation for this work: it could not fully utilize the available NVLink-C2C bandwidth. The updated results incorporate the changes. All revisions have been reflected in the PDF from the Experiment section.

The main changes are summarized below:

  • MoE execution breakdown: We conducted additional experiments to analyze the breakdown of MoE-related execution time during decoding across multiple datasets and quantized models. We have added these results as a new Section 3.4. We also compared our method against FreeToken, our primary baseline. For single-request workloads, we observed an interesting result: when static placement works effectively, the MoE execution time of our method becomes comparable to that of FreeToken. This suggests that zero-copy execution on GH200 may consume a similar amount of memory bandwidth to the LRU cache management overhead in FreeToken. We also found that our method underperforms on some datasets in a single request. We have added further discussion of this limitation in Appendix 2.

  • Single-request performance: We found that FreeToken achieves nearly the same performance as our method for single requests, and even outperforms it on some datasets. This indicates that our method still has room for improvement in single-request scenarios. We have added a discussion of this point.

  • Batch-request performance: FreeToken and other method remains less suitable for batch requests. With multiple concurrent requests, the expert locality assumed by its LRU cache breaks down, increasing the data transfer overhead associated with expert caching.

  • Analysis of Distribution Shifts and Quantization: From appendix 2.1, we have added an analysis of the impact of MoE distribution shifts under single-request workloads, as well as an evaluation using quantized models.

And we answer the questions:

Q1. What is the main technical advance beyond established NUMA memory access and frequency-based expert placement?

A1. The main technical advance beyond NUMA memory management and frequency-based expert placement is demonstrating that GH200's hardware capabilities can be leveraged to implement efficient MoE serving with offloading while reusing existing inference frameworks. We did not implement our serving systems from scratch. Our approach requires only modifications to memory placement to exploit GH200's heterogeneous memory architecture. Consequently, it can retain and reuse existing implementations of CUDA Graphs, quantization, and computational kernels without requiring framework-specific modifications to these components. This is enabled by our heterogeneous_tensor implementation, which we describe and make publicly available in Appendix 1.2.

Q2. Why use token counts and a uniform per-layer offload ratio rather than estimated memory traffic or latency savings?

A2. In our preliminary experiments with MoE Grouped GEMM kernels, we confirmed that experts that are not selected for execution incur no memory accesses and therefore do not incur additional NUMA remote-access latency. Based on this observation, we adopted static placement as a proxy for predicting MoE memory access patterns.

AuthorsOpenReview ↗

Answer to the other three questions

Q3. Can a cost model identify when direct access should give way to prefetching?

A3. We leave this as an interesting direction for future work. A potential approach is to estimate the average number of expert cache misses based on the input length, particularly during prefill, where longer input sequences are expected to increase the number of accessed experts and consequently the likelihood of cache misses. By estimating the resulting memory access latency, it may be possible to determine when direct access should give way to prefetching. However, developing a reliable prediction model requires further investigation.

Q4. How much of the FreeToken speedup remains after accounting for the zero-offload engine gap?

A4. We added a breakdown of MoE execution time for FreeToken and our method in Section 3.4 to explain the effectiveness of our method. For single-request workloads, our method achieves comparable latency to FreeToken on MMLU, ShareGPT, and Needle-in-a-Haystack. However, on SWE-bench, our method exhibits 19% higher latency due to the distribution shift in expert accesses. In contrast, for batched workloads, our method achieves more than 23% faster latency than FreeToken at batch size 8 and more than 28% faster latency at batch size 16. We attribute this advantage to the increased diversity of expert accesses across multiple requests as the batch size grows. This makes FreeToken less suitable for batched requests.

Q5. How close is TruthfulQA-based placement to workload-specific placement on each evaluation dataset?

Q5. We added experiments on expert access distributions in Appendix 2.1. The MMLU dataset exhibits an access distribution somewhat similar to TruthfulQA, whereas Needle-in-a-Haystack, which we use as our primary evaluation benchmark, shows a somewhat different distribution. The impact of these distribution differences on MoE-related kernel execution is also discussed in Section 3.4. Our results suggest that significant performance degradation is unlikely unless the expert access distribution deviates substantially, as observed with the SWE-bench dataset. Nevertheless, we acknowledge that there is still room for improvement, particularly for small batch sizes.

Appreciate the authors' prompt responses. After reading them carefully, I still have the following concerns, mainly about the robustness of the work:

  1. How well does INM perform on different hardware with NUMA support?

  2. For Q3, perhaps a heuristic cost model from a system perspective could work as you mentioned in A3.

Reviewer_D8dtOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Imbalance-aware NUMA Memory Placement (INM) for efficient Mixture-of-Experts (MoE) serving on tightly coupled CPU–GPU architectures. INM uses offline expert-access frequencies to place frequently accessed experts in GPU memory and less frequently accessed experts in CPU memory, enabling zero-copy execution over NVLink-C2C. The approach integrates with vLLM without redesigning its scheduler, KV-cache management, or execution kernels. Experiments on GH200 demonstrate improved decode throughput and end-to-end serving performance compared with several existing offloading approaches, particularly at high offload ratios.

Critical strengths

Timely and relevant problem: The paper addresses an important architectural shift in heterogeneous computing, where tightly coupled CPU–GPU systems challenge traditional MoE offloading assumptions. The motivation is compelling, as large MoE models have substantial memory requirements despite sparse expert activation.

Simple approach with practical integration: INM uses a straightforward frequency-based placement mechanism without runtime prediction or dynamic expert migration, making the design easy to understand. Its integration into vLLM without redesigning the scheduler, KV-cache management, or fused-MoE kernels further strengthens its practical relevance.

Critical weaknesses

Confounding factors in cross-system comparisons: INM is evaluated using vLLM, while ktransformers and FreeToken use different execution stacks, scheduling mechanisms, and caching policies. Consequently, the reported throughput and latency improvements may partly reflect differences in serving-engine efficiency rather than the proposed memory placement strategy itself. This makes it difficult to isolate INM's contribution to the reported performance gains.

Insufficient ablation of the proposed components: Although the random expert-placement baseline is useful, the evaluation does not sufficiently separate the benefits of zero-copy execution from those of frequency-aware placement. Additional controlled ablations and stronger placement baselines are needed to establish the individual contributions of these design choices.

Questions for authors

In Figures 4 and 5, how much of the performance improvement can be attributed specifically to INM rather than differences between serving engines? Could the authors provide normalized results relative to each system's no-offloading baseline or additional controlled comparisons?

Could the authors provide ablations that independently evaluate zero-copy execution and frequency-aware expert placement? How would INM compare with stronger static placement strategies under the same GPU memory budget?

4 comments
AuthorsOpenReview ↗

Thank you for reviewing our paper and there are updates for our paper

Thank you for your review. First, we would like to clarify the central claims of our work.

This work demonstrates that, for MoE serving on coupled CPU-GPU architectures such as the GH200, even a simple combination of two techniques can achieve substantial performance improvements on this new class of hardware:

  1. Imbalance-aware NUMA Memory placement: Placing experts according to their access frequencies.
  2. Zero-copy execution: Executing directly on CPU-resident expert weights without explicitly transferring them to GPU memory.

Our central claim is that combining these two straightforward techniques is already effective for improving MoE serving performance on emerging coupled CPU-GPU architectures.

Furthermore, another important advantage of our approach is its ease of integration into existing, highly optimized inference frameworks. Specifically, on the GH200, both techniques can be implemented relatively easily in vLLM by replacing the torch.Tensor memory, without requiring the development of a specialized serving system from scratch, such as FreeToken with LRU cache.

These are the main points we aim to demonstrate in this work.

And, there are updates for our paper:

We have updated the experimental results to reflect the latest software versions, as there have been significant software updates since the paper was submitted. In particular, the previous version of FreeToken had a critical limitation for this work: it could not fully utilize the available NVLink-C2C bandwidth. The updated results incorporate the changes. All revisions have been reflected in the PDF from the Experiment section.

The main changes are summarized below:

  • MoE execution breakdown: We conducted additional experiments to analyze the breakdown of MoE-related execution time during decoding across multiple datasets and quantized models. We have added these results as a new Section 3.4. We also compared our method against FreeToken, our primary baseline. For single-request workloads, we observed an interesting result: when static placement works effectively, the MoE execution time of our method becomes comparable to that of FreeToken. This suggests that zero-copy execution on GH200 may consume a similar amount of memory bandwidth to the LRU cache management overhead in FreeToken. We also found that our method underperforms on some datasets in a single request. We have added further discussion of this limitation in Appendix 2.

  • Single-request performance: We found that FreeToken achieves nearly the same performance as our method for single requests, and even outperforms it on some datasets. This indicates that our method still has room for improvement in single-request scenarios. We have added a discussion of this point.

  • Batch-request performance: FreeToken and other method remains less suitable for batch requests. With multiple concurrent requests, the expert locality assumed by its LRU cache breaks down, increasing the data transfer overhead associated with expert caching.

  • Analysis of Distribution Shifts and Quantization: From appendix 2.1, we have added an analysis of the impact of MoE distribution shifts under single-request workloads, as well as an evaluation using quantized models.

And we answer the questions

Q1. In Figures 4 and 5, how much of the performance improvement can be attributed specifically to INM rather than differences between serving engines? Could the authors provide normalized results relative to each system's no-offloading baseline or additional controlled comparisons?

A1. In Section 3.4, we present a comparison of the MoE components during decoding between FreeToken and our method, which are the primary approaches evaluated in this study. The results show that, for single requests, our method achieves comparable performance to FreeToken when static placement is effective. In contrast, for batched requests, FreeToken's dynamic expert placement incurs substantial overhead from LRU cache management, making it less efficient.

Q2. Could the authors provide ablations that independently evaluate zero-copy execution and frequency-aware expert placement? How would INM compare with stronger static placement strategies under the same GPU memory budget?

A2. In Figure 3, the experimental results include not only random placement but also vLLM's Unified Virtual Addressing (UVA). Please let us know if these results do not adequately address the question.

Reviewer_D8dtOpenReview ↗

Follow-up on Author Response and Additional Experiments

Thank you for the detailed response and additional experiments. The MoE execution-time breakdown provides useful insights into the performance differences between INM and FreeToken.

Regarding Q1, while the additional analysis helps explain the results, differences in serving engines may still influence the end-to-end comparisons.

Regarding Q2, the random placement and UVA baselines are useful, but they do not fully isolate the contributions of zero-copy execution and frequency-aware placement.

Overall, the revisions strengthen the paper and partially address my concerns, although clearer attribution of the performance improvements would further strengthen the evaluation. I would like to hold my current scores.

AuthorsOpenReview ↗

Follow-up on Q1

Regarding the concern raised in Q1 about the E2E comparisons for batched requests, we examined the breakdown of MoE-related kernel execution times in the revised Figure 6, particularly at batch sizes of 8 and 16. The results confirm that FreeToken's dynamic LRU cache management incurs substantial overhead during batched decode. This provides evidence that our proposed method is effective for improving the decode performance of batched requests, particularly by reducing the overhead during the decode phase. We believe this is a valuable result, especially considering that previous work reports that decoding accounts for over 90% of the execution time in applications such as ReAct-style agents [1]. Therefore, we expect our method to be effective in terms of E2E performance for decode-heavy workloads. Regarding prefill, we acknowledge that there is still room for further discussion. To address this, we added a comparison between FreeToken and our method in the Appendix. Based on this analysis, we found that the choice between prefetching and our method for prefill should depend on the Expert cache hit ratio and input sequence length. These additional experiments, also motivated by Reviewer cGdt's comment, are presented in Appendix A.4, PREDICTING TTFT WITH AN ANALYTICAL MODEL.

Although we acknowledge that differences in serving engine implementations may influence the E2E comparisons, we believe that our results demonstrate the potential of tightly coupled architectures for MoE serving. In particular, as we can see, our proposed method, implemented on a tightly coupled architecture, outperforms those existing serving engines in batch requests that incorporate optimized MoE offloading techniques and are implemented from scratch. We consider this an important contribution toward establishing tightly coupled architectures as a promising direction for building memory-efficient LLM inference systems.

[1] Yuan, Y., Nayak, A., Kundu, S., & Talati, N. (2026). Agentic AI Workload Characteristics. arXiv preprint arXiv:2605.26297.

AuthorsOpenReview ↗

Follow-up on Q2

We appreciate the reviewer's insightful comment. To better understand the intended ablation study, we would like to ask whether the following experimental configurations would address your concern:

ConfigurationExpert ExecutionExpert Placement
A (Ours)Zero-copyFrequency-aware
BZero-copyRandom
CExplicit copyingFrequency-aware
DExplicit copyingRandom

Regarding expert execution, although the serving engines differ, our existing kernel-level breakdown of decode performance already provides a comparison between zero-copy execution in our method and explicit expert copying in FreeToken. In particular, it illustrates the overhead associated with explicit expert copying during batched decode.

Regarding expert placement, we would like to clarify whether the combination of random placement and explicit copying is a meaningful experimental configuration. Our understanding is that explicit expert copying is typically coupled with a caching algorithm, such as LRU or LFU (Least Frequently Used), which dynamically determines which experts should reside in GPU memory based on their access patterns. In this sense, we are unsure how random placement would be defined when combined with explicit copying, since the caching algorithm itself determines the expert placement in GPU memory. Could you please clarify whether this is the intended experimental setting? We believe that random placement is meaningful primarily in the context of zero-copy execution with static placement.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their reviews. Below is a summary of the changes in the revised PDF.

1. Updated all measurements to the latest version of FreeToken

On October 7, after the ICLR submission deadline, an update to FreeToken was merged that improves NVLink-C2C bandwidth utilization. We consider this update significant, so we replaced our data with measurements taken on the latest version. With the update, both systems now reach nearly the same decode tokens/s on Needle-in-a-Haystack in the single-request setting. We think this happens because both systems now make full use of the NVLink-C2C bandwidth when transferring the experts offloaded to the CPU.

2. Added a breakdown of MoE-related kernels during decoding

For a more detailed analysis, we added a section on the execution time of MoE-related computation. This analysis shows that our approach has limitations on some datasets in the single-request setting, and that it holds a clear advantage in the batch-request setting.

3. Added Appendix sections on data distributions, quantization, and other model families

Data distributions:

On datasets such as SWE-bench, where the expert distribution differs greatly from the other datasets, performance drops in the single-request setting. Our approach still keeps its advantage even on SWE-bench in the batch-request setting.

Quantization:

Our method outperforms FreeToken. However, in the analysis of MoE-related kernels, this results not from MoE-related kernels but from the other optimized kernel in vLLM. This supports a key advantage of our approach: it enables MoE offloading while reusing the rest of vLLM’s software stack.

Other model families:

Experiments on Gemma 4 give results consistent with those in the main text.

4. Fixed an algorithm

We identified a discrepancy between Algorithm 1 and our actual implementation. Specifically, the previous pseudocode incorrectly described expert selection as being performed independently for each layer. In our implementation, we globally rank all experts across layers based on their router selection frequencies and select the least frequently used ⌊rLE⌋\lfloor rLE \rfloor experts for CPU offloading. We have corrected Algorithm 1 accordingly. This correction only clarifies the algorithmic description and does not affect the implementation, experimental results, or conclusions.

AuthorsOpenReview ↗

Summary of Discussion (2/2)

We would also like to highlight several updates since our initial submission.

1. Updated all measurements to the latest version of FreeToken

On October 7, after the ICLR submission deadline, an update to FreeToken was merged that improves NVLink-C2C bandwidth utilization. We consider this update significant, so we replaced our data with measurements taken on the latest version. With the update, both systems now reach nearly the same decode tokens/s on Needle-in-a-Haystack in the single-request setting. We think this happens because both systems now make full use of the NVLink-C2C bandwidth when transferring the experts offloaded to the CPU.

2. New section of a breakdown of MoE-related kernels during decoding

In response to concerns raised by all reviewers regarding the serving system, we added a dedicated section analyzing MoE-related kernels in isolation. This allows us to eliminate the effects of KV-cache management, scheduling, and FlashAttention implementations, enabling us to examine how our approach affects MoE-related computations specifically. This analysis shows that our approach has limitations on some datasets in the single-request setting, and that it holds a clear advantage in the batch-request setting.

3. Added Appendix sections on data distributions, quantization, other model families, and TTFT analysis

Data distributions:

On datasets such as SWE-bench, where the expert distribution differs greatly from the other datasets, performance drops in the single-request setting. Our approach still keeps its advantage even on SWE-bench in the batch-request setting (Please refer to the section on the breakdown analysis of MoE-related kernels).

Quantization:

Our method outperforms FreeToken. However, in the analysis of MoE-related kernels, this results not from MoE-related kernels but from the other optimized kernel in vLLM. This supports a key advantage of our approach: it enables MoE offloading while reusing the rest of vLLM’s software stack.

Other model families:

Experiments on Gemma 4 give results consistent with those in the main text. This further confirmed the effectiveness of our approach for batched requests across a different model family.

TTFT analysis:

In response to Reviewer cGdt's suggestion to develop a cost model that could help determine when to switch between prefetching and our method, we developed a theoretical execution-time model for TTFT and compared its predictions against actual measurements. Our model predicts which of the two approaches, prefetching or our method, achieves faster TTFT with over 68% accuracy. While we discuss in the main paper how the TTFT slowdown can be addressed through prefill-decode disaggregation, this appendix provides a theoretical basis for deciding which server a request should be routed to.

Finally, we believe this work is a first step toward MoE offloading that effectively exploits CPU memory on tightly coupled architectures. We demonstrated that our approach can be implemented with only minimal modifications to the existing serving framework. Furthermore, our results show that the benefits of our approach extend across different model architectures and are particularly significant for batched workloads. Although challenges remain in single-request workloads and TTFT, we believe these findings establish the potential of tightly coupled CPU-GPU architectures as an effective platform for memory-efficient and high-throughput MoE serving. We hope this work will inspire future research on adaptive MoE serving systems that address these challenges while further exploiting the characteristics of tightly coupled architectures.

Sincerely, Authors

AuthorsOpenReview ↗

Summary of Discussion (1/2)

Dear Reviewers, AC, and SAC,

We would like to thank you for reviewing our paper. We would like to highlight some important updates and our contributions as follows. It would be great if you take these into consideration for your final decision.

First, we would like to clarify the central claims of our work.

This work demonstrates that, for MoE serving on coupled CPU-GPU architectures such as the GH200, even a simple combination of two techniques can achieve substantial performance for serving on this new class of hardware:

Imbalance-aware NUMA Memory placement: Placing experts according to their access frequencies.

Zero-copy execution: Executing directly on CPU-resident expert weights without explicitly transferring them to GPU memory. Our central claim is that combining these two straightforward techniques is already effective for improving MoE serving performance on emerging coupled CPU-GPU architectures.

As also mentioned in our response to Reviewer cGdt, this system was conceived based on the bandwidth differences between conventional PCIe-based systems and NVLink-C2C-based systems.

Furthermore, another important advantage of our approach is its ease of integration into existing, highly optimized inference frameworks. Specifically, on the GH200, both techniques can be implemented relatively easily in vLLM by replacing the torch.Tensor memory, without requiring the development of a specialized serving system from scratch, such as FreeToken with LRU cache.

In this work, we demonstrate that, by leveraging a tightly coupled architecture to enable imbalance-aware NUMA memory placement and zero-copy execution, our approach offers advantages over other MoE offloading serving systems for batched requests. Although our approach has limitations for single-request workloads, particularly regarding distribution shifts and TTFT, we believe this work is valuable in demonstrating the potential to serve a larger number of requests with limited memory resources compared to existing systems. Furthermore, as we have repeatedly emphasized, our approach can be implemented simply by replacing existing torch.Tensor with NUMA memory allocations.

These are the main points we aim to demonstrate in this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.