acceptodds
Withdrawn from ICLR 2027

DyS-MoE: Virtualizing Dynamic Expert Execution with Static Windows for Resource-Constrained MoE Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-Experts (MoE) inference under limited GPU memory relies on expert offloading. Dynamic execution exploits heterogeneous resources but incurs frequent transfers and scheduling overhead, while static expert placement simplifies execution at the cost of GPU utilization. We present DyS-MoE, which uses selective expert replacement to serve dynamic demand through reusable execution windows. Its core abstraction, a Virtualized Static Execution Graph, preserves operations and dependencies while expert bindings change. Slotted Grouped Execution provides stable GPU cache and execution slots, and Importance-Aware Miss Handling maps residency misses and capacity overflow onto these resources while reserving CPU exact computation for sensitive contributions. Together, these mechanisms enable CUDA Graph replay under dynamic expert offloading. On an RTX 4090 at batch 8, DyS-MoE achieves 431.99 and 551.06 decode tokens/s on Qwen3-30B-A3B and DeepSeek-V2-Lite-Chat, respectively, reaching 2.28× and 6.90× the throughput of state-of-the-art baselines while maintaining competitive task accuracy.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, Efficient LLM Inference, Expert Offloading, CUDA Graphs, Expert Replacement, Resource-Constrained Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kNqsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DyS-MoE targets MoE inference under limited GPU memory. The core idea is to decouple dynamic expert demand from execution structure: a fixed execution window on the GPU (operator sequence and dependencies) supports CUDA Graph replay, while expert identity becomes a mutable binding inside the window. A miss (expert absent from the GPU, or slot full) is served by a similar resident expert, with a small number of sensitive requests computed exactly on the CPU. The three mechanisms are the Virtualized Static Execution Graph, Slotted Grouped Execution, and Importance-Aware Miss Handling. On an RTX 4090 at batch 8, it reaches 431.99 and 551.06 tokens/s on Qwen3-30B-A3B and DeepSeek-V2-Lite, 2.28× and 6.90× the fastest evaluated baseline, with roughly 3 points of accuracy loss under the default configuration.

Critical strengths
  • The problem is real and well-motivated. Launch and scheduling overhead is a genuine bottleneck in dynamic MoE offloading, so fixing the execution structure for reuse is a sensible goal.
  • The ablations cleanly separate the roles of the components: graph replay carries throughput, CPU exact computation carries accuracy, and capacity grouping is a minor optimization.
  • Execution consistency is verified, and the authors disclose several limitations, which is honest.
Critical weaknesses
  1. The core idea combines known techniques, and the novelty boundary is not stated. Serving misses with similar experts comes from BuddyMoE and SERE, and a fixed execution structure plus CUDA Graph replay already appears in KTransformers. The authors narrow the novelty to a "joint design" of replacement, slot capacity, and execution windows, but the paper should say what is new in the mechanism beyond engineering integration and tuning, and how it differs from closely related work.

  2. The accuracy-throughput trade-off is not reported clearly. The reported speedups hide the accuracy and memory cost. Cache ratio, CPU budget, and prefetch degree all move along the accuracy-throughput frontier, but the paper gives no joint view. A larger cache ratio lowers throughput, and removing CPU exact computation raises throughput while DeepSeek accuracy drops sharply; so the "competitive accuracy" is bought back with CPU exact computation and cache, a cost not reflected in the abstract or conclusion.

  3. Prefetch, a listed contribution, is net-negative in its own ablation and is not explained. Removing prefetch raises DeepSeek throughput by about 46% and slightly improves HumanEval, and the benefit is non-monotonic in the prefetch degree. The paper only says prefetch is optional and workload-dependent, without explaining why it hurts in the default configuration or when it helps. A component that is better turned off by default should not be listed as a core contribution.

  4. The expert-redundancy assumption the method rests on is not quantified, so the quality guarantee lacks a principled basis. The whole accuracy story depends on experts being redundant and replaceable, but the paper does not measure redundancy; the replacement error is fit on a small calibration set with no upper bound. DeepSeek collapses in accuracy at a small cache while Qwen stays stable, which is exactly a failure of the redundancy assumption, and the paper does not explain this asymmetry. Quality evaluation is also limited to GSM8K and HumanEval, so the side effects of per-token replacement are not covered.

Questions for authors
  1. What exactly is the novelty, and where does it lie beyond the combination of known replacement and graph-replay techniques?
  2. What exactly is the accuracy-throughput-memory trade-off, and how does it depend on cache ratio, CPU budget, prefetch degree, and model architecture or parameter count? Please quantify it.
  3. Why is prefetch net-negative in the default configuration? Under what conditions does it help, and what criterion decides whether to enable it?
  4. How is expert redundancy quantified? Does the replacement error have an upper bound? Why does DeepSeek collapse at a small cache while Qwen stays stable?
Reviewer_DFUaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies MoE decoding on a consumer GPU with experts offloaded to the CPU. Its main idea is to replace a missing expert with a similar resident one. This keeps the computation graph fixed across steps, so each layer can be replayed with CUDA Graphs. The authors combine this with fixed-capacity expert slots and a policy that sends a few important misses to the CPU for exact computation. On an RTX 4090 with 40% of experts resident, they report 2.28× (Qwen3-30B-A3B) and 6.90× (DeepSeek-V2-Lite) the throughput of the best baseline at batch 8. GSM8K and HumanEval accuracy stays within a few points of the original model.

Critical strengths
  1. Keeping the graph fixed by changing only expert bindings is a simple way to bring CUDA Graph replay into offloaded inference. Table 2 shows that replay alone gives 2.76× (Qwen) and 4.06× (DeepSeek), and A.4 confirms that replay and direct execution produce the same outputs.
  2. The policy comparison within the same runtime (Table 7) is well designed. Random selection drops Qwen HumanEval from 87.2% to 76.2%, so the prioritization clearly matters.
  3. The algorithm is fully specified. The appendix reports raw counts, memory use, and results that do not favor the method, which makes the work easy to assess and reproduce.
Critical weaknesses
  1. DyS-MoE replaces missing experts while the baselines compute exact ones, so part of the speedup comes from doing less work. Qwen even gets faster as residency drops (Table 1). An exact upper bound and a simple approximate baseline under the same replay are needed to attribute the gains to the execution design.
  2. Both models appear to fit on the 4090 at full residency (Table 9), so the offloading setting is emulated rather than necessary. All results also come from a single hardware setup.
  3. The 6.90× DeepSeek speedup relies on an odd KTransformers dip at batch 8 (Table 4); against neighboring batch sizes it is closer to 4.4–4.8×. The same Qwen configuration also ranges from 365 to 442 tok/s across Tables 1, 2, and 8, with no variance reported.
  4. Calibration prompts match each test task, and many accuracy differences are only a few problems out of 164, with no confidence intervals. Quality is also measured only at batch 8.
Questions for authors
  1. What are the throughput and accuracy of a fully resident exact setup? What are they for top-k restricted to resident experts with the same replay?
  2. Were the main throughput numbers measured with a single prompt repeated across lanes (Appendix C.1)? Could you report throughput on the actual task batches?
  3. What explains the KTransformers dip at batch 8? What explains the spread in DyS-MoE throughput across Tables 1, 2, and 8?
  4. How does accuracy change at other batch sizes? How often does a request’s output depend on the other requests in its batch?
  5. Why are the DeepSeek reference scores (59.06% GSM8K, 34.76% HumanEval) well below those in the DeepSeek-V2 report?
Reviewer_gSkcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DyS-MoE accelerates memory-constrained MoE inference by combining reusable execution windows, fixed-capacity expert slots, and importance-aware miss handling. These mechanisms enable CUDA Graph replay under dynamic expert offloading. Some cache misses and capacity overflows are handled through approximate expert replacement, while selected important assignments receive exact CPU computation.

Critical strengths

S1: The paper addresses an important practical problem: efficient MoE inference with limited GPU memory.

S2: It compares against relevant systems, including KTransformers, HybriMoE, and llama.cpp, and reports substantial throughput improvements. Component ablations provide evidence for the benefits of execution reuse.

S3: The appendix provides extensive implementation details, calibration procedures, execution budgets, and experimental configurations.

Critical weaknesses

W1: The performance comparisons involve different accuracy guarantees. DyS-MoE replaces some router-selected experts with approximate substitutes, whereas the compared baselines retain the requested experts. Reported accuracy drops reach 3.49 percentage points at the default cache setting and become considerably larger at lower residency. Without comparisons at matched quality and memory budgets, the headline speedups do not establish an advantage for semantics-preserving offloading.

W2: The central insight and novelty need clearer presentation. The contributions are described through broad abstractions such as virtualized execution and importance-aware handling. The paper should explain more clearly what prevents existing graph-replay and expert-replacement techniques from achieving the same result, and which new design principle resolves that limitation.

W3: The evaluation provides limited evidence of generality. Quality evaluation focuses on GSM8K and HumanEval, with an additional StrategyQA experiment in the appendix. More demanding code generation, long-context, and extended-generation workloads are needed to assess the robustness of expert replacement. Testing only Qwen3-30B-A3B and DeepSeek-V2-Lite also limits architectural and scale coverage.

Questions for authors

See my weakness discussion above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.