- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DyS-MoE targets MoE inference under limited GPU memory. The core idea is to decouple dynamic expert demand from execution structure: a fixed execution window on the GPU (operator sequence and dependencies) supports CUDA Graph replay, while expert identity becomes a mutable binding inside the window. A miss (expert absent from the GPU, or slot full) is served by a similar resident expert, with a small number of sensitive requests computed exactly on the CPU. The three mechanisms are the Virtualized Static Execution Graph, Slotted Grouped Execution, and Importance-Aware Miss Handling. On an RTX 4090 at batch 8, it reaches 431.99 and 551.06 tokens/s on Qwen3-30B-A3B and DeepSeek-V2-Lite, 2.28× and 6.90× the fastest evaluated baseline, with roughly 3 points of accuracy loss under the default configuration.
Critical strengths
- The problem is real and well-motivated. Launch and scheduling overhead is a genuine bottleneck in dynamic MoE offloading, so fixing the execution structure for reuse is a sensible goal.
- The ablations cleanly separate the roles of the components: graph replay carries throughput, CPU exact computation carries accuracy, and capacity grouping is a minor optimization.
- Execution consistency is verified, and the authors disclose several limitations, which is honest.
Critical weaknesses
-
The core idea combines known techniques, and the novelty boundary is not stated. Serving misses with similar experts comes from BuddyMoE and SERE, and a fixed execution structure plus CUDA Graph replay already appears in KTransformers. The authors narrow the novelty to a "joint design" of replacement, slot capacity, and execution windows, but the paper should say what is new in the mechanism beyond engineering integration and tuning, and how it differs from closely related work.
-
The accuracy-throughput trade-off is not reported clearly. The reported speedups hide the accuracy and memory cost. Cache ratio, CPU budget, and prefetch degree all move along the accuracy-throughput frontier, but the paper gives no joint view. A larger cache ratio lowers throughput, and removing CPU exact computation raises throughput while DeepSeek accuracy drops sharply; so the "competitive accuracy" is bought back with CPU exact computation and cache, a cost not reflected in the abstract or conclusion.
-
Prefetch, a listed contribution, is net-negative in its own ablation and is not explained. Removing prefetch raises DeepSeek throughput by about 46% and slightly improves HumanEval, and the benefit is non-monotonic in the prefetch degree. The paper only says prefetch is optional and workload-dependent, without explaining why it hurts in the default configuration or when it helps. A component that is better turned off by default should not be listed as a core contribution.
-
The expert-redundancy assumption the method rests on is not quantified, so the quality guarantee lacks a principled basis. The whole accuracy story depends on experts being redundant and replaceable, but the paper does not measure redundancy; the replacement error is fit on a small calibration set with no upper bound. DeepSeek collapses in accuracy at a small cache while Qwen stays stable, which is exactly a failure of the redundancy assumption, and the paper does not explain this asymmetry. Quality evaluation is also limited to GSM8K and HumanEval, so the side effects of per-token replacement are not covered.
Questions for authors
- What exactly is the novelty, and where does it lie beyond the combination of known replacement and graph-replay techniques?
- What exactly is the accuracy-throughput-memory trade-off, and how does it depend on cache ratio, CPU budget, prefetch degree, and model architecture or parameter count? Please quantify it.
- Why is prefetch net-negative in the default configuration? Under what conditions does it help, and what criterion decides whether to enable it?
- How is expert redundancy quantified? Does the replacement error have an upper bound? Why does DeepSeek collapse at a small cache while Qwen stays stable?