- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents AsyncEP, a distributed execution engine for MoE prefill that addresses synchronization stalls caused by imbalanced attention workloads across DP replicas. It consists of three mechanisms: asynchronous EP, which starts expert computation as soon as inputs are ready; streamFFN, which uses threshold-based batching and multiple CUDA streams to maintain FFN efficiency; and opportunistic expert weight fetching (OEWF), which allows idle GPUs to take over part of the pending FFN work. The system is implemented in SGLang and evaluated on three MoE models, two workloads, and five scheduling policies. The results show up to 1.48× speedup in p95 TTFT and 1.17× improvement in throughput.
Critical strengths
S1. The paper addresses a practical bottleneck in MoE serving. The authors provide measurements of synchronization overhead and convincingly explain why request scheduling alone cannot fully resolve attention imbalance.
S2. The proposed mechanisms form a coherent system design with a nontrivial implementation. The evaluation covers multiple models, workloads, and scheduling policies, with solid ablation results supporting the reported performance gains.
Critical weaknesses
W1. The overhead of OEWF could be analyzed in more detail. Fetching expert weights introduces additional communication and temporary memory usage, and the decision to reassign work relies on runtime estimates. While the paper reports positive results, it would be useful to see how much time is spent fetching weights and how often the reassignment is actually performed. This would help clarify when OEWF is most effective.
W2. The experiments are limited to a single eight-GPU B200 node with NVLink and a fixed DP8+EP8 configuration. It is unclear how sensitive the gains are to communication bandwidth or different parallelism settings. Also, the component ablation is performed on only one model and workload configuration. Some additional analysis would help establish how broadly the individual mechanisms apply.
Questions for authors
Q1. Could the authors provide more details on the runtime overhead of OEWF, including weight transfer latency, temporary memory consumption, and the percentage of fetch attempts that result in successful FFN reassignment?
Q2. How sensitive are the performance gains to the degree of attention imbalance and the available communication bandwidth? It would be interesting to understand under what conditions OEWF provides little or no benefit.
Thank you for your careful review and thoughtful questions. Below, we address weight-transfer costs and reassignment (Weakness 1 / Question 1), sensitivity to imbalance and bandwidth (Question 2 / Weakness 2), and parallelism settings and component contributions (Weakness 2).
Weakness 1 / Question 1: Transfer time, memory overhead, and successful FFN reassignment. For DeepSeek-V4-Flash in our DP8+EP8 setup, the transfer overhead is 0.63 ms, with 427.82 MB of temporary weight-and-scale storage per helper. These costs depend on the expert-parallel configuration. In our analysis, 21.6% of the helpers' pending token–expert assignments on their selected donors are candidates whose reassignment is predicted to shorten donor–helper completion time if the weights are already available. Of these candidates, 52.8% are successfully reassigned.
Question 2 / Weakness 2: Sensitivity to attention imbalance and communication bandwidth. We evaluate both factors for DeepSeek-V4-Flash on AgentX with the cache-aware Preble policy at 5 req/s, varying one factor at a time. Both tables report the end-to-end p95 TTFT reduction of full AsyncEP relative to Sync.
As the synchronous attention-idle ratio increases from 10% to 40%, the TTFT reduction grows from 6.4% to 14.3%.
For bandwidth sensitivity, we evaluate different numbers of active NVLink links per GPU, which NVIDIA's
--setLinkWidthinterface can control [1]. Across 2–18 active links, the TTFT reduction increases from 6.2% to 12.1%.In these experiments, limiting NVLink bandwidth to 100 GB/s (two active links per GPU) leaves OEWF with negligible additional benefit. The 6.2% p95 TTFT reduction comes primarily from asynchronous EP and streamFFN.
Weakness 2: Sensitivity to parallelism settings. We extend the comparison from DP8+EP8 to DP16+EP16. The table below reports results for DeepSeek-V4-Flash on AgentX at 10 req/s, using two eight-GPU B200 nodes connected by 400 Gb/s InfiniBand. EP communication spans nodes, while OEWF remains within each helper's local NVLink domain. AsyncEP reduces p95 TTFT by 9.6–13.5% across the five policies.
Weakness 2: Component ablation. We extend the component ablation to DeepSeek-V4-Pro and GLM-5.3 using AgentX under round-robin scheduling with DP8+EP8, at 2.5 and 2 req/s, respectively. All throughput values below are in ktokens/s, and percentages report cumulative improvements relative to synchronous EP.
References
[1] NVIDIA. nvidia-smi — NVIDIA System Management Interface program. Official documentation, n.d.