- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RouteEP, an optimization method for decode-scenario MoE inference speedup. RouteEP mainly has two contributions. First, it uses a packing-based algorithm to refine the placement of experts per layer, mitigating the probability that tokens of a small decoding batch target the same EP rank. Then it claims a "new" routing-aware EP scheme, using two allreduce instead of commonly used dispatch and combine (two alltoall premitives). The paper uses Qwen3-30B-A3B, DeepSeek-V2-Lite, and Mixtral-8x7B as the experimental models, and compares with native SGLang and LMDeploy, claiming a better performance.
Critical strengths
- The paper writes well, and is easy to follow.
- The problem is important and the motivation is clear.
- The math formulation is easy to follow.
Critical weaknesses
My concerns remain at the following two aspects. First, the so-called second contribution (i.e., the routing-aware EP scheme) is not new. Back in 2024.12, SGLang has implemented the two-allreduce-based EP scheme in case of the combination of attention-TP and MoE-EP (refer to PR#2203 and PR2371 of the upstream SGLang repository). In this way, the attention layers are sliced via tensor parallel, and MoE layers distributed experts in a EP fashion, the whole comp-comm workflow works like: attn computation -> allreduce -> topk and local experts computation -> allreduce. Therefore, the "route-aware EP scheme" can not be attributed as an inovation point. Second, in this literature, the paper announce a new-invented expert grouping algorithm. In this case, the author should compare his/her method with other expert grouping algorithm such as EPLB. But I do not see relavant evaluations in the experiments section. And baselines just include the native SGLang and LMDeploy.
Questions for authors
- Can the authors open-source the artifacts, so I can check the implementation, in case that I do not understand enough by just reading the paper.
- DeepEP has two modes: normal and low-latency. Which mode do the authors use?
- In Fig. 1, the author said "Others includes the prefill phase and the remaining decode operations." So the other parts exclude "Others" are MoE computation and communication cost in decoding phase?