acceptodds
Under review as a conference paper at ICLR 2027

RouteEP: Routing-Aware Expert Parallelism for Efficient MoE LLM Decoding

10 views

Abstract

Mixture-of-Experts (MoE) architectures enable Large Language Models (LLMs) to scale their parameter capacity while activating only a small subset of experts for each token. However, dynamic expert routing in expert parallelism (EP) and expensive all-to-all collective communication incur substantial inter-device communication overhead during model inference. Such bottlenecks become more pronounced in decoding, where sparse expert activation at small active batch sizes limits computation–communication overlap and effective parallelism. In this paper, we present RouteEP, a routing-aware expert parallelism framework to reduce inter-device communication and improve the load balance of expert execution for LLM decoding. RouteEP introduces a decode-specific parallelism design tailored to the memory- and communication-bound nature of decoding through three coordinated contributions. (1) We propose a lead-anchored offline expert grouping algorithm that selects lead experts, builds expert groups around them, and assigns the remaining experts to maximize position-wise routing affinity under capacity constraints. (2) Based on these expert groups, we design a routing-aware expert parallelism scheme for LLM decoding that replaces the communication-intensive dispatch–combine EP pattern with rank-local partial expert computations followed by a global all-reduce. (3) We further integrate intra-expert sharding into this framework to increase parallelism within the limited set of activated experts. Experiments on a range of MoE LLMs, including Qwen, DeepSeek, and Mixtral models, show that RouteEP achieves – speedups over state-of-the-art inference frameworks.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, Mixture of Experts, Expert Parallelism, Load Balance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N3PmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RouteEP, an optimization method for decode-scenario MoE inference speedup. RouteEP mainly has two contributions. First, it uses a packing-based algorithm to refine the placement of experts per layer, mitigating the probability that tokens of a small decoding batch target the same EP rank. Then it claims a "new" routing-aware EP scheme, using two allreduce instead of commonly used dispatch and combine (two alltoall premitives). The paper uses Qwen3-30B-A3B, DeepSeek-V2-Lite, and Mixtral-8x7B as the experimental models, and compares with native SGLang and LMDeploy, claiming a better performance.

Critical strengths
  1. The paper writes well, and is easy to follow.
  2. The problem is important and the motivation is clear.
  3. The math formulation is easy to follow.
Critical weaknesses

My concerns remain at the following two aspects. First, the so-called second contribution (i.e., the routing-aware EP scheme) is not new. Back in 2024.12, SGLang has implemented the two-allreduce-based EP scheme in case of the combination of attention-TP and MoE-EP (refer to PR#2203 and PR2371 of the upstream SGLang repository). In this way, the attention layers are sliced via tensor parallel, and MoE layers distributed experts in a EP fashion, the whole comp-comm workflow works like: attn computation -> allreduce -> topk and local experts computation -> allreduce. Therefore, the "route-aware EP scheme" can not be attributed as an inovation point. Second, in this literature, the paper announce a new-invented expert grouping algorithm. In this case, the author should compare his/her method with other expert grouping algorithm such as EPLB. But I do not see relavant evaluations in the experiments section. And baselines just include the native SGLang and LMDeploy.

Questions for authors
  1. Can the authors open-source the artifacts, so I can check the implementation, in case that I do not understand enough by just reading the paper.
  2. DeepEP has two modes: normal and low-latency. Which mode do the authors use?
  3. In Fig. 1, the author said "Others includes the prefill phase and the remaining decode operations." So the other parts exclude "Others" are MoE computation and communication cost in decoding phase?
Reviewer_HQT3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RouteEP targets the communication and load-imbalance problems of MoE decoding. It uses offline routing-position-aware expert placement, replaces the decode dispatch-combine path with local expert computation followed by all-reduce, and applies intra-expert sharding when the number of GPUs exceeds Top-K. Experiments on three MoE models and four B200 GPUs show lower communication time and improved TPOT and throughput.

Critical strengths
  • The paper targets a practical bottleneck and does not approximate the original routing or expert computation.
  • The end-to-end evaluation covers several MoE architectures, and the communication-time reduction is consistent.
Critical weaknesses
  • The main concern is that the decode execution path overlaps substantially with existing inference-engine support. SGLang already provides a no-A2A backend based on local expert computation and global reduction; DeepEP provides a low-latency decode path; and EPLB already supports expert remapping and replication.
  • The current comparison does not isolate whether the gain comes from RouteEP's position-aware placement or from switching to an all-reduce execution path that is already available.
  • The expert map is built offline from prefill traces, but the paper does not evaluate workload or domain shift. The implementation is also tested only on a single 4xB200 NVLink node, without clarifying CUDA Graph support or multi-node RDMA behavior.
  • The serving experiments use fixed-length greedy decoding and do not report tail latency or separately ablate placement, communication redesign, and intra-expert sharding.
Questions for authors
  1. Could the authors compare four matched variants using the same kernels and collectives: (a) SGLang no-A2A with default placement, (b) the same backend with EPLB placement, (c) the same backend with RouteEP placement, and (d) RouteEP plus intra-expert sharding? This would isolate the contribution of the placement algorithm.
  2. Is the complete routing -> local expert computation -> all-reduce path CUDA-Graph compatible? Does it require CPU synchronization, dynamic allocation, or graph recapture when routing changes?
  3. How does the design scale across nodes with RDMA, where a dense all-reduce may have a different cost relative to sparse all-to-all?
  4. How robust is the static map when calibration and serving workloads come from different domains? A variable-length workload with p95/p99 latency would also be useful.
Reviewer_WRngOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RouteEP targets communication overhead and expert imbalance in small-batch MoE decoding. It combines offline, position-wise expert grouping, rank-local expert execution with all-reduce aggregation, and intra-expert sharding. Experiments on three MoE models using four B200 GPUs report average TPOT speedups of 1.13×–1.24× over SGLang and 1.38×–1.84× over LMDeploy.

Critical strengths
  1. The paper addresses a relevant systems bottleneck: sparse expert activation during decoding exposes communication and synchronization overhead.
  2. The evaluation covers three model families and multiple workloads. The component breakdown and expert-sharding ablation provide useful evidence of practical performance gains.
Critical weaknesses
  1. The central algorithmic contribution is insufficiently established. The paper combines expert placement, all-reduce execution, and intra-expert sharding, but does not clearly establish what the proposed grouping adds to this combination. Figure 6 reports substantial communication-time reductions and smaller computation-time reductions; this breakdown alone cannot attribute the gains to grouping. A controlled comparison that changes only expert placement is needed.
  2. The grouping objective is not directly validated against decode-time workload. Equation (5) rewards routing counts at positions aligned with each rank. However, Section 3.2 states that each rank executes all selected experts it owns, irrespective of routing position. Position alignment may be a useful heuristic, but the main-text results do not show that it reduces per-step rank imbalance. Moreover, the mapping is calibrated using prefill traces, while the target workload is decoding; the effectiveness of this transfer needs direct empirical validation.
  3. The performance evidence needs stronger measurement support. Section 4.1 reports one measured run per configuration, without variability estimates. This limits confidence in the magnitude and consistency of the gains. The four-GPU evaluation also supports a narrower scope than the broader scalability motivation suggests.
Questions for authors
  1. With the communication path and sharding configuration fixed, how much does the proposed grouping improve TPOT over the original placement and a simple frequency-balanced placement?
  2. On held-out decode workloads, does the prefill-derived mapping reduce per-step rank computation time and imbalance? Please provide a direct measurement connecting the grouping objective to runtime performance.
  3. Can the main TPOT comparisons be repeated with variability reported, and the scalability claims explicitly scoped to the evaluated configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.