- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses load imbalance in MoE expert parallelism and makes the case that prefill and decode need different objectives. Prefill is bottlenecked by the busiest rank’s token load, while decode is bottlenecked by the maximum number of active experts across ranks. The paper present a system RankCast which reroutes tokens at inference time. Each token keeps a protected Top-p prefix and selects exactly k experts from a Top-L candidate set. For prefill, rank prices are subtracted from candidate scores so crowded ranks get fewer flexible assignments. For decode, the system closes low-cost experts on busy ranks to reduce peak active-expert count while preserving protected experts and at least k candidates per token. To make decisions before routing, RankCast predicts the next layer’s routing and piggybacks compact metadata on DeepEP’s existing dispatch communication, avoiding extra synchronization.
The contributions are the phase-aware formulation, a prefill balancing algorithm with distributed histograms and rank prices, a decode expert-shedding algorithm, and an SGLang/DeepEP implementation. Experiments on Qwen3-235B and Qwen3.5-397B show over 75% of the busiest rank’s excess prefill load removed, up to 19% higher prefill throughput, and up to about 10% lower decode TPOT, with quality changes under one point.
Critical strengths
This paper addresses an important performance problem in MoE serving by considering the different load-balancing bottlenecks in prefill and decode. The phase-aware design is good, and the proposed rank pricing and expert shedding methods provide practical solutions for both phases. The system implementation is a strong point of the paper. By predicting next-layer routing and overlapping the balancing computation with model execution, RankCast achieves low runtime overhead without introducing additional communication barriers. The evaluation is fairly comprehensive, e.g. using two different large Qwen MoE models with EP=8 and EP=16. The results show consistent improvements in prefill throughput and decode latency, with limited impact on model quality across multiple benchmarks.
Critical weaknesses
Insufficient validation of cross-layer routing prediction. Cross-layer routing prediction is an important component of RankCast, as it enables load-balancing decisions to be made ahead of execution while protecting the most important experts. However, Figure 3(b) only reports the recall of the true Top-8 experts within the predicted Top-8 and Top-16 sets. It does not directly evaluate the accuracy of the predicted Top-p experts used for protection (e.g. it is p=2 used in this paper). A high Top-16 recall does not necessarily imply that the actual Top-2 experts are correctly protected. The paper evaluates RankCast on two large Qwen models, but it is unclear whether the method generalizes to models with different MoE architectures and routing mechanisms. For example, DeepSeek-V3 uses a different expert routing and load-balancing design. Since RankCast relies heavily on cross-layer routing prediction and Top-p expert protection, it would be useful to evaluate another model family and examine whether the prediction accuracy, load-balancing effectiveness, and quality-performance trade-off remain consistent.
Questions for authors
- Since the main purpose of Top-p protection is to preserve model quality when changing expert routing, it would be useful to include a quality comparison across different values of p. Now in p20, D.3 only gives the ablation analysis results for system performance, such as throughput and imbalance.
- A larger p should help preserve the original expert routing, but may also limit the flexibility for load balancing. Why is p=2 chosen for both Qwen3-235B-A22B and Qwen3.5-397B-A17B? It would be useful to evaluate the trade-off between model quality and load-balancing performance across a wider range of p values, particularly during decode.
- Please directly provide the top-p prediction accuracy analysis.
- Can the author make experiment on one more MoE model which has significant architecture difference from Qwen3/3.5?
Thank you so much for providing such clear and reasonable response and questions. We will try to solve most of them.
Part 1 of 2: Top-p prediction accuracy
Top-p prediction accuracy. In prefill, the protected Top-p and the candidate set come from the target layer's own router scores, and the prediction only sets the per-rank prices. So the true Top-p is preserved exactly, whatever the prediction accuracy. In decode, the mask is decided before the target layer runs, so we report both the prediction accuracy (Table 1) and the final protection (Table 2).
Table 1. Prediction accuracy, averaged over layers. The prefill column covers 12 sampled MoE layers. The decode columns cover all layers after the first, on chat and code captures.
Table 2. Protection achieved in decode at p=2, D=∞.
The final protection is much higher than the exact top-2 accuracy, because the true top-2 almost always lies inside the predicted Top-L, and a mis-ranked pair mostly swaps experts of similar weight. Qwen3.5-397B has a much flatter router, so exact ranking is harder, but no benchmark drops by more than 0.47 pp in decode. A depth cap D=4 can further raise its top-1 retention to 99.5%, while still cutting active experts by 16–26%.
Part 2 of 2: Quality across p and another architecture
Quality across p, and why p=2. In prefill, the true Top-p is almost always preserved, so quality and throughput change little with p, but setting a reasonable p is still a safeguard. However, in decode, p matters more (Table 3). We do not have time to rerun p=0 and p=1 with quality weights (which takes hundreds of GPU hours), but the monotonic trend is very clear and can show why we choose p=2, as it is a balanced choice of qualty and throughput.
Table 3. Decode quality change in percentage points (95% CI) on Qwen3-235B with no depth cap. The p=0 and p=1 columns use the same solver and request concurrency without quality weights. The last column adds quality weights and is the setting the paper uses.
** A model with a different MoE architecture.**
Although time limited the number of models we could evaluate, Qwen3-235B-A22B and Qwen3.5-397B-A17B differ substantially in architecture. Qwen3 uses a full-attention backbone with 128 experts and activates 8 per token. Qwen3.5 alternates Gated DeltaNet and full-attention layers, with 512 experts and 10 routed experts plus one shared expert activated per token. Our results therefore cover different attention backbones and MoE configurations