acceptodds
Under review as a conference paper at ICLR 2027

Rankcast: phase-aware rank balancing in MoE serving

Abstract

Modern large language model (LLM) inference increasingly adopts disaggregated architectures that separate Prefill (P) and Decode (D) phases. Meanwhile, Expert Parallelism (EP) has been widely adopted in large Mixture-of-Experts (MoE) models serving. However, load imbalance still slows down Mixture-of-Experts (MoE) inference because each layer must wait for the busiest EP rank. The main source of latency differs between prefill and decode. Prefill is compute-bound, and its MoE Layers' latency is determined by the maximum token count of the busiest rank. At the same time, under moderate batch sizes, Decode is memory-bound, so the latency is dominated by the maximum active-expert count across ranks. To solve both problems under a unified framework, we present RankCast, a system that reduces rank-level maximum token counts during prefill and active-expert counts during decode. For every token, RankCast preserves the predicted Top- experts and chooses the remaining from its Top- candidates (). In prefill, RankCast yields a price per rank that steers flexible assignments to less loaded ranks. In decode, RankCast closes experts on the busiest ranks while preserving the constraints for each token. Both paths are implemented and tested in the state of the art inference frameworks SGLang and DeepEP. We overlap the RankCast's kernel with model execution and reuses the existing DeepEP communication channel for meta data transmission. On Qwen3-235B-A22B with EP and Qwen3.5-397B-A17B with EP, RankCast removes over 75% of the busiest rank's excess prefill load and improves prefill input throughput by up to 19%, while reducing decode time per output token by up to 11%, nearly lossless.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, LLM Serving, Expert Parallelism, Load Balancing, Token Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hTGEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses load imbalance in MoE expert parallelism and makes the case that prefill and decode need different objectives. Prefill is bottlenecked by the busiest rank’s token load, while decode is bottlenecked by the maximum number of active experts across ranks. The paper present a system RankCast which reroutes tokens at inference time. Each token keeps a protected Top-p prefix and selects exactly k experts from a Top-L candidate set. For prefill, rank prices are subtracted from candidate scores so crowded ranks get fewer flexible assignments. For decode, the system closes low-cost experts on busy ranks to reduce peak active-expert count while preserving protected experts and at least k candidates per token. To make decisions before routing, RankCast predicts the next layer’s routing and piggybacks compact metadata on DeepEP’s existing dispatch communication, avoiding extra synchronization.

The contributions are the phase-aware formulation, a prefill balancing algorithm with distributed histograms and rank prices, a decode expert-shedding algorithm, and an SGLang/DeepEP implementation. Experiments on Qwen3-235B and Qwen3.5-397B show over 75% of the busiest rank’s excess prefill load removed, up to 19% higher prefill throughput, and up to about 10% lower decode TPOT, with quality changes under one point.

Critical strengths

This paper addresses an important performance problem in MoE serving by considering the different load-balancing bottlenecks in prefill and decode. The phase-aware design is good, and the proposed rank pricing and expert shedding methods provide practical solutions for both phases. The system implementation is a strong point of the paper. By predicting next-layer routing and overlapping the balancing computation with model execution, RankCast achieves low runtime overhead without introducing additional communication barriers. The evaluation is fairly comprehensive, e.g. using two different large Qwen MoE models with EP=8 and EP=16. The results show consistent improvements in prefill throughput and decode latency, with limited impact on model quality across multiple benchmarks.

Critical weaknesses

Insufficient validation of cross-layer routing prediction. Cross-layer routing prediction is an important component of RankCast, as it enables load-balancing decisions to be made ahead of execution while protecting the most important experts. However, Figure 3(b) only reports the recall of the true Top-8 experts within the predicted Top-8 and Top-16 sets. It does not directly evaluate the accuracy of the predicted Top-p experts used for protection (e.g. it is p=2 used in this paper). A high Top-16 recall does not necessarily imply that the actual Top-2 experts are correctly protected. The paper evaluates RankCast on two large Qwen models, but it is unclear whether the method generalizes to models with different MoE architectures and routing mechanisms. For example, DeepSeek-V3 uses a different expert routing and load-balancing design. Since RankCast relies heavily on cross-layer routing prediction and Top-p expert protection, it would be useful to evaluate another model family and examine whether the prediction accuracy, load-balancing effectiveness, and quality-performance trade-off remain consistent.

Questions for authors
  1. Since the main purpose of Top-p protection is to preserve model quality when changing expert routing, it would be useful to include a quality comparison across different values of p. Now in p20, D.3 only gives the ablation analysis results for system performance, such as throughput and imbalance.
  2. A larger p should help preserve the original expert routing, but may also limit the flexibility for load balancing. Why is p=2 chosen for both Qwen3-235B-A22B and Qwen3.5-397B-A17B? It would be useful to evaluate the trade-off between model quality and load-balancing performance across a wider range of p values, particularly during decode.
  3. Please directly provide the top-p prediction accuracy analysis.
  4. Can the author make experiment on one more MoE model which has significant architecture difference from Qwen3/3.5?
1 comment

Thank you so much for providing such clear and reasonable response and questions. We will try to solve most of them.

Part 1 of 2: Top-pp prediction accuracy

Top-pp prediction accuracy. In prefill, the protected Top-pp and the candidate set come from the target layer's own router scores, and the prediction only sets the per-rank prices. So the true Top-pp is preserved exactly, whatever the prediction accuracy. In decode, the mask is decided before the target layer runs, so we report both the prediction accuracy (Table 1) and the final protection (Table 2).

Table 1. Prediction accuracy, averaged over layers. The prefill column covers 12 sampled MoE layers. The decode columns cover all layers after the first, on chat and code captures.

Metric235B prefill235B decode397B decode
True top-1 = predicted top-177.8%76–78%66%
True top-1 ∈ predicted top-291.4%92–93%84%
True top-2 set = predicted top-2 set63.1%66%48%
True top-2 ⊆ predicted Top-LL99.7%99.8%98.0–98.6%
Probability of true top-2 captured by predicted top-2—97.2–97.4%93%

Table 2. Protection achieved in decode at p=2p=2, D=∞D=\infty.

Tokens keepingQwen3-235BQwen3.5-397B
their true top-1 expert99.4–99.5%96.7–98.5%
both true top-2 experts97.6–97.8%87.9–94.5%

The final protection is much higher than the exact top-2 accuracy, because the true top-2 almost always lies inside the predicted Top-LL, and a mis-ranked pair mostly swaps experts of similar weight. Qwen3.5-397B has a much flatter router, so exact ranking is harder, but no benchmark drops by more than 0.47 pp in decode. A depth cap D=4D=4 can further raise its top-1 retention to 99.5%, while still cutting active experts by 16–26%.

Part 2 of 2: Quality across pp and another architecture

Quality across pp, and why p=2p=2. In prefill, the true Top-pp is almost always preserved, so quality and throughput change little with pp, but setting a reasonable p is still a safeguard. However, in decode, pp matters more (Table 3). We do not have time to rerun p=0 and p=1 with quality weights (which takes hundreds of GPU hours), but the monotonic trend is very clear and can show why we choose p=2, as it is a balanced choice of qualty and throughput.

Table 3. Decode quality change in percentage points (95% CI) on Qwen3-235B with no depth cap. The p=0p=0 and p=1p=1 columns use the same solver and request concurrency without quality weights. The last column adds quality weights and is the setting the paper uses.

Benchmarkp=0p=0p=1p=1p=2p=2 + quality weights
GPQA−2.90 [−5.81, +0.01]−0.38 [−3.19, +2.43]+0.18 [−0.65, +1.01]
AIME 2025 (30 questions)−14.85 [−18.53, −11.17]−1.67 [−4.71, +1.37]−0.46 [−1.29, +0.37]
IFEval−8.13 [−10.76, −5.51]−3.70 [−5.97, −1.43]−0.65 [−1.28, −0.01]
LiveCodeBench−24.77 [−28.49, −21.06]−3.52 [−6.12, −0.93]−0.59 [−1.28, +0.10]
MMLU-Redux−1.42 [−1.98, −0.86]−0.98 [−1.48, −0.48]−0.13 [−0.38, +0.11]
MATH-500−7.80 [−10.84, −4.76]−2.00 [−4.07, +0.07]−0.13 [−1.11, +0.84]
MMMLU−3.10 [−3.54, −2.65]−0.88 [−1.27, −0.50]−0.47 [−0.74, −0.20]

** A model with a different MoE architecture.**

Although time limited the number of models we could evaluate, Qwen3-235B-A22B and Qwen3.5-397B-A17B differ substantially in architecture. Qwen3 uses a full-attention backbone with 128 experts and activates 8 per token. Qwen3.5 alternates Gated DeltaNet and full-attention layers, with 512 experts and 10 routed experts plus one shared expert activated per token. Our results therefore cover different attention backbones and MoE configurations

Reviewer_EPXvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RankCast, a unified, phase-aware, rank-level load balancing framework designed specifically for Expert-Parallel (EP) Mixture-of-Experts (MoE) large language model (LLM) serving. Recognizing that optimization bottlenecks differ significantly between phases—prefill is compute-bound and bottlenecked by peak per-rank token load, while decode at moderate batch sizes is memory-bound and limited by the maximum active-expert count—RankCast handles both under a unified routing constraint framework. Specifically, it preserves each token's predicted top-pp experts as a protected prefix and restricts flexible re-routing or expert shedding within a broader top-LL candidate set (p≤k≤Lp \le k \le L). In the prefill phase, it uses compact distributed histograms and a rank-price mechanism to steer flexible token assignments away from crowded ranks. In the decode phase, it greedily closes low-cost experts on the busiest ranks to reduce active expert counts without violating token-level constraints. RankCast overlaps its lightweight metadata exchange and solver kernels with model execution by piggybacking data onto DeepEP's communication channels, avoiding extra synchronization barriers. Evaluated on Qwen3-235B and Qwen3.5-397B, RankCast improves prefill input throughput by up to 19% and reduces decode time per output token (TPOT) by up to 10%, with model accuracy degradations rigorously bounded to under 1 percentage point.

Critical strengths
  1. Phase-Aware Problem Formulation & Unified Framework: The paper provides a clear, insightful distinction between the optimization bottlenecks of prefill (compute-bound/token overload) and decode (memory-bound/excessive active experts), while cleverly solving both under a unified routing constraint framework (protected top-pp prefix + top-LL candidate set).

  2. Comprehensive & Separate Performance Statistics: The evaluation provides thorough, independent, and rigorous statistics on the efficiency gains and accuracy losses for both phases (as well as running both simultaneously under "Both"). It achieves up to a 19% increase in prefill input throughput and up to a 10% reduction in decode TPOT, while substantiating via extensive benchmarking that quality loss remains below 1 pp across all configurations.

  3. Zero-Overhead System Integration: By utilizing DeepEP's existing communication channels to transmit compact metadata and executing prediction and solver kernels asynchronously on side streams, RankCast achieves efficient computation-communication overlap with virtually no added synchronization overhead.

  4. Training-Free and Practical: As a completely training-free serving-time optimization scheme, RankCast requires no modifications to model weights or pre-training procedures, making it immediately deployable for existing open-source frontier MoE models.

Critical weaknesses
  1. Reliance on Cross-Layer Routing Prediction: RankCast relies heavily on side-stream cross-layer routing prediction to compute prices and masks ahead of time. Although online corrections are introduced, sudden domain shifts or highly unpredictable generation steps could degrade prediction accuracy and introduce suboptimal balancing decisions.

  2. Hyperparameter Sensitivity: The framework introduces several configuration choices—such as candidate width (LL), protected prefix length (pp), and the number of histogram bins (BB)—which may require careful tuning across different model architectures and scale sizes.

  3. Limited Hardware Scale in Evaluation: The experiments are restricted to a maximum of 16 GPUs (EP=16EP=16) across two nodes. The paper does not thoroughly validate how the histogram exchange and distributed price-centering mechanisms scale under ultra-large rack-scale deployments (e.g., EP≥64EP \ge 64 across multiple server racks).

Questions for authors
  1. Prediction Error Resilience: How does RankCast behave under severe domain shifts or adversarial inputs where cross-layer routing prediction accuracy temporarily drops? Are there fallback safety mechanisms to prevent performance or accuracy cliffs?

  2. Scaling to Larger EP Degrees: How do the communication overhead and convergence properties of the prefill price-centering algorithm and decode solver scale when moving to much larger expert-parallel degrees (e.g., EP≥64EP \ge 64) across rack-scale networks?

  3. Interaction with Disaggregated KV-Cache Systems: Since RankCast significantly alters prefill throughput and decode latencies, does it introduce any secondary resource contention or synchronization side effects when integrated with advanced dynamic KV-cache migration layers (such as Mooncake) in disaggregated serving architectures?

2 comments

Thank you so much for providing valuable feedback and positive assessment!

W1/Q1. Prediction errors and fallbacks.

We believe RankCast is robust to prediction errors, for two reasons. First, the prediction does not depend on past traffic. The next layer's router scores are computed from each token's current hidden state, so a domain shift does not leave a stale model behind. Only the score bias and the load offset η\eta carry history, and they are refreshed only every 32 and 8 forwards. The accuracy is also stable across domains (Table 1), and quality on the multilingual MMMLU benchmark drops by only 0.25 pp under Qwen3.5-397B decode shedding. Second, a wrong prediction has bounded consequences by construction, including on adversarial inputs. In prefill, the final selection always uses the target layer's own scores, so a wrong prediction can only produce a wrong price. It may affect balance, but each token still keeps its own Top-pp and chooses within its own Top-LL. If a price falls into an edge bin of the histogram, that rank keeps its previous price. In decode, the solver keeps at least kk predicted candidates open for every token, and the router then selects kk experts among the open ones. A row with fewer than kk valid candidates keeps that token's experts open.

Table 1. Share of tokens with both true top-2 experts inside the predicted Top-LL.

ModelChatCode (LiveCodeBench)
Qwen3-235B, L=16L=1699.8%99.8%
Qwen3.5-397B, L=20L=2098.0%98.6%

W2. Hyperparameter sensitivity.

We use the same defaults on both models (p=2p=2, D=∞D=\infty). Only LL scales with the number of experts (16 for 128 experts, 20 for 512).

  • pp. In prefill, quality and throughput change little with pp, and every quality change is within the baseline noise (Table 2). In decode, p=2p=2 reduces the largest benchmark loss from 1.32 pp at p=1p=1 to 0.45 pp.

  • LL. The predicted candidates hold 97.6%, 99.1% and 99.6% of the routing weight on Qwen3-235B at L=12L=12, 16 and 20 (App. D.3).

  • BB. Any B≥64B\geq64 works. In offline replay up to P=128P=128, B=64B=64 and B=256B=256 give the same imbalance to within 0.012 on Qwen3.5-397B.

Part 2 of 2: Scale and KV-cache systems (W3/Q2, Q3)

W3/Q2. Scaling to larger EP. We believe RankCast's design scales well.

  • Prefill. Each rank sends one small histogram per layer, 8 KiB at P=8P=8 and B=256B=256, and 32 KiB at P=128P=128 and B=64B=64. The histogram rides on DeepEP's existing notify_dispatch message, so it adds no synchronization. Each rank then computes its price from the summed histogram in a fixed number of steps, so there is no iteration between ranks and no convergence problem.

  • Decode. For each token, the ranks exchange only its LL candidate expert ids and their 2-bit weights, so the payload per token does not depend on PP. The solver is a greedy algorithm, and every round except the last lowers the busiest rank's active-expert count by at least one, so it always terminates.

We also replayed the price algorithm offline on captured scores up to P=128P=128 (Table 3). Native imbalance grows quickly with PP because each rank holds fewer experts, and RankCast still removes about half of the excess at P=128P=128.

Table 3. Offline replay: max/mean imbalance, native → RankCast, and the share of excess removed. Qwen3-235B uses prefill chunks of 16,384 tokens. Qwen3.5-397B uses decode-shaped captures combined into 16,384-token batches.

PPQwen3-235B, prefill chunkQwen3.5-397B, code
81.60 → 1.19 (68%)—
162.30 → 1.36 (70%)2.48 → 1.39 (72%)
323.45 → 1.86 (64%)3.62 → 1.87 (66%)
645.30 → 2.97 (53%)5.53 → 2.92 (57%)
128—9.34 → 5.30 (51%)

Q3. Disaggregated KV-cache systems. Our Qwen3-235B deployment is already prefill–decode disaggregated, and Mooncake transfers the KV cache (App. D.1), so all Qwen3-235B throughput and TPOT results were measured in this setting. RankCast only changes which experts a token uses inside MoE layers. It does not touch the KV cache, and all its communication stays inside the EP group. So it adds no traffic to the KV path and no synchronization with it.

Reviewer_viuYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RankCast is an inference-time load balancer for expert-parallel MoE serving: it keeps each token’s top-p predicted experts and changes only the remaining choices from its top-L candidates. In prefill it uses per-rank prices, from the dual of an assignment LP, to lower the peak token count on any rank; in decode it closes experts on the busiest ranks to lower the peak number of active experts. It is implemented in SGLang and DeepEP, and on Qwen3-235B and Qwen3.5-397B it reports up to 19% higher prefill throughput and up to 10% lower decode TPOT, with accuracy changes under 1 pp.

Critical strengths
  1. The split of objectives by phase is clear and backed by profiling: prefill is limited by the peak token count on any rank, decode by the peak number of active experts.
  2. The engineering is solid: the computation overlaps with model execution, the metadata reuses DeepEP’s existing messages, the decode path is CUDA-graph compatible, and the overhead is 0.5–0.9%.
  3. It is evaluated end to end on two large production MoE models.
Critical weaknesses
  1. This is lossy routing compared against lossless baselines. RankCast changes the model’s expert choices. EPLB and replication do not.
  2. The quality claims are not statistically supported. The paper reports the per-repeat standard deviation of the accuracy change as 0.7–1.6 pp. Some cells use only 2–3 repeats. Effects under 1 pp therefore cannot be resolved.
  3. The novelty is limited. Pricing via the dual of an assignment problem is the classic auction or transportation-dual approach, and close to the bias-based balancing in DeepSeek-V3 and Kimi K3.
Questions for authors

See weaknesses.

2 comments

Thank you so much for your concise and very insightful response. We will try my best to resolve each of your problem.

Part 1 of 3: Lossy routing

For research purposes, we believe it is important to test different kinds of methods. Traditional methods mainly focus on system-side optimization to solve the EP imbalance problem. However, these methods are either not free or limited in scope. Static-placement EPLB cannot resolve the dynamic batch problem. Other papers trade additional HBM space or communication bandwidth for EP balance, and these resources are also precious. So we believe it is meaningful that RankCast provides a new arm for solving EP balance. By using an algorithm to change the router's decisions and limiting the impact on model performance, we show that RankCast alone can provide better results than EPLB (Section 5).

Also, though not emphasized in this paper because of the page limit, in future work we want to explore how RankCast can be combined with traditional placement, prefetching and replication methods. As you can see, the algorithm RankCast uses is very classic and adaptive. After adding a few conditions to the optimization problem, it can still be solved using the same method, which may provide a balance between system resource usage and performance, or dynamic scheduling when some resources are limited. So we believe we cannot deny RankCast's potential simply because it is not lossless.

Part 2 of 3: Statistical support

W2. Statistical support. We believe the 1 pp claim is supported by the data, for two reasons. First, the cells with only 2–3 repeats have very low variance. MMLU-Redux and MMMLU are very large benchmarks with 5,700 and 19,600 questions, and MATH-500 is also stable. The per-repeat standard deviation of these three benchmarks is at most 0.23 pp, compared with 0.5–1.3 pp on the four smaller benchmarks, so 2–3 repeats are enough. On MMLU-Redux and MMMLU, the 95% confidence interval of every change stays within ±0.5 pp. Second, no measured drop reaches 1 pp, and the drops are within the noise of the baseline itself. Table 3 shows the largest drop of each benchmark across all settings. The largest one is 0.85 pp. On the four smaller benchmarks, every drop is no larger than the difference between two halves of the same baseline repeats, where the routing does not change at all. We also compare the two arms question by question with a paired McNemar test, and only 3 of the 38 comparisons are significant at the 0.05 level.

Table 1. Largest quality drop of each benchmark in percentage points, across all settings (Qwen3-235B: Prefill, Decode at D=4D=4 and D=∞D=\infty, and Both; Qwen3.5-397B: Prefill and Decode). Baseline noise is the difference between two halves of the same baseline repeats (95th percentile over all 5-vs-5 splits of 10 repeats). Benchmarks with 2–3 repeats have no such split.

BenchmarkQuestions235B largest drop235B baseline noise397B largest drop397B baseline noise
GPQA1980.851.350.471.20
AIME 2025300.540.890.091.57
IFEval5410.650.650.000.67
LiveCodeBench8800.840.890.430.66
MMLU-Redux5,7000.21—0.01—
MATH-5005000.40—0.30—
MMMLU19,6000.22—0.25—

Part 3 of 3: Novelty

RankCast does take insights from the classic assignment dual and Kimi K3's quantile balancing algorithm for prefill. However, traditionally, these algorithms are used in training, and it is not painless to shift them to inference. For training, the dynamic batch problem is averaged across many steps. We do not have to perfectly solve the problem in each forward pass as long as the algorithm finally converges. But in inference, we must achieve good balance in each forward pass, and a lot of system optimizations must be applied because inference is very sensitive to latency. The end-to-end for overhead is only 0.5–0.9% in Rankcast.

Table 2 shows why each forward pass matters. We ran only the slow per-rank offset η\eta (App. C.2), which works like a training-time bias and converges over many forward passes. On LongBench, it balances the load averaged over forward passes (the imbalance of this average falls from 1.24 to 1.04), but each individual forward pass stays unbalanced and throughput does not improve. With the per-batch prices, throughput rises by 7.4%. RankCast proved that it is possible to shift a training algorithm to inference.

Table 2. Ablation of the slow offset η\eta on Qwen3-235B prefill (one prefill and one decode node, one run per arm). The clamp on η\eta is relaxed to the full price range. Imbalance is the per-forward max/mean over EP ranks.

WorkloadArmThroughput vs. Top-kkImbalance
ShareGPTTop-kk—1.75
ShareGPTη\eta alone+14.7%1.20
ShareGPTη\eta + per-batch prices (RankCast)+18.1%1.08
LongBench v2, 32KTop-kk—1.66
LongBench v2, 32Kη\eta alone−0.8%1.65
LongBench v2, 32Kη\eta + per-batch prices (RankCast)+7.4%1.16

Secondly, there are no decode settings in training, because the training workload mainly consists of prefill-style forward passes and backward passes. Decode is therefore a new problem. RankCast minimizes the active experts on the busiest rank, and our solver closes experts on several ranks per round (Section 3.2).

Comments on the paper

1 comment

We thank the reviewers for their invaluable time and for feedback that was specific and helpful enough to make our paper better. We have uploaded the revised PDF, which carries the changes discussed in our responses. RankCast adds a routing-side lever to a problem that has so far mainly been attacked only from the system side, where placement cannot follow batch-to-batch shifts and replication or prefetching must pay for balance in HBM or bandwidth. We hope to do better in future work, especially by combining the two levers to get even better results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.