- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents BLAZE, a replica-free expert routing method that aims to reduce load imbalance in MoE inference. It uses expert-level and rank-level load feedback to redirect tokens toward less-loaded experts, while protecting high-confidence routing decisions to limit accuracy degradation. The authors also introduce an overhead-aware activation gate that bypasses the additional routing logic for small batches where its overhead may outweigh the benefits. BLAZE is integrated into TensorRT-LLM with CUDA Graph support and evaluated on multiple MoE models. The experiments show near-baseline accuracy, improvements in tail latency and SLO compliance under certain serving workloads such as large-batch inference.
Critical strengths
- Important and practical problem. Expert load imbalance is a well-known bottleneck in MoE inference, and many previous works, such as EPLB, have tried to address it. Finding a way to balance expert workloads without replicating expert weights is valuable, especially for memory-constrained serving systems. The problem itself is important and well-motivated.
- Solid system implementation and evaluation. A major strength is that BLAZE is implemented in TensorRT-LLM with CUDA Graph support, rather than being evaluated only in simulations or microbenchmarks. This makes the results more convincing and relevant to real-world serving. The authors also evaluate the method from several angles, including offline throughput, online latency (TTFT and TPOT), and SLO compliance.
Critical weaknesses
- Limited novelty. The main ideas behind BLAZE have already been explored in previous work. [1] and [2] have studied inference-time expert rerouting to reduce load imbalance, while [3] introduced load-dependent routing biases and the decoupling of expert selection from mixture weights. The additional rank-aware feedback and activation gate seem relatively incremental, making the overall methodological contribution rather limited.
- Insufficient motivation for modern MoE models. Recent MoE models have adopted more advanced load-balancing techniques during training, such as Quantile Balancing in Kimi K3. Although these techniques do not guarantee balanced routing during inference, they may reduce the severity of the problem. The paper does not provide enough evidence showing that expert load imbalance remains a major inference bottleneck in recent models, especially in terms of its actual impact on execution time.
- Insufficient ablation and kernel-level performance analysis. The paper provides several ablations and high-level latency measurements, but the connection between reduced load imbalance and actual kernel speedup is still unclear. More balanced expert token counts do not necessarily lead to faster execution, since MoE kernel performance also depends on tiling, scheduling, and hardware utilization. The effectiveness of the method may also vary significantly across different MoE backends(triton, flashinfer, cutlass, megamoe), while the current experiments do not systematically investigate this.
[1] Shahout R, Cai C, Du Y, et al. From score distributions to balance: Plug-and-play mixture-of-experts routing[J]. arXiv preprint arXiv:2510.03293, 2025.
[2] He S, Cai W, Huang J, et al. Capacity-aware inference: Mitigating the straggler effect in mixture of experts[C]//International Conference on Learning Representations. 2026, 2026: 92274-92299.
[3] Wang L, Gao H, Zhao C, et al. Auxiliary-loss-free load balancing strategy for mixture-of-experts[J]. arXiv preprint arXiv:2408.15664, 2024.
Questions for authors
- How does BLAZE differ from existing inference-time expert rerouting methods, especially [1] and [2] Expanded Drop? What are the key improvements over these approaches?
- How severe is expert load imbalance in recent MoE models that already use advanced training-time balancing techniques, such as Kimi K3? Can the authors quantify how much this imbalance actually slows down inference?
- Can the authors provide a more detailed per-layer breakdown showing how each component reduces expert/rank imbalance and how this translates into grouped-GEMM and All-to-All execution time? It would also be helpful to understand how these benefits change across different batch sizes and MoE backends.
6 comments
[W2/Q2 – Motivation – Imbalance in recent MoEs and its inference cost]
Training-time balancing equalizes load only in expectation over large batches; small, domain-skewed inference batches stay imbalanced. Kimi K3 freezes its Quantile Balancing bias at inference (§2.3.3), and its MoonEP inference system reports skewed per-expert workloads even with balanced rank totals, needing load-aware GEMM scheduling (§5.2.1). K3 report
How severe. Vanilla-router load imbalance in serving (DP=EP=8, one trace per model; K3's 1.56 TB checkpoint exceeds our 8 × H200 node):
| Model | Expert load max/mean | EP-rank load max/mean |
|---|---|---|
| GPT-OSS-20B | 4.24 | 1.80 |
| GPT-OSS-120B | 12.18 | 1.73 |
| Qwen3-30B-A3B | 8.59 | 1.70 |
| Qwen3-235B-A22B | 8.91 | 1.67 |
Imbalance grows with EP size: rank max/mean reaches 3.90 at 32 virtual EP ranks (GPT-OSS-20B) and 4.11 at 64 (Qwen3-30B-A3B), and BLAZE's reduction grows with it (9.9% → 17.9%; 8.9% → 22.3%).
How much it slows inference. Replaying GPT-OSS-20B's recorded per-step expert counts through the CUTLASS grouped GEMM, uniform counts would cut the busiest rank's GEMM time by 21–36% (pinned). End to end, with no profile and no replica, BLAZE lowers TTFT P99 by 11–15% on GPT-OSS-20B and TTFT and TPOT P99 by 17% on GPT-OSS-120B against the α=0 control (W3/Q3).
[W3/Q3 – Kernel-level breakdown – Per layer, components, batch size and backends]
Setup: GPT-OSS-120B, TRT-LLM CUTLASS, 4 × H200, DP=EP=4, pinned; BLAZE (α=0.15, τ=1.5, =64, rank penalty off) vs. its α=0 control (same code and CUDA-graph path, load penalty off: only routing differs).
Serving. At QPS 2 (identical prompts and arrivals, exactly 1,024 output tokens), BLAZE lowers TTFT P99 from 2,262 ± 23 to 1,871 ± 2 ms (−17.3%) and TPOT P99 from 80.20 ± 0.65 to 66.73 ± 0.05 ms (−16.8%); mean ± SD, three fresh-server runs per arm; a second set gives −18.9% / −15.4%.
Per layer. We recorded a real prefill+decode serving step (3,247 tokens; selection rule fixed beforehand) and replayed its 36 MoE layers under both policies (identical inputs, frozen EMA from an α=0 run; only α differs). The α=0 replay matches the recorded MoE outputs exactly (all 144 layer–rank pairs). Timing: Nsight Systems, 50 alternating pairs per layer.
| Layers | Expert max/mean (α=0 → BLAZE) | Rank max/mean (α=0 → BLAZE) | Slowest-rank GEMM time |
|---|---|---|---|
| 0–11 | 10.29 → 4.87 | 1.224 → 1.134 | −5.7% |
| 12–23 | 14.33 → 5.43 | 1.424 → 1.182 | −14.6% |
| 24–35 | 15.99 → 5.88 | 1.438 → 1.152 | −17.5% |
| All 36 | 13.54 → 5.39 | 1.362 → 1.156 | −12.9% |
GEMM time falls 11.3% even without the best layer and tracks the change in rank max/mean across the 36 layers (r = 0.98). The sum of per-layer maximum-rank mean MoE times (routing, GEMMs, communication) falls by 12.4%.
Components. With the rank penalty off, the expert penalty alone (safeguards fixed) produces these gains (busiest rank's load −15%). The gate restricts BLAZE to ranks with enough tokens (here one of four), yet load shifts on all four. Offline, BLAZE changes 7.7–22.0% of GPT-OSS-20B's expert slots per layer but replaces the top-1 for only 0.01–4.4% of tokens. Its router costs 1.0–3.2 µs more per layer than vanilla at N≤1,024 (pinned).
Workload dependence. In a 16,384-token prefill-only step (four 4,096-token MLPerf prefills), rank max/mean falls from 1.395 to 1.328, the slowest rank's GEMM time by 4.6% and the MoE time by 4.4% on every rank: the gain follows how much BLAZE unloads the busiest rank.
Communication. CUTLASS here uses AllGather/ReduceScatter semantics (NCCL Broadcast/Reduce), so message sizes do not depend on routing. The collective before the expert GEMMs is unchanged (217.4 → 216.5 µs per layer); the one after them halves (2,115.3 → 1,009.8 µs), consistent with ranks waiting less for the slowest one.
Backends. CUTLASS, GPT-OSS-20B (4 × H200, DP=EP=4, pinned, three runs per cell): BLAZE's TTFT P99 is 845 ± 3 / 1,259 ± 3 ms at QPS 1 / 3, vs. 970 ± 1 / 1,422 ± 14 ms for α=0 (−13% / −11%; −14% / −15% at QPS 2 / 5), with vanilla within 0.3% of α=0 at QPS 1 / 3. TRT-LLM as released (TRITON, DP=EP=8, unpinned, one run per cell): BLAZE raises GPT-OSS-20B's SLO-compliant load from QPS 3.0 to 4.0 and lowers TTFT P99 at QPS 3.0 from 2,336 to 273 ms; its α=0 control reaches 336 ms, so this gain is delivered by BLAZE's CUDA routing path, which largely avoids the released Triton router's recompilation. GPT-OSS-120B under the MLPerf v6.0 Server protocol with LoadGen (CUTLASS, pinned, one run per arm; not an official MLPerf result): TTFT / TPOT P99 18% / 8% below vanilla. FlashInfer and MegaMoE are unavailable as MoE backends in TRT-LLM 1.3.0rc8 on H200 (FlashInfer MoE requires SM100/103).
We will add these results to the paper. If our responses have addressed your concerns, we would be sincerely grateful if you would kindly consider raising your score.
Dear Reviewer,
The authors have posted their rebuttal. If any of your concerns remain unaddressed, or if the response raises new questions, please let authors know.
Authors can no longer respond after 11/19, so now is the time to follow up.
Thanks for your time and effort.
AC
We thank the reviewer for the detailed assessment and for recognizing the importance of replica-free balancing and BLAZE's solid system implementation and evaluation.
[W1/Q1 – Novelty – BLAZE vs. LASER [1], Expanded Drop [2] and aux-loss-free balancing [3]]
[1] and [2] are now cited. At near-matched balance BLAZE keeps 4.8 pp more reasoning accuracy than LASER and replaces the top-1 expert less frequently. On the same router, its TTFT P99 is up to 11% lower at QPS 3 and 5.
What is new. BLAZE ranks all experts by router score minus a bounded load penalty and keeps exactly K per token. Each substituted slot has score regret ≤ 2αDE+2αrankDR (Thm. 1), so the top-1 expert is replaced only when its lead is within the penalty range. Load feedback is an EMA updated off the critical path, in a single CUDA-graph-capturable pass.
Controlled comparison (GPT-OSS-20B). All policies share one graph-capturable PyTorch router in TRT-LLM 1.3.0rc8 (same EMA load, mixture weights and backend) and differ only in the selection rule. We implement LASER's rule there, capping candidates at each token's top-2K experts by score and ordering them by −load+10−4logit. Accuracy: AIME25 + GPQA-Diamond (n = 1,230), one GPU, gate off, first run of each policy paired with the same vanilla run. Serving: Nmin=128, CUTLASS, 4 × H200, DP=EP=4, pinned cores, identical prompts and arrivals, exactly 1,024 output tokens, mean of three runs (SD ≤ 2.4% at QPS 3, ≤ 9.3% at QPS 5). BLAZE's native CUDA router (W3/Q3) is faster than this shared router, so absolute latencies differ.
At near-matched balance, BLAZE keeps 4.80 ± 2.74 pp more AIME/GPQA accuracy (paired 95% CI, p = 0.001) and about 1.3 pp more on MMLU: LASER removes the top-1 expert for 16.5% of tokens, BLAZE for 0.74%; TTFT P99 is equal at QPS 3 and 6% lower at QPS 5. At similar single-GPU accuracy, BLAZE balances more (3.27 vs. 3.59) and its TTFT P99 is 6% and 11% lower. Balancing as much as BLAZE or more, LASER loses 5.4–18.0 pp and removes the top-1 for 16.5–44.5% of tokens. On Qwen3-30B-A3B, LASER removes the top-1 for 28.4% of tokens (expert max/mean 4.47), BLAZE for 0.98% (4.11), and at near-matched balance BLAZE is 1.5 ± 0.5 pp higher on MMLU.
Activation gate. The gate applies BLAZE only where a rank holds enough tokens for balancing to pay off; on GPT-OSS-120B it acts on the rank holding the prefill chunk, where the slowest rank's grouped-GEMM time falls by 12.9% (W3/Q3).
Contributions (revised). (1) A bounded-regret selection rule that keeps 4.8 pp more accuracy than LASER at near-matched balance and drops no tokens; (2) a profile-free load controller off the critical path in a single-pass, graph-capturable router; (3) an overhead-aware gate and a matched-work SLO evaluation in a production engine with an implementation-matched control (α=0).