acceptodds
Under review as a conference paper at ICLR 2027

BLAZE: Bias-Driven Load-Aware Zero-Overhead Expert Routing for MoE LLM Inference

Abstract

Expert-parallel Mixture-of-Experts (MoE) inference is usually bottlenecked by routing-induced load imbalance: overloaded experts and EP ranks create stragglers in All-to-All communication and grouped-GEMM execution. Existing placement-level approaches mitigate persistent hotspots by replicating expert weights, but this consumes GPU memory and does not directly correct transient router imbalance at inference time. We present BLAZE, a token-drop-free and replica-free router-side controller for expert-parallel MoE serving. BLAZE redirects only confidence-ambiguous expert choices away from overloaded experts or EP ranks, while locking high-confidence semantic decisions and computing mixture weights from the original router scores. To make redirection deployable in end-to-end serving, BLAZE further uses an overhead-aware activation gate that bypasses the controller on small effective MoE batches where redirection overhead is unlikely to pay off. Across GPT-OSS-20B and Qwen3-30B-A3B, BLAZE stays within 1.6 percentage points of vanilla accuracy, while random or worst-ranked replacement collapses accuracy by 44–49 percentage points; on GPT-OSS-120B, three complete MLPerf accuracy runs of BLAZE all pass the official threshold with no loss relative to vanilla. On DeepSeek-R1, the overhead-aware gate avoids a 16% TPOT regression from always-on redirection. Under MLPerf-style serving with TP = EP = 8 at SLO-feasible load, BLAZE turns a vanilla TTFT violation on GPT-OSS-20B into an SLO pass (26.7% lower TTFT P99) and, in a matched QPS sweep, still meets the latency SLO at 1.75 QPS, whereas capacity-aware routing fails from 1.25 QPS; on Qwen3-30B-A3B, all methods meet the SLO and BLAZE delivers the highest completed-token throughput. BLAZE also improves offline throughput by 8.9% on GPT-OSS-120B.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wU5vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents BLAZE, a replica-free expert routing method that aims to reduce load imbalance in MoE inference. It uses expert-level and rank-level load feedback to redirect tokens toward less-loaded experts, while protecting high-confidence routing decisions to limit accuracy degradation. The authors also introduce an overhead-aware activation gate that bypasses the additional routing logic for small batches where its overhead may outweigh the benefits. BLAZE is integrated into TensorRT-LLM with CUDA Graph support and evaluated on multiple MoE models. The experiments show near-baseline accuracy, improvements in tail latency and SLO compliance under certain serving workloads such as large-batch inference.

Critical strengths
  • Important and practical problem. Expert load imbalance is a well-known bottleneck in MoE inference, and many previous works, such as EPLB, have tried to address it. Finding a way to balance expert workloads without replicating expert weights is valuable, especially for memory-constrained serving systems. The problem itself is important and well-motivated.
  • Solid system implementation and evaluation. A major strength is that BLAZE is implemented in TensorRT-LLM with CUDA Graph support, rather than being evaluated only in simulations or microbenchmarks. This makes the results more convincing and relevant to real-world serving. The authors also evaluate the method from several angles, including offline throughput, online latency (TTFT and TPOT), and SLO compliance.
Critical weaknesses
  • Limited novelty. The main ideas behind BLAZE have already been explored in previous work. [1] and [2] have studied inference-time expert rerouting to reduce load imbalance, while [3] introduced load-dependent routing biases and the decoupling of expert selection from mixture weights. The additional rank-aware feedback and activation gate seem relatively incremental, making the overall methodological contribution rather limited.
  • Insufficient motivation for modern MoE models. Recent MoE models have adopted more advanced load-balancing techniques during training, such as Quantile Balancing in Kimi K3. Although these techniques do not guarantee balanced routing during inference, they may reduce the severity of the problem. The paper does not provide enough evidence showing that expert load imbalance remains a major inference bottleneck in recent models, especially in terms of its actual impact on execution time.
  • Insufficient ablation and kernel-level performance analysis. The paper provides several ablations and high-level latency measurements, but the connection between reduced load imbalance and actual kernel speedup is still unclear. More balanced expert token counts do not necessarily lead to faster execution, since MoE kernel performance also depends on tiling, scheduling, and hardware utilization. The effectiveness of the method may also vary significantly across different MoE backends(triton, flashinfer, cutlass, megamoe), while the current experiments do not systematically investigate this.

[1] Shahout R, Cai C, Du Y, et al. From score distributions to balance: Plug-and-play mixture-of-experts routing[J]. arXiv preprint arXiv:2510.03293, 2025.

[2] He S, Cai W, Huang J, et al. Capacity-aware inference: Mitigating the straggler effect in mixture of experts[C]//International Conference on Learning Representations. 2026, 2026: 92274-92299.

[3] Wang L, Gao H, Zhao C, et al. Auxiliary-loss-free load balancing strategy for mixture-of-experts[J]. arXiv preprint arXiv:2408.15664, 2024.

Questions for authors
  1. How does BLAZE differ from existing inference-time expert rerouting methods, especially [1] and [2] Expanded Drop? What are the key improvements over these approaches?
  2. How severe is expert load imbalance in recent MoE models that already use advanced training-time balancing techniques, such as Kimi K3? Can the authors quantify how much this imbalance actually slows down inference?
  3. Can the authors provide a more detailed per-layer breakdown showing how each component reduces expert/rank imbalance and how this translates into grouped-GEMM and All-to-All execution time? It would also be helpful to understand how these benefits change across different batch sizes and MoE backends.
6 comments

We thank the reviewer for the detailed assessment and for recognizing the importance of replica-free balancing and BLAZE's solid system implementation and evaluation.

[W1/Q1 – Novelty – BLAZE vs. LASER [1], Expanded Drop [2] and aux-loss-free balancing [3]]

[1] and [2] are now cited. At near-matched balance BLAZE keeps 4.8 pp more reasoning accuracy than LASER and replaces the top-1 expert less frequently. On the same router, its TTFT P99 is up to 11% lower at QPS 3 and 5.

What is new. BLAZE ranks all experts by router score minus a bounded load penalty and keeps exactly KK per token. Each substituted slot has score regret ≤ 2αDE+2αrankDR2\alpha D_E+2\alpha_{rank}D_R (Thm. 1), so the top-1 expert is replaced only when its lead is within the penalty range. Load feedback is an EMA updated off the critical path, in a single CUDA-graph-capturable pass.

  • LASER [1] keeps top-KK at high probability mass and otherwise takes the KK least-loaded score-thresholded candidates, ignoring score order; it specifies no load feedback and reports no serving results.
  • Expanded Drop [2] uses a variable number of experts per token and drops tokens beyond capacity; BLAZE keeps exactly KK and drops none. [2]'s Token Drop is already our baseline (Tab. 3/13/20).
  • [3] balances during training. At inference the weights are frozen: arbitrary substitution loses 44–49 pp of accuracy, and 24–34 pp even with the top-1 protected (Fig. 2). BLAZE builds on [3]'s selection–weight decoupling and adds bounded load penalties that limit routing-score regret.

Controlled comparison (GPT-OSS-20B). All policies share one graph-capturable PyTorch router in TRT-LLM 1.3.0rc8 (same EMA load, mixture weights and backend) and differ only in the selection rule. We implement LASER's rule there, capping candidates at each token's top-2K experts by score and ordering them by −load+10−4 logit-\mathrm{load}+10^{-4}\,\mathrm{logit}. Accuracy: AIME25 + GPQA-Diamond (n = 1,230), one GPU, gate off, first run of each policy paired with the same vanilla run. Serving: NminN_{min}=128, CUTLASS, 4 × H200, DP=EP=4, pinned cores, identical prompts and arrivals, exactly 1,024 output tokens, mean of three runs (SD ≤ 2.4% at QPS 3, ≤ 9.3% at QPS 5). BLAZE's native CUDA router (W3/Q3) is faster than this shared router, so absolute latencies differ.

PolicyExpert max/meanAIME/GPQA Δ vs. vanilla (pp)TTFT P99 @ QPS 3 (ms)TTFT P99 @ QPS 5 (ms)
Vanilla3.9101,4753,972
BLAZE α=0.153.27−0.571,4083,257
LASER ε=0.5, t=0.5 (near-matched balance)3.14−5.371,4083,480
LASER ε=0.95, t=0.6 (similar single-GPU accuracy; ε above its reported settings)3.59−0.491,4983,665

At near-matched balance, BLAZE keeps 4.80 ± 2.74 pp more AIME/GPQA accuracy (paired 95% CI, p = 0.001) and about 1.3 pp more on MMLU: LASER removes the top-1 expert for 16.5% of tokens, BLAZE for 0.74%; TTFT P99 is equal at QPS 3 and 6% lower at QPS 5. At similar single-GPU accuracy, BLAZE balances more (3.27 vs. 3.59) and its TTFT P99 is 6% and 11% lower. Balancing as much as BLAZE or more, LASER loses 5.4–18.0 pp and removes the top-1 for 16.5–44.5% of tokens. On Qwen3-30B-A3B, LASER removes the top-1 for 28.4% of tokens (expert max/mean 4.47), BLAZE for 0.98% (4.11), and at near-matched balance BLAZE is 1.5 ± 0.5 pp higher on MMLU.

Activation gate. The gate applies BLAZE only where a rank holds enough tokens for balancing to pay off; on GPT-OSS-120B it acts on the rank holding the prefill chunk, where the slowest rank's grouped-GEMM time falls by 12.9% (W3/Q3).

Contributions (revised). (1) A bounded-regret selection rule that keeps 4.8 pp more accuracy than LASER at near-matched balance and drops no tokens; (2) a profile-free load controller off the critical path in a single-pass, graph-capturable router; (3) an overhead-aware gate and a matched-work SLO evaluation in a production engine with an implementation-matched control (α=0).

[W2/Q2 – Motivation – Imbalance in recent MoEs and its inference cost]

Training-time balancing equalizes load only in expectation over large batches; small, domain-skewed inference batches stay imbalanced. Kimi K3 freezes its Quantile Balancing bias at inference (§2.3.3), and its MoonEP inference system reports skewed per-expert workloads even with balanced rank totals, needing load-aware GEMM scheduling (§5.2.1). K3 report

How severe. Vanilla-router load imbalance in serving (DP=EP=8, one trace per model; K3's 1.56 TB checkpoint exceeds our 8 × H200 node):

ModelExpert load max/meanEP-rank load max/mean
GPT-OSS-20B4.241.80
GPT-OSS-120B12.181.73
Qwen3-30B-A3B8.591.70
Qwen3-235B-A22B8.911.67

Imbalance grows with EP size: rank max/mean reaches 3.90 at 32 virtual EP ranks (GPT-OSS-20B) and 4.11 at 64 (Qwen3-30B-A3B), and BLAZE's reduction grows with it (9.9% → 17.9%; 8.9% → 22.3%).

How much it slows inference. Replaying GPT-OSS-20B's recorded per-step expert counts through the CUTLASS grouped GEMM, uniform counts would cut the busiest rank's GEMM time by 21–36% (pinned). End to end, with no profile and no replica, BLAZE lowers TTFT P99 by 11–15% on GPT-OSS-20B and TTFT and TPOT P99 by 17% on GPT-OSS-120B against the α=0 control (W3/Q3).

[W3/Q3 – Kernel-level breakdown – Per layer, components, batch size and backends]

Setup: GPT-OSS-120B, TRT-LLM CUTLASS, 4 × H200, DP=EP=4, pinned; BLAZE (α=0.15, τ=1.5, NminN_{min}=64, rank penalty off) vs. its α=0 control (same code and CUDA-graph path, load penalty off: only routing differs).

Serving. At QPS 2 (identical prompts and arrivals, exactly 1,024 output tokens), BLAZE lowers TTFT P99 from 2,262 ± 23 to 1,871 ± 2 ms (−17.3%) and TPOT P99 from 80.20 ± 0.65 to 66.73 ± 0.05 ms (−16.8%); mean ± SD, three fresh-server runs per arm; a second set gives −18.9% / −15.4%.

Per layer. We recorded a real prefill+decode serving step (3,247 tokens; selection rule fixed beforehand) and replayed its 36 MoE layers under both policies (identical inputs, frozen EMA from an α=0 run; only α differs). The α=0 replay matches the recorded MoE outputs exactly (all 144 layer–rank pairs). Timing: Nsight Systems, 50 alternating pairs per layer.

LayersExpert max/mean (α=0 → BLAZE)Rank max/mean (α=0 → BLAZE)Slowest-rank GEMM time
0–1110.29 → 4.871.224 → 1.134−5.7%
12–2314.33 → 5.431.424 → 1.182−14.6%
24–3515.99 → 5.881.438 → 1.152−17.5%
All 3613.54 → 5.391.362 → 1.156−12.9%

GEMM time falls 11.3% even without the best layer and tracks the change in rank max/mean across the 36 layers (r = 0.98). The sum of per-layer maximum-rank mean MoE times (routing, GEMMs, communication) falls by 12.4%.

Components. With the rank penalty off, the expert penalty alone (safeguards fixed) produces these gains (busiest rank's load −15%). The gate restricts BLAZE to ranks with enough tokens (here one of four), yet load shifts on all four. Offline, BLAZE changes 7.7–22.0% of GPT-OSS-20B's expert slots per layer but replaces the top-1 for only 0.01–4.4% of tokens. Its router costs 1.0–3.2 µs more per layer than vanilla at N≤1,024 (pinned).

Workload dependence. In a 16,384-token prefill-only step (four 4,096-token MLPerf prefills), rank max/mean falls from 1.395 to 1.328, the slowest rank's GEMM time by 4.6% and the MoE time by 4.4% on every rank: the gain follows how much BLAZE unloads the busiest rank.

Communication. CUTLASS here uses AllGather/ReduceScatter semantics (NCCL Broadcast/Reduce), so message sizes do not depend on routing. The collective before the expert GEMMs is unchanged (217.4 → 216.5 µs per layer); the one after them halves (2,115.3 → 1,009.8 µs), consistent with ranks waiting less for the slowest one.

Backends. CUTLASS, GPT-OSS-20B (4 × H200, DP=EP=4, pinned, three runs per cell): BLAZE's TTFT P99 is 845 ± 3 / 1,259 ± 3 ms at QPS 1 / 3, vs. 970 ± 1 / 1,422 ± 14 ms for α=0 (−13% / −11%; −14% / −15% at QPS 2 / 5), with vanilla within 0.3% of α=0 at QPS 1 / 3. TRT-LLM as released (TRITON, DP=EP=8, unpinned, one run per cell): BLAZE raises GPT-OSS-20B's SLO-compliant load from QPS 3.0 to 4.0 and lowers TTFT P99 at QPS 3.0 from 2,336 to 273 ms; its α=0 control reaches 336 ms, so this gain is delivered by BLAZE's CUDA routing path, which largely avoids the released Triton router's recompilation. GPT-OSS-120B under the MLPerf v6.0 Server protocol with LoadGen (CUTLASS, pinned, one run per arm; not an official MLPerf result): TTFT / TPOT P99 18% / 8% below vanilla. FlashInfer and MegaMoE are unavailable as MoE backends in TRT-LLM 1.3.0rc8 on H200 (FlashInfer MoE requires SM100/103).

We will add these results to the paper. If our responses have addressed your concerns, we would be sincerely grateful if you would kindly consider raising your score.

Dear Reviewer,

The authors have posted their rebuttal. If any of your concerns remain unaddressed, or if the response raises new questions, please let authors know.

Authors can no longer respond after 11/19, so now is the time to follow up.

Thanks for your time and effort.

AC

Reviewer_4CvwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

BLAZE is a token-drop-free, replica-free controller for expert-parallel MoE inference. It subtracts expert-load and EP-rank-load EMA penalties from routing scores, locks the original top-1 expert when its margin exceeds a threshold, and selects the remaining slots using the adjusted scores. Mixture weights are computed from the original scores over the selected set. An effective-token-count gate bypasses the controller on small MoE calls. The approach preserves the number of selected experts but changes their logical identities, making it an approximate inference method rather than a semantics-preserving placement optimization.

The main Server evaluation uses TensorRT-LLM on eight H200 GPUs with TP=EP=8. The paper reports selected TTFT/SLO improvements, near-vanilla accuracy in several evaluated settings, and avoidance of an approximately 16% TPOT regression caused by always-on control. Additional experiments examine calibration, fixed-work latency, controller overhead, request-order sensitivity, and rank-skewed workloads.

Critical strengths

S1. A practical implementation that explicitly addresses the cost of inference-time feedback

The implementation goes beyond adding a load penalty to router scores. It maintains device-resident controller state, supports parameter updates without CUDA Graph recapture, and provides a true bypass that avoids routing kernels, count writes, and controller updates when inactive. Its persistent state is reported as approximately 37 KB for the GPT-OSS-120B configuration, without allocating additional expert replicas. These are useful deployment properties for a router-side optimization.

The DeepSeek-R1 activation experiment provides concrete evidence that this overhead control matters: always-on routing increases mean TPOT from approximately 68.24 ms to 79.16 ms, while the activation gate restores it to approximately 68.18 ms. This does not demonstrate a small-batch decoding speedup, but it establishes an important failure mode and an implemented safeguard against it. That connection between controller overhead and effective batch size is a meaningful positive contribution.

S2. Substantial diagnostic effort, including controls that qualify the headline claims

The paper evaluates expert substitution through task-level accuracy rather than assuming that preserving Top-K cardinality preserves quality. The GPT-OSS-120B study includes one vanilla and three complete BLAZE executions, with paired intervals and an appropriately limited interpretation: the authors do not claim a statistically significant accuracy improvement.

The fixed-work Shadow/Active/Vanilla experiment also distinguishes controller cost from the effect of changed routing. Moreover, Appendix K’s one-output-token experiment reports a 44.2% increase in prompt throughput, providing evidence that at least one measured gain does not depend on shortening autoregressive generations. These controls strengthen the empirical case for a potentially useful implementation, even though they do not fully establish the main online-capacity claim.

Critical weaknesses

W1. The incremental quality–efficiency benefit over close inference-time routing methods is not sufficiently established

A particularly close external prior work is LASER—Shahout et al., “From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing,” arXiv:2510.03293 (2025). LASER already conditions expert substitution on gate-score confidence, broadens the candidate pool when preferences are diffuse, and selects less-loaded experts without retraining. BLAZE’s EMA/rank penalties, top-1 guardrail, and graph-compatible execution differ from LASER, but those differences need to be evaluated rather than treating confidence-aware, replica-free inference-time balancing itself as the new contribution.

Selection–weight decoupling also has a training-time precedent in Auxiliary-Loss-Free Load Balancing, which BLAZE appropriately acknowledges. The paper’s random, worst-ranked, and top-1-plus-random controls establish that arbitrary substitution is damaging; they do not establish superiority over a plausible score-constrained, load-aware alternative. The missing comparison is consequential because it could show whether the proposed combination offers a better quality–efficiency trade-off or primarily provides an implementation of an existing general idea.

The serving evidence also does not fully isolate the benefit at comparable work and quality. In Table 3, GPT-OSS-20B’s TTFT improvement accompanies lower completed-token throughput and higher TPOT. A common output cap does not guarantee equal generated work when the router changes model outputs, so generation-length and batching effects remain possible contributors. This does not invalidate the observed TTFT improvement, but it limits attribution to faster execution of comparable work. The fixed-128-token experiment addresses this issue at one operating point, where Active–Vanilla intervals include zero; it does not yet establish the benefit near the reported SLO boundary.

The quality and serving configurations should also be explicitly aligned. For example, the main GPT-OSS-20B Server configuration disables overhead-aware activation, while the Qwen Server row uses $$\tau=5.0=5.0; neither should automatically inherit the strongest quality conclusion from a different BLAZE configuration.

W2. The theoretical guarantees depend on assumptions whose applicability to the proposed controller is not established

The quality bound assumes the key semantic relationship that needs justification. Assumption A1 bounds the output change from an expert substitution by GmoeηG_{\mathrm{moe}}\eta, where η\eta is router-score regret. Similar scores alone do not constrain the distance between expert outputs: two equal-scoring experts can produce different vectors, giving a nonzero output change even at zero score regret. A1 could describe an additional empirical regularity in a restricted regime, but the current task-accuracy experiments do not establish that regularity or useful constants. Consequently, the theorem does not independently demonstrate that low-score-regret substitutions are safe.

The load-contraction assumption is stronger than ordinary negative feedback and appears incompatible with important target cases. Setting u=1u=\mathbf1 and ρ=1\rho=\mathbf1 in Equation (6) requires $ |p(\mathbf1,\mathbf1)-\mathbf1|_\infty \le \xi_N, \xi_N=O(N^{-1/2}). $ However, uniform controller states subtract the same penalty from every expert, recovering vanilla Top-K. For a traffic distribution with persistent vanilla routing skew, the expected assignment rate need not approach uniformity as the batch grows. Thus, the assumed response condition does not generally describe the imbalance that BLAZE is intended to correct. This challenges the applicability of the convergence guarantee; it is not a proof that the empirical controller cannot improve balance.

There is also a proof-level gap in Step 6: Equation (6) is stated for an expected response, but is applied directly to realized assignments by asserting that ξN\xi_N absorbs sampling error. This requires an explicit stochastic bound or a stronger stated assumption. These issues are consequential because Appendix A presents accuracy stability, load stability, and efficiency guarantees as support for the method. The empirical contribution may remain viable, but the theoretical claims need justification, revision, or narrower scope.

Questions for authors

Q1. About incremental contribution relative to LASER

How does BLAZE differ from LASER in eligible expert choices, load feedback, and the resulting quality–efficiency trade-off?

A focused comparison on one supported model, with comparable kernel/CUDA-Graph treatment and an explicit quality budget, would materially affect my novelty assessment. A targeted ablation of the top-1 lock against a credible score-constrained alternative would also be more discriminating than random or worst-expert replacement. The requested comparison is about the proposed routing policy, not whether an unoptimized Python implementation can match a fused CUDA implementation.

Q2. About applicability of the theoretical guarantee

What evidence or model property supports A1’s relationship between router-score regret and expert-output perturbation?

For A3, how can Equation (6) hold at uniform controller states when native routing has persistent expected skew? Does the intended result concern a potentially nonuniform fixed point instead, or a restricted traffic regime? Please also specify the probability or expectation interpretation needed to apply the response bound to realized assignments in Step 6. A revised or explicitly narrowed statement would help distinguish what is proved from what is supported empirically.

Q3. About performance at matched work and quality

Can the principal SLO-boundary comparison be repeated with identical prompts, arrival traces, and fixed output-token counts, reporting variation across independent runs?

I recognize that Appendix K already provides a fixed-work control; the missing connection is to the load range supporting the main capacity claim. Please identify the exact α\alpha, αrank\alpha_{\mathrm{rank}}, τ\tau, and Nmin⁡N_{\min} used in each compared serving configuration, and provide corresponding task-level quality results or clearly identify the existing measurements that match those settings. For variable-length runs, reporting actual output-length distributions would also help separate execution improvements from changes in generated work.

Q4. About token-count convention in the activation threshold

Section 2.3 estimates assignments per local expert as NK/(E/EP)NK/(E/EP), while Appendix C states that ranks in the main TP=EP configuration observe the same router stream. What exactly does NN count in this derivation and in the implementation?

If NN denotes the unique tokens in that replicated stream, the balanced expected count per expert appears to be NK/ENK/E, whereas NK/(E/EP)NK/(E/EP) would require a different per-rank input convention. Please reconcile these definitions and explain whether the distinction affects the proposed one-tile threshold, the manually selected thresholds, or only their stated derivation.

3 comments

We thank the reviewer for the detailed assessment and for recognizing BLAZE's practical implementation and diagnostic evaluation.

[W1/Q1/Q3 – Incremental benefit, matched work and quality]

On one shared router, BLAZE reaches lower expert imbalance than LASER at all three tested quality budgets and keeps 4.8 pp more AIME/GPQA accuracy at near-matched balance; at matched work, its balancing lowers TTFT P99 by 11–15% against an implementation-matched control. We now cite LASER.

Policies. LASER keeps the vanilla top-K when its probability mass is high and otherwise picks low-load, score-qualified experts; BLAZE subtracts bounded load penalties, with a top-1 safeguard. Both run in one graph-capturable PyTorch router in TRT-LLM 1.3.0rc8 with the same EMA and mixture weights; we implement LASER's rule there within its reported parameter range, capping candidates at each token's top-2K by score and ordering them by −load+10−4 logit-\mathrm{load}+10^{-4}\,\mathrm{logit}.

Quality at matched budgets (GPT-OSS-20B, one GPU, gate off; AIME/GPQA n=1,230, MMLU n=14,042). Budgets use BLAZE's multi-run mean loss where available (α=0.15: −1.5 ± 0.8 pp, 7 vs. 8 runs):

AIME/GPQA budgetLowest expert max/mean: BLAZELASER (reported range)
≈1 pp3.453.64
≈2 pp3.273.58
≈3 pp2.893.58

Vanilla's expert max/mean is 3.91. At near-matched balance (BLAZE α=0.15 at 3.27, LASER ε=0.5, t=0.5 at 3.14), BLAZE scores +4.80 ± 2.74 pp higher on AIME/GPQA (paired 95% CI, p=0.001) and about 1.3 pp higher on MMLU. LASER settings that balance as much or more lose 5.4–18.0 pp and remove the top-1 expert for 16.5–44.5% of tokens, against 0.74–2.4% for BLAZE. On Qwen3-30B-A3B at near-matched balance, BLAZE is 1.5 ± 0.5 pp higher on MMLU (p<0.001; LASER ε=0.95, above its reported settings).

Serving on the same router (BLAZE α=0.15, N_min=128; CUTLASS, 4 × H200, DP=EP=4, pinned; identical prompts and arrivals, exactly 1,024 output tokens, three runs per arm). Against LASER ε=0.95, t=0.6 (similar single-GPU accuracy), BLAZE's TTFT P99 is 6% / 11% lower at QPS 3 / 5 (1,408 / 3,257 vs. 1,498 / 3,665 ms), with equal or 1.5% higher throughput.

Lock ablation. Removing the lock changes GPT-OSS-20B's AIME/GPQA by −0.9 to +1.1 pp and MMLU by −0.3 to +0.3 pp (α=0.15–0.6), within noise; the bounded penalty already protects high-margin tokens (Thm. 1). Against a hard top-(K+1) constraint at near-matched balance, BLAZE is within noise on AIME/GPQA (−1.5 ± 2.8 pp) and higher on MMLU by 0.6 ± 0.5 pp on GPT-OSS-20B (p=0.03) and 0.8 ± 0.4 pp on Qwen3 (p<0.001).

Matched work with an implementation-matched control (native CUTLASS router, same protocol, 300 requests/cell; α=0.15, τ=1.5, N_min=64, rank penalty off; α=0 runs BLAZE's code path with vanilla routing). TTFT P99 in ms, mean ± SD:

ArmQPS 1QPS 3QPS 5
Vanilla971 ± 21,418 ± 233,889 ± 142
BLAZE α=0970 ± 11,422 ± 143,734 ± 44
BLAZE845 ± 31,259 ± 33,177 ± 354

Against α=0, BLAZE lowers TTFT P99 by 13 / 14 / 11 / 15% at QPS 1 / 2 / 3 / 5 and TPOT P99 by 4–10%, with throughput equal to vanilla's at QPS 1–2 and 0.5 / 1.3 / 2.0% higher at QPS 3 / 4 / 5. On GPT-OSS-120B (QPS 2, three runs per arm), TTFT P99 falls from 2,262 ± 23 to 1,871 ± 2 ms and TPOT P99 from 80.20 to 66.73 ms (−17.3% / −16.8%).

Released TRITON and output length (DP=EP=8, unpinned, one run per cell). BLAZE moves GPT-OSS-20B's SLO boundary from QPS 3 to 4 (TTFT P99 at QPS 3: vanilla 2,336, α=0 336, BLAZE 273 ms). These routing-path gains come from avoiding batch-bucket recompilation, and we reattribute Tab. 3's GPT-OSS rows to it. As the reviewer noted, Qwen3's outputs were ~28% shorter, so we replace its row with matched-work WIDEEP results (QPS 6 for both); GPT-OSS-20B's are similar (1,055 / 1,062 tokens), and its higher TPOT disappears at matched work.

Quality at each serving configuration (β=0.85; Δ vs. vanilla in pp; MLPerf as labeled):

Configurationα / α_rank, τ, NminN_{min}Accuracy Δ (pp)
GPT-OSS-20B Server (Tab. 3)0.15 / 0.02, 1.5, 32live +1.8 / +1.4 / −0.5; batched −3.2 / −2.0 / −2.9
GPT-OSS-120B Offline (Tab. 3)0.15 / 0, 1.5, no gate+0.47 (MLPerf, 3 runs, all pass)
Qwen3 Server (Tab. 3)0.15 / 0.02, 5.0, 64+0.6 (WIDEEP), +1.1 (TRITON); 16k-token cap
DeepSeek-R1 (Fig. 3)0.015 / 0.0025, 0.025, 256 per rank+2.0 ± 2.3
GPT-OSS-20B CUTLASS (above)0.15 / 0, 1.5, 64one-GPU proxy: −1.5 ± 0.8

At α ≤ 0.1, AIME/GPQA stays within vanilla's run range (52.6–53.2% vs. 52.5–54.7%), while expert max/mean still falls to 3.45–3.63.

[W2/Q2 – Theory – A1, A3 and Step 6]

We agree with all three observations and revise App. A into a conditional analysis. What is proved without A1 stays: every slot BLAZE changes has score regret at most Γ=2αDE+2αrankDR\Gamma=2\alpha D_E+2\alpha_{rank}D_R (using only A2, which clipping enforces), so whenever s(K)−s(K+1)>Γs_{(K)}-s_{(K+1)}>\Gamma, BLAZE selects exactly vanilla's set for the same input and candidate pool.

A1. We measured it. On Qwen3-30B-A3B (48 AIME/GPQA prompts, 96 prefill tokens × 48 layers, all 8 × 120 single-slot swaps), the relative MoE-output change ν=∥Δy∥/∥y∥\nu=\|\Delta y\|/\|y\| grows only weakly with score regret η\eta (per-layer Spearman 0.22–0.46) and does not vanish as η→0\eta\to0: ν≈0.165+0.056 η\nu\approx0.165+0.056\,\eta over 144,000 sampled swaps. This does not support A1's zero intercept, so Eqs. 9 and 12 are removed as guarantees. The data support BLAZE's design instead: which slot is swapped matters most (median ν\nu 0.62 for the top-1 slot, 0.14 for the eighth), and BLAZE removes the top-1 for only 0.18–0.74% of tokens at α=0.15.

A3. At the uniform state all experts receive the same penalty, so routing is vanilla Top-KK and ∥p(1,1)−1∥∞=δE0\|p(\mathbf 1,\mathbf 1)-\mathbf 1\|_\infty=\delta^0_E, not O(N−1/2)O(N^{-1/2}). For the joint expert/rank EMA state zz with expected response FF and a possibly nonuniform fixed point z⋆=F(z⋆)z^\star=F(z^\star) (expert component u⋆u^\star), assume stationary traffic and local contraction ∥F(z)−F(z⋆)∥∞≤(1−κ)∥z−z⋆∥∞\|F(z)-F(z^\star)\|_\infty\le(1-\kappa)\|z-z^\star\|_\infty. With EMA factor β and a finite-horizon response-error bound ξN\xi_N keeping iterates in this region, ∥ℓˉt−1∥∞≤∥u⋆−1∥∞+(1−(1−β)κ)t∥z0−z⋆∥∞+ξN/κ.\|\bar\ell^t-\mathbf 1\|_\infty\le\|u^\star-\mathbf 1\|_\infty+\big(1-(1-\beta)\kappa\big)^t\|z^0-z^\star\|_\infty+\xi_N/\kappa . After the transient, expert EMA imbalance stays within δ⋆+ξN/κ\delta^\star+\xi_N/\kappa, where δ⋆=∥u⋆−1∥∞\delta^\star=\|u^\star-\mathbf 1\|_\infty, rather than reaching uniformity (measured expert max/mean 3.91 → 3.27, GPT-OSS-20B, α=0.15). A4 and Eq. 13 are revised likewise. The efficiency bounds (Eqs. 11, 14, 15) now use δ⋆\delta^\star (and its A4 rank analogue) instead of uniform load and hold under A5; a sufficient condition for a steady-state gain (Eq. 11) is λE(δE0−δ⋆−ξN/κ)>Ω\lambda_E(\delta^0_E-\delta^\star-\xi_N/\kappa)>\Omega.

Step 6. For illustration, if token assignments within a step are independent given the controller state, Hoeffding's inequality and a union bound over experts give ∥p^(t)−p(zt)∥∞≤ξN=EKlog⁡(2E/δ)/(2N)\|\hat p^{(t)}-p(z^{t})\|_\infty\le\xi_N=\frac{E}{K}\sqrt{\log(2E/\delta)/(2N)} with probability ≥1−δ\ge1-\delta per step (1−Tδ1-T\delta over TT steps). Here pp is FF's expert component; correlated traffic needs a separate finite-horizon sampling bound.

Our accuracy and speed claims rest on measurements (W1/Q1/Q3), not on App. A.

[Q4 – Activation threshold – What NN counts, and its effect]

We thank the reviewer for this careful reading. Aligning the two conventions changes the derivation and where our TRITON thresholds fall relative to the band; it changes no measurement, and the corrected threshold shows no clear latency change (three paired runs).

What NN counts. In code, NN counts router-logit rows on the gate's rank, padding included. On WIDEEP and CUTLASS each rank routes its own NrN_r tokens (§2.3's convention); on TRITON the model all-gathers first, so every rank routes Ng=∑rNrN_g=\sum_r N_r tokens, and App. C's "same router stream" holds only there (Tab. 3's "TP=EP=8" is attention DP, DP=EP=8). Balanced expert load is NgK/EN_gK/E, equal to NrK/(E/EP)N_rK/(E/\mathrm{EP}) for equal per-rank counts; only the count compared with NminN_{min} differs, so on TRITON the band is NgN_g from 0.25BME/K0.25B_ME/K to BME/KB_ME/K (GPT-OSS-20B: 256–1,024). The auto mode is quarter-tile, corrected here to 256 gathered rows (one tile is 1,024); manual thresholds stay as measured but shift relative to the band.

Effect of the corrected threshold (GPT-OSS-20B, TRITON router with recompilation fixed, 4 × H200, DP=EP=4, pinned, QPS 3, identical prompts and arrivals, exactly 1,024 output tokens; α=0.15, α_rank=0, τ=1.5). Mean ± SD, three fresh-server runs:

PolicyNminN_{min} (gathered rows)TTFT P99 (ms)TPOT P99 (ms)
Vanilla–402.84 ± 127.787.12 ± 0.14
BLAZE, existing auto64311.86 ± 4.597.20 ± 0.13
BLAZE, corrected threshold256301.61 ± 11.997.12 ± 0.07

Paired TTFT / TPOT P99 differences are −3.3 ± 4.6% / −1.1 ± 1.5%: no clear change. A wider sweep on the same router (NminN_{min} 0–2,048, QPS 3, unpinned, one run per point) is flat too: TTFT P99 299.0–309.7 ms, TPOT P99 7.51–7.72 ms. The threshold matters where routing overhead is exposed: on DeepSeek-R1 (WIDEEP, unpinned), always-on routing adds 16% mean TPOT, which the gate at NminN_{min}=256, inside the band, removes (Fig. 3) with no accuracy change. We will correct §2.3, Eq. 28, Alg. 2 and App. C.

If our responses have addressed your concerns, we would be sincerely grateful if you would kindly consider raising your score.

The authors have addressed part of my concerns. The current version looks fine to me. I would like to change the rating score from 2 to 3.

Reviewer_UQhCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces BLAZE, a replica-free and token-drop-free router-side load balancer for expert-parallel (EP) Mixture-of-Experts (MoE) LLM inference. In distributed MoE serving, routing-induced imbalance across experts and EP ranks creates severe stragglers in All-to-All communication and grouped-GEMM execution. Existing placement-level solutions like EPLB duplicate hot expert weights across GPUs, which consumes substantial GPU memory and limits KV cache capacity.

BLAZE addresses transient routing imbalance at runtime through three tightly coupled mechanisms:

  1. Confidence-Aware Semantic Redirection: High-confidence routing decisions are protected using a semantic margin threshold on the top-1/top-2 router scores (Δ(x)=se1(x)−se2(x)>τ\Delta(x) = s_{e_1}(x) - s_{e_2}(x) > \tau). Only ambiguous top-1 choices and fallback expert slots are exposed to load-penalized scores (r=s−αℓˉ−αrankρˉr = s - \alpha \bar{\ell} - \alpha_{\mathrm{rank}} \bar{\rho}).
  2. Selection-Weight Decoupling: Expert selection is driven by the load-adjusted scores, but the mixture weights for the selected experts are computed strictly from the original unpenalized semantic scores.
  3. Overhead-Aware Activation Gating: Redirection logic is bypassed via a token-count threshold (N<NminN < N_{\mathrm{min}}) derived from the grouped-GEMM tile floor, preventing control-path and dispatch overhead from causing latency regressions on small decode batches.

The system is implemented inside TensorRT-LLM and evaluated on 8xH200/H100 nodes across GPT-OSS-20B/120B, Qwen3-30B-A3B, and DeepSeek-R1. The authors demonstrate that BLAZE preserves accuracy within 1.6 pp of vanilla (and zero degradation under the complete MLPerf v6.0 protocol on GPT-OSS-120B), turns TTFT SLA violations into passes at SLO-feasible QPS, and frees up to 0.84-1.11 GiB/GPU compared to weight-replicating baselines.

Critical strengths
  1. Practical and High-Impact System-Algorithm Co-Design: The paper tackles a major real-world bottleneck in large-scale MoE serving: expert replication (e.g., EPLB) directly consumes VRAM that would otherwise be allocated to the KV cache, while token-dropping mechanisms cause severe quality degradation. BLAZE provides a clean, replica-free solution with negligible memory footprint (~37 KB) and full CUDA Graph compatibility. The coupling of margin-based top-1 locking, selection-weight decoupling, and hierarchical (expert + EP-rank) feedback elegantly addresses transient router skew without requiring any offline weight migration or retraining.

  2. Rigorous, Transparent, and Thorough Empirical Evaluation: The authors present exceptionally honest and well-scoped experimental analyses rather than overclaiming:

  • They establish the empirical feasibility boundary of expert substitution, clearly proving that arbitrary replacement degrades accuracy by 44-49 pp and that top-1 locking alone is insufficient (24-34 pp drop), substantiating why load-aware fallback selection is strictly required.
  • They candidly ablate the small-batch penalty (e.g., showing always-on redirection causes a 16% TPOT regression on DeepSeek-R1), which justifies their overhead-aware activation gate.
  • They evaluate on comprehensive benchmarks (AIME25, GPQA-Diamond, LiveCodeBench-V6) and validate against the complete MLPerf v6.0 accuracy protocol over multiple runs with rigorous bootstrap confidence intervals.
Critical weaknesses
  1. Accuracy-Throughput Trade-off and Calibration Portability in Serving Settings: As shown in Figure 4 and Appendix E, the margin threshold tau varies drastically across model families (tau ~ 0.78 on GPT-OSS-20B vs. tau ~ 5.09 on Qwen3-30B-A3B). Although the authors propose an online quantile tracker (q = 0.90) to achieve architectural portability without accuracy loss, the primary Server-scenario evaluation in Table 3 uses a fixed threshold (tau = 5.0 for Qwen3-30B-A3B). Notably, Table 1 shows that setting tau = 5.0 on Qwen3-30B-A3B causes a 1.63 pp drop in accuracy compared to vanilla. This indicates that the serving gains reported in Table 3 for Qwen3 come at the cost of non-negligible accuracy degradation, slightly undermining the "accuracy-safe" claim unless the online tracker is verified under the same end-to-end serving conditions.

  2. Fragmented Head-to-Head Baseline Comparisons: Due to various structural and framework integration limitations (acknowledged in Appendix F), direct head-to-head comparisons against state-of-the-art balancers are fragmented:

  • On GPT-OSS-20B, EPLB is marked "n/a" due to an integration issue in TensorRT-LLM.
  • On Qwen3-30B-A3B, Capacity-Aware routing is marked "n/a" due to CUDA Graph incompatibility with dynamic nonzero() shapes.
  • Libra is evaluated solely through its released SGLang script without CUDA Graphs (achieving only ~0.51 req/s, where its fast path never activated), precluding an on-graph, fair comparison in TRT-LLM. While the authors are transparent about these engineering obstacles, it means BLAZE is compared against EPLB on only a single model and single SLO-feasible operating point.
Questions for authors
  1. Serving Performance under Online Quantile Tracking: In Table 3, the Qwen3-30B-A3B result uses a fixed tau = 5.0, which Table 1 indicates incurs a 1.63 pp accuracy degradation. When using the recommended online quantile tracker (q = 0.90), how much of the completed-token throughput and latency (TTFT P99) benefit is retained on Qwen3-30B-A3B? Does the periodic histogram reduction and quantile synchronization introduce noticeable jitter to P99 latencies under live continuous batching?

  2. Generation Quality on Non-Reasoning Tasks: The accuracy validation focuses heavily on reasoning, mathematics, and code benchmarks (AIME25, GPQA-Diamond, LiveCodeBench). How does BLAZE impact tasks that may be more sensitive to subtle token-level nuances or factual recall, such as standard multi-turn conversational benchmarks or open-domain perplexity (e.g., Wikitext-103 or C4)? Are certain layers or token types disproportionately affected by fallback expert redirection?

  3. Behavior in Multi-Node Distributed Deployments: All experiments are performed on a single 8xH200/H100 node connected via NVLink. In multi-node expert parallelism (e.g., 16 or 32 GPUs across InfiniBand/RoCE), inter-node All-to-All communication latency and cross-switch bandwidth bottlenecks are substantially more severe. Does the rank-level feedback penalty (alpha_rank) require retuning in such topologies, and does BLAZE maintain its straggler reduction benefits when inter-node communication latency dominates?

4 comments

We thank the reviewer for the detailed assessment and for recognizing BLAZE's practical, replica-free design and the rigor and transparency of its evaluation.

[W1 – Accuracy–Throughput Trade-off – Calibration portability and serving quality]

The transferable component is the quantile rule q=0.90, which adapts τ to each model’s router-score distribution (App. E). We agree that the quality evaluation should cover the online configuration used for serving, and have added this comparison on Qwen3-30B-A3B.

We evaluated fixed τ=5.0 and online q=0.90 on WIDEEP with DP=EP=8 and NminN_{min}=64. Quality was measured through the serving engine using batched evaluation at concurrency 512.

Tab. 1 reports −1.63 pp with a 4,096-token cap. In a separate TRITON re-run at this cap, 81% (996/1,230) of vanilla outputs reached the limit, motivating additional generation and evaluation with a 16,384-token budget. This substantially reduces truncation, although some outputs still reach the cap: 14.3% for vanilla and 11.5% for BLAZE on TRITON.

Table 1. Qwen3 accuracy differences relative to vanilla (percentage points; paired 95% confidence intervals, one run per configuration). The AIME25 and GPQA-Diamond results report accuracy over their combined evaluation set, using the 16,384-token cap.

EvaluationBackendFixed τ=5.0Online q=0.90
AIME25 and GPQA-Diamond, combinedWIDEEP+0.57 ± 2.35+1.71 ± 2.27
AIME25 and GPQA-Diamond, combinedTRITON+1.06 ± 2.39+1.30 ± 2.26
MMLUTRITON−0.18 ± 0.36−0.17 ± 0.36

Under the longer generation budget, neither configuration shows a statistically significant decrease in the combined AIME25 and GPQA-Diamond accuracy on either backend. The online configuration also shows a small MMLU difference of −0.17 pp. These results provide additional quality evidence for online calibration under the tested settings.

We will add these configuration-specific results. The corresponding matched-work serving results and quantile-tracker overhead are reported in Q1.

[W2 – Fragmented baselines – Shared-backend comparisons and profile sensitivity]

Adapting to online traffic. The tested static EPLB keeps its offline-profiled placement fixed: when serving traffic differs from that profile, the placement does not adjust to live load. BLAZE updates load penalties from live routing counts without an offline expert-load profile. We test this profile–traffic mismatch directly below.

We compare them using TRT-LLM 1.3.0rc8, CUTLASS, four H200s, DP=EP=4, pinned to reserved cores. EPLB uses TRT-LLM's placement algorithm without replicas.

GPT-OSS-20B: profile mismatch. EPLB is calibrated on 16 MLPerf accuracy-set prompts and evaluated on perf-set serving traffic. BLAZE uses α=0.15, τ=1.5, N_min=64, rank penalty off. All arms use identical prompts/arrivals and exactly 1,024 output tokens. Table 2a: TTFT P99 (ms), mean ± SD over three runs:

ArmQPS 1QPS 3QPS 5
Vanilla971 ± 21,418 ± 233,889 ± 142
BLAZE α=0970 ± 11,422 ± 143,734 ± 44
BLAZE (no profile)845 ± 31,259 ± 33,177 ± 354
EPLB, mismatched profile1,002 ± 31,508 ± 124,138 ± 100

Under this mismatch, EPLB is 3–6% slower than vanilla, whereas BLAZE achieves 16–23% lower TTFT P99 than EPLB and 11–18% lower than vanilla. Against α=0, which preserves BLAZE's code path with vanilla routing, the reductions are 13%/11%/15% at QPS 1/3/5. This demonstrates a limitation of fixed-profile placement on mismatched serving traffic and BLAZE's advantage in this tested setting.

Qwen3-235B-A22B. EPLB is profiled on separate MLPerf prompts never used in serving. All arms use 120 serving prompts, identical arrivals and exactly 1,024 output tokens; BLAZE α=0.15, τ=5, N_min=64, rank penalty off. Table 2b: each cell is one run; values are TTFT P99 / TPOT P99 (ms).

ArmQPS 0.5QPS 0.75
Vanilla1,684 / 50.51,749 / 61.6
BLAZE α=01,686 / 50.61,682 / 61.9
BLAZE1,669 / 50.31,656 / 61.3
EPLB1,631 / 49.61,688 / 60.3

BLAZE's TTFT P99 remains within 2.4% of EPLB without profiling: EPLB is lower at QPS 0.5, BLAZE at QPS 0.75. EPLB has lower TPOT at both loads. These single-run results show comparable TTFT, while the repeated 20B experiment establishes BLAZE's advantage under the tested profile mismatch.

Our shared-router LASER comparison additionally gives BLAZE +4.80 ± 2.74 pp AIME/GPQA accuracy at near-matched balance (paired 95% CI). Capacity-Aware's dynamic nonzero remains incompatible with CUDA Graph capture on WIDEEP; an on-graph Libra port requires further integration. We will add these results, profile definitions and implementation boundaries.

[Q1 – Online Quantile Tracking – Serving performance and tail latency]

We compared vanilla, BLAZE with fixed τ=5.0, and BLAZE with online q=0.90 on Qwen3-30B-A3B. Online calibration achieves comparable throughput to fixed τ and preserves the highest passing tested QPS on WIDEEP. Our follow-up evaluation also identified differences in generated output lengths in the original comparison. When output lengths are matched, the original Qwen3 serving advantage is not reproduced.

For the matched-work evaluation, all configurations use identical prompts and Poisson arrivals under continuous batching, with exactly 1,024 output tokens per request.

Table 3a. Qwen3 TTFT P99 on WIDEEP, DP=EP=8, Nmin⁡=64N_{\min}=64 (milliseconds; unpinned). Each cell uses 300 requests and one run.

QPSVanillaBLAZE, fixed τ=5.0BLAZE, online q=0.90
1.0363363367
3.0403417431
3.5421438386

Completed-token throughput, measured as completed output tokens divided by wall-clock time, differs by less than 1% across the three configurations at QPS ≤ 7. At QPS 6, throughput is 4,968 tokens/s for vanilla, 4,959 for fixed τ, and 4,919–4,968 across two online-q runs. Under overload at QPS 8–24, differences remain within 5%, with all three saturating near 9.2k tokens/s.

All three configurations satisfy the SLO at QPS 6 and fail at QPS 7 (TTFT P99 ≤ 3 s; TPOT P99 ≤ 80 ms). Thus, online q preserves the same highest passing tested load as fixed τ at a resolution of 1 QPS. Across the loads in Table 3a, their TPOT P99 differs by at most 0.3 ms.

Repeated runs on four H200 GPUs (CUTLASS, DP=EP=4, pinned, QPS 6): three runs per configuration, interleaved, server restarted per run, identical prompts and arrivals; all 15 runs meet the SLO. Online and fixed τ use different router builds.

Table 3b. Qwen3 on CUTLASS, DP=EP=4, QPS 6 (mean ± standard deviation, three runs). Nmin⁡=64N_{\min}=64 unless otherwise indicated.

ConfigurationTTFT P99 (ms)TPOT P99 (ms)Completed tokens/s
Vanilla1,120 ± 38422.67 ± 0.084,222 ± 32
BLAZE, fixed τ1,176 ± 10123.43 ± 0.114,190 ± 6
BLAZE, online q1,098 ± 5423.31 ± 0.084,194 ± 12
BLAZE, fixed τ, Nmin⁡=0N_{\min}=01,088 ± 16423.60 ± 0.144,142 ± 10
BLAZE, online q, Nmin⁡=0N_{\min}=0925 ± 24523.88 ± 0.094,151 ± 10

Online q's throughput is within 0.3% of fixed τ's in both gate settings.

The quantile tracker requires no inter-rank communication or CUDA Graph recapture. Each rank accumulates a 64-bin device-resident histogram, with threshold updates configured every 50 eager steps. Captured decode graphs replay the update every time or omit it, according to the captured schedule. A CUDA-event microbenchmark on one H200 measures 2.98–3.17 μs per quantile update. Setting Nmin⁡=0N_{\min}=0 collects statistics on every step without changing the configured 50-step eager interval.

In the paired repeats of Table 3b, online q minus fixed τ changes TPOT P99 by −0.12 ± 0.10 ms at Nmin⁡=64N_{\min}=64 and +0.28 ± 0.21 ms at Nmin⁡=0N_{\min}=0 (mean ± standard deviation, three pairs). Across these repetitions, online q shows no increase in the run-to-run standard deviation of TPOT P99.

GPT-OSS-20B also retains the benefit: on CUTLASS (four H200s, DP=EP=4, pinned to reserved cores; three runs), online q's TTFT P99 is 13%/12%/20% lower than the α=0 control at QPS 1/3/5, and matches fixed τ (845 ± 4 / 1,254 ± 4 vs. 845 ± 3 / 1,259 ± 3 ms at QPS 1 / 3), with identical prompts/arrivals and exactly 1,024 output tokens.

[Q2 – Non-Reasoning Tasks – Generation quality and redirection patterns]

We added evaluations of multi-turn conversation, factual knowledge, instruction following, and open-domain language modeling. We observe no statistically significant decrease in judged conversational quality or instruction following; Qwen3 perplexity changes by +0.04% / +0.07%, and factual-knowledge accuracy changes by at most 1.4 pp and shrinks with α. We also measured where expert redirection occurs across layers and token types.

Factual knowledge and instruction following. Table 4 reports differences from vanilla using TRITON, DP=EP=8, greedy decoding, and fixed τ=1.5 for GPT-OSS-20B and τ=5.0 for Qwen3-30B-A3B.

Table 4. BLAZE − vanilla (percentage points; paired 95% confidence intervals, one run per configuration).

BenchmarkGPT-OSS-20BQwen3-30B-A3B
MMLU−0.80 ± 0.48−0.18 ± 0.36
IFEval, prompt-strict−1.29 ± 3.22+1.11 ± 2.11
MMLU-Pro−0.39 ± 0.64−1.35 [−1.91, −0.81]

IFEval shows no statistically significant difference. The knowledge benchmarks show task-dependent costs, including the 1.35 pp decrease on Qwen3 MMLU-Pro. That evaluation covers all 12,032 questions with a 2,048-token budget; unparsed answers count as incorrect.

A separate one-GPU, always-on MMLU evaluation shows that reducing α from 0.15 to 0.05 reduces the observed accuracy decrease from 1.23 to 0.40 pp on GPT-OSS-20B and from 0.38 to 0.16 pp on Qwen3. At α=0.05, both paired confidence intervals include zero.

Open-domain perplexity. On WikiText-103, using teacher forcing and 1,024-token windows, Qwen3 perplexity changes from 9.691 to 9.695 (+0.04%). On the first 64 windows of a C4 validation shard, it changes from 14.828 to 14.839 (+0.07%). WikiText uses TRITON, DP=EP=8; C4 uses CUTLASS, DP=EP=4. GPT-OSS-20B perplexity decreases from 171.9 to 141.7 on WikiText and from 294.9 to 257.6 on C4. Because GPT-OSS is a chat-format model evaluated here on raw text, we report these as auxiliary comparisons rather than evidence of improved general language quality.

Multi-turn conversation. We evaluated 80 two-turn MT-Bench conversations with greedy decoding and a 1,024-token cap per turn. Each second turn uses that run’s own first-turn response. Generation uses CUTLASS, DP=EP=4, and Nmin⁡=64N_{\min}=64: BLAZE is active on 96–98% of prefill tokens, while decode steps below the gate use vanilla routing.

We use a local GPT-OSS-120B judge with vanilla routing and temperature 0, blinded to the configuration, with shuffled answers and custom MT-Bench-style single-answer grading prompts. Task references are included where available. Two vanilla generations provide a comparison for run-to-run variation.

Table 5. MT-Bench judge scores (1–10). Δ compares BLAZE with the mean of the two vanilla runs; intervals are 95% bootstrap confidence intervals over questions.

ModelVanilla 1BLAZEVanilla 2Δ [95% CI]
Qwen3-30B-A3B7.988.147.99+0.16 [+0.01, +0.30]
GPT-OSS-20B8.288.197.92+0.10 [−0.12, +0.34]

Restricting scoring to non-truncated answers gives +0.12 [−0.03, +0.28] and +0.02 [−0.19, +0.26], respectively. An always-on GPT-OSS-20B check (Nmin⁡=0N_{\min}=0) gives +0.12 [−0.19, +0.43]. These comparisons show no statistically significant decrease in judged quality under this evaluation.

Layer and token-type analysis. We replayed router logits from 65,536 teacher-forced WikiText-103 tokens. GPT-OSS-20B changes 14.9% of expert slots, with a semantic top-1 replacement rate of 1.30%; Qwen3 changes 12.7% of slots, with a top-1 replacement rate of 0.57%. Most substitutions therefore affect fallback slots.

Redirection varies by layer. Top-1 replacement rates range from 0.01–4.4% on GPT-OSS-20B, peaking at layer 14, and 0–4.35% on Qwen3, peaking at layer 38. Across measured coarse token classes, GPT-OSS rates range from 0.69% for digits to 1.45% for punctuation; Qwen3 rates range from 0.51–0.71%. On C4, the overall top-1 replacement rates are 1.26% and 0.02%, respectively, showing that the rates also depend on the workload. These statistics describe where routing changes occur; the task-level evaluations above assess their quality effects.

[Q3 – Multi-Node Deployment – α_rank and EP scaling]

BLAZE's rank-imbalance reduction grows with EP size at the same α. Real multi-node EP was beyond our resources in the rebuttal period, so we regrouped recorded single-GPU router traces under virtual EP of 8–64 ranks, which shows how rank imbalance scales with EP size.

  • Stragglers grow with EP size, and BLAZE removes more of them at the same α=0.15. Vanilla's rank max/mean rises to 3.90 at EP 32 on GPT-OSS-20B and to 4.11 at EP 64 on Qwen3-30B-A3B. Against vanilla routing of the same tokens, BLAZE cuts EP-rank max/mean by 9.9 / 13.3 / 17.9% at EP 8 / 16 / 32 on GPT-OSS-20B and by 8.9 / 12.6 / 17.0 / 22.3% at EP 8 / 16 / 32 / 64 on Qwen3.
  • α_rank. The expert term does the balancing: with vs. without α_rank=0.02, rank max/mean differs by at most 0.013 in all four tested cells (α_rank ∈ {0, 0.02}); tuning α_rank on real multi-node EP remains to be tested.
  • Communication. By design, BLAZE's only communication is one AllReduce of the L×EL\times E int32 counts per step (18 KB on GPT-OSS-120B), independent of batch size; in our runs each rank uses its local counts and BLAZE adds no collective.

On a single node, BLAZE's balancing already shortens the slowest rank's work: on GPT-OSS-120B (CUTLASS, 4 × H200, pinned), a per-layer replay of a recorded prefill-carrying step cuts the slowest rank's grouped-GEMM time by 12.9%, and end to end BLAZE lowers TTFT and TPOT P99 by 17% against the α=0 control (QPS 2, three runs per arm). We will add 16/32-GPU results (α_rank sensitivity, inter-node All-to-All timing, end-to-end latency) when multi-node resources are available.

If our responses have addressed your concerns, we would be sincerely grateful if you would kindly consider raising your score.

Comments on the paper

1 comment

Dear Area Chair,

Thank you very much for following up. We posted detailed responses to all three reviews on November 15, but unfortunately, we have not yet received any further comments from the reviewers as of November 18. Below, we briefly summarize the key strengths identified by the reviewers alongside the concerns we addressed during the rebuttal phase.

1. Strengths identified by the reviewers

  • Replica-free serving. UQhC and wU5v valued balancing expert loads without allocating replicas that consume memory needed for the KV cache.
  • Implementation. wU5v recognized the end-to-end TensorRT-LLM evaluation with CUDA Graphs. 4Cvw highlighted the device-resident controller and its small-batch bypass.
  • Evaluation. UQhC and 4Cvw recognized the task-level accuracy measurements and routing ablations. All three reviewers assigned a Soundness score of 3.

2. How we addressed the main concerns

  • EPLB under profile mismatch (UQhC). On GPT-OSS-20B, we profiled TensorRT-LLM’s replica-free EPLB placement on 16 MLPerf accuracy-set prompts, then held it fixed while serving performance-set traffic. BLAZE updated its penalties from live routing counts without an offline profile. Both used CUTLASS on four H200s, identical prompts and arrivals, and 1,024 output tokens per request. Over three runs per configuration, BLAZE reduced P99 time to first token (TTFT) by 16–23% relative to static EPLB. P99 TTFT fell from 1,002 to 845 ms at 1 request/s, 1,508 to 1,259 ms at 3, and 4,138 to 3,177 ms at 5. On Qwen3-235B-A22B (one run per configuration per load), P99 TTFT differed by at most 2.4%, and EPLB had lower P99 time per output token (TPOT).

  • Imbalance in modern MoEs (wU5v). DeepSeek’s open-source DeepInfer tracks expert loads and replicates hot experts during prefill and decode. vLLM’s Humming provides indexed and grouped GEMMs for uneven expert token counts. Kimi K3, the reviewer’s example, freezes its Quantile Balancing biases at inference and uses MoonEP to balance work across ranks during pretraining. DeepInfer · Humming · Kimi K3, §§2.3.3 and 5.2.1

    Across serving traces from GPT-OSS-20B, GPT-OSS-120B, Qwen3-30B-A3B, and Qwen3-235B-A22B, maximum-to-mean load ratios were 4.24–12.18 for experts and 1.67–1.80 for ranks. K3 exceeded our eight-H200 node’s memory. Replaying all 36 MoE layers of one GPT-OSS-120B serving step with identical inputs, we measured a 12.9% reduction in grouped-GEMM time on the slowest rank with BLAZE.

  • Latency attributable to routing (4Cvw, wU5v). With the same CUDA router and fixed work, enabling the load penalty reduced P99 TTFT by 11–15% on GPT-OSS-20B across four tested loads. On GPT-OSS-120B at 2 requests/s, P99 TTFT fell 17.3% and P99 TPOT fell 16.8% (three runs per configuration). Avoiding recompilation accounted for much of the larger gain over the released Triton router.

  • LASER comparison (4Cvw, wU5v). In the same router, BLAZE and LASER achieved comparable expert max/mean loads on GPT-OSS-20B (3.27 and 3.14). BLAZE scored 4.80 percentage points higher on AIME25/GPQA-Diamond (paired 95% CI, +2.06 to +7.54) and replaced the top-ranked expert for 0.74% of tokens versus 16.5% for LASER.

  • Limits and revisions (UQhC, 4Cvw). At fixed output length on Qwen3-30B-A3B, vanilla and both BLAZE configurations met the latency targets at 6 requests/s and missed them at 7. We will withdraw the capacity gain claimed for this setting. We found no significant MT-Bench score decrease for GPT-OSS-20B or Qwen3-30B-A3B across 80 two-turn conversations. Fixed-threshold BLAZE reduced Qwen3 MMLU-Pro accuracy by 1.35 percentage points. Across 144,000 sampled substitutions, swapping experts with nearly equal scores could substantially change the MoE layer output. We will remove the guarantees in Eqs. 9 and 12 and retain the routing-score regret bound and its sufficient condition for preserving vanilla’s top-K set.

We would be grateful if you could factor these results into your final decision. Thank you again for your valuable time and input during the discussion period.

Best regards,

The authors of BLAZE

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.