acceptodds
Under review as a conference paper at ICLR 2027

QuaL-MoE: Practical Dynamic Latent Mixture-of-Experts with Quantile Balancing

Abstract

Fixed Top- routing assigns every token in a mixture-of-experts (MoE) model the same number of experts, regardless of how much computation it needs. Dynamic routers removes this restriction, but they typically control the average budget only indirectly and handle load balancing separately. We present QuaL-MoE, a dynamic MoE router that uses expert-specific quantile thresholds to determine which tokens each expert accepts. This naturally balances expert loads while keeping the average number of activated experts close to a target, yet allowing the number of experts to vary across tokens. To make this routing practical at scale, we introduce global normalization, which uses one normalized score for both expert selection and output weighting, allowing task-loss gradients to directly shape routing decisions. We also develop distributed estimators that aggregate the thresholds across training ranks. A single inference-time coefficient scales the thresholds, letting one trained model trade quality for computation without retraining. At the 1.2B scale, QuaL-MoE outperforms Top- and DynMoE while keeping the average expert count closer to its target. In 10B-parameter pretraining, it surpasses an auxiliary-loss-balanced Top- model while activating 11.8% fewer experts at inference. Our analysis shows that allocation varies across task domains, declines with token position during both prefill and decoding, and increases with the number of reasoning steps in synthetic tasks. Finally, combining QuaL-MoE with latent experts makes allocation finer-grained: QuaL-MoE-Latent outperforms an architecture-matched Top--Latent model throughout training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, dynamic moe

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7S5sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

QuaL-MoE introduces dynamic expert routing through expert-specific quantile thresholds, allowing different tokens to activate different numbers of experts while approximately maintaining a target average budget and balanced loads. It combines global score normalization, distributed threshold estimation, inference-time threshold adjustment, and an extension to latent experts. Experiments at the 1.2B and 10B scales show comparable or slightly better average task performance with fewer expert activations.

Critical strengths
  1. Direct budget control and deployment flexibility. A single threshold multiplier adjusts the inference budget of an existing model without retraining, providing a practical mechanism for trading quality against computation.

  2. Substantial validation at scale. Evaluation includes 10B-parameter pretraining on 840B tokens and architecture-matched latent-expert comparisons, strengthening the evidence for scalability and compatibility with finer-grained experts.

Critical weaknesses
  1. Insufficient isolation of the contribution relative to closely related methods. The thresholding rule is closely related to the concurrent Expert Threshold method discussed in the paper, but direct comparisons with it and budget-controlled DTop-p are absent. This makes the incremental contribution harder to assess.

  2. Practical efficiency and statistical reliability remain unclear. The reported 11.8% reduction concerns expert activations rather than measured end-to-end speed. At that operating point, the average score exceeds Top-k by only 0.071 points.

Questions for authors

I hope authors could address my concerns in the weakness part. If authors can give clearer and more convince content, I will reconsider my score.

2 comments

We thank Reviewer 7S5s for their time and the encouraging assessment! We are glad the reviewer values direct budget control through a single coefficient and the 10B-scale and latent-expert evidence.

W1. Direct comparison with Expert Threshold and budget-controlled DTop-pp.

Please see C1 for baseline comparisons.

W2. Practical efficiency and statistical reliability.

We agree that end-to-end speed has to be measured; please see C3 for inference latency comparison.

Qual-MoE shows practical efficiency and achieves lower measured latency, and overall a robust improvement in performance compared to Top-kk (please refer results in C2). And we believe as model scales up, the improvement gap between Qual-MoE and Top-kk will widen as 10B scale shows a larger gap than 1.2B scale. We plan to scale up Qual-MoE and will release it in the near future.

We also sincerely thank the reviewer for the offer to reconsider; we hope the added comparison, efficiency measurements and clarified claims address the concerns!

I sincerely thank the authors for their thoughtful responses and the additional experiments.

The newly added comparisons with Expert Threshold (ET) and DTop-p, along with the end-to-end inference efficiency measurements, have largely addressed my previous concerns and further demonstrated the practical value of the proposed method.

However, I still find the performance improvements relatively modest. For example, compared with ET, QuaL-MoE improves Mean21 only from 32.627 to 32.884 (+0.257) and reduces PPL from 4.998 to 4.990 (−0.008), while activating slightly more experts on average (3.989 vs. 3.921). Moreover, I believe the novelty relative to existing threshold-based routing methods remains somewhat limited. The core per-expert quantile thresholding and EMA update mechanisms are highly similar to those of the existing Expert Threshold method. Although the authors introduce improvements such as global normalization, distributed threshold estimation, and inference-time budget control, these contributions primarily represent refinements and practical extensions of existing routing mechanisms rather than a fundamentally different dynamic routing approach.

Overall, I remain positive about this paper and lean toward acceptance. I believe my current rating of 3 (Weak Acceptance) already reflects my appreciation of the work. Therefore, I have decided to maintain my original rating.

I sincerely thank the authors again for their detailed responses and additional efforts.

Reviewer_99UpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QUAL-MOE, a dynamic Mixture-of-Experts (MoE) routing framework that uses expert-specific quantile thresholds to adaptively allocate different numbers of experts to individual tokens while approximately maintaining a target average computation budget. The framework introduces global normalization to couple expert selection with task-loss optimization, three distributed quantile estimators for threshold computation, and an inference-time threshold scaling mechanism for flexible computation control. This paper further integrates QUAL-MOE with latent experts to enable finer-grained computation allocation. Experiments on 1.2B and 10B parameter models show improvements over the routing baselines, while additional analyses investigate expert allocation across tasks, token positions, and reasoning complexity.

Critical strengths
  1. This paper addresses an important limitation of conventional Top-k MoE routing, where all tokens receive the same number of experts regardless of their computational requirements. Controlling average expert activation while allowing token-level flexibility is meaningful for efficient MoE training and inference.
  2. QUAL-MOE combines quantile-based routing, global normalization, distributed threshold estimation, and inference-time budget adjustment into a coherent framework. The extension to latent experts further demonstrates the flexibility of the proposed routing mechanism.
  3. This paper includes experiments at both 1.2B and 10B scales, with the standard 10B model pretrained on 840B tokens. The evaluation covers multiple downstream benchmarks, routing normalization ablations, distributed threshold estimators, and expert allocation patterns. These experiments provide useful insights into dynamic MoE routing.
Critical weaknesses
  1. Limited novelty compared with existing threshold-based routing methods. The central idea of using expert-specific quantile thresholds with exponential moving average updates closely resembles Expert Threshold Routing[1], which already supports causal token-level dynamic computation allocation and load balancing. This paper could provide a more detailed comparison of the differences between the two.
  2. The claimed inference efficiency improvement is primarily supported by reductions in the average number of activated experts. However, fewer expert activations do not necessarily translate into proportional reductions in actual inference latency or GPU computation time. Dynamic routing introduces irregular expert assignments, potentially increasing dispatch overhead, communication costs, and workload imbalance. The paper does not provide sufficient end-to-end inference throughput, latency, or GPU memory measurements against Top-k routing. Such measurements are particularly important because practical efficiency is a central motivation of the work.
  3. Relatively modest performance improvements without statistical validation. The improvements over Top-k are relatively small in several settings. For example, Table 1 reports only marginal gains at the 1.2B scale. More importantly, in Table 2, QUAL-MOE at the reduced-budget setting achieves a Mean21 score of 52.311, compared with 52.240 for Top-k, representing a very small difference.
  4. Insufficient clarification of expert allocation statistics in the latent setting. Section 4 and Table 3 describe QUAL-MOE-Latent as using 256 experts with a target of eight activated experts per token. However, Table 5 reports average expert counts close to four for QUAL-MOE-Latent on several tasks, without clearly explaining whether these values represent raw latent expert counts or full-width-equivalent normalized counts.

[1] Sun H, Liu Y, Wu Y, et al. Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing[J].

Questions for authors
  1. How much improvement does QUAL-MOE achieve in measured inference latency, tokens per second, and GPU memory usage compared with Top-k?
  2. Why does Table 5 report approximately four activated experts for QUAL-MOE-Latent when the nominal target is eight? Are these measurements normalized to full-width-equivalent expert counts, and are all latent comparisons based on matching training checkpoints and computational budgets?
  3. What proportion of tokens activate zero experts or unusually large numbers of experts, particularly after inference-time threshold scaling? How does global normalization affect these extreme routing cases, and how much of its benefit remains when compared against an otherwise identical threshold-routing baseline?
2 comments

We thank Reviewer 99Up for their time and detailed feedback! We are glad the reviewer finds the problem important and the framework coherent.

W1. Novelty relative to Expert Threshold routing.

Please see C1.

W2/Q1. Latency, tokens/s and GPU memory.

Please see C3.

W3. Modest improvements without statistical validation.

Please see C2. On an H200 GPU at 10B scale, QuaL-MoE reduces end-to-end latency to 0.98× at λ=1.0\lambda=1.0 and 0.94× at λ=1.1\lambda=1.1.

W4/Q2. Expert counts of QuaL-MoE-Latent in Table 5.

Thank you for catching this presentation ambiguity. Table 5 reports full-width-equivalent counts, i.e. the raw count of latent experts divided by the compression factor α=2\alpha=2. The raw latent counts are twice as large and consistent with the nominal target of eight: for example MMLU 8.08 (4.04 after normalization), Arabic-MMLU 10.03 (5.02) and GSM8K 6.73 (3.36). We will report both raw and normalized counts and fix the caption.

Yes, all latent comparisons are based on matching training checkpoints and computational budgets.

Q3. Fraction of zero-expert and very-large-count tokens; effect of global normalization.

Zero-expert tokens. Every MoE layer has one shared expert, and the counts we report exclude it. A token that activates zero routed experts is therefore still processed by the shared expert, in addition to attention and the residual path, so no token receives zero computation.

For a more comprehensive analysis, we report the fraction of tokens that activate only the shared expert below (10B scale, last checkpoint on held-out set):

λ\lambdaZero routed expertsMax routed experts / token
1.00.27%13
1.10.45%12
1.40.93%10
1.0 w/o global normalization0.21%128

Raising λ\lambda increases the "zero-routed" fraction, but it all stays below 1%, and the maximum count decreases.

Effect of global normalization, and how much of its benefit remains. Global normalization decreases max routed experts from all 128 experts to a reasonably smaller number. As for how much of its benefit remains compared to other baselines, please see C1.

Thank you for your detailed response. Most of my questions have been answered, and I will raise my rating.

Reviewer_WEAiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes a pretrain time method to enable dynamic routing in a an MoE model, assigning variable number of experts per token, yet maintaining a controllable way to load-balance.

  1. Each expert maintains a running quintile threshold score. Once the router scores the tokens in a sequence, a given expert accepts the tokens scoring more than its threshold. As a result, this balances experts load by construction and let each token have varying number of experts assigned.
  2. The paper includes a way to compute the running threshold per expert using different estimators: Quantile-Avg, Quantile-Hist, Quantile-BS. Moreover, they includes various analysis showing how allocation varies by domain, token positions etc.
  3. The experiments are shown on two different sized MoE models trained from scratch: 1.2B and 10B total parameters.
Critical strengths

It seems QuaL-MoE controls the average budget well (Fig 2) even though the per-token maximum is not capped, which matters for latency and load on real hardware. So, it provides a reliable topk range.

Critical weaknesses
  1. The proposed routing method in an Moe model mainly differs from the expert-choice routing on how experts choose the tokens. In expert-choice method, experts select top q tokens per sequence/batch but here they select based on its own running threshold. The expert based thresholding idea itself is not very novel as the concurrent work mentioned here (Sun et al. 2026) have also tried it. And there is no rationale about why this running thresholds being used by experts are optimal. Based on the results, they only seem to provide a balanced load target but there is no apparent quality gain by this method.

  2. The proposed method is similar to expert-choice routing, however the baselines does not include it.

  3. The quality gain of the proposed method is very marginal: QUAL-MOE over Topk (Gain of 0.10 on accuracy with experts selected as 3.99 for 1.2B–8E4A , gain of 0.19 points on accuracy with experts as 1.98 for 1.2B–8E2A ). The experts count and accuracy both are very close to the baseline topk (table 1).

  4. Similarly for the 10B model size, the gain is a bit harder to read: At λ=1\lambda=1 expert count is 3.91 and mean downstream gain is ~0.46 as compared to topk. But the metric is the mean of 21 per-task values that range from −6.8 (GSM8K) to +7.9 (HumanEval) as per table 4 and overall it seems that 15 tasks go up and 6 go down, and there are no seed or error bars, so the average gain shown for 21 tasks is small next to the per-task swings.

  5. The comparison of Qual-MoE against baseline topk (fig 3) is consistent but seems small overall. (a) Loss: the curves overlap, including the zoomed inset. (b) Loss gap is noisy and small.(c) MaxVio (load imbalance): QuaL-MoE about 0.203 vs topk about 0.211. (d) Mean experts: about 3.9 vs 4. (e) Max experts per token: QuaL-MoE reaches 15–45 out of 128, against a flat 4 for Top-k.

Hence, for this proposed method, the budget control is good but results show parity in quality compared to baseline and not a clear gain from this idea.

Questions for authors

NA

2 comments

We thank Reviewer WEAi for their time and valuable feedback! We appreciate the reviewer's recognition that QuaL-MoE controls the average expert budget.

We respond to reviewer's concerns point by point below:

W1, W2. QuaL-MoE differs from Expert Choice only in how experts choose tokens; expert-based thresholding is not novel (ET); no rationale for the running thresholds; no quality gain beyond balanced load.

  • Relation to Expert Choice and Expert Threshold: please see C1 for why standard EC is non-causal in autoregressive generation and for new controlled experiments showing QuaL-MoE outperforms both ET and DTop-pp.
  • Rationale for the running thresholds: the per-expert quantile Q1−k/nQ_{1-k/n} is the solution of the expert-load-constrained assignment problem (Eq. 3–4), and the EMA estimate (Eq. 5) is its causal online approximation. That is why it keeps both the expert load and the average expert count at target. We will state this derivation more directly in §3.1.
  • Quality gain over Top-k: please see C2.

W3, W5. The gain over Top-kk is marginal.

Please see C2, which gives the 1.2B and 10B numbers and what QuaL-MoE provides beyond the average score. We believe that Qual-MoE aims to provide flexible, dynamic and controllable routing. Additionally, QuaL-MoE enables training-free compute–quality control via λ\lambda and delivers over 6% lower end-to-end H200 latency without performance loss.

W4. At 10B the gain (+0.46) is small next to the per-task swings (−6.8 on GSM8K to +7.9 on HumanEval), and there are no seeds or error bars.

Please see Statistical validation in C2. GSM8K's drop at λ=1.0\lambda=1.0 directly reflects aggressive compute saving: QuaL-MoE allocates only 3.335 experts/token on GSM8K. Increasing budget via λ\lambda can recover some of the lost performance, while other reasoning benchmarks gain strongly even at λ=1.0\lambda=1.0.

Dear reviewer WEAi,

We'd like to check in with you and see if our rebuttal has clarified your concerns or not. If there are still any unclear parts in our work, please let us know and we would be happy to follow up before the discussion period closes. Thanks for your time and effort in reviewing this paper!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Common Response to All Reviewers

We thank all reviewers for their time and constructive feedback! We are glad that the reviewers find the problem important and our experiments valuable (WEAi-W1**,** 99Up, 7S5s).

Below, we address the three common concerns shared across reviews:

C1. Relation to Expert Choice and Expert Threshold (WEAi-W1, 99Up-W1, 7S5s-W1)

  • Expert Choice (EC). Standard Expert Choice ranks tokens within the current batch. If that includes future positions in the same sequence, the routing decision is no longer causal. It therefore cannot be used directly for autoregressive training and generation. We therefore do not compare with it directly.
  • Expert Threshold (ET)(Sun et al., 2026): While concurrent ET also uses per-expert EMA quantile thresholds, QuaL-MoE differs in four key ways:
    • QuaL-MoE uses global normalization lets task-loss gradients shape routing. The globally normalized variants reach lower training loss and PPL, achieve better downstream performance, also keeps allocation sparse.
    • QuaL-MoE provides three distributed estimators:, QuaL-MoE introduces distributed estimators (Quantile-Avg, Quantile-Hist, Quantile-BS) that synchronize thresholds accurately across multi-node ranks to accurately control experts.
    • Qual-MoE supports flexible inference-time budget control via a single coefficient λ\lambda (Table 2),
    • Qual-MoE extends to latent experts. Our experiments are validated on 10B-scale validation (840B tokens).
  • DTop-pp: Unlike QuaL-MoE, DTop-pp applies a PI controller to a global Top-pp cutoff while still requiring auxiliary load-balancing losses and controller hyperparameters. Because official code is not publicly available, we reimplemented DTop-pp following Jin et al. (2026) under its native recipe in our controlled 1.2B-8E4A setting

For completness of experiments, we trained ET in our 1.2B-8E4A setting with its own recipe:

1.2B 8E4AMean routed expertsHeld-out PPL ↓Mean$_{21}$ ↑
ET3.9214.99832.627
DTop-pp4.2285.11332.758
QuaL-MoE w/o global norm3.9864.99332.740
QuaL-MoE (Ours)3.9894.99032.884

ET has less accurate expert control and shows a higher held-out PPL and lower downstream performance. Meanwhile, DTop-pp struggles to eliminate the Top-pp budget drift (4.228 experts) and incurs higher validation PPL.

C2. Size of the gain over Top-kk (WEAi-W2, 99Up-W3, 7S5s-W2)

Though the average-quality difference at the trained budget is modest, we would like to clarify, however, that quality at a single fixed budget is not the only thing QuaL-MoE is designed for. Its advantage can be summarized as following:

  • expert count becomes a dynamic and controllable quantity, obtained at no quality cost relative to a well-tuned Top-kk baseline.
  • the loss gap between QUAL-MoE and Top-KK is obtained without an auxiliary balancing loss, and the load of Qual-MoE is better balanced (MaxVio 0.203 vs. 0.211) without tuning auxiliary balancing loss coefficients.
  • QuaL-MoE provides flexible inference budget control, so a single coefficient λ\lambda moves a trained model along the quality–cost curve without retraining (Table 2). At λ=1.1\lambda=1.1 it uses 3.529 experts per token (11.8% fewer than Top-kk) with PPL 3.789 vs. 3.789 and Mean$_{21}$ 52.311 vs. 52.240.

The same checkpoint therefore matches Top-kk quality with 11.8% fewer expert activations, can be pushed to about two-thirds of Top-kk's expert use with a modest drop, or given more compute when quality matters. The best λ\lambda also differs by task, which a fixed-kk model cannot exploit. A Top-kk model has a single operating point and would need a separately trained model for each budget.

Statistical validation. We re-train the 1.2B-scale models with three different seeds and report the average results with standard deviations as following.

1.2B 8E4AHeld-out PPL ↓Mean$_{21}$ ↑
Top-kk5.009 ± 0.01532.580 ± 0.187
QuaL-MoE4.980 ± 0.01132.975 ± 0.188
1.2B 8E2AHeld-out PPL ↓Mean$_{21}$ ↑
Top-kk5.100 ± 0.01032.318 ± 0.197
QuaL-MoE5.079 ± 0.01232.532 ± 0.182

The results suggest that Qual-MoE performs consistently better than Top-kk in both held-out PPL and average downstream tasks. The improvement doesn't come from one certain seed but is statistically robust.

At the 10B scale (840B tokens per run), training multiple full seeds within the rebuttal window is computationally prohibitive. But we do observe that increasing model to 10B from 1.2B scale widens the gap between Top-kk and Qual-MoE, and we expect this trend will continue as the model scales further. We will further validate this point in our future experiments.

C3. End-to-end inference efficiency (99Up-W2/Q1, 7S5s-W2)

We agree that fewer activated experts do not imply a proportional reduction in latency. We benchmark end-to-end inference including prefill and decode using the last 10B checkpoints on a single H200. We use 8k tokens prompt, batch size=128, with Transformer Engine's grouped GEMM, and report the average of three runs:

Modelλ\lambdaEnd-to-end inference latency RatioThroughput (token/s) RatioPeak memory Ratio
Top-kk–1.0x1.0x1.0x
QuaL-MoE1.00.98x1.02x1.0x
QuaL-MoE1.10.94x1.06x1.0x
QuaL-MoE1.40.87x1.15x1.0x

QuaL-MoE matches Top-kk's peak GPU memory (since all expert weights reside in memory) while reducing end-to-end wall-clock latency by 6% at λ=1.1\lambda=1.1 by activating 11.8% less experts. Because attention and the shared expert incur fixed overhead regardless of λ\lambda, achieving a 6% end-to-end latency reduction from an 11.8% reduction (λ=1.1\lambda =1.1) in routed experts confirms that dynamic routing overhead does not erode our theoretical savings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.