- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
QuaL-MoE introduces dynamic expert routing through expert-specific quantile thresholds, allowing different tokens to activate different numbers of experts while approximately maintaining a target average budget and balanced loads. It combines global score normalization, distributed threshold estimation, inference-time threshold adjustment, and an extension to latent experts. Experiments at the 1.2B and 10B scales show comparable or slightly better average task performance with fewer expert activations.
Critical strengths
-
Direct budget control and deployment flexibility. A single threshold multiplier adjusts the inference budget of an existing model without retraining, providing a practical mechanism for trading quality against computation.
-
Substantial validation at scale. Evaluation includes 10B-parameter pretraining on 840B tokens and architecture-matched latent-expert comparisons, strengthening the evidence for scalability and compatibility with finer-grained experts.
Critical weaknesses
-
Insufficient isolation of the contribution relative to closely related methods. The thresholding rule is closely related to the concurrent Expert Threshold method discussed in the paper, but direct comparisons with it and budget-controlled DTop-p are absent. This makes the incremental contribution harder to assess.
-
Practical efficiency and statistical reliability remain unclear. The reported 11.8% reduction concerns expert activations rather than measured end-to-end speed. At that operating point, the average score exceeds Top-k by only 0.071 points.
Questions for authors
I hope authors could address my concerns in the weakness part. If authors can give clearer and more convince content, I will reconsider my score.
2 comments
I sincerely thank the authors for their thoughtful responses and the additional experiments.
The newly added comparisons with Expert Threshold (ET) and DTop-p, along with the end-to-end inference efficiency measurements, have largely addressed my previous concerns and further demonstrated the practical value of the proposed method.
However, I still find the performance improvements relatively modest. For example, compared with ET, QuaL-MoE improves Mean21 only from 32.627 to 32.884 (+0.257) and reduces PPL from 4.998 to 4.990 (−0.008), while activating slightly more experts on average (3.989 vs. 3.921). Moreover, I believe the novelty relative to existing threshold-based routing methods remains somewhat limited. The core per-expert quantile thresholding and EMA update mechanisms are highly similar to those of the existing Expert Threshold method. Although the authors introduce improvements such as global normalization, distributed threshold estimation, and inference-time budget control, these contributions primarily represent refinements and practical extensions of existing routing mechanisms rather than a fundamentally different dynamic routing approach.
Overall, I remain positive about this paper and lean toward acceptance. I believe my current rating of 3 (Weak Acceptance) already reflects my appreciation of the work. Therefore, I have decided to maintain my original rating.
I sincerely thank the authors again for their detailed responses and additional efforts.
We thank Reviewer 7S5s for their time and the encouraging assessment! We are glad the reviewer values direct budget control through a single coefficient and the 10B-scale and latent-expert evidence.
W1. Direct comparison with Expert Threshold and budget-controlled DTop-p.
Please see C1 for baseline comparisons.
W2. Practical efficiency and statistical reliability.
We agree that end-to-end speed has to be measured; please see C3 for inference latency comparison.
Qual-MoE shows practical efficiency and achieves lower measured latency, and overall a robust improvement in performance compared to Top-k (please refer results in C2). And we believe as model scales up, the improvement gap between Qual-MoE and Top-k will widen as 10B scale shows a larger gap than 1.2B scale. We plan to scale up Qual-MoE and will release it in the near future.
We also sincerely thank the reviewer for the offer to reconsider; we hope the added comparison, efficiency measurements and clarified claims address the concerns!