- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MoRE, a low-rank routing architecture that reduces routing overhead in Mixture-of-Experts (MoE) models, enabling more experts under the same active FLOPs budget. The authors provide theoretical analyses of routing expressivity, load balancing, and expert scaling, and develop a fused Triton kernel to improve inference efficiency. Extensive experiments, including synthetic memorization tasks and language model pretraining, demonstrate improved memorization capacity and knowledge-intensive task performance while maintaining comparable reasoning performance.
Critical strengths
The paper addresses an important scalability bottleneck in MoE models through a simple and practical low-rank routing approach, enabling more experts under the same active FLOPs budget. The proposed fused Triton kernel further improves practical inference efficiency. In addition, the paper provides solid theoretical analyses of routing expressivity and load balancing, supported by extensive experiments on synthetic memorization tasks, language model pretraining, and GPU benchmarks, demonstrating improvements in memorization capacity and knowledge-intensive task performance.
Critical weaknesses
The novelty of the proposed method is somewhat limited. Low-rank routing has already been explored in prior work, including distillation-based settings. Therefore, applying this idea to pretraining may represent a relatively incremental contribution rather than a fundamentally new approach.
The experimental comparisons may not fully establish the effectiveness of the proposed method. Although the models are compared under matched active FLOPs, MoRE uses approximately 5–6× more total parameters than the MoE baselines. The observed performance improvements may therefore largely result from increased model capacity. Comparisons under matched total parameters and actual training costs would help better isolate the benefits of the proposed method.
Questions for authors
Given that low-rank routing has already been explored in prior work, particularly Boix-Adsera (2025 https://arxiv.org/pdf/2512.18452), could the authors clarify the key methodological novelty of MoRE beyond applying low-rank routing to pretraining? What are the main differences from existing approaches?
Since MoRE models have substantially more total parameters and incur higher actual training costs despite matched active FLOPs, could the authors provide comparisons under matched total parameters or end-to-end training budgets? This would help clarify whether the performance gains justify the additional resource requirements.
5 comments
Thank you for the detailed response and the additional experiments. The FSDP optimization shown in Figure 25, along with the experimental results for a fixed training duration, addresses my concerns regarding the training time budget.
Regarding novelty, thank you for the clarifications concerning expert-scaling analysis, load-balancing analysis, and the fused implementation. These contributions go beyond a simple application of low-rank routing to pre-training.
I have one further question: how does low-rank routing affect language model quality when the expert capacity is held constant? Table 12 compares models with the same number of experts and the same expert width—resulting in roughly equal total parameter counts—but it only provides data on training speed. Could you provide at least one set of results comparing the performance of such models on downstream tasks, given an identical training token budget? This would complement the synthetic experimental results presented in Figures 2 and 14, and would further clarify whether the gains in efficiency come at the expense of model quality when expert capacity is fixed.
Thanks for recognizing our reply!
At a fixed number of experts and expert width: We take the MoRE-1792 configuration of the second pair in Table 1 (, , ) and train it for 5B tokens with a full-rank router (MoE) and with low-rank routers. Only the router differs between these models, so their total parameter counts are nearly equal.
| MoE (full rank) | MoRE () | MoRE () | MoRE () | MoRE () | |
|---|---|---|---|---|---|
| Val PPL ↓ | 16.40 | 16.61 | 16.90 | 17.24 | 17.83 |
| Active FLOPs | 6.19M | 4.23M | 2.46M | 1.57M | 1.13M |
We have not evaluated these models on downstream tasks. Because at 5B tokens the scores could be within benchmark noise. A larger token budget is beyond our compute during the rebuttal period.
Validation perplexity improves as the rank grows at the expense of high active FLOPs, and also the gain becomes smaller each time the rank doubles. At , the rank used in Table 1, perplexity is 5% higher than with the full-rank router (17.24 vs. 16.40), and active FLOPs are 3.9× lower (1.57M vs. 6.19M).
This indicates a possible better tradeoff: Increasing expert count by lowering the router rank at matched active FLOPs. Figure 6 of the paper shows that validation perplexity then improves as the rank decreases.
If this addresses your concerns, we would greatly appreciate it if you would consider raising your score. Otherwise, please let us know if you have any further questions and we are happy to respond.
Thank you for your thoughtful feedback and for recognizing the strengths of our work: identifying and solving an important scalability bottleneck, and providing solid theoretical analysis and experiments. We address your concerns below.
W1 & Q1: Novelty of the proposed method
Related Work (Section 1.2, Lines 110–119) and Appendix B compare MoRE with existing methods. In particular, Boix-Adsera (2025) only provided a preliminary study of low-rank routing, in the distillation setting at an intermediate layer with a mean-squared loss. It is also written in the paper that "Further testing this proposed architecture is a promising direction".
Our work provides several important new contributions. In Section 4, we provide principled methods on how to scale the total parameters by the FLOPs saving from low-rank router. We also make hardware contributions that make MoRE feasible to make inference and training in practice. Moreover, we provide theoretical guarantees on how low-rank router preserves load balancing.
Q2: Comparison under matched training budgets
During the rebuttal period, we reduced MoRE's step-time overhead from 15–42% to 10–17% by sharding parameter updates with FSDP. We updated Table 11 and describe the details in Appendix C.2.
We added Figure 25 to the manuscript, which compares MoE and MoRE at the same training time on all four pairs of Table 1. MoRE reaches a lower validation loss on all four pairs.
Thank you for your time to read the responses. If they address your concerns, we would greatly appreciate it if you would consider raising your score. Otherwise, please let us know if you have any further questions and we are happy to respond.