acceptodds
Under review as a conference paper at ICLR 2027

MoRE: Scaling mixture of experts with hardware-aware low-rank routing

Abstract

Mixture-of-Experts (MoE) layers are central to frontier language models, and recent architectures push toward more and smaller experts. In this regime, the standard linear router becomes a bottleneck: with experts and hidden dimension , its per-token cost dominates the MoE layer once is large. We introduce MoRE (Mixture of Rank-reduced-routed Experts), which factorizes the router weight matrix at rank and reduces the routing cost to . We prove that rank logarithmic in suffices for routing expressivity when the number of active experts is fixed, and is necessary up to precision factors. We also prove that logarithmic rank preserves load balance in a Gaussian memorization model, and training on a synthetic phonebook task shows that low rank does not hurt memorization. At matched active FLOPs, the factorization allows a factor of more experts. To realize this gain in wall-clock time, we design a fused Triton kernel at inference that avoids expensive memory operations on HBM. Empirically, MoRE improves memorization on the phonebook task and performance on knowledge-intensive Q&A benchmarks after pretraining, while matching reasoning ability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Memorization, Knowledge Capacity, Theory, Hardware-efficient

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ADTeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoRE, a low-rank routing architecture that reduces routing overhead in Mixture-of-Experts (MoE) models, enabling more experts under the same active FLOPs budget. The authors provide theoretical analyses of routing expressivity, load balancing, and expert scaling, and develop a fused Triton kernel to improve inference efficiency. Extensive experiments, including synthetic memorization tasks and language model pretraining, demonstrate improved memorization capacity and knowledge-intensive task performance while maintaining comparable reasoning performance.

Critical strengths

The paper addresses an important scalability bottleneck in MoE models through a simple and practical low-rank routing approach, enabling more experts under the same active FLOPs budget. The proposed fused Triton kernel further improves practical inference efficiency. In addition, the paper provides solid theoretical analyses of routing expressivity and load balancing, supported by extensive experiments on synthetic memorization tasks, language model pretraining, and GPU benchmarks, demonstrating improvements in memorization capacity and knowledge-intensive task performance.

Critical weaknesses

The novelty of the proposed method is somewhat limited. Low-rank routing has already been explored in prior work, including distillation-based settings. Therefore, applying this idea to pretraining may represent a relatively incremental contribution rather than a fundamentally new approach.

The experimental comparisons may not fully establish the effectiveness of the proposed method. Although the models are compared under matched active FLOPs, MoRE uses approximately 5–6× more total parameters than the MoE baselines. The observed performance improvements may therefore largely result from increased model capacity. Comparisons under matched total parameters and actual training costs would help better isolate the benefits of the proposed method.

Questions for authors

Given that low-rank routing has already been explored in prior work, particularly Boix-Adsera (2025 https://arxiv.org/pdf/2512.18452), could the authors clarify the key methodological novelty of MoRE beyond applying low-rank routing to pretraining? What are the main differences from existing approaches?

Since MoRE models have substantially more total parameters and incur higher actual training costs despite matched active FLOPs, could the authors provide comparisons under matched total parameters or end-to-end training budgets? This would help clarify whether the performance gains justify the additional resource requirements.

5 comments

Thank you for your thoughtful feedback and for recognizing the strengths of our work: identifying and solving an important scalability bottleneck, and providing solid theoretical analysis and experiments. We address your concerns below.

W1 & Q1: Novelty of the proposed method

Related Work (Section 1.2, Lines 110–119) and Appendix B compare MoRE with existing methods. In particular, Boix-Adsera (2025) only provided a preliminary study of low-rank routing, in the distillation setting at an intermediate layer with a mean-squared loss. It is also written in the paper that "Further testing this proposed architecture is a promising direction".

Our work provides several important new contributions. In Section 4, we provide principled methods on how to scale the total parameters by the FLOPs saving from low-rank router. We also make hardware contributions that make MoRE feasible to make inference and training in practice. Moreover, we provide theoretical guarantees on how low-rank router preserves load balancing.

Q2: Comparison under matched training budgets

During the rebuttal period, we reduced MoRE's step-time overhead from 15–42% to 10–17% by sharding parameter updates with FSDP. We updated Table 11 and describe the details in Appendix C.2.

We added Figure 25 to the manuscript, which compares MoE and MoRE at the same training time on all four pairs of Table 1. MoRE reaches a lower validation loss on all four pairs.

Thank you for your time to read the responses. If they address your concerns, we would greatly appreciate it if you would consider raising your score. Otherwise, please let us know if you have any further questions and we are happy to respond.

Thank you for the detailed response and the additional experiments. The FSDP optimization shown in Figure 25, along with the experimental results for a fixed training duration, addresses my concerns regarding the training time budget.

Regarding novelty, thank you for the clarifications concerning expert-scaling analysis, load-balancing analysis, and the fused implementation. These contributions go beyond a simple application of low-rank routing to pre-training.

I have one further question: how does low-rank routing affect language model quality when the expert capacity is held constant? Table 12 compares models with the same number of experts and the same expert width—resulting in roughly equal total parameter counts—but it only provides data on training speed. Could you provide at least one set of results comparing the performance of such models on downstream tasks, given an identical training token budget? This would complement the synthetic experimental results presented in Figures 2 and 14, and would further clarify whether the gains in efficiency come at the expense of model quality when expert capacity is fixed.

Thanks for recognizing our reply!

At a fixed number of experts and expert width: We take the MoRE-1792 configuration of the second pair in Table 1 (M=1792M = 1792, s=56s = 56, h=512h = 512) and train it for 5B tokens with a full-rank router (MoE) and with low-rank routers. Only the router differs between these models, so their total parameter counts are nearly equal.

MoE (full rank)MoRE (r=256r=256)MoRE (r=128r=128)MoRE (r=64r=64)MoRE (r=32r=32)
Val PPL ↓16.4016.6116.9017.2417.83
Active FLOPs6.19M4.23M2.46M1.57M1.13M

We have not evaluated these models on downstream tasks. Because at 5B tokens the scores could be within benchmark noise. A larger token budget is beyond our compute during the rebuttal period.

Validation perplexity improves as the rank grows at the expense of high active FLOPs, and also the gain becomes smaller each time the rank doubles. At r=64r = 64, the rank used in Table 1, perplexity is 5% higher than with the full-rank router (17.24 vs. 16.40), and active FLOPs are 3.9× lower (1.57M vs. 6.19M).

This indicates a possible better tradeoff: Increasing expert count by lowering the router rank at matched active FLOPs. Figure 6 of the paper shows that validation perplexity then improves as the rank decreases.

If this addresses your concerns, we would greatly appreciate it if you would consider raising your score. Otherwise, please let us know if you have any further questions and we are happy to respond.

Reviewer_DswPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Mixture-of-experts (MoE) models under large number of small experts where linear routing projection becomes a computational bottleneck. The authors introduce Mixture of Rank-reduced-routed Experts (MoRE) which replaces linear routing matrix with rank r matrices to reduce the complexity from O(Mh) to O(hr + Mr) where M and h denote the number of experts and hidden dimension. They prove that when the number of active experts is fixed, logarithmic rank in M is sufficient for router expressivity, and it is necessary up to precision factors. In addition, they prove that logarithmic rank does not affect load balancing in a Gaussian memorization model and empirically show that low rank does not hurt memorization on a synthetic phonebook task on small models. The factorization enables using a factor of \theta(h/r) more experts without increasing activate computation. The paper implements a fused Triton Kernel for inference that computes scores in blocks and keeps the current top-k values and expert indices in SRAM, avoiding expensive memory operations on high-bandwidth memory. Finally, the conducted experiments shows that MoRE outperforms MoE baseline on knowledge-intensive tasks while matching it on commonsense and scientific question answering tasks.

Critical strengths

The proposed low rank factorization and the fused Triton kernel significantly reduces computational time required for routing computations when the number of experts is large and thus allows for sparser and more memory intensive MoEs without increasing the computational cost.

The conducted experiments show that MoRE outperforms MoE at matched computational budget on targeted configurations and applications.

Critical weaknesses

MoRE improves against MoE baseline at roughly matched active FLOPs, but at the cost of substantial parameter increase, similar to the trade-off between dense models and MoE models. However, the practical value of the proposal is unclear beyond the tested configurations, which seem to use significantly higher portion of their computation for routing compared the modern architecture. The ratio of route projection computation to active expert projection computation is ρ=M/(3KS)\rho=M/(3KS) .If we look at Table 2, this ratio is below 1% for eight of the nine architectures, with only Qwen3-Next-80B-A3B reaching 3.33%. On the other hand, the chosen MoE baselines have ratios of 16.67%, 33.33%, and 133.33% for the 1.27B configuration. In these configurations, the router factorization has a large potential from the beginning.

Table 1 shows that the perplexity gap between MoE and MoRE decreases with scale. Comparing the first and last configurations, MoE’s perplexity goes down from 14.01 to 12.43 while MoRE’s perplexity goes from 11.96 to only 11.92. Similarly, from second configuration to the last, the number of experts quadruples, making the configuration closer to MoRE’s motivation, but the gap between MoE and MoRE decreases. In addition, when we compare MoE from last configuration to MoRE from second configuration, we see that MoE achieves a lower perplexity with 33% less parameters while it uses more FLOPs, most of which is due to router computations. The end-to-end runtime difference between MoE-1024 against the last configuration’s MoRE-1792 might be relatively smaller at larger sequence lengths. It is unclear whether the improvements in these targeted configurations will hold for configurations where active expert computation accounts for a larger portion of the end-to-end computation.

Questions for authors

How does MoRE compare against MoE without increasing the number of experts? This would show the impact of replacing linear router alone without increasing the number of experts. It is unclear whether the findings of Figure 2 holds in general due the scale and task used for Figure 2. In addition, MoE-1024 from the second configuration achieves better perplexity than MoRE-1792 from the last configuration despite having less parameters and using similar non-routing FLOPs. It is not clear whether this is due to the router or not.

Could the authors use a more standard MoE configurations as baselines? MoRE makes the model sparser so starting with a denser MoE model (either larger S or K) might make this advantage clearer. For instance, possible configurations that are still close to the paper’s choices might be (M=256, S=128, K=8,16, h=512) or (M=256, S=256,512, K=4, h=512).

Could the authors report total FLOPS including other operations such as attention? It is not clear what the fraction of the FLOPs is spent at router and MoEs compared to the end-to-end FLOPS.

6 comments

We thank the reviewer for the detailed and useful feedback. We address the comments below.

W2: The perplexity gap between MoE and MoRE decreases with scale

The gap in validation perplexity does decrease from the first pair to the last pair of Table 1. However, this does not necessarily mean that the advantage of MoRE diminishes with scale, for two reasons.

First, because these models are expensive to train, Table 1 contains only four choices of (s,h)(s, h), and the pairs differ in ss and hh at the same time. For example, the first pair has h=1024h = 1024 and the second has h=512h = 512. MoRE's gain in expert count grows as Θ(h/r)\Theta(h/r) (Section 4.1), so a smaller gap at a smaller hh is consistent with our analysis and does not by itself indicate a trend with model scale.

Second, because of compute constraints, all configurations use the same 40B-token budget, so the larger models are trained with fewer tokens per parameter. Figure 24 (Appendix G.9) shows that for the largest pair, the perplexity gap grows with the token budget. We therefore expect that the gap would not shrink if the models were trained with the same ratio of tokens to model size.

Q1: MoRE vs. MoE with the same number of experts

We ran a rank ablation at a fixed expert configuration. We take the MoRE-1792 configuration of the second pair in Table 1 (M=1792M = 1792, h=512h = 512) and train it for 5B tokens with a full-rank router (MoE) and with low-rank routers.

MoE (full rank)MoRE (r=256r=256)MoRE (r=128r=128)MoRE (r=64r=64)MoRE (r=32r=32)
Val PPL ↓16.4016.6116.9017.2417.83
Active FLOPs6.19M4.23M2.46M1.57M1.13M

At a fixed number of experts, validation perplexity improves as the rank grows, but the drop becomes smaller each time the rank doubles. A higher rank costs more: the full-rank router uses about 4 times the active FLOPs of r=64r = 64. Because active FLOPs strongly affect model quality, MoE models are usually compared at matched active FLOPs in pretraining / inference [1, 2, 3]. At matched active FLOPs, a lower rank allows more experts, and Figure 6 shows that validation perplexity improves as the rank decreases and allowing more experts.

W1 & Q3: Share of router computation

Seem to use significantly higher portion of their computation for routing compared the modern architecture

We agree that the router does not dominate in current frontier models. However, recent models are moving toward more and smaller experts [1,4], which brings them closer to the regime where the router dominates (Figure 1).

It is not clear what the fraction of the FLOPs is spent at router and MoEs compared to the end-to-end FLOPS

We agree that they do not dominate the end-to-end FLOPs of our models (up to 22%), and attention and the LM head take the rest. This is because our compute budget limits pretraining to models far below frontier scale, with 12 layers, h≤1024h \le 1024 and s≤128s \le 128. In wall-clock time, however, the MoE layer takes a larger share than its FLOPs suggest (Figure 21, Appendix G.6), so MoRE still improves efficiency.

At frontier scale, the router costs more than the LM head once a model has about 2,000 experts. The exact number for each model in Table 2 is between about 600 and 8,400. At long context, however, attention could become the largest cost in most frontier models, because its cost per token grows with the sequence length (new Figure A.1, Appendix A). MoRE reduces the cost of the MoE layer, not of attention. We think more efficient attention is an important direction for long-context models, and it can be combined with MoRE.

Q2: More standard MoE configurations

MoRE makes the model sparser so starting with a denser MoE model (either larger S or K) might make this advantage clearer. For instance, possible configurations that are still close to the paper’s choices might be (M=256, S=128, K=8,16, h=512) or (M=256, S=256,512, K=4, h=512).

Thanks for the suggestion, in this rebuttal we ran the configuration M=256, S=128, K=8, h=512 for 40B tokens, and found MoE and MoRE have validation PPL 12.32, 10.93 respectively, having a wider gap compared with k=4k=4. MoRE also has larger improvement in benchmarks: TriviaQA (+2.8), NQ (+3.1), HellaSwag (+5.3) and ARC-e (+3.0). We also observed that the gap of MoRE and MoE increases as training goes, which also justifies our answer in W2.

Thanks for your time to read the responses. If they address your concerns, we would greatly appreciate it if you would consider raising your score. Otherwise, please let us know if you have any further questions and we are happy to respond.


[1] Scaling Laws for Fine-Grained Mixture of Experts. Krajewski et al.

[2] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Fedus et al.

[3] Unified Scaling Laws for Routed Language Models. Clark et al.

[4] Mixture of A Million Experts. He. 2024.

As the discussion period ends tomorrow, we would like to check whether our responses have addressed your concerns. If so, we would appreciate it if you would consider raising your score. We are happy to answer any further questions before the deadline.

Thank you for the new experiments and answers. The answer for Q2 is encouraging and Q1 helps us understand the trade-off between expert counts vs routing quality.

One thing that still bothers me is that MoE-1024 performs better than MoRE-1792 at same (s=64,h=512) both at validation perplexity and all tasks. MoE-1024 has fewer parameters and experts but spend more time for routing, and considering the portion of (routing + experts) will decrease as sequence length increases, I am not convinced that MoRE is a better choice for realistic LLM hyperparameter choices. The table compares only routing + expert FLOPS but that does not capture the total FLOPS, so MoE-1024 needs 2.5x FLOPS for routing but it is not clear what is the end-to-end runtime difference. Could the authors report inference throughput (tokens / sec) MoE-1024 vs MoRE-1792 at longer sequence lengths?

Reviewer_gPgTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a low-rank router to reduce the routing cost of MoEs from O(Mh) to O(M+h), where M is the number of experts and h is the hidden size. The paper also derives configurations that maximize expert parameters under an active-FLOP budget and implements fused routing kernels that avoid materializing the full token-by-expert score matrix. Experiments cover synthetic memorization, language-model pretraining on 40B tokens, and hardware measurements. The larger expert configurations enabled by low-rank routing generally improve validation perplexity and knowledge-intensive performance while maintaining approximately comparable inference latency.

Critical strengths

S1: The implementation addresses both routing arithmetic and memory traffic.

S2: Margin-aware expressivity bounds and load-balancing analysis provide useful theoretical guidance.

S3: Extensive experiments include whole-model timings, hardware comparisons, decoding, and important baseline controls.

Critical weaknesses

W1: MoRE uses substantially more parameters and takes 15–42% longer per training step. The configuration rule maximizes parameters rather than predictive quality.

W2: Headline speedups partly reflect fusion differences. With both routers fused, maximum B200 speedup is 1.72× rather than 4.54×; decoding time is approximately unchanged.

W3: The theory establishes existential routing constructions, not preservation of arbitrary routing decisions on learned representations.

W4: Phonebook comparisons give MoRE 12.5% extra FLOPs at the smallest expert width. Direct comparisons with existing low-rank routers and evaluation without downstream benchmark training sources are also needed.

Questions for authors

Q1: Does MoRE outperform well-tuned standard MoEs at matched training time and memory?

Q2: Can the main results separate gains from rank reduction and fusion, and correct the phonebook FLOP mismatch?

Q3: How does MoRE compare with existing low-rank routers on unseen tasks, and how closely do trained representations satisfy the theoretical assumptions?

6 comments

We thank the reviewer for recognizing that our work addresses the routing problem that arises when MoE scales to more experts, with useful theoretical guidance and extensive experiments. We answer the concerns below.

W1 & Q1: Number of parameters and training speed of MoRE vs. MoE

MoRE uses substantially more parameters and takes 15–42% longer per training step. The configuration rule maximizes parameters rather than predictive quality.

The larger parameter count is intended. Our goal is to efficiently scale to a larger expert count at the same active FLOPs, so at matched active FLOPs MoRE has more total parameters. During the rebuttal period, we reduced MoRE's step-time training overhead from 15–42% to 10–17% by sharding parameter updates with FSDP. We updated Table 11 and describe the details in Appendix C.2.

This overhead is small, given that MoRE has 5 to 6 times as many total parameters as its MoE counterpart. This shows the efficiency of replacing the standard router with a low-rank router. When at the same total parameters, training MoRE is actually 1.10 to 1.23× faster than training MoE (Lines 459–461 and Table 12).

Does MoRE outperform well-tuned standard MoEs at matched training time and memory?

At matched active FLOPs and the same training time, MoRE reaches a lower validation loss than MoE on all four pairs. This demonstrates MoRE's efficiency to scale to more experts. Because MoRE has larger total parameters and therefore uses more memory. An MoE with the same total parameters would have a similar memory cost. We added this comparison as Figure 25 (Appendix G.10) in the updated manuscript.

W4: Evaluation without downstream training data

We additionally evaluate 5-shot on PopQA [4] and EntityQuestions [5], whose training splits are not in our mixture. MoRE scores higher than MoE in most comparisons. The results are noisier in the remaining which is expected, because prior work shows that including the training splits of the downstream benchmarks in pretraining is necessary for reliable knowledge retrieval [1, 3]. We added this evaluation and its discussion in Appendix H.

ModelPopQAEntityQuestions
MoE-12811.37 ± 0.4713.99 ± 0.42
MoRE-153613.35 ± 0.5115.26 ± 0.43
MoE-25611.89 ± 0.4813.72 ± 0.41
MoRE-179212.28 ± 0.4913.62 ± 0.40
MoE-25612.46 ± 0.4914.98 ± 0.43
MoRE-179213.27 ± 0.5116.65 ± 0.44
MoE-102413.04 ± 0.5116.34 ± 0.44
MoRE-819212.78 ± 0.5016.47 ± 0.44

W2 & Q2: The headline speedup partly reflects fusion differences

With both routers fused, maximum B200 speedup is 1.72× rather than 4.54×; decoding time is approximately unchanged.

In Lines 351–353, we show that even when both routers are fused, the gap between MoRE and MoE grows as hh increases. This demonstrates the efficiency of MoRE at large hh, which is the typical regime of frontier models.

For decoding, we show that MoRE has a similar decoding time to MoE at the same active FLOPs, although it has many more experts (Appendix E.4). The larger expert count therefore does not increase decoding time.

W3 & Q2: The theory only establishes existential routing constructions

The theory establishes existential routing constructions, not preservation of arbitrary routing decisions on learned representations. and how closely do trained representations satisfy the theoretical assumptions?

Our theory focuses on memorization, a core function of MoE layers [1]. We prove that logarithmic rank preserves load balance and explain why balanced load maximizes memorization capacity (Section 3.2, Proposition 1). We agree that Gaussian activation is a simplifying assumption. However, similar assumption is standard in theoretical literature in memorization [1, 7].

We verify our expressivity findings empirically in training too: Lines 260–268 and Figure 2 (Left) show that a sufficient rank does not affect memorization on the phonebook task, and Figure 2 (Middle, Right) shows that MoRE and MoE reach similar load balance and most experts are activated.

W4: Extra FLOPs on Phonebook

Phonebook comparisons give MoRE 12.5% extra FLOPs at the smallest expert width.

Thank you for the question. Although MoRE uses 12.5% extra FLOPs at this width, this adds only a small error to the comparison, because MoRE memorizes 3.0× to 6.0× more entries than MoE (Figure 5, Left). Prior work has also shown that memorization depends mainly on total parameters, not on FLOPs [1, 6].

Q3: Comparison with existing low-rank routers

Related Work (Section 1.2) and Appendix B discuss prior work in detail and compare them with MoRE. In particular, Appendix B.1 gives a wall-clock comparison with PEER [2]: at matched active FLOPs, MoRE is much faster than PEER in both the forward pass and training (Table 3).

References:

[1] Mixture of Parrots: Experts Improve Memorization More than Reasoning. Jelassi et al. ICLR 2025.

[2] Mixture of A Million Experts. He. 2024.

[3] Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. Allen-Zhu & Li.

[4] When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. Mallen et al. ACL 2023.

[5] Simple Entity-Centric Questions Challenge Dense Retrievers. Sciavolino et al. EMNLP 2021.

[6] Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. Allen-Zhu & Li. ICLR 2025.

[7] Understanding Factual Recall in Transformers via Associative Memories. Nichani et al. ICLR 2025.

Reviewer_gPgTOpenReview ↗

Thank you for the additional experiments.

Figure 25 addresses my training-time concern. MoRE achieves lower validation loss at equal wall-clock time across all four pairs. Reporting peak training memory for those runs would clarify the remaining trade-off.

Please highlight the 1.72× maximum B200 speedup with both routers fused and flag the 12.5% FLOP mismatch in the smallest-width phonebook comparison. The PopQA and EntityQuestions results address my request for evaluation with their training splits excluded from pretraining. Predictive quality relative to PEER and the applicability of the theory to learned representations remain open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.