acceptodds
Under review as a conference paper at ICLR 2027

GRAFT-MOE: GROUP-GUIDED ROUTING ADAPTA- TION AND FUNCTIONAL TRANSFER FOR PRETRAINED MOE EXPANSION

Abstract

Mixture-of-Experts (MoE) expansion must add capacity without disrupting pretrained routing. We propose GRAFT-MoE, which groups pretrained experts using routed-input statistics and shared-probe responses, allocates new capacity by group traffic, selects representative parents, and applies a layerwise routed-output warm-start that jointly updates new experts and the router. During continued training, confidence-aware intra-cluster exploration allows low-confidence experts to be replaced by same-group alternatives while preserving distinct selections; it is disabled at inference. On OLMoE-1B-7B-0125 and Qwen1.5-MoE-A2.7B at , , and expansion, GRAFT-MoE achieves the best average downstream score in all six settings and the lowest perplexity in five. At , it reaches 8.86 PPL / 65.08 Avg on OLMoE and 10.97 PPL / 66.63 Avg on Qwen, and recovers the source NLL after 8.39M tokens (12.17 GPU-hours) versus 12.58M (16.29 GPU-hours) for EU-GN. It improves over EU-GN by 0.29/0.36 average-score points and over full-parameter continued pretraining by 0.79/0.77 points on OLMoE/Qwen at expansion, while reducing recovery cost by 25.3%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Large Language Models, Model Expansion, Expert Upcycling, Expert Routing, Continual Pre-training, Neural Network Growth

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3CjDOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to increase the number of experts in a pretrained MoE while retaining its native per-token activation budget. GRAFT-MoE groups existing experts using routed-input statistics and shared-probe responses, allocates additional capacity according to group traffic, and initializes new experts from representative parents. It then uses routed-output matching for warm-starting and temporary within-group routing exploration during continued training. Experiments on two MoE backbones and three expansion ratios report improved downstream performance, faster recovery of the source model’s language-modeling performance, and more sustained utilization of newly added experts.

Critical strengths

The paper addresses a practical and relevant problem: expanding a useful pretrained MoE without paying the cost of training a larger model from scratch. I find the overall pipeline coherent. Expert organization, functional transfer, and routing exploration address different aspects of integrating additional capacity, and their roles are explained clearly. The paper also appropriately distinguishes empirical output recovery from an exact function-preservation guarantee.

The evaluation goes beyond final benchmark scores. In particular, measuring new-expert utilization with exploration disabled helps establish whether the additional experts remain useful under ordinary inference routing. Including preparation costs in the recovery analysis is also a strength, since warm-starting is an additional expense rather than part of the shared continued-training budget. The component ablations provide useful evidence that the complete pipeline benefits from grouping, warm-starting, and exploration.

Critical weaknesses
  1. Related-work positioning needs some improvement, particularly around functional expert grouping. The paper already covers several important expansion methods, so I do not see a broad failure to acknowledge prior work. However, please add and discuss Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024), Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025), and Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025). The first two are relevant to organizing experts using routing information or functional similarity, while the third provides context for budget-dependent benefits from checkpoint reuse. These are primarily citation and discussion requests, not requests to compare expansion against compression algorithms experimentally.

  2. A stronger expansion baseline and a matched-training-recipe control would improve the experimental argument. An experimental comparison with the width-expansion component of Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts would be valuable. Its noisy expert duplication is directly relevant to increasing the expert pool. This should be evaluated at matched expert counts and activation budgets, with any adaptation to the original method clearly identified; adding depth expansion is unnecessary. I understand that the time is limited, so if you could add at least some demonstrations, it will be enough and you could put it in the appendix later.

Questions for authors

How does GRAFT-MoE compare with noisy expert duplication under the same expansion configuration and training budget?

How much of the advantage remains when a strong expansion baseline is given the same warm-start and comparable exploration?

Can you provide the per-seed main results and indicate whether the recovery-cost advantage is consistent across runs?

And also please refer to the weaknesses. But in general, I think this is a good paper.

Reviewer_iXxJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GRAFT-MoE, a framework for expanding pretrained MoEs while retaining the original Top-K activation count. It groups existing experts using input statistics, allocates additional experts through group-guided replication, and applies routed-output matching loss for warm-start initialization. Confidence-aware routing then encourages router adaptation during continued training. Experiments on OLMoE and Qwen report improved perplexity and average downstream scores over expansion baselines under a 33.55M-token continuation budget.

Critical strengths
  • The method offers a practical way to expand pretrained MoEs through expert replication with a small continuation-training budget, updating only new experts and routers while retaining the original Top-K activation count.
Critical weaknesses
  • The novelty appears limited. The three main components build on well-established ideas: Kmeans-based grouping in MoEfication (https://arxiv.org/abs/2110.01786), expert replication in Expert Upcycling (https://arxiv.org/abs/2604.19835), and stochastic routing (input jitter of router or Gumbel-TopK) in Switch Transformers (https://arxiv.org/abs/2101.03961). Although the specific designs differ, the experiments do not sufficiently establish their benefits over simpler alternatives.

  • The training budget is too limited. All experiments use only 33.55M continuation tokens and a rather small batch size of 8192 tokens (Section 4.1). Although GRAFT-MoE targets limited-budget adaptation, the absence of longer training runs makes it unclear whether its advantage persists as the added experts develop, especially when compared with existing methods like Expert Upcycling.

Questions for authors
  • Why is Expert Upcycling evaluated only through the adapted EU-GN baseline rather than its original training recipe? Appendix B.2 changes both the trainable parameters and load balancing strategy, leaving unclear how GRAFT-MoE compares with the original method.
  • How does EU-GN perform when given the same routed-output warm-start as GRAFT-MoE? This comparison would clarify whether output matching serves as a generally effective initialization strategy for expert expansion.
  • Do the improvements over EU-GN and Continue Original persist with substantially longer continuation training? Extending the budget to at least several billion tokens would clarify whether the added experts provide lasting benefits beyond faster initial adaptation.
  • Could the authors include the original pretrained checkpoints in Table 1 to show how much performance improves after expansion?
Reviewer_ZvVTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GRAFT-MoE, which groups pretrained experts using routed-input statistics and shared-probe responses, allocates new capacity by group traffic, selects representative parents, and applies a layerwise routed-output warm-start that jointly updates new experts and the router. GRAFT-MoE has the best average downstream score in all six settings and recovers the original model’s loss faster than an adapted EU-GN.

Critical strengths
  1. The problem is practical: growing a pretrained MoE model’s expert pool without retraining from scratch.
  2. The design is thoughtful. It matches the full routed layer output, which accounts for how adding router entries changes routing, rather than just copying parameters.
  3. The results go in a consistent direction across two model families and three expansion sizes.
Critical weaknesses
  1. Most gains are very small and may be within noise.
  2. The warm-start’s stated purpose is to transfer the source model’s behavior. However, in Figure 3, GRAFT-MoE starts with a worse loss than plain copying.
  3. No other MoE growth method is compared.
  4. The training scale is very small.
Questions for authors
  1. Can you report standard deviations for all methods in Table 1 and test whether the differences from EU-GN are significant?
  2. Why does GRAFT-MoE start with a higher loss than plain copying after warm-start?
  3. Do the gains hold with a much larger continued-training budget? Can you include a control that trains the original model with a comparable set of trainable parameters?
  4. How sensitive are the results to the number of clusters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.