- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how to increase the number of experts in a pretrained MoE while retaining its native per-token activation budget. GRAFT-MoE groups existing experts using routed-input statistics and shared-probe responses, allocates additional capacity according to group traffic, and initializes new experts from representative parents. It then uses routed-output matching for warm-starting and temporary within-group routing exploration during continued training. Experiments on two MoE backbones and three expansion ratios report improved downstream performance, faster recovery of the source model’s language-modeling performance, and more sustained utilization of newly added experts.
Critical strengths
The paper addresses a practical and relevant problem: expanding a useful pretrained MoE without paying the cost of training a larger model from scratch. I find the overall pipeline coherent. Expert organization, functional transfer, and routing exploration address different aspects of integrating additional capacity, and their roles are explained clearly. The paper also appropriately distinguishes empirical output recovery from an exact function-preservation guarantee.
The evaluation goes beyond final benchmark scores. In particular, measuring new-expert utilization with exploration disabled helps establish whether the additional experts remain useful under ordinary inference routing. Including preparation costs in the recovery analysis is also a strength, since warm-starting is an additional expense rather than part of the shared continued-training budget. The component ablations provide useful evidence that the complete pipeline benefits from grouping, warm-starting, and exploration.
Critical weaknesses
-
Related-work positioning needs some improvement, particularly around functional expert grouping. The paper already covers several important expansion methods, so I do not see a broad failure to acknowledge prior work. However, please add and discuss Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024), Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025), and Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025). The first two are relevant to organizing experts using routing information or functional similarity, while the third provides context for budget-dependent benefits from checkpoint reuse. These are primarily citation and discussion requests, not requests to compare expansion against compression algorithms experimentally.
-
A stronger expansion baseline and a matched-training-recipe control would improve the experimental argument. An experimental comparison with the width-expansion component of Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts would be valuable. Its noisy expert duplication is directly relevant to increasing the expert pool. This should be evaluated at matched expert counts and activation budgets, with any adaptation to the original method clearly identified; adding depth expansion is unnecessary. I understand that the time is limited, so if you could add at least some demonstrations, it will be enough and you could put it in the appendix later.
Questions for authors
How does GRAFT-MoE compare with noisy expert duplication under the same expansion configuration and training budget?
How much of the advantage remains when a strong expansion baseline is given the same warm-start and comparable exploration?
Can you provide the per-seed main results and indicate whether the recovery-cost advantage is consistent across runs?
And also please refer to the weaknesses. But in general, I think this is a good paper.