- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The article introduces a scaling-law-based method for growing the expert pool of an MoE during pretraining. The authors argue that holding the expert count fixed for the whole run is sub-optimal, since a large pool early in training is wasteful: the cost it adds comes not from the theoretical FLOPs, which depend only on the activated parameters, but from the overheads of high sparsity and of maintaining and updating a much larger total parameter count. EMO therefore starts small and expands in stages, with each stage's token budget derived from a sparsity-aware scaling law fit on a small-scale sweep over expert counts. The main study grows a 1.1B-activated model to 128 experts over 1.92T tokens in five stages; this ends at a slightly higher loss (1.017) than the fixed 128-expert baseline (0.994), while the authors claim a 10% saving in overall GPU hours.
Critical strengths
The paper identifies the right question. Any progressive-growth method lives or dies on scheduling, and section 3 isolates this cleanly rather than treating expansion timing as an implementation detail subordinate to the architecture.
Using a scaling law to set that schedule is the appropriate tool, and it is applied with a standard protocol: the functional form is established in the literature, the fit follows Hoffmann et al. (LBFGS under Huber loss), and the reported quality is good: RMSE 0.0085 and held-out RMSE 0.0092. The one gap is that the held-out set is never specified.
The method is well situated against prior work. section 2.3 states precisely what single-step sparse upcycling does - replicating FFN weights so the dense model's behaviour is preserved under uniform routing, and identifies where it degrades as the expert count grows. Multi-step expansion is then motivated as a direct response to that limitation rather than asserted.
The presentation is clear. Algorithm 1 gives the method compactly, Figure 2 makes the expansion step concrete at the level of expert, router-weight and router-bias initialization, and Figure 4 walks through the schedule derivation step by step.
The empirical validation is substantial for a submission of this kind: a five-stage from 8 to 128 expansion of a 1.1B-activated model over 1.92T tokens, against from-scratch fixed-expert baselines under matched token budget, hyperparameters and data mixture, with loss curves and downstream evaluation measured throughout training.
Critical weaknesses
The held-out evaluation is not specified. A held-out RMSE of 0.0092 is reported, but the paper never says how that set was chosen, which points, or how many. To measure scaling laws extrapolation accuracy the best would be to test leave lowest losses out (f.e. 10% of lowest losses models), which would say the most about the extrapolation the schedule relies on. Additionally the training grid itself is only partly described, listing the expert counts swept but not the activated sizes or token budgets.
Section 2.2 attributes the FLOPs–wall-clock gap primarily to all-to-all communication from expert parallelism, and Section 1 repeats this, but expert parallelism appears nowhere in the paper: Section 4.1 describes 4 nodes × 8 H200 with "data parallelism," and Appendix B adds no parallelism detail. Under that configuration there is no expert-parallel dispatch, so the overhead named first in the paper's own causal account cannot be what Figure 1 measures; the costs that do scale with E here are gradient-synchronization volume, optimizer state, and optimizer-step work, all tracking total parameters, which grow roughly 9× from E=8 to E=128 at constant activated FLOPs. This bears directly on the claim in Section 6 that EMO is "orthogonal" to systems optimizations. As composability that is uncontroversial, but the paper uses it in the stronger sense that EMO's gains persist undiminished on such a stack, and that does not follow: EMO's saving is determined by the shape of the realized step-time curve dependent on number of experts - T(E). Relevant optimizations: the grouped GEMM, block-sparse computation and all-to-all overlap cited in Section 6, or, under the corrected account, routine bucketed gradient overlap - act precisely to flatten that T(E) curve. Since Figure 1 is measured on a stack with no stated expert-path optimization (Section 4.1), the reported slowdowns are best read as an upper bound on the gap a well-optimized stack would exhibit, and correspondingly on EMO's advantage. Some E-dependence is irreducible, so a benefit would remain; its magnitude is what is unestablished. We would ask the authors to state the parallelism and overlap configuration and report MFU, revise Section 2.2 to name the overheads actually operative in their setup. By: demonstrating EMO's savings on top of methods from “Systems optimizations for MoE training.” (section 6), reporting MFU alongside the parallelism, would settle or clarify more measured gains.
Questions for authors
Could authors report MFU alongside the parallelism and T(E) optimization (see weakness 1) of their training, one for static E and averaged or per E after extension for EMO method training? This does not require additional computation that could most likely be read or approximated from already conducted experiments. For more precise measurement (that would be preferable), few step training with additional time measurement utility should give accurate MFU results for different E (2, 4, 8,16, 32, 64, 128) with the training method used in the article (using 4 x 8 h200 gpus).
Could you clarify the held-out methodology for reporting RMSE value. Could you compute a hold-out leaving out 10%, 20%, 30% of lowest losses out of (90%, 80%, 70% highest losses fit respectively) and test their RMSE to evaluate the quality of the fit. Leaving 10% lowest losses out would be similar to Ludziejewski et al., 2025 hold-out evaluation.
We thank Reviewer Vzfj for their time and the insightful, positive review! We are glad the reviewer finds that the paper isolates scheduling as the key question of progressive growth and that the scaling-law protocol is the appropriate tool for it.
W1/Q2. Held-out methodology and training grid.
The scaling-law experiments covers E∈1,2,4,…,256 at top-2, activated model size ranges from 65M to 657M, and training budgets from 60B to 1.2T tokens, with 50 runs in total. The original held-out set is randomly sampled 10%. Following reviewer's suggestion, we fit on the highest-loss 90/80/70% of runs and evaluate on the held-out lowest-loss 10/20/30%:
Held-out error stays within about 2× the fit error even when extrapolating to the lowest 30% of losses.
W2/Q1. Parallelism, the causal account of T(E), and MFU.
Our training stack binds EP with TP, but the reported A1.1B experiments use FSDP across 32 H200 GPUs with TP=EP=1. Thus, Figure 1 contains no inter-GPU expert-token all-to-all. The all-to-all discussed in Sections 1 and 2.2 is a potential cost when EP is enabled, rather than an explanation of the measured A1.1B slowdown. We will make this distinction explicit in the revision.
In the measured configuration, increasing E increases FSDP parameter all-gather, gradient reduce-scatter, and optimizer work; expert-kernel efficiency can also change. Parameter all-gathers are issued asynchronously, and backward prefetch overlaps communication with gradient computation. G1 specifies our sharding and communication-overlap configuration. The reported MFU values across E=8,16,32,64,128 are 33.2%, 32.1%, 31.1%, 29.7%, and 28.0%. At the tested E=8 shape, our expert GEMM benchmark is comparable to Transformer Engine grouped GEMM (92.1 vs. 92.3 ms). We observe that grouped GEMM would not flatten T(E): the two kernels also match at E=128.
Orthogonality. EMO can be combined with expert-kernel and communication optimizations: it changes how long each expert-pool size is used. The magnitude of its saving depends on the resulting T(E): our measured GPU-hour reduction is obtained on the stated FSDP configuration, while an EP configuration would change both parameter communication and token dispatch costs.