acceptodds
Under review as a conference paper at ICLR 2027

EMO: Frustratingly Easy Progressive Training of Extendable MoE

Abstract

Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on active experts rather than the total pool of experts. Yet, this asymmetry creates an MoE efficiency paradox in practice: adding more experts balloons memory and communication costs, making actual training inefficient. We argue that this bottleneck arises in part because current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity. Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training. EMO explicitly models sparsity in scaling law to derive stage-wise compute-optimal token budgets for progressive expansion. Empirical results show that EMO matches the performance of a fixed-expert setup in large-scale experiments while improving wall-clock efficiency. It offers a surprisingly simple yet effective path to scalable MoE training, preserving the benefits of large expert pools while significantly reducing both training time and GPU cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of expert, efficient pre-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VzfjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The article introduces a scaling-law-based method for growing the expert pool of an MoE during pretraining. The authors argue that holding the expert count fixed for the whole run is sub-optimal, since a large pool early in training is wasteful: the cost it adds comes not from the theoretical FLOPs, which depend only on the activated parameters, but from the overheads of high sparsity and of maintaining and updating a much larger total parameter count. EMO therefore starts small and expands in stages, with each stage's token budget derived from a sparsity-aware scaling law fit on a small-scale sweep over expert counts. The main study grows a 1.1B-activated model to 128 experts over 1.92T tokens in five stages; this ends at a slightly higher loss (1.017) than the fixed 128-expert baseline (0.994), while the authors claim a 10% saving in overall GPU hours.

Critical strengths

The paper identifies the right question. Any progressive-growth method lives or dies on scheduling, and section 3 isolates this cleanly rather than treating expansion timing as an implementation detail subordinate to the architecture.

Using a scaling law to set that schedule is the appropriate tool, and it is applied with a standard protocol: the functional form is established in the literature, the fit follows Hoffmann et al. (LBFGS under Huber loss), and the reported quality is good: RMSE 0.0085 and held-out RMSE 0.0092. The one gap is that the held-out set is never specified.

The method is well situated against prior work. section 2.3 states precisely what single-step sparse upcycling does - replicating FFN weights so the dense model's behaviour is preserved under uniform routing, and identifies where it degrades as the expert count grows. Multi-step expansion is then motivated as a direct response to that limitation rather than asserted.

The presentation is clear. Algorithm 1 gives the method compactly, Figure 2 makes the expansion step concrete at the level of expert, router-weight and router-bias initialization, and Figure 4 walks through the schedule derivation step by step.

The empirical validation is substantial for a submission of this kind: a five-stage from 8 to 128 expansion of a 1.1B-activated model over 1.92T tokens, against from-scratch fixed-expert baselines under matched token budget, hyperparameters and data mixture, with loss curves and downstream evaluation measured throughout training.

Critical weaknesses

The held-out evaluation is not specified. A held-out RMSE of 0.0092 is reported, but the paper never says how that set was chosen, which points, or how many. To measure scaling laws extrapolation accuracy the best would be to test leave lowest losses out (f.e. 10% of lowest losses models), which would say the most about the extrapolation the schedule relies on. Additionally the training grid itself is only partly described, listing the expert counts swept but not the activated sizes or token budgets.

Section 2.2 attributes the FLOPs–wall-clock gap primarily to all-to-all communication from expert parallelism, and Section 1 repeats this, but expert parallelism appears nowhere in the paper: Section 4.1 describes 4 nodes × 8 H200 with "data parallelism," and Appendix B adds no parallelism detail. Under that configuration there is no expert-parallel dispatch, so the overhead named first in the paper's own causal account cannot be what Figure 1 measures; the costs that do scale with E here are gradient-synchronization volume, optimizer state, and optimizer-step work, all tracking total parameters, which grow roughly 9× from E=8 to E=128 at constant activated FLOPs. This bears directly on the claim in Section 6 that EMO is "orthogonal" to systems optimizations. As composability that is uncontroversial, but the paper uses it in the stronger sense that EMO's gains persist undiminished on such a stack, and that does not follow: EMO's saving is determined by the shape of the realized step-time curve dependent on number of experts - T(E). Relevant optimizations: the grouped GEMM, block-sparse computation and all-to-all overlap cited in Section 6, or, under the corrected account, routine bucketed gradient overlap - act precisely to flatten that T(E) curve. Since Figure 1 is measured on a stack with no stated expert-path optimization (Section 4.1), the reported slowdowns are best read as an upper bound on the gap a well-optimized stack would exhibit, and correspondingly on EMO's advantage. Some E-dependence is irreducible, so a benefit would remain; its magnitude is what is unestablished. We would ask the authors to state the parallelism and overlap configuration and report MFU, revise Section 2.2 to name the overheads actually operative in their setup. By: demonstrating EMO's savings on top of methods from “Systems optimizations for MoE training.” (section 6), reporting MFU alongside the parallelism, would settle or clarify more measured gains.

Questions for authors

Could authors report MFU alongside the parallelism and T(E) optimization (see weakness 1) of their training, one for static E and averaged or per E after extension for EMO method training? This does not require additional computation that could most likely be read or approximated from already conducted experiments. For more precise measurement (that would be preferable), few step training with additional time measurement utility should give accurate MFU results for different E (2, 4, 8,16, 32, 64, 128) with the training method used in the article (using 4 x 8 h200 gpus).

Could you clarify the held-out methodology for reporting RMSE value. Could you compute a hold-out leaving out 10%, 20%, 30% of lowest losses out of (90%, 80%, 70% highest losses fit respectively) and test their RMSE to evaluate the quality of the fit. Leaving 10% lowest losses out would be similar to Ludziejewski et al., 2025 hold-out evaluation.

1 comment

We thank Reviewer Vzfj for their time and the insightful, positive review! We are glad the reviewer finds that the paper isolates scheduling as the key question of progressive growth and that the scaling-law protocol is the appropriate tool for it.

W1/Q2. Held-out methodology and training grid.

The scaling-law experiments covers E∈1,2,4,…,256E\in1,2,4,\dots,256 at top-2, activated model size ranges from 65M to 657M, and training budgets from 60B to 1.2T tokens, with 50 runs in total. The original held-out set is randomly sampled 10%. Following reviewer's suggestion, we fit on the highest-loss 90/80/70% of runs and evaluate on the held-out lowest-loss 10/20/30%:

Held-out setFit RMSEHeld-out RMSE
lowest 10%0.00840.0105
lowest 20%0.00820.0128
lowest 30%0.00790.0163

Held-out error stays within about 2× the fit error even when extrapolating to the lowest 30% of losses.

W2/Q1. Parallelism, the causal account of T(E)T(E), and MFU.

Our training stack binds EP with TP, but the reported A1.1B experiments use FSDP across 32 H200 GPUs with TP=EP=1. Thus, Figure 1 contains no inter-GPU expert-token all-to-all. The all-to-all discussed in Sections 1 and 2.2 is a potential cost when EP is enabled, rather than an explanation of the measured A1.1B slowdown. We will make this distinction explicit in the revision.

In the measured configuration, increasing EE increases FSDP parameter all-gather, gradient reduce-scatter, and optimizer work; expert-kernel efficiency can also change. Parameter all-gathers are issued asynchronously, and backward prefetch overlaps communication with gradient computation. G1 specifies our sharding and communication-overlap configuration. The reported MFU values across E=8,16,32,64,128E=8,16,32,64,128 are 33.2%, 32.1%, 31.1%, 29.7%, and 28.0%. At the tested E=8E=8 shape, our expert GEMM benchmark is comparable to Transformer Engine grouped GEMM (92.1 vs. 92.3 ms). We observe that grouped GEMM would not flatten T(E)T(E): the two kernels also match at E=128E=128.

Orthogonality. EMO can be combined with expert-kernel and communication optimizations: it changes how long each expert-pool size is used. The magnitude of its saving depends on the resulting T(E)T(E): our measured GPU-hour reduction is obtained on the stated FSDP configuration, while an EP configuration would change both parameter communication and token dispatch costs.

Reviewer_KhWCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EMO, progressive training framework that increases the number of MoE experts during pretraining while keeping the number of activated experts per token fixed. The motivation is that increasing the expert pool raises memory and system overhead even when theoretical FLOPs remain approximately unchanged. EMO therefore uses smaller expert pools early in training and expands capacity as training progresses.

The method uses an expert-aware scaling law to determine stage-wise token allocations. At each expansion, new experts and router weights are randomly initialized, router biases and optimizer states are reset, and training resumes with a short warmup. The main experiment progressively expands from 8 to 128 experts over 1.92T tokens, reaching 9.6B total parameters with 1.1B activated parameters. EMO achieves a final training loss of 1.017 versus 0.994 for the fixed-128-expert baseline, while reporting approximately 10% lower GPU-hour consumption in practice. Additional experiments examine expansion timing, initialization, expert utilization, representation collapse, and the relationship between expert capacity and downstream capabilities.

Critical strengths
  1. The paper addresses a practical training-efficiency problem: The distinction between theoretical FLOPs and actual training cost is relevant to MoE deployment at scale. The profiling results show that increasing expert count can substantially increase step time despite fixed activated capacity, providing a concrete motivation for progressive expansion.
  2. The proposed framework is simple and compatible with standard MoE architectures: EMO requires relatively limited architectural changes and can potentially complement existing routing and systems optimizations. The initialization experiments suggest that sophisticated expert initialization is unnecessary under the evaluated training budget, which improves the method’s practical appeal.
  3. The main experiment provides substantial empirical evidence for feasibility: Training a five-stage model on 1.92T tokens demonstrates that repeated expansion can remain stable over a substantial pretraining run. Comparisons with fixed expert pools use the same token budget and similar training settings, and show that EMO recovers much of the performance benefit of a larger expert pool.
  4. The analysis examines several potential failure modes: The paper goes beyond final loss by studying expansion transients, initialization choices, optimizer-state handling, routing statistics, and representation collapse. The near-unity new-to-old expert load ratios provide useful evidence that newly introduced experts are incorporated into routing rather than remaining systematically underused.
Critical weaknesses
  1. The central token-allocation derivation is incomplete as written: Equation (2) optimizes over D while fixing both N_{act} and F, subject to F=6N_{act}D. These conditions uniquely determine D, leaving no freedom to obtain expert-dependent optimal token counts. The connection between this formulation and the cumulative budgets in Figure 4 therefore requires clarification. Furthermore, fitting a loss model to fixed-expert runs does not establish that differences between their optimal token budgets yield an optimal progressive schedule. Expansion transients and actual wall-clock costs are not explicitly included in this optimization.
  2. The experiments do not sufficiently isolate the benefit of the proposed schedule: The timing study evaluates only one transition, 16→32, at three expansion times. It shows that the predicted timing lies near a reasonable trade-off region, but does not establish the advantage of the scaling-law allocation over simpler schedules in the full multistage setting. The one-shot 8→128 comparison is an extrapolation rather than a measured baseline, and transfers recovery behavior from a twofold expansion to a sixteenfold expansion. Consequently, the respective contributions of progressive growth, multiple intermediate stages, and the specific allocation rule remain unclear.
  3. The reported efficiency gain needs a more transparent accounting: The approximately 10% GPU-hour reduction is not supported by a sufficiently detailed end-to-end timing breakdown. If the A1.1B step-time ratios in Figure 1 apply directly to the main experiment, weighting them by the reported stage allocations suggests approximately 3.5% savings relative to fixed E=128. Different measurement conditions could explain this discrepancy, but they should be documented. Absolute GPU hours, per-stage throughput, expansion overhead, and the cost of obtaining the scaling-law fit would make the efficiency claim easier to assess.
  4. Some performance claims are stronger than the reported results support: EMO’s final loss is higher than that of fixed E=128, and several downstream differences are substantial: GSM8K decreases from 64.73 to 57.09, and BoolQ from 74.89 to 70.40. These results demonstrate a quality–cost trade-off; performance equivalence is not established. A comparison at matched GPU-hour budgets would better determine whether EMO provides a superior trade-off. The paper also does not report sufficient uncertainty estimates to assess small downstream differences.
  5. The generality of the allocation strategy remains insufficiently validated: The scaling-law experiments use top-2 routing, whereas the main experiment uses top-8. The transfer of the fitted allocation across these settings is not directly validated. Complete progressive training is also demonstrated at only one activated model scale and hardware configuration; the larger 4B setting is used for efficiency profiling. Dependence on routing sparsity, model scale, and training hyperparameters remains an open question.
Questions for authors
  1. Could you clarify the optimization underlying Equation (2)? Which variables are actually optimized, and is the compute budget shared across expert configurations or chosen separately for each configuration? Please provide a reproducible derivation of the cumulative token budgets in Figure 4, including any joint optimization over activated parameters and tokens.
  2. How much does the scaling-law schedule improve over simpler expansion strategies? Could you provide measured comparisons with a one-shot expansion and a simple uniformly spaced multistage schedule, preferably under matched token and GPU-hour budgets? A controlled experiment at a smaller representative scale would also help isolate these contributions.
  3. How is the reported 10% GPU-hour saving calculated? Please report absolute training costs and per-stage timing, explain whether Figure 1 uses the same measurement conditions, and specify whether expansion overhead and scaling-law fitting costs are included.
  4. What is the performance comparison at equal actual training cost? For example, how does EMO compare with a fixed-128-expert model trained for the same GPU-hour budget with an appropriately adjusted learning-rate schedule? Can you also provide uncertainty estimates for the downstream comparisons?
  5. How sensitive is the allocation to routing and training settings? What supports transferring a top-2 scaling-law fit to top-8 training? Have you tested alternative routing sparsities, activated model sizes, or learning-rate and optimizer-state schedules?
1 comment

We thank Reviewer KhWC for their time and the detailed, constructive feedback! We are glad the reviewer finds that EMO addresses a practical training-efficiency problem.

W1/Q1. The optimization underlying Eq. (2) and the derivation of the cumulative token budgets in Figure 4.

Eq. (2) is the joint compute-optimal problem over both NN and DD:

(N∗(F,E), D∗(F,E))=arg⁡min⁡N,D L(N,E,D)s.t.6ND=F.(N^*(F,E),\ D^*(F,E)) = \arg\min_{N,D}\ L(N,E,D) \quad \text{s.t.} \quad 6ND = F.

Given fixed NN, DD and final desired total experts EE decided for training, the compute budget is chosen separately for each expert configuration:

  1. For each stage EsE_s, we find the budget FsF_s at which the compute-optimal activated size equals our model: N∗(Fs,Es)=NactN^*(F_s,E_s) = N_{\text{act}}.
  2. We set Ds∗=Fs/(6Nact)D_s^* = F_s/(6N_{\text{act}}), the token count at which an NactN_{\text{act}}-sized model with EsE_s experts is compute-optimal.
  3. Eqs. (3)–(4) convert these cumulative optima into incremental, normalized stage budgets.

We will state this explicitly in the revision, with the closed-form solution and a script reproducing Figure 4.

From fixed-expert fits to a progressive schedule. Each EMO stage is a standard fixed-EE MoE, so Ds∗D_s^* is the cumulative data at which capacity level EsE_s is compute-optimal, and each stage receives the additional data its capacity justifies. This gives a schedule with no tuned hyperparameters. Expansion transients recover within ~10K steps, small relative to every stage budget, and the timing ablation (G3) confirms the predicted expansion point empirically.

W2/Q2. Isolating the benefit of the schedule; measured one-shot and uniform baselines.

Please see G3.

W3/Q3. How is the 10% GPU-hour saving calculated? Are expansion overhead and scaling-law fitting costs included?

Please see G2 for the measurement conditions, absolute GPU-hours and per-stage timing. In addition, expansion overhead only use 0.2% of the run in total. Scaling-law fitting costs are not considered, as this is a one-time cost per model family and data mixture, can be reused by every subsequent run.

W4/Q4. Performance claims; comparison at equal training cost; uncertainty estimates.

Please see G4 for matched gpu hours comparison.

Uncertainty (paired bootstrap, 10K resamples): EMO improves the 13-task average over FIXED_E=32 by +1.99, 95% CI [+1.4, +2.6]. Against FIXED_E=128, the per-task 95% CI includes zero on 10 of 13 tasks.

W5/Q5. Sensitivity of the allocation to routing and training settings.

  • Top-2 → top-8. Please see G5.
  • Optimizer state. Carrying vs. resetting optimizer states at expansion makes little difference after the 500-step warmup (§5).
  • Other routing sparsities and activated sizes. We thank the reviewer for this suggestion. We report a top-8 scaling-law re-fit in G5 to assess sensitivity to routing sparsity. We agree full progressive-training experiments across additional activated model sizes, learning-rate schedules, and optimizer-state policies are valuable directions for future work; they are beyond the scope of the current large-scale study. However, we remain positive to explore them in the future!
Reviewer_BJ2BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EMO, a progressive training framework for Mixture-of-Experts (MoE) models that gradually increases the number of experts during pretraining while keeping the number of activated experts fixed. The motivation is that increasing the total expert count can introduce substantial memory and communication overhead, even when the theoretical per-token FLOPs remain unchanged.

EMO uses a sparsity-aware scaling law to determine the token budget for each expansion stage. At each stage, new experts and router parameters are initialized, and training continues with the expanded model. The authors evaluate EMO by progressively increasing the expert count from 8 to 128 over 1.92T training tokens. Compared with training a fixed E=128 model, EMO achieves a final loss of 1.017 versus 0.994 while reportedly reducing GPU hours by approximately 10%.

Critical strengths

The paper proposes the interesting idea of progressively increasing the expert count, which is a straightforward approach to reducing the cost of large-scale MoE training. The method requires relatively few architectural changes and may be useful when larger expert pools introduce significant training overhead.

Critical weaknesses

The practical training efficiency gains are not sufficiently established.

  • The paper reports approximately 10% GPU-hour savings, but does not provide a comparison of model quality under the same GPU-hour budget. Since EMO achieves worse final loss than the fixed E=128 baseline at the same token budget, it remains unclear whether EMO actually provides a better quality–training-cost trade-off.
  • Additionally, the reported efficiency gains may depend heavily on the underlying MoE implementation. Appendix B.1 mentions data parallelism but does not specify whether expert parallelism is used or provide details about gradient synchronization and MoE communication. This makes it difficult to assess how well the results generalize to well-optimized large-scale MoE training systems.
Questions for authors
  • Could the authors provide more details about the MoE implementation used in the experiments?

    • Are expert computations implemented using efficient grouped GEMM kernels?
    • Appendix B.1 mentions data parallelism. Is expert parallelism (EP) or any other parallelism strategy used to distribute expert parameters and reduce parameter-related communication overhead, such as gradient synchronization?
    • What are the main sources of the increased step time as E grows (e.g., expert computation, gradient synchronization, or token dispatch/communication)?
    • This would help clarify whether the reported overhead is inherent to increasing the expert count or depends on the particular implementation.
  • Figure 1 reports a 1.08× step-time increase from E=8 to E=128 for the 1.1B activated model. However, EMO reports approximately 10% GPU-hour savings despite spending around 28.6% of training at E=128. Could the authors clarify this discrepancy? Are the step-time measurements and GPU-hour savings obtained under the same hardware and training configurations?

  • How does EMO compare with a fixed E=128 model trained under the same GPU-hour budget, rather than the same number of tokens? A GPU-hour-matched comparison would help establish whether progressive expansion actually improves training efficiency.

  • Why is the scaling law fitted using top-k=2 while the main experiments use top-k=8? Have the authors validated whether the resulting token allocation generalizes across different top-k configurations?

  • How much of the improvement comes from delaying the use of a large expert pool versus the proposed scaling-law-based token allocation? How does the proposed schedule compare with simpler strategies, such as uniform stage allocation or a one-shot E=8→128 expansion, at comparable training costs?

1 comment

We thank Reviewer BJ2B for their time and valuable feedback! We are glad the reviewer finds progressive expert expansion an interesting and straightforward way to reduce large-scale MoE training cost.

W1/Q1. MoE implementation

Please see G1. In short, our training stack is highly optimized for MoE training.

Q2. Figure 1 reports a 1.08× step-time increase at A1.1B, while EMO reports ~10% GPU-hour savings.

Please see G2.

Q3. EMO vs. FIXED_E=128 under the same GPU-hour budget.

Please see G4: at FIXED_E=128's GPU-hours, EMO reaches 0.992 vs. 0.994 for FIXED_E=128, with average downstream scores outperforms FIXED_E=128 (59.02% vs. 58.15%).

Q4. Why top-k=2 for the scaling law and top-k=8 for the main experiments?

Please see G5.

Q5. Delaying the large expert pool vs. the scaling-law allocation; uniform allocation or one-shot E=8→128.

Please see G3: the scaling-law expansion point sits in the flat region of the quality–cost curve in the timing ablation, and EMO (1.017) outperforms both uniform allocation (1.026) and one-shot expansion (1.039).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to reviewers

We thank all reviewers for their time and valuable feedback! Several questions were shared across reviewers; we answer them here.

G1. MoE implementation, sources of EE-dependent step time, and MFU (BJ2B-Q1, Vzfj-Q1)

All experiments run on a highly optimized training stack: BF16 mixed precision, FlashAttention-3, fused transformer-block kernels, selective activation recomputation and sequence packing. The measurements below quantify this.

  • Parallelism. FSDP data parallelism on 4 × 8 H200, with all parameters, gradients and optimizer states fully sharded across the 32 GPUs (ZeRO-3 style), one FSDP unit per Transformer layer. Communication overlaps with computation: all-gathers are issued asynchronously, and in the backward pass the next layer's parameters are prefetched before the current layer's gradient computation. Parameters are gathered in BF16 and gradients are reduce-scattered in FP32. Expert parallelism is not used, so there is no token dispatch.
  • Expert kernels. Expert computation uses per-expert cuBLAS GEMMs over expert-sorted tokens. At the exact shapes of our runs (262K tokens per GPU, d=2048d=2048, expert dim 768, top-8; forward + backward per MoE layer), this matches Transformer Engine grouped GEMM in our experiments:
EOurs (per-expert cuBLAS)TE grouped GEMMPyTorch grouped_mm
892.1 ms (65% of peak)92.3 ms (65%)102.3 ms (59%)
  • MFU (BF16, seq 8192; FLOPs include attention with causal masking; peak 989 TFLOPs).
E8163264128
MFU33.2%32.1%31.1%29.7%28.0%

The static E MFU is almost identical to EMO per stage. Published BF16 MFU for comparable fine-grained MoEs is 13–28% [1–3], including 28% for NVIDIA NeMo's Qwen3-30B-A3B, which has the same expert shape as ours, suggesting our training stack is efficiently optimized.

  • Sources of EE-dependent step time, at fixed activated parameters:

    1. FSDP all-gather / reduce-scatter of expert parameters and gradients (volume ≈9× from E=8E=8 to E=128E=128);
    2. optimizer-step work and optimizer state over total parameters;
    3. more, smaller per-expert GEMMs.

    Expert-GEMM time grows by only 8% from E=8E=8 to E=128E=128 and accounts for only about 10% of the step-time increase. The rest comes from (1) and (2).

The measured T(E)T(E) is therefore dominated by costs that scale with the total parameter count. They are inherent to a larger expert pool, persist under any expert kernel, and are exactly what EMO removes from the early stages.

[1] NVIDIA NeMo AutoModel Performance Summary. https://docs.nvidia.com/nemo/automodel/latest/performance/performance-summary.md [2] Muennighoff et al. OLMoE: Open Mixture-of-Experts Language Models. 2024. [3] TorchTitan DeepSeek-V3 benchmarks. https://github.com/pytorch/torchtitan/pull/2107


G2. GPU-hour accounting (BJ2B-Q2, KhWC-Q3)

StageEToken shares/stepGPU-hours
1823.5%5.952,988
2169.5%6.141,241
33216.5%6.342,233
46421.8%6.663,117
512828.6%7.194,377

From E=8E=8 to E=128E=128, step time increases 1.21× at A1.1B. EMO saves 9.0% of GPU-hours relative to FIXED_E=128, in line with the ~10% reported in the paper. Weighting the per-stage step times by the stage allocation reproduces this number. Stage 5 runs the FIXED_E=128 configuration, so the savings come from Stages 1–4, which cover 71.4% of the token budget. Figure 1 was measured in a preliminary experimental setting, and we will update it with the timings above in the revision. We still observe that the wall-clock slowdown grows quickly as the model scales up.

G3. Scaling-law allocation vs. simpler schedules (BJ2B-Q5, KhWC-Q2)

Allocation. The timing ablation (E=16→32E=16\rightarrow32, Figure 5) shows that the scaling-law expansion point sits in the favorable region of the quality–cost curve. Moving the expansion from 25% to 50% of training changes the final loss by only 0.002 (1.069 → 1.071), while delaying it further to 75% costs another 0.005 (1.076). The scaling-law allocation (~45%) therefore keeps nearly all of the quality of early expansion while expanding as late as the flat region allows.

Multi-stage vs. one-shot. At matched token and GPU-hour budgets, we report the one-shot experiment E=8→128E=8\rightarrow128 final loss is 1.039, vs. 1.017 for EMO, suggesting that EMO's multi-stage is outperforming one-shot experiment.

Measured baselines at the same token budget:

ScheduleStage token sharesFinal loss
EMO (scaling law)23.5 / 9.5 / 16.5 / 21.8 / 28.6%1.017
Uniform multi-stage20 / 20 / 20 / 20 / 20%1.026
One-shot 8→1288\rightarrow128 at 50%50 / 50%1.039

The uniform schedule loses 0.009 in loss, and one-shot expansion loses 0.022. Both multi-stage growth and the scaling-law allocation contribute.

G4. Comparison with FIXED_E=128 at equal GPU-hours (BJ2B-Q3, KhWC-Q4)

  • EMO extended to FIXED_E=128's GPU-hours. With the WSD schedule extended accordingly, EMO reaches 0.992, vs. 0.994 for FIXED_E=128, with average downstream scores outperformS FIXED_E=128 (59.02% vs. 58.15%).

G5. Top-2 scaling sweep vs. top-8 main runs (BJ2B-Q4, KhWC-Q5)

We fit the scaling law at top-2 because it lets the sweep cover sparsity from dense (E=1E=1) to 128× at tractable cost. EMO takes only one quantity from this fit: the relative growth of the compute-optimal token count across EE, which sets the stage proportions. It does not use absolute loss values. Re-fitting the law on a top-8 sweep gives nearly the same allocation:

Stage (E)8163264128
Top-2 fit (paper)23.5%9.5%16.5%21.8%28.6%
Top-8 fit24.6%9.1%15.8%21.3%29.2%

Every expansion point moves by at most 1.1 points of the token budget. Shifts of this size do not matter: in the timing ablation (G3), moving the expansion point from 25% to 50% changes the final loss by only 0.002.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.