acceptodds
Under review as a conference paper at ICLR 2027

Optimal Hyperparameter Scaling Laws Across MoE Sparsity

Abstract

Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes hyperparameter transfer challenging. In this work, we show that conventional hyperparameter scaling laws are insufficient for ultra-sparse MoEs: optimal learning rate and batch size vary with activation ratio, and neither total nor activated parameter count explains these shifts. To characterize this dependence, we conduct 1,800 pre-training runs spanning six activated-parameter scales and models with up to 6B total non-embedding parameters, processing approximately 20 trillion tokens at a cost of 200,000 equivalent H800 GPU-hours. Our results reconcile conflicting findings in prior work by revealing two scaling regimes. At fixed sparsity, optimal batch size follows a power law in training tokens , whereas optimal learning rate scales with training compute and remains robust to the allocation between model size and data. Across sparsity levels, activation ratio enters both relationships as a multiplicative power-law factor, yielding unified laws that transfer across MoE sparsity levels. Large-scale evaluation shows that the scaling form outperforms alternative functional forms. On a held-out ultra-sparse MoE with 12B total parameters and only 1/64 of its experts activated, the predicted hyperparameters remain close to the observed optima, supporting joint extrapolation across model scale and sparsity. Further experiments demonstrate transfer across expert granularities and isolate the effect of activation ratio from total expert count.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Scaling Laws, MoE

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sTgZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies hyperparameter scaling laws in MoE pretraining, as a function of activation ratio of the experts. The authors report that they conducted large scale experiments, comparing different parametric scaling law forms. The main contribution is a the sparsity-dependent correction to existing hyperparameter scaling laws.

Critical strengths
  • The paper is very-well written, and easy to follow (including the appendix).
  • The experimental effort is substantial (authors reported conducting more than 1800 pretraining runs). The joint LR-BS sweeps across size scales and several sparsity levels definitely requires significant compute (grid search cost grows exponentially with the number of hyperparameters).
  • The conclusions from the empirical results are well stated as well.
Critical weaknesses

While the experimental results cover many key components of scaling, there are several issues that limit the significance of the contributions.

  • Significance of sparsity scaling: Figure 2 shows relatively flat minima with modest LR shifts across sparsity levels. The results in Fig 4a also show an exponent of LR scaling laws with sparsity is very small (roughly 0.13), which is basically indistinguishable from using a constant LR at the practical sparsity levels. For instance, a 64x change in activation ratio changes the predicted LR by only a factor of 1.76, which is very small given that optimal LR is usually within a flat region of the loss. The same can be said for BS. A small exponent does not invalidate the scaling law necessarily, but the paper should at least demonstrate meaningful excess validation loss when LR is held fixed across sparsity.

  • Theoretical contribution: Appendix C assumes a BS loss model and that η∗∼(AB∗)ρ\eta* \sim (AB^*)^\rho. These assumptions are not well-justified and seem like they’re introduced to fit an observation, rather than provide an explanation to the observed laws.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Weaknesses

Weakness 1: Significance of sparsity scaling.

Response: Thank you for raising this point. We agree that the key question is whether holding LR fixed across sparsity levels incurs excess validation loss, rather than whether the optimal LR shifts. To address this directly, we added three controlled comparisons: at each training scale, we use the LR predicted by our law at A=1A=1 and hold it fixed when transferring to A=1/64A=1/64, then compare it with the sparsity-aware prediction for the target configuration. Within each comparison, the model, compute budget, BS, and other training settings are identical, with a sequence length of 8192. Validation loss reduction = validation loss with the shared LR − validation loss with the sparsity-aware LR.

Model size NNFLOPs CCActivation ratio AABatch sizeShared LRSparsity-aware LRValidation loss: shared LRValidation loss: sparsity-aware LRValidation loss reduction
80M1e191/642560.001950.001111.67571.6629+0.0128
158M3e191/643200.001670.000951.55191.5469+0.0050
324M1e201/644160.001420.000801.44851.4371+0.0114

The sparsity correction achieves lower validation loss at all three model scales, with absolute reductions of 0.0128, 0.0050, and 0.0114, respectively. These controlled comparisons show that an approximately 1.76-fold LR adjustment yields actual loss improvements in all three configurations. Thus, flat minima do not imply that holding LR fixed across sparsity incurs no excess validation loss in these configurations.

We therefore disagree that the sparsity correction is “basically indistinguishable from using a constant LR” solely because its exponent is small. Modest multiplicative LR adjustments are also consistent with industrial pretraining practice: DeepSeek-V4 [1] Flash (284B total parameters, 13B activated) and Pro (1.6T total parameters, 49B activated) are trained on 32T and 33T tokens, respectively, with peak LRs of 2.7×10−42.7\times10^{-4} and 2.0×10−42.0\times10^{-4}. Their total parameter counts differ by approximately 5.6-fold, whereas their LRs differ by only 1.35-fold. The exponent describes the magnitude of the change in optimal LR; it does not directly determine the corresponding loss difference. The DeepSeek example provides an industrial reference for the magnitude of LR adjustments, while the table above provides direct loss evidence for our sparsity correction.

Therefore, by correcting the prediction bias caused by ignoring sparsity, our work improves the reliability of transfer across sparsity levels and reduces the risk of hyperparameters falling outside the near-optimal region. This is particularly valuable for costly, large-scale pretraining.

 

Weakness 2: Theoretical contribution.

Response: Thank you for pointing out the need to clarify the theoretical scope. Our methodology is empirical: we first identify trends in hyperparameters across training scales and sparsity levels through experiments, and then test the proposed laws through comparisons of candidate functional families, grouped cross-validation, and held-out prediction. Appendix C is intended to provide a simplified mechanistic model for understanding these trends, rather than a first-principles derivation of the empirical scaling laws.

We agree that the BS loss model and η∗∝(AB∗)ρ\eta^* \propto (AB^*)^\rho in Appendix C are modeling assumptions, rather than independently established explanations of the observed laws. Under balanced routing, each expert receives an average of AB tokens, which provides a mechanistic motivation for the effective expert batch size, but is insufficient to directly derive the assumed loss function or the power-law response of the optimal LR. Thus, the signs of the exponents obtained in the appendix are conditional deductions under these assumptions, rather than an independent proof of the empirical laws.

We will explicitly present Appendix C as “a simplified model for understanding sparsity trends,” distinguish assumptions from deductions, and remove wording that suggests an independent derivation of the empirical laws. These assumptions were not used to select the functional form or fit the empirical laws; they were introduced only to provide a simplified interpretation of the observed trends. The empirical fits and held-out predictions do not depend on this model; our core contribution is a set of empirically tested hyperparameter scaling relationships.

 


References:

[1] DeepSeek-AI. "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence." arXiv preprint arXiv:2606.19348 (2026).

Reviewer_sTgZOpenReview ↗

Response

Thank you for the comments and additional experiments.

I remain however unconvinced by the results:

  1. Significance of sparsity scaling: as I mentioned previously the exponents are extremely small. The reported excess risk results are also very small (unclear if it's within seed noise margin). The authors claim that production level training adjust the LRs with sparsity. While this is true, the evidence in this paper does not apply to production scale (the exponents might not extrapolate to that scale), so claiming that production scale results support the findings in this paper is a big stretch.

  2. Theoretical contributions: I understand that the paper is primarily empirical. That does not justify using any convenient assumptions to explain the observed patterns. In that case, it might be better not to have any theory than have a set of unjustified and loose assumptions.

1. Significance of sparsity scaling.

  • Thank you for highlighting this point. We acknowledge that the current results are based on single runs. Nevertheless, the sparsity correction reduces validation loss at all three model scales, providing consistent evidence of its benefits. Given the high cost of pretraining and our limited resources, we are currently unable to conduct sufficient multi-seed runs.

  • We would also like to clarify that the production-model examples were cited only to provide a practical reference for the magnitude of LR adjustments , not to validate our laws or the accuracy of extrapolating their exponents to production scale. We will revise the wording to clearly distinguish this reference from the conclusions supported by our experiments.

2. Theoretical contributions.

Thank you for your suggestion. We will remove the simplified model in Appendix C and the mechanistic explanations in the main text that rely on it, leaving the development of an independently justified theoretical explanation to future work. We appreciate your rigorous review, which helps us define our contribution more clearly.


Thank you again for your careful review and constructive feedback.

Dear reviewer sTgZ,

As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your support if we have fully resolved all outstanding issues. Thank you for your time and consideration.

Reviewer_JBe7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how the optimal learning rate and batch size can be predicted with training scale in MoE pretraining, and argues that the activation ratio A = E_act/E_tot (that models sparsity) must be incorporated explicitly. From ~1,800 runs (six activated scales, A in {1, 1/4, 1/16, 1/32}, ~20T tokens, ~200k H800-hours) it claims: at fixed sparsity the optimal learning rate follows a power law in compute (C) and the optimal batch size a power law in tokens (D). Across sparsity levels A they used both as a multiplicative power-law factor, giving h*(X, A) = k_h X^g_h A^d_h. The form is selected against three alternative families by grouped cross-validation (Table 3) and validated against DeepSeek Law and Step Law.

Critical strengths

Important and under-served problem. Hyperparameter transfer into the ultra-sparse regime matters for frontier MoEs and is not covered by existing hyperparameter scaling studies.

Clean formalization and clearly stated methodology. Section 2.1 defines the target and the candidate variables without presupposing which matter, then compares them empirically.

Modern, well-controlled setup: hybrid linear-attention/MLA backbone, Muon, WSD schedule, and computed analytically as C = MD rather than via 6ND.

The Section 4.4 controls are the good design: a 2x2 that separates A from absolute expert count and from total capacity (Figure 6).

Section 6 is honest. It names the overlapping cross-validation intervals, the inability to rule out exponents varying with A, the single extrapolation target, and the absence of seed repeats. Section 3.2 likewise concedes that four slices cannot rule out varying base-scale exponents.

Critical weaknesses

Novelty of the learning-rate result is overstated. Ludziejewski et al. (2025) already gives LR = exp(8.39 - 0.81 ln N_act - 0.25 ln E), i.e. LR ~ N^-0.81 E^-0.25. With non-granular top-1 routing E = 1/A, so this is LR ~ N^-0.81 A^+0.25 - the same multiplicative power law in sparsity, same sign, as this paper's A^+0.1361. That work also ablates the E term and validates by extrapolation to E = 32. What is genuinely new here is narrower: C in place of N_act, and higher sparsity extrapolation, and granular experts. The introduction's claim that "existing scaling laws do not characterize how the optima vary continuously with sparsity" should be revised. Section 3.1 cites that work for activated parameter count while omitting that it already carries a sparsity term.

The claim that A modulates the optimal batch size is weakly supported. Claim that A term in formula is a key element for fitting capabilities is given equally to the bs and lr - "A enters both relationships" or "an additional key predictor ... of both laws". Section 3.2 takeaway and the conclusion - and the "unified" framing depends on it. Table 3 does not bear this out. Adding a sparsity term improves batch-size prediction over the scale-only law by only 0.013 log2 under LONO and 0.050 under LOAO; the same step gains 0.140 and 0.202 for the learning rate, four to eleven times more. More telling, the same families split in opposite directions on the two hyperparameters: the additive law predicts batch size worse than scale-only under both schemes for (for LOAO 0.312 and 0.276 against 0.295 and 0.271) while predicting the learning rate far better (for LOAO 0.216 and 0.201 against 0.316 and 0.355), similar for LONO. For the same formula, folds, and data that gives opposite outcomes - the sparsity term contributes with iits parameters on the learning rate and fails on the batch size. Section 4.2 concedes that "differences in BS prediction are smaller and depend on the grouping scheme". The fitted d_B = -0.0841 is correspondingly small: it moves B* by 1.33x across the entire range A = 1 to 1/32. This bears on the framing: of the two laws making up the unified form, the sparsity term in the learning-rate law has clear precedent (weakness 1) while the batch-size one does not, so the batch-size law carries much of the weight of the "unified" claim - and it is the law the data does not establish definitively.

The choice of C over N is not shown. Figure 3a compares N, D and C with no A term and is judged qualitatively - "a clearer relation with C than with N or D alone"; Table 3 then fixes X = C and varies only how A enters. The natural competitor h*(N, A) = k N^b A^d is never fit, although Table 1 lists a published instance of it.

Three of the five methods in Table 1 are never fitted to the researchers data. Only DeepSeek and Step Law are evaluated. The stated reason (Section 4.3) - that the comparison needs a joint LR+BS law - follows from making the only baseline comparison a joint one; it is no obstacle in Table 3, where LR and BS errors are in separate columns.

Limited scale and a single-point test. All fits use 10M-324M activated parameters and at most ~6B total; the holdout is 324M activated / 12B total. The held-out evaluation is one configuration with no seed repeats, and the refitted baselines are effectively tied: +1.37 and +1.03 per mille (Section 4.3). Only the published-coefficient transfers lose meaningfully (+7.22, +9.02), and those were expected to be less accurate across a different optimizer, schedule and architecture.

Questions for authors

Table 1 records which variables each law uses, but not how much the switch from activated parameter count to compute actually improves prediction. Could the learning-rate law of Ludziejewski et al. (2025), LR = N_act^b E^-d, which under their non-granular top-1 routing is the family kN^bA^d - be refitted and added as a fifth candidate in Table 3 in the lr column evaluation? It requires only a refit of data, so no new training.

2 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions (1/2)

Weakness 1: Novelty of the learning-rate result.

Response: Thank you for accurately identifying the connection to prior work. We agree that, under the non-granular top-1 setting of Joint MoE [1], E=1/AE=1/A, and its LR law already includes a multiplicative sparsity term. Our novelty is not the first introduction of a sparsity power-law term for LR. We will revise the introduction and Section 3.1, replacing “existing laws do not characterize continuous sparsity dependence” with “prior work characterizes the dependence of LR on expert count under a specific routing setting; our work further examines hyperparameter transfer when training budget, activation ratio, and expert granularity vary jointly.”

This extension is particularly relevant to long training runs: with the model and expert count fixed, the Joint MoE LR formula predicts the same LR even when training is extended from 100B to 50T tokens, creating a risk of departing from the near-optimal region for long training horizons. Our formulation responds to changes in training budget through C=MDC=MD, and we test its practical benefits through comparisons across budgets (see our response to Reviewer-wWdS's Weakness 1 for the relevant comparisons). The capacity-matched controls in Figure 6 further support transfer across expert granularities.

 

Weakness 2: Evidence for sparsity-dependent batch size and the unified framing.

Response: Thank you for your detailed analysis. We agree that the sparsity effect and the supporting evidence are weaker for BS than for LR. We would like to further clarify its out-of-fold predictive value. Figure 2b shows that the BS optimum still shifts with AA when LR, DD, NN, and CC are fixed; Figure 4b shows the trend across sparsity levels at fixed DD; Table 3 further evaluates out-of-fold predictive value.

BS prediction error (absolute log⁡2\log_2 ratio)Scale-onlyMultiplicativeRelative error reduction
LONO0.2950.2824.4%
LOAO0.2710.22118.5%

The degradation of the additive form that you noted indicates that the specific form of the sparsity term affects predictive performance. Our selected multiplicative form achieves the lowest BS prediction error under both grouping schemes , supporting its predictive value within the tested range.

“Unified” refers to using the same separable form to characterize LR as a function of (C,A)(C,A) and BS as a function of (D,A)(D,A); it does not require equal sensitivity to AA. We will revise Section 3.2 and the conclusion to describe the BS sparsity term as a correction with out-of-fold predictive benefits, while making clear that its supporting evidence is weaker than for LR.

AuthorsOpenReview ↗

Response to Weaknesses and Questions (2/2)

Weakness 3 & 4 and Question: Choice of compute over parameter count and missing refitted baselines.

Response: Thank you for your suggestion. The original held-out comparison focused on joint LR–BS prediction, and Joint MoE does not provide a corresponding BS scaling law , so it was not included in that comparison. Your suggestion allows us to further evaluate LR separately and directly quantify the predictive gain from replacing NN with CC.

Following your suggestion, we refitted kNbAdkN^bA^d using the same development set, near-optimal target definition, LOAO/LONO splits, and error metric as kCγAδkC^\gamma A^\delta. All coefficients are refitted using only the training surfaces in each fold. Both forms have three fitted parameters, allowing a direct comparison of the base scale variable. Under the fixed top-2 routing in our main experiments, E=2/AE=2/A, and the constant 22 can be absorbed into the coefficient; thus, the refitted Joint MoE LR formula belongs to the same functional family as kNbAdkN^bA^d.

LR candidate familyLONO errorLOAO error
Scale-only: kCγkC^\gamma0.3160.355
Joint MoE (refitted): kNbAdkN^bA^d0.2140.203
Ours: kCγAδkC^\gamma A^\delta0.1760.153

With three fitted parameters in both forms and the same cross-validation protocol, our method reduces LR prediction error relative to refitted Joint MoE by 17.8% under LONO and 24.6% under LOAO. This shows that, on the current development set, replacing active parameter count with compute improves LR prediction while retaining the same sparsity term.

We also added controlled training comparisons to test whether prediction differences translate into actual loss improvements. We refitted Joint MoE on the same development set and compared the two LR predictions for the 80M configuration at A=1/64A=1/64. Each paired comparison uses a common BS selected with guidance from our BS law, with all other training settings held identical.

Training tokens DDFLOPs CCBatch sizeJoint MoE LRSparsity-aware LRValidation loss: Joint MoE LRValidation loss: sparsity-aware LRValidation loss reduction
2.0B1e18640.001250.001522.02072.0195+0.0012
19.6B1e192560.001250.001111.66091.6629-0.0020
196.0B1e207680.001250.000801.52231.5122+0.0101

Our method does not outperform at every budget, but reduces validation loss by 0.0101 at the longest budget of 196B tokens , supporting more accurate LR selection in the tested long-training configuration and providing actual loss evidence for extrapolation to longer token horizons. This capability is particularly relevant to production training: even with an unchanged model architecture, the training token budget may increase substantially, requiring LR to adapt to the budget. Explicitly modeling this dependence helps reduce hyperparameter-selection bias from reusing short-training settings and lowers the cost of trial-and-error tuning in expensive long training runs. We will add these results and their scope to the revised manuscript.

 

Weakness 5: Limited scale and a single-point extrapolation test.

Response: Thank you for pointing out the scope of the extrapolation evaluation. Under resource constraints, in addition to systematic searches covering six model scales and approximately 1,800 training runs, we completed a long-horizon held-out evaluation with 12B total parameters, 324M active parameters, and 159B tokens , but could not further cover multiple large-scale targets and repeated seeds. At A=1/64A=1/64 and C=3×1020C=3\times10^{20} FLOPs, this configuration tests joint extrapolation in sparsity, training tokens, and compute beyond the fitting ranges.

The value of this experiment is to test whether predictions frozen in advance transfer to an unseen configuration. Target losses were excluded from functional-family selection and coefficient fitting. We also refitted the baselines on the same development set, avoiding judging functional forms solely by gaps arising from transferring published coefficients across architectures. Our method achieves lower loss on this configuration, but the gaps of 1.37‰1.37‰ and 1.03‰1.03‰ remain single-run observations and are not evidence of a statistically significant advantage.

The added comparisons across budgets provide further evidence for long training horizons. Our conclusions are supported jointly by grouped out-of-fold prediction, long-horizon held-out evaluation, and controlled training comparisons , while larger active-parameter scales and repeated-seed validation remain areas for further evaluation.

 


References:

[1] Ludziejewski, Jan, et al. "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient." Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:41056–41073, 2025.

Reviewer_wWdSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how the optimal learning rate and batch size vary with training scale and MoE sparsity pretraining. It proposes empirical power laws in which learning rate depends on training compute C and activation ratio A, while batch size depends on training tokens D and activation ratio A. The proposed laws are evaluated through comparisons with alternative functional forms, grouped cross-validation, a frozen joint-extrapolation target, and expert-granularity controls under a fixed architecture family and training recipe.

Critical strengths
  1. The authors systematically investigate the power-law relationships between optimal batch size and training tokens (D) and between optimal learning rate and training compute (C) at fixed sparsity. It also examines how both hyperparameters vary with the MoE activation ratio.
  2. The authors propose unified hyperparameter scaling laws across multiple MoE sparsity levels and provide empirical support for their fit through comparisons with alternative functional forms.
Critical weaknesses
  1. Joint MoE Scaling Laws (Ludziejewski et al., 2025) already investigates the relationship between expert count and optimal learning rate. Although the additional sparsity controls provide further support, the current comparisons do not sufficiently establish the added predictive value of the proposed learning rate law over this prior formulation.
  2. The paper does not report evaluations on public downstream benchmarks, leaving it unclear whether the proposed laws translate into improvements in downstream model performance.
  3. Key details about the training-data distribution, validation-set construction, and parts of the fitting and evaluation procedures remain unclear, limiting the reproducibility of the results.
Questions for authors
  1. Could you clarify the added predictive value of the proposed learning rate law over Joint MoE Scaling Laws, and provide supporting comparative evidence?
  2. Are any downstream evaluation results available to assess whether the proposed laws translate into improvements in downstream model performance?
  3. Could you clarify the training-data composition, validation-set construction, and key steps of the fitting and evaluation procedures to facilitate reproducibility?
3 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions (1/2)

Weakness 1 & Question 1: Added predictive value over Joint MoE Scaling Laws.

Response: Thank you for the comment. Joint MoE [1] relates expert count to optimal LR. We further address LR transfer across training budgets and LR/BS selection across sparsity levels, with three main differences:

  1. Dependence on training budget. The Joint MoE LR formula depends only on the active parameter count and expert count, yielding the same prediction for different training horizons of the same model. Optimal LR varies with training horizon [2]; we capture this through C=MDC=MD.
  2. Joint selection of LR and BS. Joint MoE provides a batch-size ramp-up strategy, but no optimal BS law across training horizons and sparsity levels. Based on joint LR–BS searches, we establish predictive relationships in (C,A)(C,A) and (D,A)(D,A), respectively. We therefore did not include Joint MoE in the joint LR–BS prediction comparison in the original manuscript; here, we add an LR comparison at fixed BS to directly address your question.
  3. Transfer across expert granularities. In Figure 6, the configurations with “64 experts, 2 activated” and “128 experts, 4 activated” maintain the same A=1/32A=1/32, active capacity, and total capacity by adjusting expert width, and exhibit similar optimal LRs and BSs. This supports transfer across the tested expert granularities using activation ratio; absolute expert count alone cannot explain hyperparameter changes.

On the same development set, we pooled data across training budgets with 2 experts activated per token, refitted the Joint MoE LR formula, and compared predictions at three budgets for the same 80M model with A=1/64A=1/64. Each comparison uses the same integer BS guided by our BS law and identical remaining settings, isolating LR predictions under a common BS.

Model size NNTraining tokens DDFLOPs CCActivation ratio AABatch sizeJoint MoE LR (refitted)Sparsity-aware LRValidation loss: Joint MoE LRValidation loss: sparsity-aware LRValidation loss reduction
80M2.0B1e181/64640.001250.001522.02072.0195+0.0012
80M19.6B1e191/642560.001250.001111.66091.6629-0.0020
80M196.0B1e201/647680.001250.000801.52231.5122+0.0101

Sequence length is 8192; BS is measured in samples. Validation loss reduction = Joint MoE validation loss − our validation loss; positive values favor our method.

Our method reduces validation loss by 0.0012 and 0.0101 at 101810^{18} and 102010^{20} FLOPs, respectively, while its loss is 0.0020 higher at 101910^{19} FLOPs. The largest gain occurs at the longest horizon. This difference is particularly important for large-scale industrial pretraining: for a production training run with a fixed model architecture and expert count, even if the training horizon is extended to, for example, 50T tokens, the Joint MoE formula still predicts the same LR. It cannot respond to changes in training budget, creating an extrapolation risk that the LR may fall outside the near-optimal region for long training horizons. We will add these results and their scope to the revised manuscript.

 

Weakness 2 & Question 2: Downstream model performance.

Response: Thank you for the suggestion. Existing scaling-law and hyperparameter studies commonly use language-modeling loss for fitting or hyperparameter selection [1–5], because it directly corresponds to the pretraining objective and provides a continuous metric suitable for comparison and fitting. However, whether loss improvements translate into downstream gains still requires direct validation.

We evaluated four hyperparameter-selection approaches on public benchmarks using the same target model. All laws are refitted on the same development set, with the model, data, training budget, and other settings held identical. DeepSeek, StepFun, and our method use their respective predicted LR/BS settings; Joint MoE uses its predicted LR together with the BS predicted by our law.

Hyperparameter-selection lawHellaSwagARC-EasyARC-ChallengeAverage
DeepSeek Law [3]44.860.231.545.50
StepFun Law [4]45.259.532.245.63
Joint MoE [1]44.259.830.244.73
Ours45.460.532.045.97

All three benchmarks measure multiple-choice accuracy by selecting the answer with the lowest PPL given the question and context.

Our method achieves the highest average score of 45.97 and leads on two of the three tasks. Scores are subject to variability, and our method is not best on every task (ARC-Challenge: 32.0, versus 32.2 for StepFun). These results support higher average downstream performance in this configuration, but not superiority on every task or statistical significance. We will add the results and full protocol to the revised manuscript.

AuthorsOpenReview ↗

Response to Weaknesses and Questions (2/2)

Weakness 3 & Question 3: Reproducibility of data, fitting, and evaluation.

Response: Thank you for pointing out the insufficient detail. We provide additional clarification on data composition, validation-set construction, and the fitting protocol below.

  • Training data. The training data comes from an internal multilingual corpus, primarily in English and Chinese. Each experiment uses this corpus according to its training budget, with the following composition.

    Data typeWebCodeMathOthers
    Proportion46.0%25.0%6.0%23.0%

    This corpus is also used for our internal flagship model. Due to confidentiality requirements, we cannot release the data publicly, but we commit to disclosing further details of the data mixture, cleaning, and processing procedures in a subsequent technical report.

  • Validation set. The validation set contains 1B tokens independently held out at the document level from the same source corpus, with the same category proportions as the training set. The validation and training sets are deduplicated against each other, and validation documents are excluded from training. All experiments share this fixed validation set and evaluate on the entire set each time, using mean cross-entropy over valid tokens as the validation loss.

  • Fitting and evaluation. The development set contains only six active-parameter scales and A∈{1,1/4,1/16,1/32}A\in\{1,1/4,1/16,1/32\}. For each LR–BS loss surface, grid points whose loss is no more than 1.0011.001 times the observed minimum form the near-optimal set, followed by least-squares fitting in log space. LOAO holds out one activation ratio, while LONO holds out one active-parameter scale. Both split the data at the level of entire loss surfaces, and only the training fold is used to extract fitting targets and re-estimate coefficients. The final joint-extrapolation target is N=324MN=324\mathrm{M}, A=1/64A=1/64, D=159BD=159\mathrm{B}, and C=3×1020C=3\times10^{20}; its losses are excluded from functional-family selection and fitting, and all predictions are frozen before inspecting the target losses.

These details will be consolidated in the experimental setup and appendix of the revised manuscript.

 


References:

[1] Ludziejewski, Jan, et al. "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient." Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:41056–41073, 2025.

[2] Bjorck, Johan, et al. "Scaling Optimal LR Across Token Horizons." International Conference on Learning Representations, 2025.

[3] Bi, Xiao, et al. "DeepSeek LLM: Scaling Open-Source Language Models with Longtermism." arXiv preprint arXiv:2401.02954, 2024.

[4] Li, Houyi, et al. "Predictable Scale: Part I -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining." arXiv preprint arXiv:2503.04715, 2025.

[5] Kaplan, Jared, et al. "Scaling Laws for Neural Language Models." arXiv preprint arXiv:2001.08361, 2020.

Dear reviewer wWdS,

As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your support if we have fully resolved all outstanding issues. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.