- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies hyperparameter scaling laws in MoE pretraining, as a function of activation ratio of the experts. The authors report that they conducted large scale experiments, comparing different parametric scaling law forms. The main contribution is a the sparsity-dependent correction to existing hyperparameter scaling laws.
Critical strengths
- The paper is very-well written, and easy to follow (including the appendix).
- The experimental effort is substantial (authors reported conducting more than 1800 pretraining runs). The joint LR-BS sweeps across size scales and several sparsity levels definitely requires significant compute (grid search cost grows exponentially with the number of hyperparameters).
- The conclusions from the empirical results are well stated as well.
Critical weaknesses
While the experimental results cover many key components of scaling, there are several issues that limit the significance of the contributions.
-
Significance of sparsity scaling: Figure 2 shows relatively flat minima with modest LR shifts across sparsity levels. The results in Fig 4a also show an exponent of LR scaling laws with sparsity is very small (roughly 0.13), which is basically indistinguishable from using a constant LR at the practical sparsity levels. For instance, a 64x change in activation ratio changes the predicted LR by only a factor of 1.76, which is very small given that optimal LR is usually within a flat region of the loss. The same can be said for BS. A small exponent does not invalidate the scaling law necessarily, but the paper should at least demonstrate meaningful excess validation loss when LR is held fixed across sparsity.
-
Theoretical contribution: Appendix C assumes a BS loss model and that . These assumptions are not well-justified and seem like they’re introduced to fit an observation, rather than provide an explanation to the observed laws.
Questions for authors
Please see weaknesses.
4 comments
Response
Thank you for the comments and additional experiments.
I remain however unconvinced by the results:
-
Significance of sparsity scaling: as I mentioned previously the exponents are extremely small. The reported excess risk results are also very small (unclear if it's within seed noise margin). The authors claim that production level training adjust the LRs with sparsity. While this is true, the evidence in this paper does not apply to production scale (the exponents might not extrapolate to that scale), so claiming that production scale results support the findings in this paper is a big stretch.
-
Theoretical contributions: I understand that the paper is primarily empirical. That does not justify using any convenient assumptions to explain the observed patterns. In that case, it might be better not to have any theory than have a set of unjustified and loose assumptions.
Response
1. Significance of sparsity scaling.
-
Thank you for highlighting this point. We acknowledge that the current results are based on single runs. Nevertheless, the sparsity correction reduces validation loss at all three model scales, providing consistent evidence of its benefits. Given the high cost of pretraining and our limited resources, we are currently unable to conduct sufficient multi-seed runs.
-
We would also like to clarify that the production-model examples were cited only to provide a practical reference for the magnitude of LR adjustments , not to validate our laws or the accuracy of extrapolating their exponents to production scale. We will revise the wording to clearly distinguish this reference from the conclusions supported by our experiments.
2. Theoretical contributions.
Thank you for your suggestion. We will remove the simplified model in Appendix C and the mechanistic explanations in the main text that rely on it, leaving the development of an independently justified theoretical explanation to future work. We appreciate your rigorous review, which helps us define our contribution more clearly.
Thank you again for your careful review and constructive feedback.
Dear reviewer sTgZ,
As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your support if we have fully resolved all outstanding issues. Thank you for your time and consideration.
Response to Weaknesses
Response: Thank you for raising this point. We agree that the key question is whether holding LR fixed across sparsity levels incurs excess validation loss, rather than whether the optimal LR shifts. To address this directly, we added three controlled comparisons: at each training scale, we use the LR predicted by our law at A=1 and hold it fixed when transferring to A=1/64, then compare it with the sparsity-aware prediction for the target configuration. Within each comparison, the model, compute budget, BS, and other training settings are identical, with a sequence length of 8192. Validation loss reduction = validation loss with the shared LR − validation loss with the sparsity-aware LR.
The sparsity correction achieves lower validation loss at all three model scales, with absolute reductions of 0.0128, 0.0050, and 0.0114, respectively. These controlled comparisons show that an approximately 1.76-fold LR adjustment yields actual loss improvements in all three configurations. Thus, flat minima do not imply that holding LR fixed across sparsity incurs no excess validation loss in these configurations.
We therefore disagree that the sparsity correction is “basically indistinguishable from using a constant LR” solely because its exponent is small. Modest multiplicative LR adjustments are also consistent with industrial pretraining practice: DeepSeek-V4 [1] Flash (284B total parameters, 13B activated) and Pro (1.6T total parameters, 49B activated) are trained on 32T and 33T tokens, respectively, with peak LRs of 2.7×10−4 and 2.0×10−4. Their total parameter counts differ by approximately 5.6-fold, whereas their LRs differ by only 1.35-fold. The exponent describes the magnitude of the change in optimal LR; it does not directly determine the corresponding loss difference. The DeepSeek example provides an industrial reference for the magnitude of LR adjustments, while the table above provides direct loss evidence for our sparsity correction.
Therefore, by correcting the prediction bias caused by ignoring sparsity, our work improves the reliability of transfer across sparsity levels and reduces the risk of hyperparameters falling outside the near-optimal region. This is particularly valuable for costly, large-scale pretraining.
Response: Thank you for pointing out the need to clarify the theoretical scope. Our methodology is empirical: we first identify trends in hyperparameters across training scales and sparsity levels through experiments, and then test the proposed laws through comparisons of candidate functional families, grouped cross-validation, and held-out prediction. Appendix C is intended to provide a simplified mechanistic model for understanding these trends, rather than a first-principles derivation of the empirical scaling laws.
We agree that the BS loss model and η∗∝(AB∗)ρ in Appendix C are modeling assumptions, rather than independently established explanations of the observed laws. Under balanced routing, each expert receives an average of AB tokens, which provides a mechanistic motivation for the effective expert batch size, but is insufficient to directly derive the assumed loss function or the power-law response of the optimal LR. Thus, the signs of the exponents obtained in the appendix are conditional deductions under these assumptions, rather than an independent proof of the empirical laws.
We will explicitly present Appendix C as “a simplified model for understanding sparsity trends,” distinguish assumptions from deductions, and remove wording that suggests an independent derivation of the empirical laws. These assumptions were not used to select the functional form or fit the empirical laws; they were introduced only to provide a simplified interpretation of the observed trends. The empirical fits and held-out predictions do not depend on this model; our core contribution is a set of empirically tested hyperparameter scaling relationships.
References:
[1] DeepSeek-AI. "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence." arXiv preprint arXiv:2606.19348 (2026).