- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies the joint tuning of peak learning rate and decay fraction using the functional scaling law (FSL) framework. It predicts a shared critical batch size separating pure decay with increasing peak LR from WSD with saturated peak LR and shorter decay. LLM pre-training experiments broadly support this structure and motivate a practical rule for predicting schedules across token budgets.
Critical strengths
- A simple, practical rule predicts near-optimal schedules across token budgets.
- Theory and experiments connect peak LR and decay duration through a shared critical batch size.
Critical weaknesses
- The shared-CBS formulation is novel, but its importance is not clear to me. The comparisons with pure decay and WSD with a fixed decay ratio do not show how much it improves over existing schedule-prediction methods.
- The paper is difficult to follow overall. The introduction refers to FSL results before explaining them, and the role of effective learning rate and norm control becomes clear only later. The distinction between theoretical results and empirical rules also needs clearer explanation.
Questions for authors
- Does K-scaling retain its functional form when other optimizer hyperparameters, such as momentum or weight decay, change, or is it specific to the configuration studied?
3 comments
Thank you for the additional analysis and experiments. They clarify the connection between theory and empirical rules and the optimizer dependence of K-scaling. I will raise my soundness. However, my concern about practical significance remains. The authors should discuss more concretely how K-scaling can help pre-training design, especially with AdamW or MuonW, where the quantitative scaling does not hold. While the results under norm control are useful, it remains unclear how practitioners should use these findings to choose better schedules in standard settings. The presentation also remains difficult to follow. The added appendices are helpful, but the overall text still needs a clearer introduction to FSL, effective learning rate, and norm control.
For these reasons, I will maintain my main rating.
Practical Value of K-Scaling: ELR-First Design and Guidance for Current Training
We thank the reviewer for this careful follow-up. We clarify two practical implications.
1. Predictable schedule design through ELR-based training.
Recent methods, including nGPT [1], Hyperball [2], and Training nGPT [3], demonstrate growing interest in explicit norm and update control. Complementing these methods, Liu et al. [4] show that matching effective learning rate (ELR) aligns loss trajectories across different nominal LRs and parameter norms, motivating ELR-first training: designing the ELR schedule directly and implementing it through norm control.
Beyond potential improvements in final loss, our work demonstrates an additional benefit of ELR-based training: predictable schedule transfer. Optimal peak LR and decay ratio follow a simple joint scaling structure with batch size and data budget across MuonH, Adam-P, and SGD-P. For MuonH, calibration on 1B/2B-token runs predicts both hyperparameters at 50B/100B tokens, achieving losses close to grid-search references without further tuning; see Figure 16 of the revised manuscript. These results support ELR-first design as a promising direction for more predictable and controllable pretraining.
2. Actionable guidance for current training practice with weight decay
Our theory predicts that smaller batches favor pure decay, while larger batches favor shorter-decay WSD. Figure 13 supports this qualitative trend for AdamW and MuonW. Practitioners can therefore use batch size to guide the search over decay duration, jointly tune peak LR, and reconsider fixed decay fractions when scaling batch size.
Quantitative transfer additionally requires accounting for LR–weight-decay coupling. Microsoft AI recently addressed this interaction by searching over and found that the optimal is more predictable, which depends primarily on training steps [5]. Under an approximate norm-equilibrium model, serves as a surrogate ELR scale, providing a connection between ELR-based schedule design and conventional weight-decay training.
Motivated by this connection, we are investigating joint calibration over , where is the weight-decay coefficient and is the decay ratio. This extension is ongoing, and its quantitative transfer accuracy remains to be established.
We will revise the manuscript to explain these implications and introduce FSL, ELR, and norm control more clearly. We believe training with a clean predictable hyperparameter scaling is promising in large-scale training. Please let us known if this still does not fully clarify your concern.
References
[1] Loshchilov et al. nGPT: Normalized Transformer with Representation Learning on the Hypersphere. ICLR, 2025.
[2] Wen et al. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization, 2026.
[3] Loshchilov and Ginsburg. Training nGPT, 2026.
[4] Liu et al. Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining, 2026.
[5] Microsoft AI. Frontier Pre-training at Microsoft AI, 2026.
Response to Reviewer L8U9
We thank the reviewer for the constructive feedback. We clarify the scientific contribution below and add optimizer-dependent theory and cross-optimizer validation in Appendices G and H.1.
Our focus is how optimal peak LR and decay ratio jointly depend on batch size and token budget, and whether K-scaling captures this dependence. The comparisons address three questions:
We clarify the distinction through the additional analysis in Appendix G and experiments in Appendix H:
Theoretical predictions. Building on SGD-based FSL [1] and its recent SignSGD extension [2], Appendix G derives optimizer-dependent K-scaling profiles under explicit assumptions, including the peak-LR ceiling. These yield linear and square-root subcritical LR scaling, respectively.
Empirical validation. Appendix H.1 tests these predictions with SGD-P and Adam-P and compares norm-controlled training with AdamW and MuonW, examining the role of effective learning-rate parameterization.
Practical approximation. The clipped power law approximates the theoretical implicit decay profile, with its exponent calibrated empirically. Appendix G derives the power-law tail motivating this approximation, while Appendix H.4 examines profile selection using only small-budget data.
We add experiments across optimizer configurations to examine which aspects of K-scaling persist.
Shared structure across configurations. SGD-P (momentum 0.98), Adam-P (β1=0.9, β2=0.95), and MuonH (momentum 0.95) all exhibit K-scaling (Figures 11 and 12). Here, “-P” denotes projecting updated weights onto fixed-radius spheres. The shared scheduling structure persists, while the subcritical LR scaling is approximately linear for SGD-P and square-root for Adam-P and MuonH. The CBS–budget relation and decay-profile parameters are calibrated separately for each configuration.
Behavior with weight decay. AdamW and MuonW retain the qualitative transition from near-pure decay to shorter decay (Figure 13), but their nominal-LR profiles do not quantitatively follow K-scaling. This is consistent with evolving weight norms reshaping the effective schedule.
References
[1] Li et al. Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS, 2025. arXiv:2509.19189.
[2] Wang et al. When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective. NeurIPS, 2026. arXiv:2611.16560.