acceptodds
Under review as a conference paper at ICLR 2027

K-Scaling: Critical Batch Size Governs Learning Rate Scheduling Across Token Budgets in LLM Pretraining

13 views

Abstract

We study learning rate (LR) scheduling in large language model pretraining by jointly tuning peak LR and decay ratio, the fraction of post-warmup training allocated to terminal decay. Using the functional scaling law framework with an admissibility constraint on peak LR, we establish : when expressed through the normalized batch size (BS) , both optimal hyperparameters follow budget-independent scaling profiles, where is the token budget and is a shared critical batch size (CBS). The CBS marks a from pure decay with increasing peak LR in the regime () to warmup–stable–decay (WSD) with saturated peak LR and decreasing decay ratio in the regime (). Experiments on dense and MoE models spanning 0.1B–1B parameters, trained with MuonH, confirm approximate K-scaling collapse and support a practical four-parameter model for jointly predicting peak LR and decay ratio. Fitted on small token budgets, the model extrapolates to budgets up to the largest fitting budget, yielding schedules with final losses close to grid-search optima without further tuning. The theory also explains why longer training favors larger decay ratios at fixed-BS training, helping reconcile differing schedule preferences in prior studies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling collapse, critical batch size, learning rate scheduling, phase transition, scaling laws, hyperparameter transfer

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_L8U9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies the joint tuning of peak learning rate and decay fraction using the functional scaling law (FSL) framework. It predicts a shared critical batch size separating pure decay with increasing peak LR from WSD with saturated peak LR and shorter decay. LLM pre-training experiments broadly support this structure and motivate a practical rule for predicting schedules across token budgets.

Critical strengths
  • A simple, practical rule predicts near-optimal schedules across token budgets.
  • Theory and experiments connect peak LR and decay duration through a shared critical batch size.
Critical weaknesses
  • The shared-CBS formulation is novel, but its importance is not clear to me. The comparisons with pure decay and WSD with a fixed decay ratio do not show how much it improves over existing schedule-prediction methods.
  • The paper is difficult to follow overall. The introduction refers to FSL results before explaining them, and the role of effective learning rate and norm control becomes clear only later. The distinction between theoretical results and empirical rules also needs clearer explanation.
Questions for authors
  • Does K-scaling retain its functional form when other optimizer hyperparameters, such as momentum or weight decay, change, or is it specific to the configuration studied?
3 comments
AuthorsOpenReview ↗

Response to Reviewer L8U9

We thank the reviewer for the constructive feedback. We clarify the scientific contribution below and add optimizer-dependent theory and cross-optimizer validation in Appendices G and H.1.

[W1-1] The shared-CBS formulation is novel, but its importance is not clear to me.

  • Mechanistic significance. Our FSL analysis predicts a shared boundary between two responses to increasing batch size: raising peak LR while retaining pure decay, then shortening decay once the LR ceiling is reached. The shared CBS makes this coupling an explicit, testable prediction.
  • Predictive value. A common normalized batch size, x=B/Bc(D)x=B/B_c(D), describes how both transitions shift with token budget and provides a compact rule for jointly predicting peak LR and decay ratio.
  • Shared versus separate CBSs. Figure 9 tests the precision of this description. Separate CBSs improve fitting accuracy and peak-LR extrapolation, but slightly worsen decay-ratio extrapolation. We therefore interpret the shared CBS as an approximate empirical coupling motivated by theory, with separate CBSs providing a more flexible refinement.

[W1-2] The comparisons with pure decay and WSD with a fixed decay ratio do not show how much it improves over existing schedule-prediction methods.

Our focus is how optimal peak LR and decay ratio jointly depend on batch size and token budget, and whether K-scaling captures this dependence. The comparisons address three questions:

  • Does decay-ratio selection matter? Pure decay and fixed-ratio WSD have separately tuned peak LRs in our joint-transfer experiments. Their performance gaps demonstrate the benefit of adapting decay ratio even after peak-LR tuning.
  • Can K-scaling guide this selection? Comparison with joint grid-search optima evaluates whether small-budget calibration predicts effective schedules at larger budgets.
  • Does the trend depend on decay shape? The batch-dependent preference for decay duration persists across different decay shapes (Figure 8), supporting the broader relevance of decay-ratio selection.

[W2] The distinction between theoretical results and empirical rules also needs clearer explanation.

We clarify the distinction through the additional analysis in Appendix G and experiments in Appendix H:

  • Theoretical predictions. Building on SGD-based FSL [1] and its recent SignSGD extension [2], Appendix G derives optimizer-dependent K-scaling profiles under explicit assumptions, including the peak-LR ceiling. These yield linear and square-root subcritical LR scaling, respectively.

  • Empirical validation. Appendix H.1 tests these predictions with SGD-P and Adam-P and compares norm-controlled training with AdamW and MuonW, examining the role of effective learning-rate parameterization.

  • Practical approximation. The clipped power law approximates the theoretical implicit decay profile, with its exponent calibrated empirically. Appendix G derives the power-law tail motivating this approximation, while Appendix H.4 examines profile selection using only small-budget data.

[Q1] Does K-scaling retain its functional form when other optimizer hyperparameters, such as momentum or weight decay, change, or is it specific to the configuration studied?

We add experiments across optimizer configurations to examine which aspects of K-scaling persist.

  • Shared structure across configurations. SGD-P (momentum 0.98), Adam-P (β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95), and MuonH (momentum 0.95) all exhibit K-scaling (Figures 11 and 12). Here, “-P” denotes projecting updated weights onto fixed-radius spheres. The shared scheduling structure persists, while the subcritical LR scaling is approximately linear for SGD-P and square-root for Adam-P and MuonH. The CBS–budget relation and decay-profile parameters are calibrated separately for each configuration.

  • Behavior with weight decay. AdamW and MuonW retain the qualitative transition from near-pure decay to shorter decay (Figure 13), but their nominal-LR profiles do not quantitatively follow K-scaling. This is consistent with evolving weight norms reshaping the effective schedule.

References

[1] Li et al. Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS, 2025. arXiv:2509.19189.

[2] Wang et al. When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective. NeurIPS, 2026. arXiv:2611.16560.

Thank you for the additional analysis and experiments. They clarify the connection between theory and empirical rules and the optimizer dependence of K-scaling. I will raise my soundness. However, my concern about practical significance remains. The authors should discuss more concretely how K-scaling can help pre-training design, especially with AdamW or MuonW, where the quantitative scaling does not hold. While the results under norm control are useful, it remains unclear how practitioners should use these findings to choose better schedules in standard settings. The presentation also remains difficult to follow. The added appendices are helpful, but the overall text still needs a clearer introduction to FSL, effective learning rate, and norm control.

For these reasons, I will maintain my main rating.

AuthorsOpenReview ↗

Practical Value of K-Scaling: ELR-First Design and Guidance for Current Training

We thank the reviewer for this careful follow-up. We clarify two practical implications.

1. Predictable schedule design through ELR-based training.

Recent methods, including nGPT [1], Hyperball [2], and Training nGPT [3], demonstrate growing interest in explicit norm and update control. Complementing these methods, Liu et al. [4] show that matching effective learning rate (ELR) aligns loss trajectories across different nominal LRs and parameter norms, motivating ELR-first training: designing the ELR schedule directly and implementing it through norm control.

Beyond potential improvements in final loss, our work demonstrates an additional benefit of ELR-based training: predictable schedule transfer. Optimal peak LR and decay ratio follow a simple joint scaling structure with batch size and data budget across MuonH, Adam-P, and SGD-P. For MuonH, calibration on 1B/2B-token runs predicts both hyperparameters at 50B/100B tokens, achieving losses close to grid-search references without further tuning; see Figure 16 of the revised manuscript. These results support ELR-first design as a promising direction for more predictable and controllable pretraining.

2. Actionable guidance for current training practice with weight decay

Our theory predicts that smaller batches favor pure decay, while larger batches favor shorter-decay WSD. Figure 13 supports this qualitative trend for AdamW and MuonW. Practitioners can therefore use batch size to guide the search over decay duration, jointly tune peak LR, and reconsider fixed decay fractions when scaling batch size.

Quantitative transfer additionally requires accounting for LR–weight-decay coupling. Microsoft AI recently addressed this interaction by searching over (ηpeak,ληpeak)(\eta_{\mathrm{peak}}, \sqrt{\lambda\eta_{\mathrm{peak}}}) and found that the optimal ληpeak\sqrt{\lambda\eta_{\mathrm{peak}}} is more predictable, which depends primarily on training steps [5]. Under an approximate norm-equilibrium model, ληpeak\sqrt{\lambda\eta_{\mathrm{peak}}} serves as a surrogate ELR scale, providing a connection between ELR-based schedule design and conventional weight-decay training.

Motivated by this connection, we are investigating joint calibration over (ηpeak,ληpeak,r)(\eta_{\mathrm{peak}}, \sqrt{\lambda\eta_{\mathrm{peak}}}, r), where λ\lambda is the weight-decay coefficient and rr is the decay ratio. This extension is ongoing, and its quantitative transfer accuracy remains to be established.

We will revise the manuscript to explain these implications and introduce FSL, ELR, and norm control more clearly. We believe training with a clean predictable hyperparameter scaling is promising in large-scale training. Please let us known if this still does not fully clarify your concern.

References

[1] Loshchilov et al. nGPT: Normalized Transformer with Representation Learning on the Hypersphere. ICLR, 2025.

[2] Wen et al. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization, 2026.

[3] Loshchilov and Ginsburg. Training nGPT, 2026.

[4] Liu et al. Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining, 2026.

[5] Microsoft AI. Frontier Pre-training at Microsoft AI, 2026.

Reviewer_oxEAOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how batch size and token budget jointly determine the optimal peak learning rate and decay ratio in LLM pretraining. Using functional scaling laws with a peak-learning-rate constraint, it derives a critical batch size that separates pure decay with increasing peak learning rate from warmup–stable–decay with saturated peak learning rate and shorter decay. The authors then introduce an empirically calibrated, four-parameter K-scaling model based on normalized batch size, using square-root learning-rate scaling and a clipped power-law decay ratio. Experiments cover dense and MoE models trained on FineWeb-Edu with MuonH. Parameters fitted on 1B- and 2B-token runs predict schedules at budgets up to 20B tokens, achieving validation losses close to evaluated grid-search references and improving over independently tuned fixed-ratio baselines. Joint hyperparameter transfer is tested on the smaller models, whereas the 1B-model study transfers only the decay ratio at a fixed peak learning rate. The framework also interprets batch-dependent schedule preferences and the tendency toward longer decay when training longer at fixed batch size.

Critical strengths
  1. Treating peak learning rate and decay duration as coupled variables reveals interactions that fixed-schedule comparisons can miss. The normalized batch-size perspective provides a coherent explanation for changing schedule preferences across training budgets and batch sizes.

  2. The intrinsic-time formulation yields a clear mechanism connecting learning-rate saturation to decay shortening. Appendices B–E provide derivations, uniqueness arguments, and convergence from the full FSL objective, while explicitly qualifying finite-budget and asymptotic claims.

  3. Calibration on 1B/2B-token runs produces schedules whose reported 20B-token losses are within approximately 0.001 of the searched references. Independently tuning baseline peak learning rates for the smaller models strengthens the comparison against pure decay and fixed-ratio WSD (Figure 5).

  4. Dense and MoE experiments, alternative decay shapes, and comparisons between shared and separate critical batch scales probe several aspects of the proposed structure. The manuscript also acknowledges quantitative discrepancies between theory and experiments rather than presenting the empirical rule as an exact theoretical consequence.

Critical weaknesses
  1. Theorem 1 predicts linear subcritical learning-rate scaling, whereas the empirical model requires square-root scaling. The theoretical decay profile also fits worse than the empirical alternative: 10.45% versus 5.00% MAPE (Figure 3). Moreover, the assumed budget- and batch-size-independent ceiling is not independently established as an instability boundary. An optimizer-specific derivation or direct measurements of the proposed constraint would strengthen the mechanistic interpretation.

  2. Section 5 selects the square-root and clipped-power profiles using all six Llama-120M token budgets, including budgets subsequently designated as held out. Section 6 holds out these budgets when fitting numerical parameters, but not when selecting the functional form. This still demonstrates parameter extrapolation, but a stronger validation would select both profiles and parameters using only small-budget data and evaluate on untouched budgets.

  3. Separate critical scales improve both all-budget fits and held-out peak-learning-rate prediction, reducing the latter MAPE from 8.54% to 6.22%, although decay-ratio prediction worsens (Figure 9). These results support a useful parsimonious approximation, but do not decisively establish a common empirical transition. Uncertainty estimates and a complexity-aware model comparison would clarify the strength of this central claim.

  4. All experiments use FineWeb-Edu and norm-controlled MuonH. Full joint transfer is validated only for the smaller dense and MoE models; the 1B-model experiments fix peak learning rate and transfer decay ratio alone. Additional datasets, optimizers, and joint-transfer experiments at larger model scales are needed to establish broader applicability.

  5. The reported 47%/73% token savings are inferred from fitted curves at three budgets along one fixed-update, increasing-batch-size path for Llama-120M (Figure 6). Their uncertainty and direct matching-loss validation are not provided. Furthermore, total calibration cost is unreported despite substantial small-budget searches. Reporting these costs, accounting for checkpoint reuse, would clarify whether the method delivers net tuning savings.

Questions for authors

see weaknesses

5 comments

Overall, I like this paper, and I hope the authors can try to address the concerns raised by the other reviewers.

AuthorsOpenReview ↗

Response to Reviewer oxEA (1/3)

We thank the reviewer for the strong support. We would like to address the concerns here.

[W1-1] Theorem 1 predicts linear subcritical learning-rate scaling, whereas the empirical model requires square-root scaling.

We thank the reviewer for highlighting this discrepancy. Appendix G provides an optimizer-dependent theoretical extension, supported by the additional experiments in Appendix H.1.

  • Optimizer-dependent theory. The original theorem uses SGD-based FSL [1]. Building on recent SignSGD-FSL [2], we extend the analysis to account for the different intrinsic times:

    tSGD(z)=∫0zη(u) du,tSignSGD(z)=B∫0zη(u) du.t_{\mathrm{SGD}}(z)=\int_0^z\eta(u)\,du,\qquad t_{\mathrm{SignSGD}}(z)=\sqrt{B}\int_0^z\eta(u)\,du.

    Under the corresponding FSL assumptions and our fixed peak-LR ceiling, these formulations yield linear and square-root subcritical LR scaling, respectively. Muon’s matrix-sign updates motivate the connection to SignSGD, whose scaling prediction agrees with our empirical square-root branch.

  • Quantitative validation. We add joint peak-LR and decay-ratio searches for SGD-P and Adam-P, variants that project updated weights onto fixed-radius spheres, at 1B, 2B, and 5B tokens. Across these budgets, the subcritical peak LR approximately follows ηpeak⋆∝B\eta^\star_{\mathrm{peak}}\propto B for SGD-P and ηpeak⋆∝B\eta^\star_{\mathrm{peak}}\propto\sqrt{B} for Adam-P (Figure 10), consistent with the respective theoretical predictions.

[W1-2] The theoretical decay profile also fits worse than the empirical alternative: 10.45% versus 5.00% MAPE (Figure 3).

We thank the reviewer for highlighting this comparison. We report an additional controlled comparison below and explain the theoretical motivation for the power-law approximation.

  • Validation in linear regression. In SGD linear regression, we compare the FSL-implicit and power-law profiles fitted to the same jointly optimized schedules:

    SettingFSL implicit MAPEPower-law MAPE
    SGD linear regression0.584%3.021%
    MuonH LLM experiments (Figure 3)10.45%5.00%

    The implicit profile fits more accurately in this controlled setting, providing quantitative support for the theoretical decay profile.

  • Role of the power-law approximation. Appendices C and G derive the implicit profile’s power-law tail, motivating the clipped power law as an explicit approximation to the full profile, with its exponent calibrated empirically.

[W1-3] The assumed configuration-independent learning-rate ceiling is not independently established as an instability boundary.

We thank the reviewer for this suggestion. We clarify the motivation for the ceiling and add supporting evidence.

  • Stability motivation. For GD on a quadratic objective, the stability threshold is 2/λmax⁡(H)2/\lambda_{\max}(H). Related work establishes the relevance of dynamical stability to SGD and neural-network training [3,4], motivating a stability-based LR constraint.

  • Controlled validation. In SGD linear regression, we independently derive a mean-square stability boundary that approaches a constant at large batch sizes. These experiments exhibit sharply increasing expected excess risk beyond this boundary (Figure 17a), supporting an approximately fixed stability limit in this regime.

  • LLM evidence. For MuonH, searches across four large batch sizes identify a common preferred peak LR near 0.015. Higher peak LRs worsen final validation loss even after reoptimizing the decay ratio (Figure 17b). This supports a common effective LR scale, which our fixed peak-LR ceiling approximates within the studied configuration.

[W2] This still demonstrates parameter extrapolation, but a stronger validation would select both profiles and parameters using only small-budget data and evaluate on untouched budgets.

We thank the reviewer for this suggestion. We add a profile-selection analysis using only 1B/2B-token data (Figure 18). The fitted LR exponents support square-root scaling, while the candidate decay profiles achieve comparable accuracy, supporting the clipped power law as a simple explicit choice. This analysis shows that the chosen forms are supported by small-budget data alone and complements the larger-budget extrapolation results in Figures 4–5.

AuthorsOpenReview ↗

Response to Reviewer oxEA (2/3)

[W3] Separate critical scales improve both all-budget fits and held-out peak-learning-rate prediction, reducing the latter MAPE from 8.54% to 6.22%, although decay-ratio prediction worsens (Figure 9).

We thank the reviewer for highlighting this distinction between theoretical coupling and empirical precision.

  • Why share the critical scale? Under the stated FSL assumptions and fixed peak-LR ceiling, peak-LR saturation and departure from pure decay occur at the same boundary. Below it, increasing batch size is compensated by increasing peak LR; above it, the optimum shortens decay. A shared CBS expresses this coupling through one normalized batch size, x=B/Bc(D)x=B/B_c(D).
  • What does Figure 9 establish? Allowing separate CBSs increases the practical model from four to six parameters. It improves all-budget fits and held-out peak-LR MAPE (8.54% → 6.22%), while held-out decay-ratio MAPE worsens (5.32% → 6.13%). Thus, the additional flexibility offers a tradeoff in extrapolation accuracy.

We interpret the LLM results as supporting approximate coupling and retain the shared CBS as a compact, theoretically motivated description, with separate CBSs providing a more flexible empirical refinement.

[W4] Additional datasets, optimizers, and joint-transfer experiments at larger model scales are needed to establish broader applicability.

We thank the reviewer for this suggestion. The revised manuscript expands validation across optimizers and substantially longer training budgets.

  • Quantitative K-scaling across norm-controlled optimizers. New joint searches with SGD-P and Adam-P, which project updated weights onto fixed-radius spheres, both exhibit approximate K-scaling alongside MuonH (Figures 11 and 12). SGD-P shows approximately linear subcritical LR scaling, while Adam-P and MuonH show approximately square-root scaling. The critical scale and profile parameters are calibrated separately for each configuration.
  • Qualitative structure beyond norm control. Independent joint searches with AdamW and MuonW also show a transition from near-pure decay to shorter decay as batch size increases (Figure 13). Their nominal-LR profiles exhibit weaker quantitative agreement, consistent with evolving parameter norms reshaping the effective schedule.
  • Joint transfer to longer budgets. We extend MuonH training to 50B/100B tokens, approximately 400/800 tokens per parameter, and observe approximate K-scaling collapse across 0.5–100B tokens (Figures 14 and 15). Calibrated on 1B/2B-token runs, K-scaling jointly predicts peak LR and decay ratio at both longer budgets without refitting, using batch size 32,768. These schedules outperform separately peak-LR-tuned vanilla WSD and pure decay and closely match joint grid-search results (Figure 16).

[W5-1] The reported 47%/73% token savings are inferred from fitted curves at three budgets along one fixed-update, increasing-batch-size path for Llama-120M (Figure 6).

Figure 6 fixes the number of optimizer updates NN and varies batch size BB, with token budget D=NBD=NB. Directly identifying the batch size that matches a target final loss would require additional complete training runs across candidate batch sizes. We therefore fit L(B)L(B) and invert it to estimate the budget required to match each baseline’s fitted loss at 20B tokens, obtaining the reported 47%/73% estimated token savings.

[W5-2] Total calibration cost is unreported despite substantial small-budget searches. Reporting these costs, accounting for checkpoint reuse, would clarify whether the method delivers net tuning savings.

We report calibration and search costs below, accounting for checkpoint reuse.

  • Calibration cost and checkpoint reuse. The 1B/2B-token searches across 19 (D,B)(D,B) configurations cost 11,871.1 GPU-hours. Each shared training prefix is counted once, and each decay branch is charged only for its executed continuation.
  • Net tuning savings. The calibration is reused across target configurations. Direct-search overhead across the evaluated 5B/10B/20B configurations totals 44,179.6 GPU-hours, excluding one selected full training run per configuration. Charging the full calibration cost once yields 32,308.5 GPU-hours of net tuning savings (73.1%) relative to these executed searches.
Reviewer_8vLuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

It is an open question whether LLM training benefits more from LR decay or WSD. This paper suggests that it depends on both batch size and token budget, in addition to LR schedule itself. It proposes theoretical analysis based on the functional scaling law and jointly optimize peak learning rate and decay ratio, and proposes a shared critical batch size. The eval setup is small models under 1B, with up to 20B token budget. It finds that predicted schedules approximately match the searched references and outperform fixed-ratio baselines.3

Findings:

  • There is a critical batch size. Smaller batches favor pure decay, while larger batches favor WSD.
  • Small experiments can transfer to larger scales.
Critical strengths
  • Selecting an LR scheduling given BS and token budget is an important issue.

  • It provides a clear theoretical mechanism of decay vs WSD. This is interpretable despite the quantitative theory–experiment gaps.

Critical weaknesses
  • It has limited validation at larger model scales and data budgets, especially for a scaling law paper.

  • It proposes better data efficiency, but on a larger scale training, data is usually saturated and models are overtrained. The practical benefit is limited.

Questions for authors

Overall I like this paper.

  • I hope this work can clarify whether it scales. It doesn't have to be larger models. Small models with like 100B or more token budget may be useful.
  • Quantify the cost savings with wall-clock time if the benefit is training efficiency.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8vLu

We thank the reviewer for the helpful suggestions. Appendix H.2 adds validation at 50B and 100B tokens, and we report wall-clock estimates below.

[W1, W2 & Q1] Scalability and practical benefits at larger token budgets.

We appreciate the reviewer’s suggestion to evaluate longer training horizons. For context, our original 20B-token experiments already correspond to approximately 160 tokens per parameter for Llama-120M, roughly eight times the commonly used Chinchilla reference ratio [1] and comparable to Llama-7B’s training ratio [2]. Following the reviewer’s suggestion, we further extend training to 50B and 100B tokens, corresponding to approximately 400 and 800 tokens per parameter.

  • K-scaling at longer budgets. The expanded joint-search results exhibit approximate K-scaling collapse across 0.5–100B tokens (Figures 14 and 15), supporting the shared normalized-batch structure at substantially longer horizons.
  • Practical benefits at high TPP. K-scaling calibrated on 1B/2B-token runs jointly predicts peak LR and decay ratio at 50B/100B tokens without refitting, both at batch size 32,768 (Figure 16). The predicted schedules outperform vanilla WSD and pure decay, with each baseline’s peak LR separately tuned, and closely match joint grid-search results.

These results show that scheduling remains important at high TPP: the preferred decay ratio depends on batch size relative to the CBS, and K-scaling remains predictive at long training horizons.

[Q2] Quantify the cost savings with wall-clock time if the benefit is training efficiency.

We thank the reviewer for this suggestion. For the token-efficiency experiments in Figure 6, we report wall-clock estimates from training logs below. On the same eight-GPU setup, K-scaling achieves comparable or lower final validation loss with 51.3% less training time than vanilla WSD and 73.8% less than pure decay:

ComparisonK-scaling: tokens / loss / timeBaseline: tokens / loss / timeTime savings
vs. Vanilla WSD10B / 3.2708 / 3.60 h20B / 3.2709 / 7.39 h51.3%
vs. Pure decay5B / 3.2922 / 2.02 h20B / 3.2933 / 7.71 h73.8%

References

[1] Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS, 2022. arXiv:2203.15556.

[2] Touvron et al. LLaMA: Open and Efficient Foundation Language Models. 2023. arXiv:2302.13971.

Thanks for the extra experiments. They do improve the presentation and soundness for me. After reading other reviewers' comments, I'm leaning toward reviewer L8U9 and will maintain my main score.

Reviewer_Lp8ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the joint selection of peak learning rate and terminal decay ratio in LLM pretraining. Under the functional scaling law framework and a fixed peak-learning-rate constraint, it derives a shared critical batch size separating two regimes: pure decay with increasing peak learning rate, and warmup–stable–decay with saturated peak learning rate and decreasing decay ratio. The authors then introduce an empirically calibrated four-parameter model and evaluate its ability to transfer schedules across token budgets.

The problem is practically relevant, and the experiments reveal interesting batch-dependent schedule preferences. However, the theoretical contribution is best understood as an application and extension of the existing FSL framework to joint schedule optimization across batch sizes and token budgets. While FSL has theoretical guarantees in stylized regression models and growing empirical support, its quantitative applicability to the MuonH setting studied here requires further validation.

Critical strengths
  1. Relevant problem and sensible formulation. Jointly tuning peak learning rate and decay ratio addresses an important limitation of comparisons between schedules with fixed decay fractions. Independently tuning baseline peak learning rates makes the main comparisons more informative.

  2. Useful empirical observations. The reversal of schedule preferences with batch size is interesting. The experiments cover both dense and MoE architectures.

Critical weaknesses
  1. The quantitative connection between FSL theory and MuonH training remains insufficiently established. FSL has theoretical guarantees in stylized regression models and existing empirical support. However, the assumptions needed for the present analysis are not directly validated in the MuonH setting. Theorem 1 predicts linear growth of the optimal peak learning rate below the critical batch size, whereas the experiments support approximately square-root growth. The theoretical decay-ratio profile is also replaced by an empirical power law. Consequently, the experiments support the qualitative structure more strongly than the quantitative theoretical predictions. The manuscript should more clearly distinguish these contributions.

  2. Optimizer dependence and learning-rate parameterization. The empirical results rely exclusively on MuonH, whose norm control makes the nominal learning rate proportional to the relative update magnitude. This relationship does not generally hold for standard Muon or AdamW, whose update transformations give nominal learning rates different operational meanings. While the paper explicitly holds the optimizer configuration fixed, it remains unclear whether the observed shared-scale structure extends beyond this particular normalization scheme. The relevant question is whether the structural predictions persist after separate calibration for another optimizer, rather than whether the same numerical learning rates transfer directly. Experiments with standard Muon or AdamW, together with measurements of layerwise relative update magnitudes, would help distinguish a broader scheduling principle from behavior specific to MuonH.

Questions for authors

n/a

2 comments
AuthorsOpenReview ↗

Response to Reviewer Lp8R

We thank the reviewer for the thoughtful comments on the connection between FSL theory and LLM training. We add an optimizer-dependent theoretical analysis in Appendix G and additional experimental validation in Appendix H to strengthen this connection.

[W1] The quantitative connection between FSL theory and MuonH training remains insufficiently established.

  • Optimizer-dependent K-scaling. The original theorem (Thm. 1) is derived from SGD-based FSL [1]. Recent work extends FSL to SignSGD [2]. Both formulations can be expressed as

    L[η]=S(T)+∫0TK(T−t)nB(t) dt,\mathcal{L}[\eta]=S(T)+\int_0^T K(T-t)n_B(t)\,dt,

    tB(z)=Bκ∫0zη(u) du,nB(t)=Bκ−1η(z(t)),T=tB(N),t_B(z)=B^\kappa\int_0^z\eta(u)\,du,\qquad n_B(t)=B^{\kappa-1}\eta(z(t)),\qquad T=t_B(N),

    where κ=0\kappa=0 for SGD and κ=1/2\kappa=1/2 for SignSGD. The signal-learning function S(t)∼t−sS(t)\sim t^{-s} and noise-forgetting kernel K(t)∼t−νK(t)\sim t^{-\nu} have optimizer-dependent exponents ss and ν\nu.

    Under the stated FSL assumptions and fixed peak-LR ceiling, Theorem 2 in Appendix G yields

    ηpeak⋆=ηcritmin⁡(1,x1−κ),r⋆=min⁡(1,R(x)),x=BBc(D),\eta^\star_{\mathrm{peak}}=\eta_{\mathrm{crit}}\min(1,x^{1-\kappa}),\qquad r^\star=\min(1,R(x)),\qquad x=\frac{B}{B_c(D)},

    where RR is the optimizer-dependent implicit decay profile. Both profiles transition at the same critical batch size, while the subcritical LR scaling is linear for SGD and square-root for SignSGD. The latter provides a theoretical motivation for the square-root behavior observed with Muon, given its matrix-sign updates.

  • Validation across optimizers. We add joint peak-LR and decay-ratio searches for SGD-P and Adam-P, which project updated weights onto fixed-radius spheres, at token budgets of 1B, 2B, and 5B. Across these budgets, the optimal peak LR approximately follows ηpeak⋆∝B\eta^\star_{\mathrm{peak}}\propto B for SGD-P and ηpeak⋆∝B\eta^\star_{\mathrm{peak}}\propto\sqrt{B} for Adam-P in the subcritical regime (Figure 10). These results support the linear and square-root predictions of the SGD- and SignSGD-based analyses, respectively.

  • Decay-profile approximation. Appendices C and G derive the power-law tail of the FSL-implicit decay profile for the original and optimizer-dependent formulations, respectively. This motivates the clipped power law as an explicit approximation to the full profile, with its exponent calibrated empirically.

[W2] Optimizer dependence and learning-rate parameterization.

  • Why ELR? In LLM training, evolving weight norms change the effective schedule induced by a nominal LR schedule. Liu et al. [3] show that matching ELR closely aligns loss trajectories despite different nominal LRs and weight norms. They also demonstrate that ELR-based FSL transfers accurately to unseen Hyperball runs, whereas nominal-LR-based FSL incurs a 12.9× larger prediction error in that evaluation. This motivates studying K-scaling in an ELR-aligned setting.

  • Why norm control? Hyperball [4] controls weight and update norms, making nominal LR proportional to the relative update magnitude before projection. MuonH therefore provides direct control of the effective schedule. The additional SGD-P and Adam-P results (Figures 11 and 12) further show that quantitative K-scaling is not confined to MuonH.

  • Comparison with AdamW and MuonW. We add joint peak-LR and decay-ratio searches for AdamW and MuonW (Figure 13). Both retain the qualitative trends of increasing peak LR and shortening decay with larger batch sizes, but their nominal-LR profiles do not quantitatively follow K-scaling. This is consistent with evolving weight norms reshaping the effective schedule.

References

[1] Li et al. Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS, 2025. arXiv:2509.19189.

[2] Wang et al. When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective. NeurIPS, 2026. arXiv:2611.16560.

[3] Liu et al. Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining. 2026. arXiv:2608.24814.

[4] Wen et al. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization. 2026. arXiv:2606.16899.

Dear reviewer Lp8R,

As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your suggestions to ensure we have fully resolved any outstanding issues. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.