acceptodds
Under review as a conference paper at ICLR 2027

AdamW Scaling Laws from First Principles

Abstract

Scaling language-model pre-training to larger token budgets and model sizes requires retuning optimizer hyperparameters, yet existing transfer rules are predominantly empirical and offer limited theoretical guidance. We study this problem through the convergence behavior of SGD with momentum (SGDM) and AdamW under smoothness, bounded stochastic-gradient variance, and the -Kurdyka-Lojasiewicz condition. For both algorithms, we establish convergence guarantees and derive their optimization error under a fixed token budget . In the long-horizon regime, our analysis yields token-budget-aware BST scaling rules: the batch size should grow as , while the learning rate and momentum parameters remain approximately constant. We further prove a matching lower bound for zero-respecting stochastic first-order algorithms, showing that the deterministic and stochastic terms in our upper bounds are unimprovable, demonstrating the tightness of our approach. Experiments on FineWeb with Transformer language models ranging up to 1.4B parameters support these predictions. In particular, after scaling the batch size according to the BST rule, the optimal learning rate transfers across token budgets within each model scale. Together, our results provide a theoretically grounded prescription for transferring hyperparameters to longer pre-training runs without requiring expensive empirical fits.

Primary area:
optimization
Keywords:
Scaling Laws, AdamW, Optimization Bounds

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eio7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper derives for an LLM trained with AdamW a scaling law for the training loss and hyperparameter transfer as a function of training tokens. Unlike existing approaches, both the loss scaling law and the hyperparameter transfer are derived from theory, rather than fitted empirically. The derived prescription is to scale batch size as T^(2/3) and keep learning rate and momentum fixed. Experiments on up to 1.4B LLMs provide weak empirical validation of the theory.

Critical strengths

Sweeping hyperparameters over a dense grid search is exponentially expensive. This paper derives hyperparameter transfer rules from theory, obviating the need for expensive scaling experiments.

Beyond hyperparameter transfer, the theory predicts that the loss gap should decay as T^(-1/3), which is empirically verified at roughly 1 OOM scale in Figure 2.

Finally, Table 1 provides a useful survey of existing scaling heuristics, showing just how much disagreement there is in the literature, motivating a more careful study.

Critical weaknesses

My main concerns with this work is the lack of robustness of the empirical results. The empirical results show only that the theory is self-consistent, not that it's optimal.

For example, Figure 2 (top row) shows that if batch size is scaled as T^(2/3), then the optimal LR is roughly constant. However, there is no experiment to justify that this is the globally optimal scaling. It might be the case that under a different batch size and learning rate scaling rule, we get a lower final loss.

Hyperparameters are only swept in isolation along specific 1D and 2D slices (for example [LR x BS] in Figure B.1 and [β1 x β2] in Figure B.4). This would miss any joint scaling rule not captured by these slices. For example [1, 2] show that β2 should be scaled jointly with BS. This paper only sweeps BS without β2 and β2 without BS. This means that any conclusions about the optimality of the found β2 and BS should be questioned – maybe the found β2 is optimal for the given BS and the BS is only optimal for the given β2. Jointly scaling β2 with BS might reveal a better joint optimum. This point is analogous to my concern about the T^(2/3) scaling above.

Further exacerbating this issue is the fact that the literature is conflicted on scaling heuristics. The authors themselves note this in Table 1. It would therefore be extremely useful to have stronger, more discriminative empirical results.

Finally, I find the main comparison of BST against other scaling heuristics (Figure 3) slightly misleading and underwhelming. At a first glance, it looks like BST converges much "faster" than the baselines. However, upon reading the figure more carefully, the x-axis is not compute adjusted. BST merely converges in fewer steps because it uses a larger batch size. The fact that the method with the largest batch size converges first is completely expected. All methods use the same FLOPs. One area where the methods differ is the final validation loss, and BST finishes in the middle.

Since I consider these issues to be more severe than "minor", I rated the soundness with a score of only 1.

Questions for authors
  1. What's your motivation for scaling batch size? Is the goal to find an "optimum" batch size that minimizes loss, or is the goal to always use the largest batch size smaller than the critical batch size? [3]
  2. How does a constant β2 square with [1, 2], which suggest scaling β2 with batch size?

References:

  • [1] Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful (arXiv:2507.07101)
  • [2] The Best Optimizer Depends on Batch Size (arXiv:2610.08975)
  • [3] How Does Critical Batch Size Scale in Pre-training? (arXiv:2410.21676)
2 comments

Thank you for your thoughtful review. We are happy to hear your appreciation of our hyperparameter transfer rules derived from theory, obviating the need for expensive scaling experiments, our prediction of the T−1/3T^{-1/3} loss gap scaling, and our useful survey of existing scaling heuristics. We now address your main concerns and answer your questions. We would be happy to address any remaining questions.

W1: My main concerns with this work are the lack of robustness of the empirical results...

To address the robustness concerns, we refer to Figure 1, where we estimate the confidence intervals based on a 162M model size. Similarly, we updated Figure 3 and included confidence intervals for the performance of all scaling laws.

To address the optimality concern, we report batch-size sweeps in Figure 4. They show that the BST-recommended batch size lies near the upper end of the empirically optimal range.

W2: For example, Figure 2 (top row) shows that if batch size is scaled as T^(2/3), then the optimal LR is roughly constant...

Our analysis predicts no faster convergence rate than T−1/3T^{-1/3}. A smaller batch-size exponent, B∝TγB\propto T^\gamma with γ<2/3\gamma<2/3, can retain this rate if the other hyperparameters are adjusted accordingly. Given the agreement with theory in Figure 2, we expect comparable performance to the γ=2/3\gamma=2/3 rule. We leave the empirical verification of this claim to future work.

W3: Hyperparameters are only swept in isolation along specific 1D and 2D slices...

We refer to our response to W4 and Q2.

W4: Further exacerbating this issue is the fact that the literature is conflicted on scaling heuristics...

We refer to the general response that summarizes the significance of our experiments. As the reviewer notes, an exhaustive joint sweep grows rapidly with the number of hyperparameters. Even for our 124M- and 162M-parameter base models, it would require substantially more compute than we have available.

To resolve the conflicting guidelines of scaling laws, we study the problem from first principles. Our scaling ladder in Figure 1 demonstrates that the principled BST scaling law is competitive in practice, and improves with model size.

W5: Finally, I find the main comparison of BST against other scaling heuristics (Figure 3) slightly misleading and underwhelming...

Our main experimental result is presented in Figure 1 (and not in Figure 3), which we additionally expanded with larger-scale experiments: BST’s relative performance against other scaling laws improves as token budgets and model sizes increase, the regimes most relevant in practice.

We respectfully disagree with the characterization “misleading" and "underwhelming” of Figure 1. The right-hand panels of Figure 1 explicitly plot convergence against optimizer steps. Since the scaling laws are compared at the same model and token horizon, the compute adjustments are not needed. The step axis makes the effect of batch size visible: BST uses a larger batch and reaches competitive loss in fewer optimizer updates. A larger batch can improve data parallelization and GPU utilization (higher MFU) when the hardware configuration supports it.

Q1: What's your motivation for scaling batch size? Is the goal to find an "optimum" batch size that minimizes loss...

Our analysis identifies the largest batch size that preserves the T−1/3T^{-1/3} convergence rate, a theoretical analogue of the critical batch size: smaller batches retain the rate, while larger ones degrade it. As in [3], the resulting rule scales batch size primarily with token horizon.

Larger batches require fewer optimizer steps at a fixed token budget and can improve model FLOPs utilization (MFU), a key consideration in large-scale pretraining. Importantly, our batch-size exponent is derived from the analysis.

The new sweeps in Figure 4 show that the BST batch size performs near the best tested value and lies close to the upper end of the empirically near-optimal range.

Q2: How does a constant β2\beta_2 square with [1, 2], which suggest scaling β2\beta_2 with batch size?

To address this concern, we conducted additional experiments with the 124M-parameter base model at a 1.3B-token horizon. We transferred the hyperparameters tuned at B0=128B_0=128 to B∈64,256,512B\in\\{64,256,512\\} using five scaling rules based on [1, 2]:

  • η∝B/128\eta\propto\sqrt{B/128} and β2∝0.95B/128\beta_2\propto0.95^{\sqrt{B/128}};
  • η∝B/128\eta\propto\sqrt{B/128} and β2∝0.95B/128\beta_2\propto0.95^{B/128}
  • η∝B/128\eta\propto\sqrt{B/128} and β2=const\beta_2=\mathrm{const};
  • η=const\eta=\mathrm{const} and β2∝0.95B/128\beta_2\propto0.95^{\sqrt{B/128}};
  • η=const\eta=\mathrm{const} and β2∝0.95B/128\beta_2\propto0.95^{B/128};

We report the results in Section A.3 of the revised paper. None of the runs using these rules outperformed the final validation loss of our tuned baseline. This provides further evidence that the baseline is sufficiently well tuned for our experiments.

AuthorsOpenReview ↗

Rebuttals (Part II)

References:

[1] Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful (arXiv:2507.07101)

[2] The Best Optimizer Depends on Batch Size (arXiv:2610.08975)

[3] How Does Critical Batch Size Scale in Pre-training? (arXiv:2410.21676)

Reviewer_z3okOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work derives optimal batch size and weight-decay scaling rules with respect to the token budget for SGDM and AdamW optimizers. Under assumptions of L-smoothness, $ \ mu$-KL, and bounded stochastic gradient variance, they prove that as the token budget TT increases, the batch size should be scaled as B∝T2/3B \propto T^{2/3} while keeping the learning rate constant and scaling the weight decay λ∝T−1/3\lambda \propto T^{-1/3}. They illustrate these results on LLMs of 124M parameters and show that the scaling law fit of the loss agrees with the predicted exponent and that the optimal learning rate under this batch size scaling rule is indeed constant across scales. Finally, they compare this scaling rule with others from the literature, showing a competitive final loss, although using fewer iterations.

Critical strengths

The main strength of that work is the clarity of the assumptions used in the theoretical results, Theorems 1, 2, 3. Assumptions 1, 2, 3, 4 are clearly formulated, which makes the setting of the theorems understandable and clear. Additionally, a summary and comparative table of previous works and the literature in Table 1 significantly improved the clarity of the results. Finally, the theoretical results on hyperparameters and loss scaling in Theorems 1, 2 are strong and provide an interesting lens through a scaling literature that is mostly empirically driven and lacks principled understanding.

Critical weaknesses

The main weakness concerns the experimental results. Experiments are done at a relatively small scale and are not extensive enough to strongly support the theoretical results of the paper: the sweep over T is done only over a factor ∼20\sim 20 from 1.4B to 26.7B in Figure 2, which is relatively small to talk about scaling behaviour (this is only about one order of magnitude). Additionally, Figure 3 does not show confidence intervals on the losses and whether the loss gaps are significant and do not show which efficiency gains in terms of compute does a scaling method bring over the other. Finally, the power-law fits in Figure 2 are done on the loss during training is very non-standard in the literature and seems arbitrary. The fitted exponent from this fit hence cannot prove or disprove the theoretical results, which are done at the end of training.

A second concern relates to the novelty of the weight-decay scaling, which seems to have already been evidenced in prior works [Bergsma et al. 2025], see my question 5.

A third, more minor concern is about the clarity and some definitions in the paper. I could not find what "BST" stands for in the manuscript, although a definition is provided in Definition 1. Similarly, dd (line 75) was defined only late in the paper. Finally, the origin of the second part of Definition 1, the equality ϵA=M(c)T−1/3\epsilon_A = M(c) T^{-1/3}, is not explained: is it a definition or a known result? If the latter, it would be good to provide references for it.

Questions for authors
  1. The power-law fits in Figure 2, bottom, are done on the loss during training and before the cooldown phase, not at the end of training, as used in the theoretical results. It was not clearly explained why such a choice was made, and while the fitted exponent is aligned with theory, this choice appears arbitrary and does not reflect the theoretical results.

  2. The discussion on ϕ(c)\phi(c) in Section 3.2 is unclear. Indeed, the assumption that the constants c=(L,μ,σ∗)c=(L, \mu, \sigma_*) are independent of model size and scale seems strong, and it is hard to understand the intuition behind it. Additionally, the authors acknowledge that previous work by Islamov et al. (2026) actually fit these as power laws in model depth and width. Additionally, it seems even unclear why the authors need to assume that these constants cc do not depend on scale. Indeed, in the formula on θABST=ϕ(c)Tκdω\theta_A^{BST}=\phi(c)T^{\kappa}d^{\omega} in Definition 1, fitting ϕ(c)\phi(c) as power-law of dd and TT would only change the exponents κ\kappa and ω\omega which wouldn't change the analysis by itself. Could the authors, hence, explain the discussion in Section 3.2 and why the authors need to assume that ϕ(c1)/ϕ(c0)=1\phi(c_1)/\phi(c_0)=1?

  3. Could the authors clarify the definition of BST as "using the largest feasible batch size"? First, what is a feasible batch size in this context, and second, how would the results be affected if this assumption that the largest feasible batch size is used was removed? Could smaller batch sizes, under different hyperparameter scaling rules, improve the final validation loss?

  4. Do the authors have intuition on how the BST scaling rule would be changed if instead of a μ−KL\mu-KL condition, a μ−PL\mu-PL condition was used on the loss?

  5. The derived scaling rule on weight decay seems already present in the literature. Indeed, previous works (see e.g. {1}) have shown that λ∝1K\lambda \propto \frac{1}{K} when using K iterations under a fixed learning rate. Here, the number of iterations is T/BT/B \propto T^{1/3}$, which recovers the proposed weight-decay scaling. Could the authors comment on the novelty of their derived weight-decay scaling?

[1] Bergsma, Shane, et al. "Power lines: Scaling laws for weight decay and batch size in LLM pre-training." Advances in Neural Information Processing Systems 38 (2026): 125153-125188.

7 comments

Thank you for your thoughtful review. We are happy to hear your appreciation that our theoretical results on hyperparameters and loss scaling are strong, and to have an interesting lens through which to view a scaling literature that is mostly empirically driven and lacks a principled understanding. We now address your main concerns and answer your questions. We would be happy to address any remaining questions.

W1: The main weakness concerns the experimental results...

Our general response to all reviewers summarizes the experiments, including those added during revision, and provides compute-usage calculations that show the scale of our empirical evaluation.

Size of experiments. Additionally, we compare the scale of our experimental setup to other works in the table below.

StudyModel SizesToken HorizonsTPP
Ours (Figure 1)162M to 1.8B, 5 rungs20B and 40B14 to 247
Ours (Figures 2,3)124M fixed1.3B to 40.1B10 to 323
Power Lines [Bergsma et al., 2025]111M to 2.1B, plus one 3.3B2.2B to 549B20 to 1280
Step Law [Li et al., 2025a]60M to 1B dense, plus a 6.5B MoE2B to 100B (130B for the MoE)~19 to 465
Critical batch size [2]85M to 1.2B1.7B to 24.6B (to ~61B in ablations)~20 (6 to 81 in ablations)
Small-batch training [1]30M to 1.3B (active)0.6B to 10B8 to 20

We added a 1.8B model, larger than the largest dense models in other batch size studies from academic institutions [1,2]. Our largest per-model budget, 40B tokens, is four times the 10B used for the largest model in [1], and our ladder lies within Power Lines’ main model-size range. Power Lines spans a wider tokens-per-parameter range (20–1,280 versus our 14–247), while Step Law includes runs up to 100B tokens and reports nearly one million H800 GPU-hours across its full experimental program. Thus, our experiments reach a comparable model scale with a small fraction of Step Law’s compute budget.

Moreover, Figure 3 now reports the mean over three random seeds per rule with ±2σ\pm2\sigma shaded bands. The spread among the best-performing rules lies within that band, so we do not claim a significant difference in final loss at this scale. The difference we do claim in this case is in the number of optimizer steps and a larger batch to reach the comparable loss.

The power-law fit is taken at the end of the constant-learning-rate phase because that is the regime our analysis covers: the theorem assumes a constant learning rate, so this is where its prediction applies. We now also fit the end-of-training losses, and Theorem 6 (Appendix D.4) shows why a nonincreasing learning-rate phase preserves the T−1/3T^{-1/3} rate. Both fits are close to 1/31/3.

[1] Marek, Martin, et al., Small batch size training for language models: When vanilla SGD works, and why gradient accumulation is wasteful., NeurIPS 2025.

[2] Zhang, Hanlin, et al., How does critical batch size scale in pre-training?." ICLR 2025.

W2: A second concern relates to the novelty of the weight-decay scaling...

See response to Q5

W3: A third, more minor concern is about the clarity and some definitions in the paper...

We thank the reviewer for pointing out these unclear parts of the paper. We revised the text and provide clarifications below.

  • BST stands for Batch-Sequence-Token-Budget-aware scaling laws derived from convergence upper bounds. We adopted the name from [Islamov et al., 2026], who applied a similar approach to spectral algorithms.
  • In line 31, dd was mentioned for the first time, and it denotes the dimension of the model parameter vector xx. We will make this definition clearer.
  • In line 218, εA(T,c,θ)\varepsilon_{\mathcal{A}}(T,c,\theta) denotes a convergence upper bound from our theoretical analysis. We show that it scales as εA(T,c,θ)=M(c)T−1/3\varepsilon_{\mathcal{A}}(T,c,\theta)=\mathcal{M}(c)T^{-1/3}, where M(c)\mathcal{M}(c) captures the dependence on the problem constants cc (Theorems 1 and 2). We omit the explicit form of M(c)\mathcal{M}(c) because it is complex and unnecessary for the main argument. This result says that for a fixed model, the convergence rate is T−1/3T^{-1/3}.

Q1: The power-law fits in Figure 2, bottom, are done on the loss during training and before the cooldown phase...

We kindly disagree with this statement. We fitted the exponent at the end of the constant learning rate phase, since it is the setting considered in the analysis. The goal of Figure 2 is to support the analysis.

We updated Figure 2 with a fit to end-of-training losses, which shows decay close to the predicted T−1/3T^{-1/3} rate. Appendix D.4 provides Theorem 6 explaining why a nonincreasing learning-rate schedule preserves this rate.

AuthorsOpenReview ↗

Rebuttals (Part II)

Q2: The discussion on ϕ(c)\phi(c) in Section 3.2 is unclear...

Thank you for bringing up this concern. Let us clarify it in more detail here. There are several reasons why we decided to assume ϕ(c1)/ϕ(c0)=1\phi(c_1)/\phi(c_0)=1.

  • We agree that fitted exponents for ϕ(c)\phi(c) could be used without invalidating our analysis. Estimating them, however, would require additional sweeps, which our approach aims to avoid. Moreover, there is currently no reliable way of estimating constants (It’s unclear how tight the approximations are in [Islamov et al. (2026)]. We instead use CompleteP scaling to make training more predictable across model sizes.
  • [Islamov et al., 2026] report that the problem constants grow slowly with model dimensions. However, their training setup relies on NanoGPT codebase that does not use CompleteP model scaling.
  • Currently, we do not know how problem constants should change with model size in the analysis. This requires substantial work that we leave to future work.

We therefore use ϕ(c1)/ϕ(c0)=1\phi(c_1)/\phi(c_0)=1 as a practical approximation that makes the scaling rules easier to apply, while leaving its theoretical justification for future work.

Q3: Could the authors clarify the definition of BST...

We thank the reviewer for raising this concern. We believe that the word "feasible" was not appropriate in Definition 3.1. We have now removed it and added a clarification that we want the largest batch size that preserves the optimal theoretical convergence rate. We now explain what happens if we remove such a restriction.

We refer to the general comment regarding a smaller batch size exponent. Shortly, according to our theory a smaller batch size exponent cannot improve the convergence rate beyond the optimal T−1/3T^{-1/3}.

Q4: Do the authors have intuition on how the BST scaling rule would be changed if instead of a μ\mu-KL condition, a μ\mu-PL condition was used on the loss?

For simplicity, we explain the results under μ\mu-PL condition based on the convergence upper bound of SGD, which can be obtained following Theorem 4 in [3]. One can derive the convergence bound for SGD of the form

E[f(xK)−f∗]≤(1−ημ)K(f(x0)−f⋆)+Lησ⋆22μB,(2)\mathbb{E}[f(x_K)-f^*] \leq (1-\eta\mu)^K (f(x_0)-f^\star) + \frac{L\eta\sigma_\star^2}{2\mu B},\quad (2)

where η≤1/L\eta \le 1/L. To make the second term in (2) smaller than ε/2\varepsilon/2, we should choose η\eta such that η≤μεBLσ⋆2.\eta \le \frac{\mu\varepsilon B}{L\sigma_\star^2}. To make the first term in (2) smaller than ε/2\varepsilon/2, we should choose

K≥1ημlog⁡2(f(x0)−f∗)ε.K\ge \frac{1}{\eta\mu}\log\frac{2(f(x_0)-f^*)}{\varepsilon}.

Choosing

\eta=\min\left\\{1/L,\frac{\mu\varepsilon B}{L\sigma_\star^2}\right\\}, \quad K=\frac{L}{\mu}\max\left\\{1,\frac{\sigma_\star^2}{\mu\varepsilon B}\right\\}\log\frac{2(f(x_0)-f^*)}{\varepsilon},

we obtain E[f(xK)−f∗]≤ε\mathbb{E}[f(x_K)-f^*]\le\varepsilon. This implies that T=KB=\tilde{\mathcal{O}}\left(\frac{L}{\mu}\max\left\\{B,\frac{\sigma_\star^2}{\mu\varepsilon}\right\\}\right).

Since TT is fixed, the best optimization error is ε=O~(Lσ⋆2μ2T).\varepsilon=\tilde{\mathcal{O}}\left(\frac{L\sigma_\star^2}{\mu^2T}\right). Choosing the bath size as large as possible, so that we still achieve this error gives the scaling B∝σ⋆2με=O~(μTL).B\propto \frac{\sigma_\star^2}{\mu\varepsilon}=\tilde{\mathcal{O}}\left(\frac{\mu T}{L}\right). This choice of BB ensures that the two terms in (2) are equal. This also gives the choice of η∝1L=Const\eta\propto\frac{1}{L}=Const. As we can see, under the μ\mu-PL condition, batch size scaling becomes B∝TB\propto T.

We would like to highlight that μ\mu-PL is a much stronger condition in comparison to μ\mu-KL as shown in Appendix E, specifically a clear demonstration of this statement is Example 8. While μ\mu-KL holds in part of the trajectory of the optimization procedure always, μ\mu-PL only does when all of the matrices are well conditioned, which is not usually the case.

AuthorsOpenReview ↗

Rebuttals (Part III)

The derived scaling rule on weight decay seems already present in the literature

We kindly disagree with this claim for the following reasons.

  • Our weight-decay rule is derived, not assumed, and we do not claim that weight-decay scaling was not studied before: Table 1 cites several studies of it. In our AdamW analysis, the convergence bound gives, for a fixed model weight decay scaling λ∼T−1/3∝B/T=1/K\lambda\sim T^{-1/3}\propto B/T=1/K. Its agreement with the empirical relation in [2] supports the predictive power of the analysis.

  • The theorem in [2] does not establish an optimal λ∝1/K\lambda\propto 1/K rule on its own. For a fully scale-invariant network, [2] proves trajectory invariance under appropriate rescaling. Though many parameter groups are scale invariant in modern architectures, the full language-model loss generally does not meet the theorem’s global scale-invariance assumption. [1] builds on [2]’s AdamW timescale interpretation. Obtaining the optimal rule additionally requires the assumption that the optimal timescale scales with token-per-parameter, which [2] examines empirically.

  • The fitted rule in [1] differs from λ∝1/K\lambda\propto 1/K. [1] defines AdamW’s normalized timescale as τ=B/(ηλT)=1/(ηλK)\tau=B/(\eta\lambda T)=1/(\eta\lambda K), so λ∝K−1\lambda\propto K^{-1} across scales only if ητ\eta\tau is constant. Instead, [1] fits τ∝(T/d)−0.527\tau\propto(T/d)^{-0.527} and B∝T0.383B\propto T^{0.383}, which imply λopt∝η−1d−0.527T−0.090.\lambda_{\mathrm{opt}}\propto\eta^{-1}d^{-0.527}T^{-0.090}. At fixed model size and learning rate, its token-budget exponent is −0.090-0.090, compared with −0.617-0.617 for K−1=B/TK^{-1}=B/T.

References

[1] Bergsma, Shane, et al. "Power lines: Scaling laws for weight decay and batch size in LLM pre-training." Advances in Neural Information Processing Systems 38 (2026): 125153-125188.

[2] Wang, Xi, and Laurence Aitchison. "How to set AdamW's weight decay as you scale model and dataset size." arXiv preprint arXiv:2405.13698 (2024).

[3] Karimi, Hamed, Julie Nutini, and Mark Schmidt. "Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition." Joint European conference on machine learning and knowledge discovery in databases. Cham: Springer International Publishing, 2016.

[4] Hoffmann, Jordan, et al., Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022).

Reviewer_Fg7WOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the convergence of SGDM and AdamW under the µ-KŁ condition and derives scaling rules for optimizer hyperparameters. The main result suggests scaling batch size as T^(2/3), keeping the learning rate and momentum constant, and scaling weight decay as d^(-1/2)T^(-1/3). The paper also provides matching lower bounds for stochastic first-order methods. The experiments include language models up to 1.4B parameters and show that the proposed rules perform competitively with existing scaling laws.

Critical strengths
  1. The theoretical analysis is interesting and provides a simple rule for transferring hyperparameters across token budgets. The matching lower bound is also useful.

  2. The experimental results are promising. In particular, Figure 2 shows that the optimal learning rate stays roughly constant across different token budgets when the batch size is scaled as T^(2/3).

  3. The paper provides good comparisons with existing scaling laws. The proposed method achieves competitive results while using fewer optimizer steps.

Critical weaknesses
  1. I am not entirely convinced how much the theoretical results tell us about the optimal hyperparameters for AdamW. The AdamW analysis essentially uses SGDM with a bounded preconditioner, and the resulting scaling rules are similar. Moreover, the bound depends on the ratio pu/pl, which becomes very large for the small stability constants used in practice. It is unclear how informative these bounds are for actual LLM training.

  2. The proposed scaling rules do not necessarily characterize the optimal hyperparameters. For example, B ∝ T^(2/3) corresponds to the largest batch size that preserves the convergence rate, but smaller batches can also achieve the same rate. Similarly, the weight decay scaling is a sufficient condition in the analysis rather than an optimality result. The weight decay experiments in Figure B.3 are also almost flat.

  3. The experiments do not sufficiently validate the proposed batch-size exponent. Figure 2 only considers B ∝ T^(2/3), without comparing it against other exponents. Also, the theoretical analysis assumes a constant learning rate, whereas the experiments use WSD schedules. In Figure 1, Step Law also performs better than BST at the smallest model size.

Questions for authors
  1. How should we interpret the AdamW bounds when the stability constant is very small, given that the ratio pu/pl becomes extremely large? Are these bounds still informative for practical training?

  2. Is the constant learning rate in the theoretical result essentially determined by the stability limit? Does the analysis provide any prediction for how the optimal learning rate changes with token budget when the batch size is fixed?

  3. Can the authors compare the learning rate transfer results in Figure 2 using different batch-size scaling exponents, such as T^(1/2) and T? This would help establish whether T^(2/3) is actually important for the observed transfer.

2 comments

Thank you for your thoughtful review. We are happy to hear your appreciation of our theoretical analysis, which provides a principled HP transfer rule across token horizons, including the lower bounds, promising experimental results, and a good comparison with existing scaling laws showing competitive performance with fewer steps. We now address your main concerns and answer your questions. We would be happy to address any remaining questions.

W1: I am not entirely convinced how much the theoretical results tell us about the optimal hyperparameters...

We agree that the AdamW and SGDM analyses share a common structure. However, the AdamW proof must also control a preconditioner built from current and past gradients and a weight-decay term involving λ2E∣xk∣2\lambda^2\mathbb{E}\\|x_k\\|^2, both of which require careful treatment. Moreover, recent work [1] finds a T−1/3T^{-1/3} convergence rate for five algorithms, including SGDM and AdamW, suggesting that some similarities in their analyses are to be expected.

A small ν\nu can make pu/plp_u/p_l large, making the bounds less informative and quite pessimistic for choosing hyperparameters at a given token budget. BST instead uses ratios across budgets. For a fixed model and problem constants, the prefactor cancels in the ratio: B(T1)/B(T0)=(T1/T0)2/3B(T_1)/B(T_0)=(T_1/T_0)^{2/3}. We use these ratios to transfer tuned hyperparameters from a specific model size and token horizon to larger ones. Figure 2 tests the predictive power of the rules.

[1] Lee, Hyunseok, et al., Optimizer-dependent training dynamics converge to the same one-third optimal data scaling., arXiv preprint arXiv:2609.37745 (2026).

W2: The proposed scaling rules do not necessarily characterize the optimal hyperparameters...

Indeed, our convergence theory does not provide a unique choice of optimal hyperparameters. To restrict this choice, BST selects the largest batch that preserves the optimal T−1/3T^{-1/3} rate, and thus does the fewest updates among those choices. These choices are motivated by the potential for larger batches to improve model FLOPs utilization (see [Chowdhery et al., 2023], [Enskat and Wiesner, 2026]) and data parallelization. We make this distinction explicit in the short remark after Corollary 2.

Similar reasoning applies to weight decay. In practice, it matters, especially in long training: at 26.7B tokens, smaller values worsen validation loss (Figure A.3), consistent with the broader ablations in Moonlight [2]. We therefore choose the largest weight decay that preserves the convergence rate.

[2] Liu, Jingyuan, et al., Muon is scalable for llm training. arXiv preprint arXiv:2502.16982 (2025).

W3: The experiments do not sufficiently validate the proposed batch-size exponent...

Verification of Batch Size Scaling.

To evaluate the BST-recommended batch size, we added sweeps at 10B, 20B, and 40B tokens (Figure 4). Each shows a broad range of batch sizes with low loss, with BST near the upper end of optimal batch sizes, consistent with its rule of selecting the largest suitable batch size, preserving the rate.

Constant learning rate in the analysis.

Indeed, our analysis assumes a constant learning rate. As discussed in Section 7, we left the analysis to learning-rate schedules for further work. To address this concern, we provide Theorem 6 in Appendix D.4. It shows that the convergence rate remains the same, even with learning rate decay. Updated Figure 2 confirms this: the fit at the end of training is close to the predicted T−1/3T^{-1/3}.

Comparison against Step Law

Step Law slightly outperforms BST at 162M parameters, but their ranking reverses at larger model sizes, the regime of greater practical interest. Importantly, BST is derived from our analysis, whereas the Step Law study used roughly one million H800 GPU-hours.

Q1: How should we interpret the AdamW bounds when the stability constant is very small...

A small ν\nu can make pu/plp_u/p_l large, making the bounds less informative for choosing hyperparameters at a given token budget. BST instead uses ratios across budgets. For a fixed model and problem constants, the prefactor cancels: B(T1)/B(T0)=(T1/T0)2/3B(T_1)/B(T_0)=(T_1/T_0)^{2/3}. We use these ratios to transfer tuned hyperparameters. Figure 2 tests the predictive power of the rules.

AuthorsOpenReview ↗

Rebuttals (Part II)

Q2: Is the constant learning rate in the theoretical result essentially determined by the stability limit?..

The BST batch size rule can also be understood as choosing the batch size that balances the stochastic and deterministic terms in the convergence rate. If we choose B∝TγB\propto T^{\gamma} for some γ∈[0,2/3)\gamma\in[0,2/3), the corresponding learning rate follows \eta(T)\sim\min\left\\{c_1,c_2T^{\gamma-2/3}\right\\}. With the BST exponent γ=2/3\gamma=2/3, the learning rate is constant across token horizons. If γ<2/3\gamma<2/3, then the learning rate should decay as Tγ−2/3T^{\gamma-2/3} for large enough token horizons. In particular, a constant batch size means γ=0\gamma=0 and yields η∝T−2/3\eta\propto T^{-2/3} rule.

Q3: Can the authors compare the learning rate transfer results in Figure 2 using different batch-size scaling exponents...

We agree that this experiment would help assess how well the analysis generalizes. However, resource constraints prevent us from running learning-rate sweeps for other exponents as extensive as those in Figure 2.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading and constructive feedback, which has helped us improve the paper. We particularly appreciate their recognition of the following contributions:

  • A principled scaling analysis. Reviewer eio7 valued the theory-based hyperparameter transfer rules when dense sweeps across scales are too costly. Reviewer z3ok called the hyperparameter and loss-scaling results in Theorems 1 and 2 a strong contribution to a largely empirical literature.

  • Clear assumptions and a matching lower bound. Reviewer z3ok found that Assumptions 1–4 clarify Theorems 1–3. Reviewer Fg7W valued the analysis, its simple transfer rule across token budgets, and the matching lower bound.

  • Empirical support and practical competitiveness. Reviewer Fg7W found the experiments promising, noting that the optimal learning rate stays roughly constant for B∝T2/3B\propto T^{2/3} and that BST is competitive with fewer optimizer steps. Reviewer eio7 highlighted empirical support for ε∝T−1/3\varepsilon\propto T^{-1/3} over the tested range.

  • Useful comparisons with prior work. Reviewers z3ok and eio7 valued Table 1’s summary of existing scaling rules and their disagreements. Reviewer Fg7W highlighted the comparisons with existing methods.

We thank the reviewers for identifying unclear passages. We have revised the manuscript for clarity, marked the changes in red, and made the following additions:

  • Extended scaling ladder. We added the comparison at 40B tokens across 5 model sizes, up to 1.8B parameters (Figure 1). We estimate confidence intervals across scaling laws at 162M model size. BST’s relative ranking improves with longer training for this model and with larger model sizes.
  • Optimality of BST batch size. For the 304M model, we swept batch sizes at 10B, 20B, and 40B tokens with other hyperparameters set by BST (Figure 4), averaging three seeds for all batch sizes at 10B and for the BST batch size only at 20B and 40B. The BST batch size is near the best tested value and the upper end of the near-optimal range at each budget.
  • Additional seeds. Figure 3 now reports results averaged over three seeds.
  • Testing tuned hyperparameters. For the 124M base model, we compared the tuned hyperparameters with those suggested by batch-size dependent rules (Section A.3). The results support that our tuning is close to optimal.
  • GPU Use. Including the revision experiments, it consumed approximately 22k H200 GPU-hours, 800 H100 GPU-hours, and 11.9k A100 GPU-hours.
  • Fit at the end of training. We updated Figure 2, where we add the fits at the end of the training. The fitted exponents remain close to −1/3-1/3 for post-cooldown phase.

Moreover, we would like to provide some clarifications to our approach.

  • Our analysis does not provide a unique choice of hyperparameters. To restrict the choice, we choose the largest batch size and weight decay that preserve the rate, allowing high model FLOPs utilization (MFU), greater data parallelism, and performance at longer horizons.

    For a fixed model and sufficiently large token budget TT, AdamW converges as E[f(xT/B)−f∗]≤max⁡aB/T,bT−1/3\mathbb E[f(x_{T/B})-f^*]\leq \max\\{aB/T,bT^{-1/3}\\} with the prescribed learning rate, moment parameters, and weight decay. The constants aa and bb depend on the problem. At the crossover B⋆=(b/a)T2/3B_\star=(b/a)T^{2/3}, the progress and stochastic terms are equal. Smaller batch size exponents preserve the same rate with corresponding changes to the learning rate and moment parameters: for all B≤B⋆B\leq B_\star, the bound is still bT−1/3bT^{-1/3}. For a batch schedule B=b0TγB=b_0T^\gamma that stays at or below B⋆B_\star, with 0≤γ≤2/30\leq\gamma\leq2/3, we can set η(T)=min⁡c1,c2Tγ−2/3\eta(T)=\min\\{c_1,c_2T^{\gamma-2/3}\\} and adjust the moment-refresh weights 1−β11-\beta_1 and 1−β21-\beta_2 accordingly.

Batch size exponent γ\gammaBatch size BBLearning rate η\eta1st-order Momentum1−β11-\beta_12nd-order 1−β21-\beta_2Weight decay λ\lambda
1/31/3∝T1/3\propto T^{1/3}∝T−1/3\propto T^{-1/3}∝T−1/3\propto T^{-1/3}∝T−1/3\propto T^{-1/3}∝d−1/2T−1/3\propto d^{-1/2}T^{-1/3}
1/21/2∝T1/2\propto T^{1/2}∝T−1/6\propto T^{-1/6}∝T−1/6\propto T^{-1/6}∝T−1/6\propto T^{-1/6}∝d−1/2T−1/3\propto d^{-1/2}T^{-1/3}

Compared with other batch-size exponents, BST’s γ=2/3\gamma=2/3 rule offers a simple transfer strategy: scale batch size and weight decay across token horizons, keep the learning rate and momentum parameters fixed, and favor larger batches for training efficiency.

AuthorsOpenReview ↗

The summary of the rebuttals

Dear Area Chair,

The paper answers a practical question in language-model training. As the token budget grows, how large can the batch become while preserving the training-gap rate? For a fixed model under the stated assumptions, our analysis derives B∝T2/3B\propto T^{2/3} as the largest batch growth preserving the T−1/3T^{-1/3} rate, together with constant learning rate and momentum parameters and a sufficient AdamW weight-decay rule. A matching lower bound for zero-respecting stochastic first-order methods establishes the worst-case tightness of the leading optimization terms.

All three original reviews recognized value in this derivation. Fg7W highlighted the transfer rule and matching lower bound, eio7 emphasized the cost of discovering such rules through empirical searches, and z3ok explicitly described Theorems 1 and 2 as strong. The revision tests the practical implication of the theory. In Figure 4, at 10B, 20B, and 40B tokens, BST batch size lies near the largest tested batch, achieving near-best final loss. In Figure 3, at the same model size and 40.1B-token budget, BST reaches comparable final loss to Power Lines with about 38% as many optimizer updates.

We addressed specific concerns from Fg7W and eio7. For Fg7W’s question about pessimistic AdamW constants, we showed that the prefactor cancels in the batch-size ratio across horizons for a fixed model and fixed problem constants. For eio7’s question about batch-dependent momentum, we tested five alternative learning-rate and second-moment transfer rules at the base horizon. The tuned baseline matched or surpassed each on final validation loss. Figure 3 now also reports three seeds per rule.

On z3ok’s novelty concern, the original review asked whether [Bergsma et al.] had already established our weight-decay rule. [Bergsma et al.] empirically fit batch size and decay to validation performance. Our analysis derives the largest batch growth preserving the guaranteed rate, the resulting update budget K∝T1/3K\propto T^{1/3}, and a sufficient AdamW decay along that path. Even granting an empirical relation λ∝1/K\lambda\propto 1/K, the token-budget exponent remains undetermined until K(T)K(T) is derived. The reviewer’s follow-up asks why the fitted and derived exponents differ. That is a question about practical interpretation. The discussion identifies no prior result establishing the joint prescription.

On z3ok’s empirical concern, the reviewer requested final-checkpoint fits and larger-scale experiments. We supplied both, extended the evaluation to 1.8B parameters, and added Theorem 6 establishing rate preservation after cooldown. The reviewer explicitly credited the new evidence in raising their recommendation from clear to weak reject. Their remaining observation about the largest-horizon point in Figure 2 qualifies the precision of the fitted loss curve. Figure 4 addresses the central batch prediction independently by varying batch size at each horizon and measuring final loss directly. Its evidence is independent of the fitted exponent in Figure 2.

We believe the remaining weak-reject recommendation gives insufficient weight to the acknowledged contributions. z3ok raises legitimate questions about fit precision and accessibility, and reported lower confidence particularly in judging readability. Their final assessment identifies no error in the main derivation, prior work establishing the same joint rule, or experimental result contradicting the central batch prediction. Fg7W and eio7 have posted no author-visible follow-up to our 14 November responses. We therefore ask you to weigh the revised evidence directly. In our view, the joint derivation, the matching complexity bound, and the direct empirical tests of transfer warrant acceptance.

AuthorsOpenReview ↗

Summary of the main concerns and how the authors addressed them

ReviewerAssessment / appreciationMain concernsOur response and evidence
Fg7WValues the theoretical analysis, simple transfer rule, matching lower bound, and competitive empirical performance with fewer optimizer updates.Pessimistic AdamW constants and practical relevance of the bounds. Optimality of BST hyperparameters. Limited evidence for the batch exponent. Constant learning rate in theory versus cooldown in experiments.Explained that the prefactor cancels in transfer ratios for a fixed model and problem constants. Clarified that BST selects the largest batch preserving the guaranteed rate, not a unique empirical optimum (Section 3.1). Added batch sweeps at 10B, 20B, and 40B tokens (Figure 4) and Theorem 6 for cooldown.
eio7Values deriving hyperparameter transfer rules from theory rather than costly empirical searches. Recognizes the evidence for the predicted scaling rate and the useful comparison with prior work in Table 1.Reliability of empirical comparisons. Asks whether alternative learning-rate and momentum rules perform better. Asks whether fewer optimizer updates lead to computational savings.Tested five alternative learning-rate and second-moment transfer rules at the base horizon. The tuned baseline matched or surpassed each in final validation loss (Appendix A.3). Added three seeds to Figure 3 and added 40B token horizon and 1.8B model size in Figure 1. Clarified that comparisons use the same model and token budget, while fewer updates can offer practical efficiency benefits (higher MFU, greater data parallelization).
z3okCalls Theorems 1 and 2 strong, praises the clear assumptions and Table 1, and acknowledges the new experiments and final-checkpoint fits when raising the recommendation.Experimental scale and statistical uncertainty. Pre-cooldown fits and precision of the fitted exponent. Novelty of weight decay scaling relative to [Bergsma et al.]. Differences between empirical and theoretical exponents. Readability of Definition 3.1 and explanation around it.Added experiments to 1.8B parameters and 40B tokens (Figure 1). Added final-checkpoint fits (Figure 2), three-seed comparisons (Figure 3), direct batch sweeps (Figure 4), and Theorem 6 for cooldown. Clarified how the joint derivation of B∝T2/3B \propto T^{2/3}, K∝T1/3K \propto T^{1/3}, and weight decay scaling differs from empirical fits in [Bergsma et al.]. Revised Definition 3.1 and added a step-by-step scaling explanation with a numerical example (page 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.