- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper derives for an LLM trained with AdamW a scaling law for the training loss and hyperparameter transfer as a function of training tokens. Unlike existing approaches, both the loss scaling law and the hyperparameter transfer are derived from theory, rather than fitted empirically. The derived prescription is to scale batch size as T^(2/3) and keep learning rate and momentum fixed. Experiments on up to 1.4B LLMs provide weak empirical validation of the theory.
Critical strengths
Sweeping hyperparameters over a dense grid search is exponentially expensive. This paper derives hyperparameter transfer rules from theory, obviating the need for expensive scaling experiments.
Beyond hyperparameter transfer, the theory predicts that the loss gap should decay as T^(-1/3), which is empirically verified at roughly 1 OOM scale in Figure 2.
Finally, Table 1 provides a useful survey of existing scaling heuristics, showing just how much disagreement there is in the literature, motivating a more careful study.
Critical weaknesses
My main concerns with this work is the lack of robustness of the empirical results. The empirical results show only that the theory is self-consistent, not that it's optimal.
For example, Figure 2 (top row) shows that if batch size is scaled as T^(2/3), then the optimal LR is roughly constant. However, there is no experiment to justify that this is the globally optimal scaling. It might be the case that under a different batch size and learning rate scaling rule, we get a lower final loss.
Hyperparameters are only swept in isolation along specific 1D and 2D slices (for example [LR x BS] in Figure B.1 and [β1 x β2] in Figure B.4). This would miss any joint scaling rule not captured by these slices. For example [1, 2] show that β2 should be scaled jointly with BS. This paper only sweeps BS without β2 and β2 without BS. This means that any conclusions about the optimality of the found β2 and BS should be questioned – maybe the found β2 is optimal for the given BS and the BS is only optimal for the given β2. Jointly scaling β2 with BS might reveal a better joint optimum. This point is analogous to my concern about the T^(2/3) scaling above.
Further exacerbating this issue is the fact that the literature is conflicted on scaling heuristics. The authors themselves note this in Table 1. It would therefore be extremely useful to have stronger, more discriminative empirical results.
Finally, I find the main comparison of BST against other scaling heuristics (Figure 3) slightly misleading and underwhelming. At a first glance, it looks like BST converges much "faster" than the baselines. However, upon reading the figure more carefully, the x-axis is not compute adjusted. BST merely converges in fewer steps because it uses a larger batch size. The fact that the method with the largest batch size converges first is completely expected. All methods use the same FLOPs. One area where the methods differ is the final validation loss, and BST finishes in the middle.
Since I consider these issues to be more severe than "minor", I rated the soundness with a score of only 1.
Questions for authors
- What's your motivation for scaling batch size? Is the goal to find an "optimum" batch size that minimizes loss, or is the goal to always use the largest batch size smaller than the critical batch size? [3]
- How does a constant β2 square with [1, 2], which suggest scaling β2 with batch size?
References:
- [1] Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful (arXiv:2507.07101)
- [2] The Best Optimizer Depends on Batch Size (arXiv:2610.08975)
- [3] How Does Critical Batch Size Scale in Pre-training? (arXiv:2410.21676)
2 comments
Rebuttals (Part II)
References:
[1] Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful (arXiv:2507.07101)
[2] The Best Optimizer Depends on Batch Size (arXiv:2610.08975)
[3] How Does Critical Batch Size Scale in Pre-training? (arXiv:2410.21676)
Rebuttals
Thank you for your thoughtful review. We are happy to hear your appreciation of our hyperparameter transfer rules derived from theory, obviating the need for expensive scaling experiments, our prediction of the T−1/3 loss gap scaling, and our useful survey of existing scaling heuristics. We now address your main concerns and answer your questions. We would be happy to address any remaining questions.
To address the robustness concerns, we refer to Figure 1, where we estimate the confidence intervals based on a 162M model size. Similarly, we updated Figure 3 and included confidence intervals for the performance of all scaling laws.
To address the optimality concern, we report batch-size sweeps in Figure 4. They show that the BST-recommended batch size lies near the upper end of the empirically optimal range.
Our analysis predicts no faster convergence rate than T−1/3. A smaller batch-size exponent, B∝Tγ with γ<2/3, can retain this rate if the other hyperparameters are adjusted accordingly. Given the agreement with theory in Figure 2, we expect comparable performance to the γ=2/3 rule. We leave the empirical verification of this claim to future work.
We refer to our response to W4 and Q2.
We refer to the general response that summarizes the significance of our experiments. As the reviewer notes, an exhaustive joint sweep grows rapidly with the number of hyperparameters. Even for our 124M- and 162M-parameter base models, it would require substantially more compute than we have available.
To resolve the conflicting guidelines of scaling laws, we study the problem from first principles. Our scaling ladder in Figure 1 demonstrates that the principled BST scaling law is competitive in practice, and improves with model size.
Our main experimental result is presented in Figure 1 (and not in Figure 3), which we additionally expanded with larger-scale experiments: BST’s relative performance against other scaling laws improves as token budgets and model sizes increase, the regimes most relevant in practice.
We respectfully disagree with the characterization “misleading" and "underwhelming” of Figure 1. The right-hand panels of Figure 1 explicitly plot convergence against optimizer steps. Since the scaling laws are compared at the same model and token horizon, the compute adjustments are not needed. The step axis makes the effect of batch size visible: BST uses a larger batch and reaches competitive loss in fewer optimizer updates. A larger batch can improve data parallelization and GPU utilization (higher MFU) when the hardware configuration supports it.
Our analysis identifies the largest batch size that preserves the T−1/3 convergence rate, a theoretical analogue of the critical batch size: smaller batches retain the rate, while larger ones degrade it. As in [3], the resulting rule scales batch size primarily with token horizon.
Larger batches require fewer optimizer steps at a fixed token budget and can improve model FLOPs utilization (MFU), a key consideration in large-scale pretraining. Importantly, our batch-size exponent is derived from the analysis.
The new sweeps in Figure 4 show that the BST batch size performs near the best tested value and lies close to the upper end of the empirically near-optimal range.
To address this concern, we conducted additional experiments with the 124M-parameter base model at a 1.3B-token horizon. We transferred the hyperparameters tuned at B0=128 to B∈64,256,512 using five scaling rules based on [1, 2]:
We report the results in Section A.3 of the revised paper. None of the runs using these rules outperformed the final validation loss of our tuned baseline. This provides further evidence that the baseline is sufficiently well tuned for our experiments.