- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Large Language Models (LLMs) are increasingly pretrained in continual or open-ended regimes where the total token budget is unknown in advance. However, the standard cosine decay learning-rate (LR) schedule requires specifying the terminal horizon N upfront, and a cosine schedule tuned for a long horizon performs poorly at intermediate checkpoints. While Warmup-Stable-Decay (WSD) serves as a "nearly-anytime" alternative, it requires saving checkpoints at sparse intervals and branching off into separate decay phases. This paper suggests horizon-free schedulers and polynomial decay, combined with iterate weights, allowing anytime large language model pretraining.
Critical strengths
- The paper tackles a major pain point in modern LLM pretraining—the compute rigidity of cosine decay and the checkpointing/cooldown overhead of WSD in continual pretraining.
- The authors provide both theoretical and empirical studies to support claims in the research paper.
- The authors conduct extensive experiments across different scenarios.
Critical weaknesses
- There is a disconnection between theory (Section 2) and practice (Section 3). In Section 2, the author set up the study with a batch size of 1; however, in Section 3, the batch size is much larger.
- The theory is investigated under a batch size of 1, which reduces the motivation for the scheduler; especially in practice, the batch size used to pretrain large language models is much higher. For example, GPT -2 [1] uses a batch size of 512 for the pretraining task.
- The paper should not contain the informal word "Chinchilla" at the start without any explanation. This should be used later in the introduction section with a clear definition, which will prevent the reader from getting lost.
Questions for authors
- Can the author justify the use of batch size 1 for theoretical study while using a much larger one for empirical study?
- The practical schedule relies on a tuned offset in , and Tables 3 and 4 show that the largest tested value () is always preferred for horizons of Chinchilla and above. If you test even larger values of , does simply reduce to the Constant + EMA schedule? Given that Constant + EMA already achieves nearly identical loss to + EMA, is polynomial decay actually necessary in practice?
4 comments
Thank you, authors, for your response.
What if the batch size becomes very large?
and , which putting low effect on the learning rate, respectively. Might this phenomenon affect the behaviour of your proposed method, right?
I think the theoretical analysis needs to investigate this perspective as well.
Best
Reply
Thank you for further engaging in the rebuttal process. In the infinite batch limit, the process becomes GD, as opposed to SGD. For GD, no decay or averaging is necessary, and one simply needs to use a fixed learning rate \eta = 1/\lambda_\max in order to converge. We exemplify this:
Moreover, as there is no noise, to get excess risk convergence we just want to ensure contraction in each eigendirection, for which a constant \eta = 1/\lambda_\max suffices (Corollary 2.1.2 in [2]).
We focus our analysis on SGD as this is the setting where practical optimization operates, through minibatch stochastic gradient descent. Moreover, in our experiments we set the batch size to be the critical batch size [1] for our model size and data size, so the noise and bias term have comparable contribution in the recursion.
[1] Zhang, Hanlin, et al. "How does critical batch size scale in pre-training?." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Nesterov, Yurii. Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, 2013.
Thank you, authors, for your response. The response does answer my question. I will raise my score.
However, since I am not working in this field, Area Chair, please give my review a small weight.
Reply
We address the critical weaknesses first:
We would like to point out that we wrote the theory in batch size 1 for simplicity, but the extension to arbitrary batch size is straightforward and does not impact the results at all. Our analysis uses the same methodology as [1] and [2], where you can see that the only difference would be a constant factor of 1/B appended in several places in the bound. To be precise, line 887 in the manuscript would be replaced by:
mt+1=[I−2ηtΛ+ηt2(1+B1)Λ2+Bηt2λλ⊤]mt+Bηt2σ2λ
following Appendix B in [2]. As this is a simple constant factor, it does not change our theoretical results at all. We are happy to rewrite the bound in terms of arbitrary batch size B for the final version of the paper, and we would like to emphasize that presenting SGD analysis at batch size 1, when the batch size is held constant during training, is a common practice in literature [3, 4].
Thank you for pointing this out and we apologize for not specifying. We will introduce the notion of compute optimal pretraining and the Chinchilla rule in our introduction section. In our context, we are referring to Chinchilla scaling [5], which means that the number of tokens used during training is 20× number of model parameters.
We have answered in the weakness section.
In the limit, yes for a very large α the schedule recovers constant with averaging, however note that our contributions are two-fold. First, we theoretically extend the regime for horizon free schedules in regression with power law source and capacity covariance, from b≤a regime established by [1] which study constant with averaging, to α≤β≤2α, for which 1/tγ for γ=1−a/b is optimal and anytime. Second, from an empirics perspective, we show that these 2 viable horizon-free schedulers achieve competitive performance in pretraining, matching a well tuned cosine envelope.
Furthermore, we believe that learning how to choose the scheduler is an important future work, and requires taking into account nonconvex phenomena such as progressive sharpening [8], and the effect of sharper minima on posttraining. In our work, both schedulers perform similarly for pretraining loss, however recent work has suggested a correlation between small learning rates and sharper minima, which leads to worse post trainability [6, 7], thus it would seem plausible that less decay is better for end-to-end LLM training.
[1] Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham M. Kakade. How Does Critical Batch Size Scale in Pre-training? ICLR 2025.
[2] Meterez, Alexandru, et al. "A simplified analysis of sgd for linear regression with weight averaging." arXiv preprint arXiv:2506.15535 (2025).
[3] Zou, Difan, et al. "Benign overfitting of constant-stepsize sgd for linear regression." arXiv preprint arXiv:2103.12692 (2021).
[4] Lin, Licong, et al. "Scaling laws in linear regression: Compute, parameters, and data." Advances in Neural Information Processing Systems 37 (2024): 60556-60606.
[5] Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).
[6] Watts, Ishaan, et al. "Sharpness-aware pretraining mitigates catastrophic forgetting." arXiv preprint arXiv:2605.02105 (2026).
[7] Yano, Kazuki, et al. "Pre-training LLM without learning rate decay enhances supervised fine-tuning." arXiv preprint arXiv:2603.16127 (2026).
[8] Cohen, Jeremy M., et al. "Gradient descent on neural networks typically occurs at the edge of stability." arXiv preprint arXiv:2103.00065 (2021).