acceptodds
Under review as a conference paper at ICLR 2027

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging

12 views

Abstract

Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance. Despite this, most existing pretraining recipes are not anytime: they rely on horizon-dependent learning rate schedules and extensive tuning under a fixed compute budget. In this work, we provide a theoretical analysis demonstrating the existence of anytime learning schedules for overparameterized linear regression, and we highlight the central role of weight averaging—also known as model merging—in achieving the minimax convergence rates of stochastic gradient descent. We show that these anytime schedules polynomially decay with time, with the decay rate determined by the source and capacity conditions of the problem. Empirically, we evaluate 150M and 300M parameter language models trained for up to 32 Chinchilla scale, comparing constant and schedules with weight averaging against a well-tuned cosine schedule. Across the full training range, the anytime schedules achieve comparable final loss to cosine decay. Taken together, our results suggest that weight averaging combined with simple, horizon-free step sizes offers a practical and effective anytime alternative to cosine learning rate schedules for large language model pretraining.

Primary area:
optimization
Keywords:
anytime, pretraining, optimization, llm, horizon, free, scheduling, averaging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FH7JOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Large Language Models (LLMs) are increasingly pretrained in continual or open-ended regimes where the total token budget is unknown in advance. However, the standard cosine decay learning-rate (LR) schedule requires specifying the terminal horizon N upfront, and a cosine schedule tuned for a long horizon performs poorly at intermediate checkpoints. While Warmup-Stable-Decay (WSD) serves as a "nearly-anytime" alternative, it requires saving checkpoints at sparse intervals and branching off into separate decay phases. This paper suggests horizon-free schedulers and polynomial decay, combined with iterate weights, allowing anytime large language model pretraining.

Critical strengths
  • The paper tackles a major pain point in modern LLM pretraining—the compute rigidity of cosine decay and the checkpointing/cooldown overhead of WSD in continual pretraining.
  • The authors provide both theoretical and empirical studies to support claims in the research paper.
  • The authors conduct extensive experiments across different scenarios.
Critical weaknesses
  • There is a disconnection between theory (Section 2) and practice (Section 3). In Section 2, the author set up the study with a batch size of 1; however, in Section 3, the batch size is much larger.
  • The theory is investigated under a batch size of 1, which reduces the motivation for the scheduler; especially in practice, the batch size used to pretrain large language models is much higher. For example, GPT -2 [1] uses a batch size of 512 for the pretraining task.
  • The paper should not contain the informal word "Chinchilla" at the start without any explanation. This should be used later in the introduction section with a clear definition, which will prevent the reader from getting lost.
Questions for authors
  • Can the author justify the use of batch size 1 for theoretical study while using a much larger one for empirical study?
  • The practical 1/t1/\sqrt{t} schedule relies on a tuned offset α\alpha in ηt=ηα/(t+α)\eta_t = \eta\sqrt{\alpha/(t+\alpha)}, and Tables 3 and 4 show that the largest tested value (α=51,200\alpha = 51,200) is always preferred for horizons of 4×4\times Chinchilla and above. If you test even larger values of α\alpha, does 1/t1/\sqrt{t} simply reduce to the Constant + EMA schedule? Given that Constant + EMA already achieves nearly identical loss to 1/t1/\sqrt{t} + EMA, is polynomial decay actually necessary in practice?
4 comments

We address the critical weaknesses first:

disconnection between theory (Section 2) and practice (Section 3)... The theory is investigated under a batch size of 1,

We would like to point out that we wrote the theory in batch size 11 for simplicity, but the extension to arbitrary batch size is straightforward and does not impact the results at all. Our analysis uses the same methodology as [1] and [2], where you can see that the only difference would be a constant factor of 1/B1/B appended in several places in the bound. To be precise, line 887 in the manuscript would be replaced by:

mt+1=[I−2ηtΛ+ηt2(1+1B)Λ2+ηt2Bλλ⊤]mt+ηt2Bσ2λm_{t+1} = \left[ I - 2\eta_t \Lambda + \eta_t^2 \left(1 + \frac{1}{B}\right) \Lambda^2 + \frac{\eta_t^2}{B} \lambda \lambda^\top \right] m_t + \frac{\eta_t^2}{B} \sigma^2 \lambda

following Appendix B in [2]. As this is a simple constant factor, it does not change our theoretical results at all. We are happy to rewrite the bound in terms of arbitrary batch size BB for the final version of the paper, and we would like to emphasize that presenting SGD analysis at batch size 11, when the batch size is held constant during training, is a common practice in literature [3, 4].

The paper should not contain the informal word "Chinchilla" at the start 

Thank you for pointing this out and we apologize for not specifying. We will introduce the notion of compute optimal pretraining and the Chinchilla rule in our introduction section. In our context, we are referring to Chinchilla scaling [5], which means that the number of tokens used during training is 20×20 \times number of model parameters.

Can the author justify the use of batch size 1 for theoretical study while using a much larger one for empirical study?

We have answered in the weakness section.

Given that Constant + EMA already achieves nearly identical loss 

In the limit, yes for a very large α\alpha the schedule recovers constant with averaging, however note that our contributions are two-fold. First, we theoretically extend the regime for horizon free schedules in regression with power law source and capacity covariance, from b≤ab \leq a regime established by [1] which study constant with averaging, to α≤β≤2α\alpha \leq \beta \leq 2\alpha, for which 1/tγ1/t^\gamma for γ=1−a/b\gamma = 1-a/b is optimal and anytime. Second, from an empirics perspective, we show that these 2 viable horizon-free schedulers achieve competitive performance in pretraining, matching a well tuned cosine envelope.

Furthermore, we believe that learning how to choose the scheduler is an important future work, and requires taking into account nonconvex phenomena such as progressive sharpening [8], and the effect of sharper minima on posttraining. In our work, both schedulers perform similarly for pretraining loss, however recent work has suggested a correlation between small learning rates and sharper minima, which leads to worse post trainability [6, 7], thus it would seem plausible that less decay is better for end-to-end LLM training.

[1] Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham M. Kakade. How Does Critical Batch Size Scale in Pre-training? ICLR 2025.

[2] Meterez, Alexandru, et al. "A simplified analysis of sgd for linear regression with weight averaging." arXiv preprint arXiv:2506.15535 (2025).

[3] Zou, Difan, et al. "Benign overfitting of constant-stepsize sgd for linear regression." arXiv preprint arXiv:2103.12692 (2021).

[4] Lin, Licong, et al. "Scaling laws in linear regression: Compute, parameters, and data." Advances in Neural Information Processing Systems 37 (2024): 60556-60606.

[5] Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).

[6] Watts, Ishaan, et al. "Sharpness-aware pretraining mitigates catastrophic forgetting." arXiv preprint arXiv:2605.02105 (2026).

[7] Yano, Kazuki, et al. "Pre-training LLM without learning rate decay enhances supervised fine-tuning." arXiv preprint arXiv:2603.16127 (2026).

[8] Cohen, Jeremy M., et al. "Gradient descent on neural networks typically occurs at the edge of stability." arXiv preprint arXiv:2103.00065 (2021).

Thank you, authors, for your response.

What if the batch size becomes very large?

ηt2(1+1B)→ηt2\eta_t^2(1 + \frac{1}{B})\rightarrow\eta_t^2 and ηtB→0\frac{\eta_t}{B}\rightarrow0, which putting low effect on the learning rate, respectively. Might this phenomenon affect the behaviour of your proposed method, right?

I think the theoretical analysis needs to investigate this perspective as well.

Best

Thank you for further engaging in the rebuttal process. In the infinite batch limit, the process becomes GD, as opposed to SGD. For GD, no decay or averaging is necessary, and one simply needs to use a fixed learning rate \eta = 1/\lambda_\max in order to converge. We exemplify this:

mt+1=(I−2ηtΛ+ηt2Λ2)mt=(I−ηtΛ)2mtm_{t+1} = (I - 2 \eta_t \Lambda + \eta_t^2 \Lambda^2)m_t = (I - \eta_t \Lambda)^2 m_t Moreover, as there is no noise, to get excess risk convergence we just want to ensure contraction in each eigendirection, for which a constant \eta = 1/\lambda_\max suffices (Corollary 2.1.2 in [2]).

We focus our analysis on SGD as this is the setting where practical optimization operates, through minibatch stochastic gradient descent. Moreover, in our experiments we set the batch size to be the critical batch size [1] for our model size and data size, so the noise and bias term have comparable contribution in the recursion.

[1] Zhang, Hanlin, et al. "How does critical batch size scale in pre-training?." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Nesterov, Yurii. Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, 2013.

Thank you, authors, for your response. The response does answer my question. I will raise my score.

However, since I am not working in this field, Area Chair, please give my review a small weight.

Reviewer_AoCPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies learning-rate schedules that do not require a fixed training horizon. It derives a risk bound for SGD with iterate averaging in Gaussian linear regression. Under power-law source and capacity assumptions, it obtains the decay exponent (\gamma^\star=1-a/b) and an optimal risk exponent up to logarithmic factors. Experiments on 150M- and 300M-parameter language models show that constant and inverse-square-root learning rates, combined with weight averaging, can remain competitive with cosine schedules tuned separately for each training horizon.

Critical strengths

The transfer experiment supports extending training without retuning the schedule. Hyperparameters selected at 4× Chinchilla remain competitive with cosine schedules tuned separately for later horizons. The selection uses only losses available at 4×. This makes the result useful when the final training budget is unknown.

Critical weaknesses

Hägele et al. already study constant learning rates with weight averaging for LM pretraining without a fixed horizon. This paper extends that work to inverse-square-root decay, a growing averaging window, and longer training horizons. These experiments are useful, but they leave unclear how much of the improvement comes from the schedule or the averaging window.

There are two substantial gaps between the theoretical assumptions and the LLM experiments. First, the power-law convergence result assumes a source exponent β>1\beta>1, whereas Meterez et al. report estimates below 1 for LLM pretraining. Although those estimates concern local, preconditioned quadratic models, they raise doubts about whether the assumed regime describes the experiments. Second, the theory analyzes SGD, while the experiments use AdamW. SGD updates scale with gradient magnitude; AdamW rescales gradients using their estimated second moments. This difference can change how the learning rate should decrease during training. The SGD analysis therefore does not establish that the proposed schedule is appropriate for AdamW. These gaps limit the theory’s relevance to the empirical results.


References

Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. NeurIPS 2024.

Meterez et al. A Defense of the Quadratic Model. 2026.

Questions for authors

Can you separate the effects of inverse-square-root decay and the growing averaging window? In particular, how do constant and inverse-square-root learning rates compare under both fixed and growing averaging windows, with comparable tuning budgets?

4 comments

We address the reviewer’s comments pointwise:

These experiments are useful, but they leave unclear how much of the improvement comes from the schedule or the averaging window.

From a theoretical point of view, averaging is necessary in order to obtain the best rates for constant learning rate and 1/t1/\sqrt{t} decay. Without averaging, none of these schedules have optimal sample complexity rates - and constant would not even converge. Regarding the window itself, for the experiments present in the paper we have kept EMAs over the last 1,2,4,8,161, 2, 4, 8, 16% of iterates, as well as last iterate, and plotted the best one, which in this case was 44%. In line with the theoretical analysis, across all windows there is an improvement over the last iterate loss:

ModelScheduleHorizonLast iterateEMA 1%EMA 2%EMA 4%EMA 8%EMA 16%
150MConstant32×2.779792.749712.748192.747652.748972.75616
150M1/t1/\sqrt{t}32×2.771202.750082.749012.748852.750822.75885
300MConstant16×2.665752.633642.631762.631572.636382.66125
300M1/t1/\sqrt{t}16×2.652182.630492.629042.628502.629642.63401

Also, note that our guarantees require tail averaging (in line with prior work on linear regression theory [1], [2], [3]), which is a fixed fraction of the dataset. Thus, to ensure this requirement, it is natural to consider an EMA window that grows as a fraction of the number of samples.

Regarding the gaps mentioned:

First, the power-law convergence result assumes a source exponent (\beta>1), whereas Meterez et al. report estimates below 1 for LLM pretraining. 

From a theory perspective, the goal of our work is to extend the regime for horizon free schedules in regression with power law source and capacity covariance, from b≤ab \leq a regime established by [1], to α≤β≤2α\alpha \leq \beta \leq 2\alpha. Having α,β≥1\alpha, \beta \geq 1 is a common assumption in these types of analyses (e.g. [2], [4], [5]) as it allows the tails of the power law to be bounded by a constant. Indeed, extending to β≤1\beta \leq 1 is an interesting future work, because now there would be a dependency on the number of parameters coming from the tail sums, and an interaction between samples and number of params coming from the Chinchilla scaling. Finally, we’d like to conjecture that while β≤1\beta \leq 1 is indeed not covered by our theory, it is not clear whether the β\beta observed by Meterez et al. is the theoretical correspondent of the power law in w∗w^*, or if it contains hidden dependencies coming from momentum, preconditioning, and EMA.

Second, the theory analyzes SGD, while the experiments use AdamW. 

We do not claim that the theory perfectly captures the empirical setup - on the contrary, we work in a toy theoretical setup in order to understand if horizon free learning rate schedules exist and and how do they perform in this toy setup, and then we make a leap to empirics in LLM pretraining. Note however that this is a standard approach and numerous papers have shown that quadratics are a representative toy model for neural network training, and even LLM pretraining: [1], [4], [6], [7], [8]. Thus, while the agreement is indeed surprising and we do not claim to explain why this simplified theory immediately transfers to LLMs and believe that this is a fundamental future work, we do emphasize that prior work has shown that such a simplified setting carries a lot of predictive power, which also applies for our results showing that our schedules are indeed a competitive horizon free alternative evaluated against the cosine envelope.

Can you separate the effects of inverse-square-root decay and the growing averaging window? In particular, how do constant and inverse-square-root learning rates compare under both fixed and growing averaging windows, with comparable tuning budgets?

We refer the reviewer to the explanation provided in the weaknesses section. Due to time constraints for the rebuttal period, we cannot rerun these baselines.

[1] Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham M. Kakade. How Does Critical Batch Size Scale in Pre-training? ICLR 2025.

[2] Zou, Difan, et al. "Benign overfitting of constant-stepsize sgd for linear regression." arXiv preprint arXiv:2103.12692 (2021).

[3] Jain, Prateek, et al. "Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification." Journal of machine learning research 18.223 (2018): 1-42.

[4] Bordelon, Blake, and Francesco Mori. "Theory of optimal learning rate schedules and scaling laws for a random feature model." arXiv preprint arXiv:2602.04774 (2026).

[5] Lin, Licong, et al. "Scaling laws in linear regression: Compute, parameters, and data." Advances in Neural Information Processing Systems 37 (2024): 60556-60606.

[6] Meterez, Alexandru, et al. "Seesaw: Accelerating training by balancing learning rate and batch size scheduling." arXiv preprint arXiv:2510.14717 (2025).

[7] Meterez, Alexandru, et al. "A defense of the quadratic model." arXiv preprint arXiv:2607.21716 (2026).

[8] Li, Binghui, et al. "Functional scaling laws in kernel regression: Loss dynamics and learning rate schedules." Advances in Neural Information Processing Systems 38 (2026): 101211-101269.

Thank you for the response and the additional EMA results.

I agree that averaging is necessary for optimal rates, but this is well known and was not my question. I asked how much of the improvement comes from the schedule and how much from the averaging window. The new table compares growing EMA windows against the last iterate, with no fixed-window baseline (as in Hägele et al.), so it does not separate the two effects. I understand the rebuttal time constraints, but the question remains open.

The main value of a quadratic-model analysis in a paper like this should be a prediction that the LLM experiments test, or a new insight into practice. The new result is that 1/tγ⋆1/t^{\gamma^\star} with γ⋆=1−a/b\gamma^\star = 1 - a/b is horizon-free and rate-optimal for a<b≤2aa < b \le 2a, extending Zhang et al. (2024b). However, the experiments do not test this. γ=1/2\gamma = 1/2 comes from a separate heuristic, and constant learning rate with averaging, already covered by prior theory, performs on par with 1/t1/\sqrt{t}.

What does your analysis predict about LLM pretraining that was not already known, and which experiment tests it?

Thanks for the follow up!

I asked how much of the improvement comes from the schedule and how much from the averaging window. The new table compares growing EMA windows against the last iterate, with no fixed-window baseline (as in Hägele et al.), so it does not separate the two effects. I understand the rebuttal time constraints, but the question remains open.

We would like to point out that our results implicitly cover the fixed-window baseline, albeit not a uniformly weighted average. For any horizon TT, as the window is a percentage of TT, one can compare fixed windows across the different 1-16% EMA values we have attached in the table. So comparing, for example, at T equal to 32x Chinchilla (roughly 105B tokens in our setup), the window sizes are ~1B (1%), up to ~16B (16%) tokens, plus the the last iterate baseline.

Morevoer, we also provide synthetic experiments in Figure 3, where we do full iterate averaging from the beginning of training, showing that the increasing EMA window is not what is necessary for polynomial decay to achieve the predicted exponent.

The main value of a quadratic-model analysis in a paper like this should be a prediction that the LLM experiments test, … What does your analysis predict about LLM pretraining that was not already known, and which experiment tests it?

Our analysis predicts that for quadratics, a polynomial decay schedule is horizon-free, and optimal, for capacity aa and source bb, such that a<b≤2aa < b \leq 2a. We agree that γ=½\gamma = ½ is a heuristic choice, and it’s possibly suboptimal, and one would need to sweep over it as an extra hyperparameter. Our experiments are designed to show that the theoretical predictions made by the quadratic model surprisingly transfer to LLM pretraining even when evaluated under a very hard baseline e.g. the cosine envelope, and moreover to test the 2 possibilities for horizon-free training predicted by the linear regression with power law covariance analysis: polynomial decay (for a<b≤2aa < b \leq 2a, our theoretically novel result) and constant with averaging (for b≤ab \leq a, covered by Zhang et al for this class of problems), in Figure 2.

Moreover, we also stress test the prediction from the quadratic model in Figure 5, for large batch sizes. For quadratics, For very large batch, the variance terms start have vanishingly small contribution, and the recursion starts being dominated by the bias term, which would indicate no need for decay. This is consistent with the observation from Figure 5, which shows that all schedules outperform cosine after the 1x horizon, and the constant learning rate schedules outperform the polynomial decay.

Reviewer_2aMdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a horizon-free learning rate schedule with the weight-average technique. The method does not depend on a predefined training horizon and can achieve training efficiency matching the envelope of the cosine LR schedules. This paper also provides a theoretical analysis in power spectrum scenario.

Critical strengths
  1. This paper solves a very practical and important problem, that the training horizon is not always defined before training starts.
  2. It provides theoretical analysis and compared with a strong baseline, the loss envelope of cosine schedule training.
Critical weaknesses
  1. Further ablation studies are needed, such as the Muon optimizer and other datasets, and some important baselines need to be compared.
  2. The impact of horizon-free schedules is unclear, such as how to decide whether to use a constant LR or a 1/t schedule, or another horizon-free LR schedule?
Questions for authors
  1. If we keep ending LR=1e-5 for the baseline, can the horizon-free method still match the baseline envelope?
  2. How to decide whether to use a constant LR or a 1/t schedule, or another horizon-free LR schedule?
  3. Can you provide more ablation studies, like (1) on the Muon optimizer, (2) larger models but with less data (in case you have limited compute)?
  4. Can you provide a comparison with some similar and important work like [https://arxiv.org/abs/2405.15682], and explain the advantage?
3 comments

We first begin by addressing the critical weaknesses:

  1. We do not cover the Muon optimizer and focus only on Adam due to limited compute, and because it is standard in large scale training of models. Note however that theoretically, we analyze SGD on linear regression, and make no assumptions on the preconditioner, so we conjecture the results for Muon to be similar to Adam. From an empirics point of view, we wanted to show that there exist horizon-free alternatives to cosine that are able to compete with cosine across a wide range of horizons, broadly based on doing iterate EMA/averaging. In fact, note that for Muon, related literature has also shown that averaging can achieve horizon free optimal rates [1], both in theory and in practice.

  2. The main point of our work is not to show how to choose a horizon-free schedule, but to show that these alternatives to a horizon-tuned cosine exist, and are competitive. We believe that learning how to choose the scheduler is an important future work, and requires taking into account nonconvex phenomena such as progressive sharpening [2], and the effect of sharper minima on posttraining performance. In our work, both constant with averaging and 1/(t)1/\sqrt(t) with averaging perform comparably for pretraining loss, however recent work has suggested a correlation between small learning rates and sharper minima, which leads to worse post trainability [5, 6]. In such scenarios it would appear less decay is an appealing choice.

Regarding the questions:

  1. We tune our baselines extensively, with the sweeps being shown in Section 3. Due to limited time for the rebuttals, we cannot rerun the entire sweep with a smaller final learning rate for cosine, but we would like to point out that decaying to 10% of the peak learning rate is standard in literature, following the convention used in the Chinchilla scaling experiments.

  2. We have answered in the weaknesses section.

  3. While we cannot provide more ablation studies during the rebuttal period, but we refer the reviewer to to related work [1] for an anytime study on Muon. Regarding more data, our experiments are up to 32×32\times Chinchilla (roughly 100B tokens) for 150M and up to 16×16 \times Chinchilla (roughly 50B tokens) for 300M, which is much larger than any previous existing studies.

  4. Prior work [3] has shown that the Schedule-Free update can be rewritten into a combination of weight averaging (which is one of the main ingredients in our rates and empirics, making the anytime schedule work), and accelerated-SGD (which is a 2 buffer momentum, used to accelerate at smaller batch sizes). Our work extends the theoretically known regime for anytime optimal schedules on quadratics with power law data with source and capacity exponents bb and aa, from b≤ab \leq a established in [4] (for which constant with averaging is optimal), to a<b≤2aa < b \leq 2a (for which 1/t1/\sqrt{t} with averaging is optimal).

[1] Apte, Anuj, et al. "Anytime Training with Schedule-Free Spectral Optimization." arXiv preprint arXiv:2605.23061 (2026).

[2] Cohen, Jeremy M., et al. "Gradient descent on neural networks typically occurs at the edge of stability." arXiv preprint arXiv:2103.00065 (2021).

[3] Morwani, Depen, et al. "Connections between schedule-free optimizers, ademamix, and accelerated sgd variants." arXiv preprint arXiv:2502.02431 (2025).

[4] Zhang, Hanlin, et al. "How does critical batch size scale in pre-training?." International Conference on Learning Representations. Vol. 2025. 2025.

[5] Watts, Ishaan, et al. "Sharpness-aware pretraining mitigates catastrophic forgetting." arXiv preprint arXiv:2605.02105 (2026).

[6] Yano, Kazuki, et al. "Pre-training LLM without learning rate decay enhances supervised fine-tuning." arXiv preprint arXiv:2603.16127 (2026).

Thanks for your rebuttal. I will keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.