acceptodds
Under review as a conference paper at ICLR 2027

Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons

22 views

Abstract

Learning-rate warmup is a standard technique in language-model training, yet its duration remains largely heuristic. Common approaches use either a fixed number of updates or a fixed fraction of the training horizon, two choices that imply very different scaling as training gets longer. When should warmup stay fixed, and when should it grow with the horizon? We address this question with a quadratic model whose modes respond differently to the peak learning rate. Warmup slows progress in directions that already contract well at the peak rate, but can remove persistent error in directions near the stability edge, with higher peak rates shifting the balance toward longer warmup durations. This yields a compact horizon scaling law that captures regimes ranging from essentially no warmup, through fixed-duration warmup, to durations that grow with the training horizon, and explains how the preferred regime changes with peak learning rate. Because the law captures the tradeoff between giving up early progress and improving the trajectory that follows, it can be fit using shorter runs and used to predict warmup at substantially longer horizons. Together, our results explain several familiar properties of warmup through a single tradeoff and suggest treating warmup duration as a horizon-dependent hyperparameter rather than a fixed training heuristic.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Learning-rate warmup, Optimization, Language model training, Scaling laws, Hyperparameter tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hZe8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how learning-rate warmup duration should change with the planned training horizon. It develops simplified theoretical models of the tradeoff between slower initial progress and persistent benefits from warmup, then proposes a parametric loss function over training horizon and warmup duration. The function is fitted to observed training trajectories and used to recommend warmup durations for longer runs.

Critical strengths

1 A relevant and clearly defined problem.

Choosing warmup duration for a long training run from inexpensive preliminary experiments is practically useful. The paper distinguishes the warmup needed to avoid divergence from the warmup that achieves the best eventual loss, and investigates how this distinction depends on the training horizon.

2 Substantial controlled experimentation.

The evaluation covers 33 model-size/peak-learning-rate combinations, each with multiple warmup settings, together with additional batch-size, warmup-shape, optimizer, and decay experiments. These experiments provide meaningful evidence for the reported tradeoffs within the tested configurations.

Critical weaknesses
  1. The specific fitted function is only partially justified by the theory.

Its persistent-error component relies on an assumed distribution of effective relaxation rates, followed by a local approximation that produces the multiplicative power-law form. These assumptions are not directly established for the trained Transformers. Moreover, the practical fit allows exponents to vary independently across configurations. The theoretical results therefore support a candidate predictive model, but do not establish a general law of neural-network training. The default parameter bounds also constrain the inferred asymptotic behavior, so fitted growth trends should not be interpreted as entirely determined by the observations.

  1. Validation at larger model scales is missing.

The main experiments use approximately 60M–350M parameter models, and the longest-horizon validation uses only a 100M model. The training trajectories are substantial—approximately 16.8B tokens in the main setting and up to 32.8B tokens in the extended experiment—but they establish temporal extrapolation primarily within small-model settings. The single corpus, short training context, and zero-weight-decay recipe further limit the evidence for broader applicability. A multi-billion-parameter experiment would considerably strengthen the practical claims.

Questions for authors

1 Can the recommendations be validated on a substantially larger model using only short prefixes for fitting and competitive tuned warmup baselines?

2 How stable are the recommended warmup durations across random seeds, fitting windows, and relaxed exponent bounds?

2 comments

We thank the Reviewer for the comments. We have incorporated changes into our current draft and respond below.

Weakness 1. The quadratic is not meant as an accurate description of Transformer dynamics, but as a model of the trade-off that warmup duration creates. It motivates our law because it captures the peak-rate-dependent regimes we observe. The exponents vary across peak rates by design, since each configuration is fit from its own runs. Fixing s=1/2s=1/2 gives similar selection regret (Table 23), and sharing q+sq+s across peak rates keeps fit quality nearly unchanged (Table 25). We address the exponent bounds under Q2.

Weakness 2 and Q1. Our scope is scaling across horizons. The best warmup duration depends on the training horizon, and that dependence is the subject of the paper. Transfer across model scale is outside this scope, although Appendix E.7 gives some insight. To test the method at a larger size, we added a horizon-extrapolation experiment with a 600M model and 0.1 weight decay, fitting only on short prefixes and comparing against baselines from the same runs.

600M model, peak LRs {5e-4, 3e-3}, fitting runs up to 50k updates. Validation loss at each target horizon, averaged over the two peak learning rates.

Horizon200k400k
1k warmup2.9912.958
10% warmup2.9852.951
Ours2.9632.940

Q2. Fitting-window dependence is in Appendix E.5. Our Table 2 results already average over 3 seeds. To further test sensitivity to seeds, we refit on two different pairs of seeds. We also relax the exponent bounds to [0.05,5][0.05, 5].

Full-data fits as in Table 2. R2R^2 and RMSE are medians across families; regret is the mean loss above the best tested warmup. RMSE and regret in 10−310^{-3} loss units.

FitMedian R2R^2RMSEMean regret
Default: all seeds, bounds [0.05,1][0.05, 1]0.9964.922.00±1.512.00 \pm 1.51
Seed pair A0.9964.912.07±1.422.07 \pm 1.42
Seed pair B0.9964.941.94±1.591.94 \pm 1.59
Bounds [0.05,5][0.05, 5]0.9955.662.60±1.812.60 \pm 1.81

Thanks for the feedback. When asking results for larger models, it does not mean I wish the result is transferrable. Instead, my question is if the modeling approach proposed in this paper will work for larger, practical models. The 600M experiments is a one step towards this direction and is appreciated. It is understoodable that academia may not have the compute power to train larger models, so I am ok with this point.

However, in this experiment, you compare with 1k and 10% warmup updates, but I have asked to have comparison with tuned warmup baselines, which you have done in your small scale experiments, but is missing in the longest validation run.

Reviewer_Cpp4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper examine how the optimal learning rate warmup duration varies with the total training duration and peak learning rate. Using a quadratic model, the paper illustrate the trade-off between slow initial learning and error reduction later in training, and propose a loss model to predict the optimal warmup period. Experiments demonstrate that fitting this model to multiple short runs helps in selecting the warmup period for longer runs.

Critical strengths
  • The paper provides a systematic analysis of the interaction between the warmup duration, peak learning rate, and training horizon, offering a theoretical model based on a quadratic model.
  • The proposed loss model has been evaluated beyond its fitting range. Experiments using three runs show performance that outperforms simple rules based on the best short-run configuration, and experiments with long horizons involve training new runs using the predicted warmup duration. This provides evidence that the model can guide warmup selection rather than merely describing existing loss curves.
Critical weaknesses
  • The practical computational benefits are unclear. The method performs preliminary runs for each setting of model size and peak learning rate. While this is less costly than an exhaustive search over long runs, which consumes additional computation beyond the target runs, the reported loss improvements do not establish whether this method is preferable under the same total computational budget that includes these preliminary runs.
  • It is difficult to assess the practical significance of the loss improvements. Regarding the reduction in validation loss, the improvement over competing baselines is small, and the paper does not demonstrate whether this leads to meaningful performance gains in downstream tasks or to improved performance on post-trained models.
Questions for authors

See weaknesses.

2 comments

We thank the Reviewer for the comments. We have incorporated changes into our current draft and respond below.

Weakness 1. As in all hyperparameter scaling works, small trial runs pay off because the right setting is not known in advance. A fixed rule costs nothing, but no single rule works across peak rates. Our method needs three trial runs of as few as 50k updates. Nonetheless, to compare at equal compute, we give each fixed rule the extra 150k updates our trial runs cost.

Loss gap to our method, 100M model (positive = ours better).

BaselineOurs at 500k vs. baseline at 650kOurs at 850k vs. baseline at 1M
1k warmup, mean over peak LRs+0.019+0.023
10% warmup, mean over peak LRs+0.002+0.010

Even after paying for its trial runs, our method is ahead on average, and its lead grows with the horizon because the trial cost is fixed.

Weakness 2. Our goal is not to beat a fully tuned warmup, which is the ceiling we measure regret against, but to get close to it cheaply. With a full warmup grid, simple tuned rules can match our law (Table 16), but running that grid is what we want to avoid. To also test downstream performance, we evaluated the 1M-update models of Table 5 zero-shot on five downstream tasks.

Difference to our method (75k trial runs) at 1M updates, mean over the three peak LRs (positive = ours better).

BaselineVal. lossLAMBADAHellaSwagPIQAARC-EasySciQAvg.
1k warmup+0.028+2.6+0.6+0.8+1.1+1.0+1.2
10% warmup+0.014+0.9+0.5+0.7+0.4+0.7+0.6

Thank you for the additional experiments! The compute comparison addresses my concern about the cost of the preliminary runs, and the downstream results provide evidence that the gains extend beyond validation loss. I will raise my soundness accordingly.

My remaining concern is about the practical significance of the gains. The new results show improvements in pre-trained models, but it remains unclear whether these differences persist after the post-training procedure and lead to better performance on generation tasks. I do not view this as a flaw in the warmup analysis, or post-training evaluation as a requirement for its validity. However, such evidence would make the practical contribution more convincing to me.

I therefore maintain my current overall rating, while acknowledging that the response has addressed concern.

Reviewer_MVvpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the need for warmup in pretraining across several model sizes, and shows how warmup length should scale across duration and the interplay with the peak learning rate.

Critical strengths

The main strengths of the paper are its empirics. The authors clearly show empirically that longer warmup allows for larger peak learning rates (in line with previous literature), and while this sacrifices some initially progress (due to the warmup itself), it is better in the long run since it allows larger lrs to be stable. In general, this paper has a fairly strong empirical setup and interesting ablations.

Critical weaknesses

Unfortunately, where this paper falls short (and it is the main reason behind my grade) is in the theoretical setup. My gripe is with the authors trying to use the quadratic model to explain the need for warmup, which I claim is a fundamentally wrong point of view.

First, in section 3, the authors study GD for a quadratic model, and build their theory for why warmup is useful in this setup. This is incorrect, because for GD, the contractions (where by contraction I mean let et=wt−w∗e_t = w_t - w^* be the error vector, and the linear recursion it follow is et+1=(I−ηtH)ete_{t+1} = (I - \eta_t H) e_t) have the same left and right basis, and thus commute. Therefore, one can reorder the learning rates in any way and the final iterate loss will be the same, thus any sort of motivation for warmup does not go through. The authors do mention this in the paragraph from line 263, but then why add this whole section to begin with?

Second, the authors do also go through an example in B.5 for SGD with multiplicative noise in quadratics. However, the example they construct is very contrived: they create a 2D problem (line 1333) that has no initial error in the bottom (small) eigendirection. In this setup because of the noise the modes couple, so the commuting argument fails. However, the reason for warmup helping in this example is because there is initially no error in the bottom, and then trainign with a constant lr introduces noise there (while making progress in the top), whereas an initially smaller lr introduces less noise (at the cost of less progress in the top direction).

I am calling this example artificial because the assumption of having no initial error in one direction is very strong and asymptotically both of these schedulers are the same. Moreover, if noise is the concern, then a more fair comparison would be to compare warmup+constant vs constant with averaging (or decay). With additive noise, warmup+constant would stagnate at the noise floor and not even converge to the minimizer.

Questions for authors

Would you consider revamping the paper into an empirical paper, and removing (or coming up with a better model) the theoretical sections? In my opinion this would add a lot of value as the empirics are solid.

3 comments

We thank the Reviewer for the comments. We have incorporated changes into our current draft and respond below.

We intended the paper to be empirical, our goal is not to explain why warmup is needed, recent work already addresses that question (Alimisis et al., 2026). Our focus is instead on how long warmup should last. The theory exists to capture the trade-off of longer warmup durations and to inspire the development of the scaling law. We have adjusted the wording accordingly in the text and shortened the theory section.

  • We need a model of the trade-off that warmup length controls. Our experiments show that the loss-optimal warmup depends on the horizon and the peak learning rate. To predict it, we need a model of what a longer warmup gives up (early progress) and what it buys (lower error later).

  • A quadratic with an ordered schedule captures this trade-off. As you note, since its updates commute, it cannot explain why small learning rates should come first. But this is not our intention, once the schedule is ordered, as in warmup-stable, it does show how the best length of the off-peak phase depends on the horizon and the peak rate and it reproduces the regimes we observe, which is our goal.

  • The final scaling law is built on this trade-off and not on the quadratic dynamics. The progress term is the power law in accumulated learning rate, which is independent of the order and already used for LLM loss curves (Luo et al., 2025).

Because the updates commute, the quadratic cannot distinguish whether off-peak steps occur at the beginning or end of training. However, our goal is not to predict the location of the trade-off, but to explain why it exists. Our experiments show that this trade-off depends on warmup length, which is why we focus on warmup. If a similar dependence were observed for the decay length, a similar reasoning would apply.

So, we use the quadratic model to underpin the law's functional form, which depends only on the accumulated learning rate and WW. Section 3 provides a simple characterization of the empirical warmup-duration trade-offs; as you point out, it does not describe the actual training dynamics. It does, however, capture how large the set of off-peak step sizes should be, which we believe is still an interesting result worth keeping. In addition, we agree that the B.5 example is contrived. As it was never meant to be a main result and was only added in the appendix as a complement, we have decided to remove it.

Reviewer_MVvpOpenReview ↗

Rebuttal reply

Thank you for the reply. However, I still disagree with your arguments regarding the quadratic explaining warmup.

Because the updates commute, the quadratic cannot distinguish whether off-peak steps occur at the beginning or end of training. However, our goal is not to predict the location of the trade-off, but to explain why it exists. 

The off-peak steps need not occur at the beginning or the end of training. Because of the commutativity, any permutation of the learning rate schedule ηt\eta_t will achieve the same risk in this setup - it need not even be monotonic.

Moreover, the example in Section 3, line 209, is for deterministic ("full batch") GD on a quadratic with Hessian HH. This model does not capture any need for warmup or decay, as there is no noise present. For a constant learning rate, the optimal excess risk rate is achieved for \eta^* = 2/(\lambda_\max + \lambda_\min) (or \eta = 1/\lambda_\max is sufficient for convergence), without having access to additional spectral information. Allowing nonconstant lrs, one can use Chebyshev step sizes to recover the accelerated rate for GD.

We agree that the quadratic does not justify warmup and this was never our intention, the revised text no longer claims it does. Independent of the order of the step sizes, the quadratic shows how the number of off-peak steps should depend on the horizon and the peak rate.

This is consistent with deterministic-GD, as below ηc=2/(λmin⁡+λmax⁡)\eta_c = 2/(\lambda_{\min}+\lambda_{\max}) no warmup is needed at long horizons. A warmup (or a set of off-peak rates) that grows with the horizon appears only when the peak rate is fixed above it and the horizon is finite.

Since we observe the same tradeoff empirically for warmup length in neural networks, we use it to inspire the a law for the length of warmup, not warmup itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.