- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how learning-rate warmup duration should change with the planned training horizon. It develops simplified theoretical models of the tradeoff between slower initial progress and persistent benefits from warmup, then proposes a parametric loss function over training horizon and warmup duration. The function is fitted to observed training trajectories and used to recommend warmup durations for longer runs.
Critical strengths
1 A relevant and clearly defined problem.
Choosing warmup duration for a long training run from inexpensive preliminary experiments is practically useful. The paper distinguishes the warmup needed to avoid divergence from the warmup that achieves the best eventual loss, and investigates how this distinction depends on the training horizon.
2 Substantial controlled experimentation.
The evaluation covers 33 model-size/peak-learning-rate combinations, each with multiple warmup settings, together with additional batch-size, warmup-shape, optimizer, and decay experiments. These experiments provide meaningful evidence for the reported tradeoffs within the tested configurations.
Critical weaknesses
- The specific fitted function is only partially justified by the theory.
Its persistent-error component relies on an assumed distribution of effective relaxation rates, followed by a local approximation that produces the multiplicative power-law form. These assumptions are not directly established for the trained Transformers. Moreover, the practical fit allows exponents to vary independently across configurations. The theoretical results therefore support a candidate predictive model, but do not establish a general law of neural-network training. The default parameter bounds also constrain the inferred asymptotic behavior, so fitted growth trends should not be interpreted as entirely determined by the observations.
- Validation at larger model scales is missing.
The main experiments use approximately 60M–350M parameter models, and the longest-horizon validation uses only a 100M model. The training trajectories are substantial—approximately 16.8B tokens in the main setting and up to 32.8B tokens in the extended experiment—but they establish temporal extrapolation primarily within small-model settings. The single corpus, short training context, and zero-weight-decay recipe further limit the evidence for broader applicability. A multi-billion-parameter experiment would considerably strengthen the practical claims.
Questions for authors
1 Can the recommendations be validated on a substantially larger model using only short prefixes for fitting and competitive tuned warmup baselines?
2 How stable are the recommended warmup durations across random seeds, fitting windows, and relaxed exponent bounds?
2 comments
Thanks for the feedback. When asking results for larger models, it does not mean I wish the result is transferrable. Instead, my question is if the modeling approach proposed in this paper will work for larger, practical models. The 600M experiments is a one step towards this direction and is appreciated. It is understoodable that academia may not have the compute power to train larger models, so I am ok with this point.
However, in this experiment, you compare with 1k and 10% warmup updates, but I have asked to have comparison with tuned warmup baselines, which you have done in your small scale experiments, but is missing in the longest validation run.
We thank the Reviewer for the comments. We have incorporated changes into our current draft and respond below.
Weakness 1. The quadratic is not meant as an accurate description of Transformer dynamics, but as a model of the trade-off that warmup duration creates. It motivates our law because it captures the peak-rate-dependent regimes we observe. The exponents vary across peak rates by design, since each configuration is fit from its own runs. Fixing s=1/2 gives similar selection regret (Table 23), and sharing q+s across peak rates keeps fit quality nearly unchanged (Table 25). We address the exponent bounds under Q2.
Weakness 2 and Q1. Our scope is scaling across horizons. The best warmup duration depends on the training horizon, and that dependence is the subject of the paper. Transfer across model scale is outside this scope, although Appendix E.7 gives some insight. To test the method at a larger size, we added a horizon-extrapolation experiment with a 600M model and 0.1 weight decay, fitting only on short prefixes and comparing against baselines from the same runs.
600M model, peak LRs {5e-4, 3e-3}, fitting runs up to 50k updates. Validation loss at each target horizon, averaged over the two peak learning rates.
Q2. Fitting-window dependence is in Appendix E.5. Our Table 2 results already average over 3 seeds. To further test sensitivity to seeds, we refit on two different pairs of seeds. We also relax the exponent bounds to [0.05,5].
Full-data fits as in Table 2. R2 and RMSE are medians across families; regret is the mean loss above the best tested warmup. RMSE and regret in 10−3 loss units.