- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies limitations of fixed-learning-rate Schedule-Free (SF) training and proposes ORA (Online Learning Rates for Anytime Training). It rewrites SF using the pending update, the difference between the base iterate and its running average, and uses its norm to control the learning rate online without specifying a training horizon. In a local two-layer linear regression model, the authors prove that fixed-rate SF-Muon generally converges to a nonoptimal point, whereas ORA-Muon converges to a zero-risk solution. Language-model experiments evaluate practical performance, learning-rate sensitivity, weight decay, and loss extrapolation.
Critical strengths
-
The paper provides meaningful pretraining evidence, including a controlled improvement over SF. The experiments train 124M and 720M language models for approximately 16.8B and 48.8B tokens. In the 124M comparison with the same peak learning rate, changing only the post-warmup learning-rate rule yields a reported 42% token saving to match the specified SF baseline's final loss. Additional 124M experiments cover larger batches, AdamW, AMUSE, and Polyak-based SF variants. These are substantive experiments rather than only illustrative small problems.
-
There are comparisons against conventional scheduled training. Figure 2 evaluates Muon with cosine decay at several training horizons. The starred ORA configuration in that figure reports 23% and 33% token savings relative to the longest cosine runs at 124M and 720M, respectively. The 720M run uses the ORA hyperparameters selected at 124M, with only the warmup length rescaled.
-
The simple-model analysis makes a concrete distinction between SF and ORA. The local result identifies a nonvanishing error for fixed-rate SF-Muon and proves that the proposed rule removes it under the stated assumptions. The pending-update formulation also motivates useful practical investigations. Figure 4 reports actual learning-rate trajectories and reduced late-training sensitivity to the peak learning rate, while the weight-decay experiments expose a remaining short-versus-long-horizon tradeoff. The paper also tests loss extrapolation against later observations and checks sensitivity to the fitting start.
Critical weaknesses
1. The practical evidence does not yet establish the full range of ORA's competitiveness.
This is my main concern. The 720M experiment compares the complete ORA configuration against cosine, but does not include a same-scale SF baseline or isolate the learning-rate change. Thus, the central controlled evidence for improving SF remains at 124M. Both model sizes use 12-layer models from the same family and corpus, with a context length of 512 in the main experiments. These are meaningful pretraining runs, but the breadth of evidence is still limited for a broadly useful training method. In addition, WSD is absent. I think it is acceptable even if ORA does not outperform cosine or WSD at every budget. However, adding WSD under a transparent tuning protocol would help establish the performance tradeoffs across training budgets.
2. The connection between the theoretical mechanism and realistic training remains incomplete.
I value an explanation in a simple model and do not require a general convergence theorem for transformers. However, the theorem uses exact gradients, no internal Muon momentum or weight decay, and a local initialization region. It establishes a specific failure mode under these assumptions. The LLM learning-rate and decay studies provide relevant evidence, but they do not directly establish that the same failure mechanism, or the energy-balance heuristic used to design ORA, explains the gains in language-model training.
Questions for authors
-
Can the authors strengthen the practical comparisons with SF at 720M and WSD across representative training budgets? At 720M, a comparison of SF-Muon, ORA with unchanged weight decay, and ORA with the modified weight decay, keeping the remaining settings fixed, would separate the contributions of learning-rate control and weight decay, as already done at 124M. Comparisons with WSD under a transparent tuning protocol would complement the existing cosine results and clarify where ORA is competitive across training budgets.
-
Can the authors provide evidence that the proposed mechanism also operates in language-model training? The current LLM experiments show performance improvements and reduced sensitivity to the peak learning rate. What observations connect these results to the SF failure mechanism in Section 4 or to the pending-update energy balance in Section 3.1?
-
Can the authors briefly explain the motivation for Equation (14) in Section 5.3, page 7? In particular, why is the loss modeled as , with exponent and horizontal shift ? The authors cite prior work, and Appendix E provides extrapolation and fitting-start checks. A short explanation next to Equation (14) would help readers understand the choice of this empirical model and its relation to the paper's theoretical results.
-
Can the authors expand the comparison with earlier work on learning-rate design? Relevant examples include recent LLM training methods [1,2] and analyses of learning-rate schedules and iterate averaging [3-6]. This discussion would help clarify what ORA adds and which conclusions from prior work concern a different setting.
[1] Changxin Tian, Jiapeng Wang, Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training. ICLR, 2026.
[2] Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu, and Wenguang Chen. How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining. ICLR, 2026.
[3] Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, and Lei Wu. Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay. COLT, 2026.
[4] Aymeric Dieuleveut and Francis Bach. Nonparametric stochastic approximation with large step-sizes. The Annals of Statistics, 44(4):1363-1399, 2016.
[5] Rong Ge, Sham M. Kakade, Rahul Kidambi, and Praneeth Netrapalli. The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure for Least Squares. NeurIPS, 2019.
[6] Prateek Jain, Dheeraj Nagaraj, and Praneeth Netrapalli. Making the Last Iterate of SGD Information Theoretically Optimal. COLT, 2019.
3 comments
3. Why we use this loss model (Q3)
How quickly loss decreases depends on the problem. The relation is an empirical scaling law observed in language model pretraining. It has no direct theoretical link to the convergence analysis of the two-layer linear model in Section 4. Bu et al. (2026) found that, at fixed model size, the final losses of cosine runs of different lengths are approximately linear in . This motivates the exponent . We use the horizontal shift introduced by Defazio (2026) to reflect the effects of early training. We now clarify just before Equation (14) that we use this empirical formula for our language model experiments.
We added longer 124M runs to test how well this formula extrapolates. For both ORA-Muon and Muon cosine, fits using only the original data through 128k steps predicted the 256k validation loss to within 0.001 (revised Appendix E.4).
4. Learning rate design and iterate averaging (Q4)
The works you cite study learning rate schedules and checkpoint averaging in pretraining, as well as iterate averaging and convergence of the last iterate in SGD. As discussed in our introduction, the key distinction between SF and ordinary iterate averaging is whether the average affects subsequent training. Ordinary iterate averaging takes a weighted average of a sequence whose updates do not depend on that average. This corresponds to in SF. In practice, SF typically uses close to one, and our main experiments use . Here is an average of the iterates and in turn affects how is computed. ORA uses this structure to choose the learning rate online and control the size of relative to the pending update . Their results on learning rate schedules and iterate averaging therefore do not apply directly to SF or ORA.
For our external averaging comparison, we used WSM, the first work you cite in Q4. We tested four checkpoint merging methods (mean, cosine, 1-sqrt, and EMA) in 124M Muon training. We used windows covering the final 10%, 20%, or 40% of training and three EMA settings, giving 18 combinations per training length. ORA-Muon achieved lower validation loss than the best WSM setting at each of the three training lengths (revised Appendix G.3, Table 9).
We hope these additions address your main questions and help you reconsider our paper.
Thanks for the clarifications and extra experiments. My concerns have been largely addressed. I'll keep my positive score.
Thank you for your detailed review and suggestions.
1. Comparisons at 720M and with WSD (W1, Q1)
(a) 720M comparisons — We added 720M runs to repeat the original 124M comparisons. The runs differ only in the settings shown below. As at 124M, changing only SF-Muon's learning rate rule lowers validation loss. Raising ηmax and reducing λP each bring further gains (revised Appendix G.1, Figure 20).
(b) WSD — For the 124M model, we tested WSD settings at 128k steps, varying the learning rate across {0.0005,0.001,0.002} and the cooldown fraction across {10%,20%,40%} with both linear and 1-sqrt cooldowns. A learning rate of 0.001 with a 1-sqrt cooldown over the final 40% gave the lowest loss. We used this setting at all three training lengths. ORA-Muon achieves lower validation loss than WSD at all three endpoints (revised Appendix G.2, Table 9).
2. How the toy experiments and theory relate to language model training (W2, Q2)
Apte et al. (2026) showed that SF-NorMuon can become unstable with Y decay in long runs. We ran SF-Muon for 256k steps using the original 124M settings, replacing Z decay with Y decay. We found that ∣∣Pt∣∣ began to rise slowly around 150k steps. Loss was still decreasing and only began to rise around 205k steps. With ORA's learning rate rule as the only change, both ∣∣Pt∣∣ and loss continued to decrease (revised Appendix F.3, Figure 18).
Our interpretation is that a fixed learning rate that works well for SF early in language model training may be too large later on. With ORA, loss can keep decreasing once SF's fixed learning rate starts to limit progress. In the noisy experiments in Section 4, ORA's risk continues its earlier downward trend after SF's risk begins to rise. The local theory in Section 4 further shows that averaging Zt alone does not resolve the problem of a fixed learning rate that is too large late in training.
Late in training, Pt+1≈Pt+St. As ∣∣Pt∣∣ decreases, a fixed learning rate need not produce a corresponding decrease in ∣∣St∣∣. Adding St may then cause the pending update norm to rise again, which could in turn hurt performance. One intuition behind ORA is to let ∣∣St∣∣ vary with ∣∣Pt∣∣ to avoid this late deterioration.