acceptodds
Under review as a conference paper at ICLR 2027

When Schedule-Free Fails: Online Learning Rates for Anytime Training

Abstract

Schedule-free (SF) optimization avoids a predefined training horizon, yet its performance can worsen with continued training at a fixed learning rate. We introduce a new view of SF based on the pending update, which accumulates new updates before they are gradually applied to the output iterate. This view motivates **ORA, Online Learning Rates for Anytime Training**, which selects learning rates to control the size of new updates relative to the pending update. Our local analysis in an aligned subspace of a deterministic two-layer linear model shows that SF-Muon generally converges to a nonstationary point while ORA-Muon converges to an optimum. In noisy two-layer regression, smaller fixed learning rates delay SF-Muon's risk increase, while ORA closely tracks the lower envelope of SF-Muon curves. In 124M language model pretraining, ORA reaches the same validation loss as SF-Muon with **42-51% fewer tokens**. Compared with SF-Muon, ORA more closely follows the empirical loss scaling law under cosine decay and yields more accurate extrapolation. These results support studying anytime methods through scaling laws rather than only at a single horizon. ORA also matches the loss of strong Muon cosine baselines with **23% and 33% fewer tokens** on 124M and 720M models, respectively. Extrapolation suggests that these token savings grow further with longer training.

Primary area:
optimization
Keywords:
schedule-free optimization, online learning rates, anytime training, Muon, language model pretraining, scaling laws

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_iW4JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies limitations of fixed-learning-rate Schedule-Free (SF) training and proposes ORA (Online Learning Rates for Anytime Training). It rewrites SF using the pending update, the difference between the base iterate and its running average, and uses its norm to control the learning rate online without specifying a training horizon. In a local two-layer linear regression model, the authors prove that fixed-rate SF-Muon generally converges to a nonoptimal point, whereas ORA-Muon converges to a zero-risk solution. Language-model experiments evaluate practical performance, learning-rate sensitivity, weight decay, and loss extrapolation.

Critical strengths
  1. The paper provides meaningful pretraining evidence, including a controlled improvement over SF. The experiments train 124M and 720M language models for approximately 16.8B and 48.8B tokens. In the 124M comparison with the same peak learning rate, changing only the post-warmup learning-rate rule yields a reported 42% token saving to match the specified SF baseline's final loss. Additional 124M experiments cover larger batches, AdamW, AMUSE, and Polyak-based SF variants. These are substantive experiments rather than only illustrative small problems.

  2. There are comparisons against conventional scheduled training. Figure 2 evaluates Muon with cosine decay at several training horizons. The starred ORA configuration in that figure reports 23% and 33% token savings relative to the longest cosine runs at 124M and 720M, respectively. The 720M run uses the ORA hyperparameters selected at 124M, with only the warmup length rescaled.

  3. The simple-model analysis makes a concrete distinction between SF and ORA. The local result identifies a nonvanishing error for fixed-rate SF-Muon and proves that the proposed rule removes it under the stated assumptions. The pending-update formulation also motivates useful practical investigations. Figure 4 reports actual learning-rate trajectories and reduced late-training sensitivity to the peak learning rate, while the weight-decay experiments expose a remaining short-versus-long-horizon tradeoff. The paper also tests loss extrapolation against later observations and checks sensitivity to the fitting start.

Critical weaknesses

1. The practical evidence does not yet establish the full range of ORA's competitiveness.

This is my main concern. The 720M experiment compares the complete ORA configuration against cosine, but does not include a same-scale SF baseline or isolate the learning-rate change. Thus, the central controlled evidence for improving SF remains at 124M. Both model sizes use 12-layer models from the same family and corpus, with a context length of 512 in the main experiments. These are meaningful pretraining runs, but the breadth of evidence is still limited for a broadly useful training method. In addition, WSD is absent. I think it is acceptable even if ORA does not outperform cosine or WSD at every budget. However, adding WSD under a transparent tuning protocol would help establish the performance tradeoffs across training budgets.

2. The connection between the theoretical mechanism and realistic training remains incomplete.

I value an explanation in a simple model and do not require a general convergence theorem for transformers. However, the theorem uses exact gradients, no internal Muon momentum or weight decay, and a local initialization region. It establishes a specific failure mode under these assumptions. The LLM learning-rate and decay studies provide relevant evidence, but they do not directly establish that the same failure mechanism, or the energy-balance heuristic used to design ORA, explains the gains in language-model training.

Questions for authors
  1. Can the authors strengthen the practical comparisons with SF at 720M and WSD across representative training budgets? At 720M, a comparison of SF-Muon, ORA with unchanged weight decay, and ORA with the modified weight decay, keeping the remaining settings fixed, would separate the contributions of learning-rate control and weight decay, as already done at 124M. Comparisons with WSD under a transparent tuning protocol would complement the existing cosine results and clarify where ORA is competitive across training budgets.

  2. Can the authors provide evidence that the proposed mechanism also operates in language-model training? The current LLM experiments show performance improvements and reduced sensitivity to the peak learning rate. What observations connect these results to the SF failure mechanism in Section 4 or to the pending-update energy balance in Section 3.1?

  3. Can the authors briefly explain the motivation for Equation (14) in Section 5.3, page 7? In particular, why is the loss modeled as L^(T)=b+A/T+s\widehat L(T)=b+A/\sqrt{T+s}, with exponent 1/21/2 and horizontal shift ss? The authors cite prior work, and Appendix E provides extrapolation and fitting-start checks. A short explanation next to Equation (14) would help readers understand the choice of this empirical model and its relation to the paper's theoretical results.

  4. Can the authors expand the comparison with earlier work on learning-rate design? Relevant examples include recent LLM training methods [1,2] and analyses of learning-rate schedules and iterate averaging [3-6]. This discussion would help clarify what ORA adds and which conclusions from prior work concern a different setting.

[1] Changxin Tian, Jiapeng Wang, Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training. ICLR, 2026.

[2] Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu, and Wenguang Chen. How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining. ICLR, 2026.

[3] Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, and Lei Wu. Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay. COLT, 2026.

[4] Aymeric Dieuleveut and Francis Bach. Nonparametric stochastic approximation with large step-sizes. The Annals of Statistics, 44(4):1363-1399, 2016.

[5] Rong Ge, Sham M. Kakade, Rahul Kidambi, and Praneeth Netrapalli. The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure for Least Squares. NeurIPS, 2019.

[6] Prateek Jain, Dheeraj Nagaraj, and Praneeth Netrapalli. Making the Last Iterate of SGD Information Theoretically Optimal. COLT, 2019.

3 comments

Thank you for your detailed review and suggestions.

1. Comparisons at 720M and with WSD (W1, Q1)

(a) 720M comparisons — We added 720M runs to repeat the original 124M comparisons. The runs differ only in the settings shown below. As at 124M, changing only SF-Muon's learning rate rule lowers validation loss. Raising ηmax⁡\eta_{\max} and reducing λP\lambda_P each bring further gains (revised Appendix G.1, Figure 20).

Methodηmax⁡\eta_{\max}λP\lambda_PValidation loss at 48k steps ↓
SF-Muon0.010.052.7875
ORA-Muon0.010.052.7751
ORA-Muon0.050.052.7580
ORA-Muon (original submission)0.050.026252.7515

(b) WSD — For the 124M model, we tested WSD settings at 128k steps, varying the learning rate across {0.0005,0.001,0.002} and the cooldown fraction across {10%,20%,40%} with both linear and 1-sqrt cooldowns. A learning rate of 0.001 with a 1-sqrt cooldown over the final 40% gave the lowest loss. We used this setting at all three training lengths. ORA-Muon achieves lower validation loss than WSD at all three endpoints (revised Appendix G.2, Table 9).

Training stepsWSDORA-Muon
32k3.29403.2665
64k3.22553.2004
128k3.17423.1544

2. How the toy experiments and theory relate to language model training (W2, Q2)

Apte et al. (2026) showed that SF-NorMuon can become unstable with YY decay in long runs. We ran SF-Muon for 256k steps using the original 124M settings, replacing ZZ decay with YY decay. We found that ∣∣Pt∣∣||P_t|| began to rise slowly around 150k steps. Loss was still decreasing and only began to rise around 205k steps. With ORA's learning rate rule as the only change, both ∣∣Pt∣∣||P_t|| and loss continued to decrease (revised Appendix F.3, Figure 18).

Our interpretation is that a fixed learning rate that works well for SF early in language model training may be too large later on. With ORA, loss can keep decreasing once SF's fixed learning rate starts to limit progress. In the noisy experiments in Section 4, ORA's risk continues its earlier downward trend after SF's risk begins to rise. The local theory in Section 4 further shows that averaging ZtZ_t alone does not resolve the problem of a fixed learning rate that is too large late in training.

Late in training, Pt+1≈Pt+StP_{t+1}\approx P_t+S_t. As ∣∣Pt∣∣||P_t|| decreases, a fixed learning rate need not produce a corresponding decrease in ∣∣St∣∣||S_t||. Adding StS_t may then cause the pending update norm to rise again, which could in turn hurt performance. One intuition behind ORA is to let ∣∣St∣∣||S_t|| vary with ∣∣Pt∣∣||P_t|| to avoid this late deterioration.

3. Why we use this loss model (Q3)

How quickly loss decreases depends on the problem. The relation L^(T)=b+A/T+s\widehat L(T)=b+A/\sqrt{T+s} is an empirical scaling law observed in language model pretraining. It has no direct theoretical link to the convergence analysis of the two-layer linear model in Section 4. Bu et al. (2026) found that, at fixed model size, the final losses of cosine runs of different lengths are approximately linear in 1/T1/\sqrt{T}. This motivates the exponent 1/21/2. We use the horizontal shift ss introduced by Defazio (2026) to reflect the effects of early training. We now clarify just before Equation (14) that we use this empirical formula for our language model experiments.

We added longer 124M runs to test how well this formula extrapolates. For both ORA-Muon and Muon cosine, fits using only the original data through 128k steps predicted the 256k validation loss to within 0.001 (revised Appendix E.4).

4. Learning rate design and iterate averaging (Q4)

The works you cite study learning rate schedules and checkpoint averaging in pretraining, as well as iterate averaging and convergence of the last iterate in SGD. As discussed in our introduction, the key distinction between SF and ordinary iterate averaging is whether the average affects subsequent training. Ordinary iterate averaging takes a weighted average of a sequence whose updates do not depend on that average. This corresponds to β=0\beta=0 in SF. In practice, SF typically uses β\beta close to one, and our main experiments use β=0.95\beta=0.95. Here XtX_t is an average of the ZZ iterates and in turn affects how Zt+1Z_{t+1} is computed. ORA uses this structure to choose the learning rate online and control the size of StS_t relative to the pending update Pt=Zt−XtP_t=Z_t-X_t. Their results on learning rate schedules and iterate averaging therefore do not apply directly to SF or ORA.

For our external averaging comparison, we used WSM, the first work you cite in Q4. We tested four checkpoint merging methods (mean, cosine, 1-sqrt, and EMA) in 124M Muon training. We used windows covering the final 10%, 20%, or 40% of training and three EMA settings, giving 18 combinations per training length. ORA-Muon achieved lower validation loss than the best WSM setting at each of the three training lengths (revised Appendix G.3, Table 9).

We hope these additions address your main questions and help you reconsider our paper.

Thanks for the clarifications and extra experiments. My concerns have been largely addressed. I'll keep my positive score.

Reviewer_m2PKOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides the ORA method to adapt the learning rate online. It provides a theoretical analysis for the two-layer linear model case and proves that the ORA-Muon optimizer can converge to an optimal point. In their empirical experiments, they show that the ORA method can achieve a 42% improvement compared to SF-Muon baseline.

Critical strengths
  1. It provides a novel online learning rate method and provides theoretical analysis to support the design.
  2. The ORA method shows non-trivial improvements in results and training efficiency compared to the baseline method.
Critical weaknesses
  1. The experimental scale is limited, and a larger scale, like at least 1.7B models, can help in supporting the claims.
  2. The experimental setup compared to the cosine baseline is not fully matched, leaving the benefit source questionable.
Questions for authors
  1. Can you provide larger-scale experiments? Like at least 1.7B models with 1x Chinchilla to support the benefits of ORA.
  2. Can you provide a more rigorous comparison with the cosine schedule baseline? For example, more hyperparameter tuning, using a Muon + linear decay schedule, using a peak LR from 0.003 to 0.005, and decaying to 1e-5, for example.
  3. Since ORA equivalently reduces the ending LR, I believe it deserves a comparison of the performance after SFT.
2 comments

Thank you for your positive assessment of the method and theoretical results, and for suggesting additional experiments.

1. Experiments on larger models (W1, Q1)

We agree that larger models would provide a further test of ORA. Our compute budget did not allow us to run the 1.7B comparison.

2. Comparisons with cosine decay and other learning rate schedules (W2, Q2)

ORA builds on the SF iteration, so changing Muon's learning rate schedule alone does not give ORA. In Section 5.2 of the original submission, we changed only SF-Muon's learning rate rule to isolate the effect of the online learning rate. To compare overall performance with Muon cosine, we use the same model, data, and evaluation for both methods. The cosine baseline uses the settings tuned in the optimizer benchmark of Semenov et al. (2026).

At 124M, we also varied the cosine learning rate over {0.001,0.002,0.004} and weight decay over {0.05,0.1,0.2} separately. Neither sweep improved on the original setting (0.002,0.1) at 128k steps.

We added linear decay runs at peak learning rates of 0.002 and 0.004. For WSD, we varied the learning rate and cooldown fraction with both linear and 1-sqrt cooldowns. We also tested WSM checkpoint merging (revised Appendices G.2–G.3). At 128k steps, ORA-Muon achieved lower validation loss than Muon with cosine decay, linear decay, WSD, or WSM.

3. Performance after SFT (Q3)

Following the Alpaca SFT experiments of Liu et al. (2026), we fine-tuned the 48k checkpoints from the 720M Muon cosine and ORA runs on Alpaca for two epochs. We used Muon with cosine decay for SFT. Only the pretrained checkpoint differs. To evaluate general capabilities after SFT, we used HellaSwag, WinoGrande, and ARC-Challenge, the three tasks in Figure 1 of Liu et al..

The table shows results before SFT and after two epochs at each learning rate. Each cell reports cosine / ORA, with accuracy in percent.

SFT learning rateHellaSwag ↑WinoGrande ↑ARC-Challenge ↑Alpaca validation loss ↓
Before SFT53.11 / 55.4555.96 / 57.3827.13 / 28.502.041 / 2.026
3×10−53\times10^{-5}54.10 / 56.1456.67 / 57.6229.18 / 29.691.565 / 1.545
5×10−55\times10^{-5}54.96 / 56.6256.83 / 58.4129.52 / 30.291.564 / 1.531
10−410^{-4}55.80 / 57.6656.20 / 58.2529.61 / 30.291.598 / 1.537
3×10−43\times10^{-4}54.34 / 58.0553.99 / 55.9630.55 / 30.801.747 / 1.681
5×10−45\times10^{-4}50.83 / 56.3852.96 / 55.4128.41 / 29.951.840 / 1.729

At all five SFT learning rates, the ORA checkpoint achieves lower Alpaca validation loss and higher accuracy on all three tasks. Its pretraining advantage persists after SFT (revised Appendix G.4, Figure 21).

I will maintain my positive score.

Reviewer_Mo99OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study schedule-free optimization (Defazio et al., 2024) with a time varying learning rate. Their algorithm is motivated by the fact that the evaluation iterate XtX_t does not immediately get the full first order update StS_t in every step. This motivates ORA, an online learning-rate rule that matches the current update norm with the pending update norm. For a deterministic two-layer linear network, the authors prove that with constant lr, SF-Muon may converge to a nonstationary output, whereas ORA-Muon converges to the optimum under the correct initialization. The authors experiment with noisy regression and language-model pretraining at 124M and 720M parameters and show meaninful gains over SF-Muon.

Critical strengths
  • Unlike SF-Muon which matches the independently trained cosine runs only on smaller budgets, ORA matches the independently trained cosine runs at smaller and larger budgets demonstrating its advantage over a constant learning rate.
  • The theoretical analysis shows that SF-Muon need not converge to an optima, whereas under the right initialization conditions, ORA will.
  • The experiments test two model sizes, several relevant baselines and trainig budgets, and demonstarte ORA's superior performance.
Critical weaknesses
  • The lr and weight decay for independently trained cosine runs and SF-Muon are not swept over, however, for ORA the lr and λP\lambda_P are swept over. The comparsions must be fair. Also, it would be great if the authors report tokens in Chinchilla multiples, and it does not look like the authors go beyond 8x Chinchilla. Also, comparions with WSD should be reported.
  • The theoretical analysis does not state how one would enter the initialization subspace assumed. Also, Muon's momentum and weight decay have been removed. The analysis demonstartes failure cases for SF-Muon, however the experiments need to be done in a fair manner to estabilish this rigorously.
  • The motivation for using PtP_t is not very clear, why is this the quantity to look at and why the authors adjust the lr to match the update's norm with PtP_t.
  • λp\lambda_p depends on the horizon, which is a limitation the authors acknowledge.
Questions for authors
  • Can the hyperparameter tuning be done in a fair fashion, with the same budget for every reported method?
  • Will the theoretical analysis hold with momentum and weight decay and can it be done on quadratics? The udpate remain the same even with R(x)=12∑i(xi−ri)2{\mathcal R}(x)=\frac12\sum_i(x_i-r_i)^2.
  • Can entering into the theoretially required initialization subspace be verified empirically?
  • How can λP\lambda_P be selected without knowing the training horizon?
3 comments

Thank you for your review and comments.

1. Fairness and training length in the language model experiments (W1, Q1)

Our original submission used baseline settings tuned in prior work. The Muon cosine settings come from the optimizer benchmark of Semenov et al. (2026), and the SF-Muon settings come from the AMUSE paper by Kim et al. (2026). In the revision, we separately varied the cosine learning rate over {0.001,0.002,0.004} and weight decay over {0.05,0.1,0.2}. Neither sweep improved on the original setting (0.002,0.1) at 128k steps (Appendix G.2).

(a) Learning rate — In the original Figure 2(a), we changed only the learning rate rule for a fair comparison, using AMUSE's tuned SF-Muon settings. With both methods at the same peak rate of 0.01, ORA reached SF-Muon's 128k validation loss with 42% fewer tokens. Figure 4(a) of the original submission shows that the preferred fixed learning rate changes with training length. Further tuning can improve SF-Muon's loss at a given horizon, but in schedule-free training we want to choose the rate without knowing that horizon in advance.

(b) Weight decay — Our original comparison was already fair because SF-Muon and ORA used the same ZZ decay. We adjusted λP\lambda_P only for the comparison with Muon cosine. In SF, λP\lambda_P controls the decay of PtP_t. The cosine baseline applies weight decay directly to the model parameters, so its weight decay coefficient does not correspond directly to λP\lambda_P. We added SF-Muon runs at the four λP\lambda_P values in the original Figure 5. ORA still outperforms SF-Muon at all four values (Appendix F.2, Figure 17(b)).

(c) WSD — We added WSD runs with learning rates {0.0005,0.001,0.002} and linear or 1-sqrt cooldowns over the final 10%, 20%, or 40% of training. ORA-Muon achieves lower validation loss than the best WSD run at each training length (32k, 64k, and 128k steps) (Appendix G.2, Table 9).

(d) Training length — The original 124M and 720M runs reached 6.8×6.8\times and 3.4×3.4\times Chinchilla, respectively (revised Appendix D.2). We added 124M runs at 13.6×13.6\times Chinchilla. ORA-Muon's token saving over cosine rose from 23% at 6.8×6.8\times to 25%, close to the 24% predicted in the original Appendix E.4. We used the fitting method in Section 5.3 of the original paper to extrapolate from data through 6.8×6.8\times to 13.6×13.6\times. Both endpoint predictions were within 0.001 of the observed losses (revised Appendix E.4, Figure 16(a)). The original runs were long enough to support the reported long-term trend.

2. Theoretical setting and experimental fairness (W2, Q2, Q3)

Our main observation comes from the noisy toy experiments. Our theory partly explains it by analyzing a special case of the same two-layer model. We never claim that the theory describes the full dynamics.

(a) Experimental fairness (W2, Q2) — In the original noisy experiments, both methods use Muon's internal momentum and differ only in their learning rate rule. The theory omits internal momentum for simplicity. The corresponding experiments again use the same initialization and settings for both methods. In each comparison, both methods use the same ηmax⁡\eta_{\max} values. Both comparisons are therefore fair. Since we seek exact recovery of the regression target, we omit weight decay for both methods to avoid introducing bias.

(b) Subspace and local region (Q3) — The subspace defines the scope of the analysis. The local region specifies where the convergence results hold. We do not claim that trajectories from general initializations enter this subspace, and our results do not rely on this. In Appendix C.4 of the original paper, we verified that each method's trajectories from random initializations within the subspace eventually satisfy its local conditions.

(c) Momentum, weight decay, and quadratic objectives (Q2) — We can use the same analysis for the corresponding quadratic problem. With internal momentum or weight decay, the proof no longer applies directly.

3. Why use the pending update as a reference? (Summary, W3)

We would like to clarify the motivation for ORA as described in your summary: ORA's design is based on more than the fact that updates are applied gradually. The key point of the pending-update view is that it gives us the algorithm's own scale for new updates.

Without new updates from step tt onward, XX gradually approaches Zt=Xt+PtZ_t=X_t+P_t. We can therefore view ZtZ_t as the output's current target. As a heuristic, we can also view ZtZ_t as the algorithm's rough estimate of an optimum. Then ∣∣Pt∣∣||P_t|| is the distance from the output to that estimate. Choosing the learning rate with this distance as a reference is natural and common in optimization. Section 3.1 of the original paper explains how we choose the update size.

4. Horizon dependence and choice of λP\lambda_P (W4, Q4)

ORA's core contribution is its online learning rate rule. λP\lambda_P reflects SF's existing choice between ZZ and YY decay. Our new comparisons show that SF already has this early–late tradeoff, so ORA did not introduce it (Appendix F.2, Figure 17(a)).

We do not tune λP\lambda_P for a particular training horizon. We use a value between those for ZZ and YY decay to balance early and late performance and keep it fixed across horizons. Reducing the horizon dependence of λP\lambda_P is a separate research question for SF itself, beyond the main focus of this paper.

We hope these clarifications and additional experiments will be helpful as you reconsider our paper.

I thank the authours for their rigorous response. I have raised my score.

Comments on the paper

2 comments
Area_Chair_ysH8OpenReview ↗

Reviewers: please respond to the author rebuttal

Dear reviewers,

The authors have posted responses to your reviews. The author-reviewer discussion closes on November 18 (AoE).

Please read the responses and reply here before then: let the authors know whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Engaging with the other reviews is also welcome.

Thank you, Area Chair

AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Area Chair and Reviewers,

We thank the reviewers for their helpful feedback and suggestions for additional experiments, and the Area Chair for coordinating the discussion. One reviewer raised their score, while the other two maintained their positive scores.

The additional experiments strengthen the original conclusions in three ways:

  • Longer training validates the predictions. In the added 124M runs, ORA-Muon's token saving over Muon cosine increased from 23% at 128k to 25% at 256k. Fits using only the original data through 128k predicted the 256k loss to within 0.001 for both methods (Appendix E.4).

  • Additional comparisons support ORA's gains. We tested multiple hyperparameter settings for cosine decay, linear decay, WSD, and WSM. ORA-Muon achieves the lowest loss at 128k (Appendices G.2–G.3). The 720M comparisons show the same pattern as at 124M: changing only SF-Muon's learning rate rule lowers loss (Appendix G.1).

  • The pretraining advantage persists after SFT. Under the same SFT procedure, the ORA checkpoint achieves higher accuracy than the cosine checkpoint on all three evaluation tasks at all five SFT learning rates (Appendix G.4).

These additions are included in the revised manuscript, with added text marked in red.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.