acceptodds
Under review as a conference paper at ICLR 2027

Anytime Training with Schedule-Free Spectral Optimization

Abstract

Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines. We propose SF-NorMuon, a schedule-free spectral optimizer that closes this gap: with a single hyperparameter configuration, SF-NorMuon matches or exceeds tuned AdamW on 125M and 772M parameter language models across – Chinchilla horizons, and trails horizon-aware, cosine-scheduled NorMuon by only nats, substantially narrowing the gap between anytime and scheduled spectral optimizers. On the theoretical side, we prove a stationarity guarantee for schedule-free spectral dynamics and identify weight decay at the fast iterate as essential for long-horizon stability. SF-NorMuon enables practitioners to obtain high-quality checkpoints at any point during training without committing to a horizon in advance. Fully matching horizon-aware spectral optimizers without a schedule remains an open problem, which we highlight as a direction for future work. By closing the performance gap with tuned AdamW baselines, SF-NorMuon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.

Primary area:
optimization
Keywords:
spectral optimization, schedule-free, anytime training, muon

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GLsZOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new anytime training method by using a muon-variant as a baseline optimizer in a schedule-free scheme. Analytical results in a relatively simplified case are given to support the proposal, and empirics provide strong validation that the method works. Authors also provide analytical and empirical arguments to argue why weight decay is needed in long horizon training.

Critical strengths

I think this is a very good paper within the standards of this subfield. The main strength is the empirical results. The proposed method seem to work well in a reasonable set of benchmarks.

Critical weaknesses

The proofs are done using what authors call SF-SpectralSGD with Momentum, but the performant method is SF-NorMuon which has extra features. There is no SF-SpectralSGD with Momentum benchmark in simulations as far as I can see to judge how much gap there is between theory and practice.

Questions for authors
  1. How big is the a gap between theory and practice? Which of the extra features introduced in SF-NorMuon over SF-SpectralSGD with Momentum is necessary to make the method work? How much do each of these contribute?

  2. Relatedly, the logic behind the theory could be better explained. The authors claim that their SF-SpectralSGD with Momentum has the same convergence rates with muon. What is the significance of this result? Is the idea that the muon variant that is being cited is not schedule-free?

  3. Corollary 2.1 is a big-O-with-tilde result. Should we not worry about the coefficients? Since the goal of this kind of a study is to beat the sota, and coefficients can matter, what is the value of this kind of a theory? I know the authors are not the only ones who are doing this in this literature, this is just me trying to see how they think about this.

4 comments

Dear Reviewer GLsZ,

Thank you for your valuable comments and feedback. We are encouraged that you found our paper very good within the standards of the subfield, with strong empirical results. We also kindly refer you to our new result: combined with the time-varying interpolation of the concurrent work AMUSE [1] and a norm-preserving variant of our update scaling, SF-NorMuon outperforms both per-horizon tuned NorMuon and AMUSE (125M model, validation loss):

ChinchillaSF-NorMuonAMUSE [1]NorMuon (cosine, tuned per horizon)AdamW (cosine, tuned per horizon)SF-AdamW
1×1\times3.2743.2783.2923.3543.399
2×2\times3.1933.1983.2003.2313.282
4×4\times3.1373.1413.1393.1703.205
8×8\times3.0983.1013.1033.1243.141

SF-NorMuon uses a single configuration for all horizons, with all hyperparameters set to the values tuned for AMUSE except for the learning rate. AMUSE is run with its released code and the best configuration reported in its paper for the 124M model (η=0.01\eta=0.01, βinit=0.6\beta_{\rm init}=0.6, ρ=0.8\rho=0.8), with its 6,000 warmup steps converted to 1,500 steps at our 4×4\times larger batch (the same number of warmup tokens). NorMuon, AdamW, and SF-AdamW are from Table 1.

Due to limited GPU availability during the rebuttal period, we report the 125M model here; we will include the 772M results with the new setting in the final manuscript.

Q: How big is the gap between theory and practice? Which features of SF-NorMuon are necessary, and how much does each contribute?

A: We kindly refer the reviewer to Figure 3, which is exactly this ablation (125M model, 8×8\times Chinchilla):

  • Row-wise normalization. Removing it from SF-NorMuon recovers SF-Spectral Descent with Momentum, i.e., the method of Theorem 2.1 (up to warmup, Newton–Schulz, and weight decay at ZZ). It reaches a final loss of 3.142, compared to 3.136 for SF-NorMuon, which reaches the same loss ~12% faster. The gap between theory and practice is therefore small.
  • Explicit momentum. It is essential: removing it (μ=0\mu=0) increases the final loss from 3.14 to 3.28.
  • Weight decay at ZZ. It is necessary for long-horizon stability (Figure 2, Section 3).

We have made the connection between Figure 3 and Theorem 2.1 explicit in Section 2.2 of the revision, and added the discussion of the rates below after Corollary 2.1 (in blue).

Q: What is the significance of matching Muon's rates? Is the cited Muon variant not schedule-free?

A: Yes, the cited analyses are for Muon, which is not schedule-free. Our result shows that the schedule-free mechanism preserves Muon's convergence rate. That mechanism, evaluating gradients at interpolated points and averaging online, is what enables anytime training without a learning-rate schedule in practice. This is in the same spirit as Defazio et al. [2], where Schedule-Free SGD attains the optimal rate of SGD without a schedule. One cannot hope for a faster rate: the rates T−1/4T^{-1/4} (stochastic) and T−1/2T^{-1/2} (deterministic) match the lower bounds in TT for first-order methods on smooth non-convex problems [3, 4]. The point is that the schedule-free modification comes at no cost in the rate, up to a mild logarithmic factor (see next answer).


Q: Corollary 2.1 is an O~\widetilde{O} result. What is the value of such theory when constants matter?

A: Thank you for this question. All coefficients are explicit in Theorem 2.1, and the precise rate is O((log⁡T/T)1/4)O\big((\log T/T)^{1/4}\big) (Corollary C.2). Concretely, the leading term is 4r1−β(σ2LFATBT)1/4\frac{4\sqrt r}{1-\beta}\big(\frac{\sigma^2 L_F A_T}{BT}\big)^{1/4} with AT=Δ+2LFDF2(log⁡(eT)+π2/6)A_T=\Delta+2L_FD_F^2\big(\log(eT)+\pi^2/6\big). The O~\widetilde O therefore hides three things:

  • Problem constants: Δ\Delta, LFL_F, DFD_F, σ2/B\sigma^2/B, and the rank r=min⁡(m,n)r=\min(m,n).
  • The interpolation factor 1/(1−β)1/(1-\beta).
  • A logarithmic factor from online averaging, at most (log⁡(eT)+π2/6)1/4(\log(eT)+\pi^2/6)^{1/4}. This equals 1.83–1.95 across all our runs (T≈T\approx 5k–118k steps) and vanishes for β=0\beta=0 (Corollary C.2(iii)).

Hence the logarithmic factor is mild, whereas the problem constants cannot be measured for LLMs, as in essentially all such analyses. We fully agree that such bounds cannot rank optimizers. Even for smooth convex minimization, the exact worst-case constants of gradient descent and the exactly optimal first-order method were established only through the performance-estimation line of work [5–8], which is largely separate from rate analyses. Therefore, the role of our theory is soundness, and SOTA comparisons are made empirically (Section 4 and the new results above). More broadly, the gap between worst-case theory and deep learning practice remains a major open problem. Notably, where constants do matter in practice, our analysis provides them: Lemma 3.2 predicts the steady-state norm of ZZ including its constant, which closely matches the measured values (Figure 4).


We hope the above answers clarify your questions. Thank you again for your time in reviewing our manuscript.

[1] Kim et al. (2026) "AMUSE: Anytime Muon with Stable Gradient Evaluation", arXiv:2605.22432

[2] Defazio et al. (2024) "The Road Less Scheduled", NeurIPS 2024

[3] Arjevani et al. (2023) "Lower Bounds for Non-Convex Stochastic Optimization", Math. Programming

[4] Carmon et al. (2020) "Lower Bounds for Finding Stationary Points I", Math. Programming

[5] Drori and Teboulle (2014) "Performance of first-order methods for smooth convex minimization: a novel approach", Math. Programming

[6] Kim and Fessler (2016) "Optimized first-order methods for smooth convex minimization", Math. Programming

[7] Drori (2017) "The exact information-based complexity of smooth convex minimization", J. Complexity

[8] Rotaru et al. (2024) "Exact worst-case convergence rates of gradient descent: a complete analysis for all constant stepsizes over nonconvex and convex functions", arXiv:2406.17506

Thanks for your answers.

Thank you for reading our responses. We appreciate your feedback and engagement throughout the discussion; your questions helped us clarify the connection between our theory and practice in the revised manuscript. Thank you again for taking the time to carefully review our paper.

Reviewer_t9zsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Schedule-Free spectral optimizers and proposes SF-NorMuon based on the new optimizer NorMuon. They empirically verify that SF-NorMuon improves over SF-AdamW, matches AdamW, and comes within 0.025–0.036 nats of cosine-scheduled NorMuon on 125M and 772M parameter language models. They identify the key role of weight decay in schedule-free optimization weight decay directly applied to ztz_t leads to stable training.

Critical strengths

This paper studies Schedule-Free spectral optimizers and proposes SF-NorMuon based on the new optimizer NorMuon. They empirically verify that SF-NorMuon improves over SF-AdamW, matches AdamW, and comes within 0.025–0.036 nats of cosine-scheduled NorMuon on 125M and 772M parameter language models.

Critical weaknesses
  1. This paper lacks comparisons with key related algorithms.

The paper [1] (21 May 2026) also studies Schedule-Free spectral optimizers and proposes an anytime Muon optimizer. It is strongly related to this paper. I suggest adding comparisons with [1] regarding algorithm development and experimental performance.

[1] AMUSE: Anytime Muon with Stable Gradient Evaluation, arXiv:2605.22432, 2026.

  1. Assumption 2.3 requires ∣∣Zt∣∣F≤DF2||Z_t||_F\leq \frac{D_F}{2} for all tt, which is a strong assumption.

  2. Theorem 2.1 only establishes convergence for SF-Spectral Descent with Momentum, rather than for the proposed SF-NorMuon.

Questions for authors

No

6 comments

Dear Reviewer t9zs,

Thank you for your comments and feedback. We are glad you found that SF-NorMuon improves over SF-AdamW and matches AdamW, and that weight decay at ZZ leads to stable training. We address your concerns below.

Q: Comparison with AMUSE [1] is missing.

A: Thank you for pointing us to AMUSE, which we now cite in Section 4 and discuss in Appendix E.5 of the revision (in blue). Per the ICLR 2027 reviewer guidelines, authors are not required to compare with papers available only on arXiv, such as AMUSE (May 2026); we nonetheless compare with it below.

  • Algorithm. AMUSE combines Schedule-Free with Muon and uses a time-varying interpolation coefficient βt\beta_t. The main algorithmic difference is the neuron-wise normalization of the polar factor [2] in SF-NorMuon; our submission also used a fixed β\beta.
  • Experiments. Since βt\beta_t is orthogonal to the base update, we combined it with SF-NorMuon, together with a norm-preserving variant of our update scaling (Appendix E.6), and compared with AMUSE in our setup. SF-NorMuon outperforms AMUSE at every horizon, even though all hyperparameters other than the learning rate are set to the values tuned for AMUSE, and now also outperforms horizon-tuned NorMuon (125M model, validation loss):
ChinchillaSF-NorMuonAMUSE [1]NorMuon (cosine, tuned per horizon)AdamW (cosine, tuned per horizon)SF-AdamW
1×1\times3.2743.2783.2923.3543.399
2×2\times3.1933.1983.2003.2313.282
4×4\times3.1373.1413.1393.1703.205
8×8\times3.0983.1013.1033.1243.141

SF-NorMuon uses a single configuration for all horizons, with all hyperparameters set to the values tuned for AMUSE except for the learning rate. AMUSE is run with its released code and the best configuration reported in its paper for the 124M model (η=0.01\eta=0.01, βinit=0.6\beta_{\rm init}=0.6, ρ=0.8\rho=0.8), with its 6,000 warmup steps converted to 1,500 steps at our 4×4\times larger batch (the same number of warmup tokens). NorMuon, AdamW, and SF-AdamW are from Table 1.

Due to limited GPU availability during the rebuttal period, we report the 125M model here; we will include the 772M results with the new setting in the final manuscript.

  • Core findings beyond AMUSE. AMUSE provides no convergence guarantee or analysis of weight decay. In contrast, we (i) show that weight decay at YY, the default of the original Schedule-Free method [3], eventually diverges in long-horizon training (Figure 2), whereas weight decay at ZZ provably keeps all iterates bounded (Lemma 3.1) and the same argument fails for decay at YY (Remark 2); (ii) characterize the steady-state norm of ZZ in closed form, which closely matches the training dynamics (Lemma 3.2, Figure 4); and (iii) prove a stationarity guarantee for schedule-free spectral descent (Theorem 2.1). While AMUSE also applies weight decay at zz, it does so as an implementation choice, without identifying the instability of decay at yy or analyzing either choice.

Q: Assumption 2.3 (∥Zt∥F≤DF/2\lVert Z_t\rVert_F\le D_F/2 for all tt) is strong.

A: Bounded iterates are a standard assumption in the analysis of spectral methods, e.g., Shampoo [4, Theorem 7] and ASGO [5, Theorem 1], as we note below Assumption 2.3. Moreover, it is not restrictive for SF-NorMuon:

  • With weight decay at ZZ, Lemma 3.1 proves ∥Zt∥F≤∥Z0∥F(1−ηλ)t+κmn/λ\lVert Z_t\rVert_F\le\lVert Z_0\rVert_F(1-\eta\lambda)^t+\kappa\sqrt{mn}/\lambda for all tt, so the iterates of SF-NorMuon stay in a ball of explicit radius.
  • Empirically, the norm of ZtZ_t changes by less than 1% per step after warmup and tracks the steady state predicted by Lemma 3.2 (Figure 4).
  • If needed, the assumption can be enforced by projecting ZtZ_t onto a Frobenius-norm ball.

Q: Theorem 2.1 covers SF-Spectral Descent with Momentum, not SF-NorMuon.

A: This is standard for spectral optimizers. NorMuon [2] itself comes without a convergence analysis, and convergence analyses of Muon study its idealized SVD-based update rather than the Newton–Schulz implementation used in practice [6, 7]. Similarly, the theory of Schedule-Free [3] covers the convex setting, while SF-AdamW is what is used in deep learning. Our main contribution is SF-NorMuon itself and its strong empirical performance. The theory supports it in two ways:

  • Theorem 2.1 certifies that its core dynamics, schedule-free spectral descent with momentum, converge at the rate of Muon, up to a logarithmic factor.
  • Lemmas 3.1–3.2 are stated for the actual SF-NorMuon update, including the row-wise normalization and weight decay at ZZ.

We hope the above answers clarify your concerns, and if so, we hope you could consider reflecting on the score. Thank you again for your time in reviewing our manuscript.

[1] Kim et al. (2026) "AMUSE: Anytime Muon with Stable Gradient Evaluation", arXiv:2605.22432

[2] Li et al. (2025) "NorMuon: Making Muon more efficient and scalable", arXiv:2510.05491

[3] Defazio et al. (2024) "The Road Less Scheduled", NeurIPS 2024

[4] Gupta et al. (2018) "Shampoo: Preconditioned Stochastic Tensor Optimization", ICML 2018

[5] An et al. (2025) "ASGO: Adaptive Structured Gradient Optimization", arXiv:2503.20762

[6] Li and Hong (2025) "A Note on the Convergence of Muon", arXiv:2502.02900

[7] Shen et al. (2025) "On the Convergence Analysis of Muon", arXiv:2505.23737

Thanks for the response. I would like to note that the AMUSE paper has been accepted to the ICML 2026 Workshop HiLD. I believe this submission should be compared with papers that appeared four or five months before the ICLR deadline, especially because the ideas and algorithms in the two papers are very similar. The main difference is that one uses the original Muon, while the other uses the NorMuon extension. I suggest that the SAC and AC read the paper ''AMUSE: Anytime Muon with Stable Gradient Evaluation'' before making a decision. Given the similarity between the two papers, I am lowering my score to 1.

Reviewer_W2HjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce SF-NorMuon, a schedule-free optimizer that achieves performance comparable to well-tuned AdamW baselines without requiring explicit learning-rate schedules. Their method integrates the Schedule-Free framework [Defazio et al., 2025] with the NorMuon optimizer [Li et al., 2025], with principled modifications such as applying weight decay to the fast iterate ztz_t to ensure stability during long-horizon training. The authors further provide theoretical support for their empirical findings, including convergence guarantees for schedule-free spectral optimization and a steady-state analysis.

Critical strengths
  1. Anytime training algorithms are increasingly important, particularly in the context of single-pass language model training.
  2. The proposed SF-NorMuon is supported by various theoretical results, including convergence guarantees and a quasi steady-state analysis that justifies the choice of weight decay.
Critical weaknesses
  1. Although the proposed SF-NorMuon outperforms tuned AdamW baselines without explicit learning-rate schedules, it still lags behind tuned NorMuon, which serves as a recent baseline for language model pretraining. The authors mention that this gap between SF-NorMuon and tuned NorMuon is expected, since tuned NorMuon can benefit from knowing the training horizon in advance. I wonder to what extent this gap could be reduced through additional algorithmic improvements and whether there is a fundamental, unavoidable performance gap between schedule-free methods and tuned methods.
Questions for authors
  1. There have been some recent revisions to the original schedule-free framework to address its reported limitations and make it more scalable for language model pretraining. For example, Defazio (2026) introduces several techniques, including adding inner momentum and annealing the schedule-free parameter βt\beta_t, to improve scalability to larger batches. In closely related work, Kim et al. (2026) also combine the schedule-free framework with the Muon optimizer, annealing βt\beta_t in a different way to further improve their optimizer. I wonder whether these techniques could also be applied to SF-NorMuon to improve its performance in language model training.
  • Defazio, A., ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models, 2026.
  • Kim et al., AMUSE: Anytime Muon with Stable Gradient Evaluation, 2026.
8 comments

Dear Reviewer W2Hj,

Thank you for your valuable comments and feedback. We are glad you found anytime training increasingly important, and our theory (convergence guarantee and quasi steady-state analysis) supportive of SF-NorMuon. Your suggestion led to our main new result, which we detail below.

Q: SF-NorMuon lags behind tuned NorMuon. Can the gap be reduced, or is it fundamental?

A: The gap can be closed, and SF-NorMuon is complementary to such algorithmic improvements. Following your suggestion (next question), we combined SF-NorMuon with the time-varying interpolation coefficient βt\beta_t of AMUSE [1], together with a norm-preserving variant of our update scaling (Appendix E.6). With a single configuration for all horizons and no schedule, SF-NorMuon now outperforms NorMuon tuned separately for each horizon, as well as AMUSE (125M model, validation loss):

ChinchillaSF-NorMuonAMUSE [1]NorMuon (cosine, tuned per horizon)AdamW (cosine, tuned per horizon)SF-AdamW
1×1\times3.2743.2783.2923.3543.399
2×2\times3.1933.1983.2003.2313.282
4×4\times3.1373.1413.1393.1703.205
8×8\times3.0983.1013.1033.1243.141

SF-NorMuon uses a single configuration for all horizons, with all hyperparameters set to the values tuned for AMUSE except for the learning rate. AMUSE is run with its released code and the best configuration reported in its paper for the 124M model (η=0.01\eta=0.01, βinit=0.6\beta_{\rm init}=0.6, ρ=0.8\rho=0.8), with its 6,000 warmup steps converted to 1,500 steps at our 4×4\times larger batch (the same number of warmup tokens). NorMuon, AdamW, and SF-AdamW are from Table 1.

Hence, the gap between schedule-free and horizon-tuned spectral optimizers is not fundamental. The revision adds these results to the abstract, introduction, Section 4, and Appendix E.5 (in blue). Due to limited GPU availability during the rebuttal period, we report the 125M model here; we will include the 772M results with the new setting in the final manuscript.


Q: Can the techniques of ScheduleFree+ [2] and AMUSE [1] be applied to SF-NorMuon?

A: Yes, both are compatible with SF-NorMuon:

  • Inner momentum [2]. SF-NorMuon already uses inner momentum, i.e., the explicit momentum buffer MtM_t in Algorithm 1. Removing it (μ=0\mu=0) degrades the 8×8\times loss from 3.14 to 3.28 (Figure 3, left).
  • Annealing β\beta [1, 2]. Our new results above use AMUSE's schedule 1−βt=(1−βinit)(T0/t)ρ1-\beta_t=(1-\beta_{\rm init})(T_0/t)^{\rho} after T0T_0 warmup steps; we set its two hyperparameters βinit\beta_{\rm init} and ρ\rho to the values tuned for AMUSE. ScheduleFree+ uses a similar idea, annealing β\beta during training.

Both works appeared on arXiv in May 2026; the revision cites both in Section 4 and discusses them in Appendix E.5 (in blue).


We hope the above answers clarify your questions, and if so, we hope you could consider reflecting on the score. Thank you again for your time in reviewing our manuscript.

[1] Kim et al. (2026) "AMUSE: Anytime Muon with Stable Gradient Evaluation", arXiv:2605.22432 [2] Defazio (2026) "ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models", arXiv:2605.19095

Thank you for the response. I have also reviewed the authors' responses to the other reviewers, but I become increasingly skeptical about the novelty of this paper, considering the updated abstract, arguments, and experiments.

In the previous draft, the authors argued that scheduled NorMuon serves as an upper bound on the performance of its schedule-free variant, since scheduled NorMuon can benefit from knowing the training horizon. I had hoped the authors would further clarify this claim, but instead, they revised their argument without addressing my questions, relying on a time-varying βt\beta_t, which is not even an original idea from the authors (Defazio et al., 2026; Kim et al., 2026). This is a serious concern, as most of the performance improvements appear to come from this technique, as can be seen by comparing Table 1 and Table 9. Given that, I believe it is hard to find any algorithmic novelty in this paper.

To be more specific, according to the appendix E.5 and Table 10, the revised SF-NorMuon follows the βt\beta_t scheduling of AMUSE (Kim et al., 2026), merely replacing the base optimizer Muon with NorMuon. Could the authors clarify whether there are any algorithmic contributions beyond those already introduced in AMUSE? Without a clear answer to this question, I am inclined to recommend rejection of this paper.

Dear Reviewer W2Hj,

Thank you for your follow-up. We appreciate your concern about novelty and would like to clarify the following points.


1. Our revisions directly address your original questions.

In your original review, you asked (i) whether techniques from AMUSE and ScheduleFree+ could improve SF-NorMuon, and (ii) whether the gap between schedule-free and horizon-tuned methods could be reduced through additional algorithmic improvements, or whether it was fundamentally unavoidable.

Our additional experiments were conducted precisely to answer these questions. Following your suggestion, we incorporated AMUSE's time-varying interpolation coefficient βt\beta_t, with explicit attribution to Kim et al. (2026). We also investigated a norm-preserving update scaling, detailed in Appendix E.6.

Our answer is that the gap can indeed be closed in the tested setting. With these modifications, SF-NorMuon outperforms horizon-tuned NorMuon at all four evaluated horizons on the 125M model, using a single configuration without a learning-rate schedule (Table 9).

Regarding our earlier reference to horizon-tuned NorMuon as an "upper bound," we meant this in an empirical rather than a formal mathematical sense: horizon-aware methods can exploit knowledge of the training horizon through horizon-specific tuning. Our revised results show that SF-NorMuon can outperform this strong baseline despite not having access to the horizon, directly addressing your original question.


2. The improvement is not simply due to AMUSE's βt\beta_t: Appendix E.6 is important.

We would particularly like to emphasize Appendix E.6 (Norm-Preserving Update Scaling), which was already included in our revision and seems to have been overlooked in your follow-up.

The improved SF-NorMuon results use two distinct modifications: (i) AMUSE's time-varying βt\beta_t, and (ii) the norm-preserving scaling of Appendix E.6.

Simply incorporating AMUSE's βt\beta_t does not reproduce the improved results in Table 9. Our initial experiment with time-varying βt\beta_t yielded slightly worse performance at 1×1\times, 2×2\times, and 4×4\times, and only a modest improvement at 8×8\times (3.131 vs. 3.136 in Table 1). Since time-varying βt\beta_t changes the gradient evaluation points throughout training, we also revisited the hyperparameters and investigated the norm-preserving scaling of Appendix E.6. We obtained the following results:

Method1×2×4×8×
SF-NorMuon (Table 1)3.3183.2293.1703.136
SF-NorMuon + βt\beta_t3.3223.2383.1773.131
SF-NorMuon + βt\beta_t + hyperparameter tuning3.2883.2083.1513.110
SF-NorMuon + βt\beta_t + hyperparameter tuning + E.63.2743.1933.1373.098
AMUSE3.2793.1993.1423.102

125M model, validation cross-entropy.

Even after incorporating AMUSE's interpolation and tuning the hyperparameters, SF-NorMuon without E.6 remains behind AMUSE at every horizon. The norm-preserving variant of Appendix E.6, together with the final configuration, closes this gap.

Appendix E.6 retains NorMuon's neuron-wise adaptive update direction while preserving the Frobenius norm of Muon's Newton–Schulz update, rather than enforcing a fixed update RMS. This allows the update magnitude to reflect the Newton–Schulz output without introducing additional hyperparameters.

Comparing Table 1 with Table 9 therefore cannot isolate the contribution of βt\beta_t, since both the optimization hyperparameters and update scaling change. The experiments above demonstrate that the improvement cannot simply be attributed to AMUSE's interpolation. We hope this clarifies the role of Appendix E.6 in our revised method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.