acceptodds
Under review as a conference paper at ICLR 2027

AMO: Operator-level Adaptive Muon Orthogonalization

Abstract

Muon has recently emerged as a competitive alternative to AdamW for large-scale pre-training, with orthogonalization via Newton–Schulz (NS) iteration as its core operation. Standard Muon applies a uniform NS schedule to all parameter matrices, overlooking possible differences in orthogonalization difficulty and its impact on performance. Through a systematic empirical study, we show that this per-matrix heterogeneity is pervasive and strongly associated with matrix geometry, which evolves dynamically across operator types, training stages, and network depths. Therefore, uniform NS schedules can lead to uneven orthogonalization quality across the model. Motivated by these findings, we propose Operator-level Adaptive Muon Orthogonalization (AMO), an observe-then-commit method that measures weight geometry by operator type early in training and then uses these signals to allocate the NS budget for the remainder of training. AMO delivers consistent improvements over uniform-schedule Muon across standard, prolonged, and continual pre-training, surpassing the strongest baseline by +0.76 on Llama3.1-1.4B and +0.51 on Qwen3-1.7B in average downstream performance of 12 evaluation tasks, with gains persisting at Llama3.1-4B scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pre-training, optimizer, newton-schultz iteration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Zmr5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper argues that a uniform Newton–Schulz schedule in Muon leaves orthogonalization quality uneven across matrices, and that this heterogeneity is organized by operator type. It proposes AMO, which measures a per-type spectral statistic during an observation phase, reallocates a fixed budget of Newton–Schulz steps across the seven operator types using Polar Express coefficients, and then locks the schedule for the rest of training. On Llama3.1 and Qwen3 models up to 4B, it reports gains of 0.2 to 0.8 points in average downstream accuracy over the strongest Muon variants.

Critical strengths
  1. AMO is a practical, drop-in improvement over strong Muon baselines. It adds no optimizer state, matches the step time of Muon once the schedule is locked, and beats every baseline including Muon-PE in all main comparisons across two model families, several scales and three training regimes. Paired multi-seed runs confirm the gain over Muon.
  2. The geometry analysis gives the design a sound empirical basis. It shows that conditioning is organized by operator type, varies little across layers and keeps a stable ordering after early training. This matters because it makes operator type a measured choice of adaptation unit rather than a guess, and the finding is useful beyond this method.
Critical weaknesses
  1. The mechanism behind the gain is not established. The paper attributes the improvement to more uniform orthogonalization, but it never measures the per-type residual after the switch or relates the residual to loss. It also leaves update size uncontrolled. With a fixed output scale, changing the steps and coefficients of an operator type changes the size of its updates, so AMO also acts as a per-type learning-rate change. Two results in the paper fit this reading. Raising the NS budget by 20% improves the downstream average by only about 0.1 points, so more accurate orthogonalization helps little here. Yet reaching the same final schedule in one step rather than over 300 steps removes the entire gain. The paper attributes this to the momentum buffer, but the buffer does not depend on the schedule, which suggests that what matters is how the update itself changes. If the gain comes from update scaling, the right comparison is per-type learning-rate tuning rather than other NS schedules.
  2. The evaluation is too noisy to support gains this small. The results come from single runs scored on a twelve-task average, with no validation loss. Five of those tasks (MMLU, MMLU-Pro, CommonsenseQA, AGIEval and OpenBookQA) are close to random guessing for models of this size, yet they account for much of AMO’s lead over Muon-PE. Muon-PE is left out of the multi-seed and 4B runs, and AMO’s settings were chosen by sweeping on this same average, so the reported margin is likely optimistic. Since the gains are only a few tenths of a point, these issues make it hard to tell how much of the improvement is real.
Questions for authors
  1. Does the gain survive when update size is held fixed? Please report per-type update norms and NS residuals after the lock, and rerun AMO with each type’s output rescaled to the norm it has under the uniform schedule. If the gain remains, it supports the orthogonalization account. If it disappears, the gain is better explained by update scaling.
  2. How large is the gain under a more precise evaluation? Please report held-out validation loss for AMO and Muon-PE in the main comparisons, and add Muon-PE to the paired multi-seed study. Excluding the five tasks at chance, AMO and Muon-PE are tied at 760M, so the smaller margins need this support.
7 comments
AuthorsOpenReview ↗

Response to Reviewer Zmr5 [part 1]

We thank the reviewer for the careful review and for recognizing AMO's practicality and geometric analysis. We address the mechanism and evaluation concerns below.

W1(a,d) (momentum buffer and schedule transition)

We agree that the momentum buffer accumulates raw stochastic gradients and does not directly depend on the NS schedule. Our intended explanation concerns the orthogonalized update computed from this buffer and applied to the weights: an abrupt schedule switch changes the transformation of the buffer, potentially affecting both the singular-value profile and the update magnitude. We have corrected the wording in Secs. 3.2 and 4.4.

We also agree that the transition ablation alone does not distinguish improved orthogonalization from update scaling. The Δ=1\Delta = 1 result (37.27 vs. 37.33 for Muon-PE) shows that an abrupt transition does not retain the observed gain, but does not establish which mechanism accounts for it. We therefore address update magnitude separately below.

W1(b) / Q1 (per-type residuals and update-norm control)

We agree that changing the NS steps and coefficients can change per-type update norms. We applied the uniform Muon-PE schedule and AMO's locked schedule to the same post-lock momentum matrices from each run's final checkpoint, and we report per-type NS residuals and update norms for all four models in the new App. G.3. The main findings are:

  1. Residuals. AMO lowers the residual for 23 of the 28 type–model pairs. For the most ill-conditioned attention projections, which receive an extra step, the residual drops by 41–43% on every model. The total over the seven types drops by 31–49% on three models and by 6% on Qwen3-1.7B.
  2. Update norms. Relative to Muon-PE, AMO changes ∥O∥F\lVert O \rVert_F by at most 5% for 27 of the 28 type–model pairs (the single exception changes it by 8%).
  3. These norm changes are part of the orthogonalization error, not an independent scale. The exact polar factor has ∥UV⊤∥F=r\lVert UV^\top \rVert_F = \sqrt{r} regardless of the schedule. Muon-PE undershoots this value on the ill-conditioned types (∥O∥F/r=0.93\lVert O \rVert_F/\sqrt{r} = 0.93–0.970.97), and AMO brings them to 0.98–0.99. AMO's output norm lies within 2.5% of r\sqrt{r} for 27 of the 28 type–model pairs.

This offline analysis shows that the norm changes are small and are a by-product of better orthogonalization, but it cannot exclude a contribution from them. We will include a norm-controlled retraining experiment in the final version.

W1(c) (additional NS accuracy vs. budget reallocation)

We agree that increasing the NS budget beyond the baseline yields only a small downstream improvement in this setting: +0.11 points for 20% more budget. By comparison, AMO improves over Muon-PE by +0.44 points on Qwen3-0.6B while keeping the total NS budget fixed.

These results suggest that allocating NS effort across operator types matters more than uniformly increasing it in this setting. They do not, by themselves, determine whether the gain comes from improved orthogonalization, changes in update magnitude, or both.

W2(a) (evaluation excluding near-chance tasks)

We agree that the near-chance tasks warrant a separate evaluation. After excluding the five identified tasks, AMO and Muon-PE are effectively tied on Llama3.1-760M (50.36 vs. 50.38), which we now state explicitly.

The gains at the other scales remain on this seven-task subset. AMO exceeds the strongest baseline by +0.50 points on Llama3.1-1.4B, +0.39 on Qwen3-0.6B, and +0.38 on Qwen3-1.7B, and exceeds Muon by +0.66 at 4B. In the paired multi-seed study, AMO also exceeds Muon on this seven-task subset in all three seeds (+0.81 / +0.47 / +0.12). Thus, the improvements in these settings are not solely attributable to the five near-chance tasks.

W2(b) / Q2 (held-out validation loss)

We now report held-out validation loss in Fig. 15 (Page 39). AMO achieves the lowest final validation loss among all nine optimizers in each of the four settings, improving over Muon-PE by 0.0015–0.0039. For context, Muon-PE improves over Muon-KJ by 0.0020–0.0040.

The smallest AMO–Muon-PE gap occurs on Llama3.1-760M, consistent with the downstream tie on the seven-task subset. These results provide complementary evidence beyond downstream accuracy.

AuthorsOpenReview ↗

Response to Reviewer Zmr5 [part 2]

W2(c) / Q2 (Muon-PE in the multi-seed and 4B comparisons)

We acknowledge that Muon-PE is missing from the multi-seed experiment and the 4B run due to compute constraints. The multi-seed study does include a PE-based static per-type schedule (Best Static), which AMO outperforms in all three seeds (+0.58 ± 0.08, mean ± s.e. over seeds). We will add the Muon-PE runs in the final version if compute permits.

W2(d) (hyperparameter selection and sensitivity)

We thank the reviewer for raising this concern, and would like to clarify that AMO's default hyperparameters were fixed before the ablation study. The subsequent sweep examined sensitivity around those defaults and was not used to select the configuration with the highest evaluation average. For example, nsmp=16n_{\mathrm{smp}} = 16 achieves a higher score than the default nsmp=8n_{\mathrm{smp}} = 8 (37.83 vs. 37.77), but we retained the latter for its lower capture cost. The same defaults were used unchanged across all other models and settings (Tab. 3).

Similarly, the learning rate was not tuned separately for AMO. For each model, the base learning rate was computed from the empirical scaling rule of Zhou et al. (2026), using model size and token budget, and shared by AMO and all Muon-style baselines (App. F.1).

The improvement is not confined to the default configuration: eight of the nine non-default adaptive configurations outperform Muon-PE (37.33), with a median score of 37.63. We have revised the caption of Fig. 4.

Thank you for the detailed response and the additional analyses. My main concern, however, is the same as in my review: the improvement is small relative to the evaluation noise.

Your response confirms part of this. After the five near-chance tasks are removed, AMO and Muon-PE are tied on Llama3.1-760M, and the margin over the strongest baseline shrinks elsewhere. The new validation-loss results show the same picture: AMO improves over Muon-PE by 0.0015–0.0039, roughly 0.02–0.05 in perplexity, from single-seed curves without error bars. For a pretraining paper, validation perplexity is a standard and less noisy measure of optimization quality, so I would also suggest reporting it for the main comparisons in the main text rather than only in an appendix figure.

With gains of this size, the paper needs one of two things: (1) a controlled experiment showing that the gain comes from AMO itself, for example AMO with per-type update norms rescaled to those of Muon-PE; or (2) paired multi-seed runs showing that the gain over Muon-PE is stable. Neither is in the revision. I am not asking for large-scale runs. The smallest main setting, Qwen3-0.6B with 12B tokens, would be enough for the discussion period. For either experiment, please report at least the validation perplexity, in addition to the downstream results.

I would appreciate a response to each of these points. If the additional evidence shows that the gain is attributable to AMO or is stable across seeds, I would be glad to raise my score.

Reviewer_qMYLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a heterogeneous allocation of Newton-Schultz (NS) steps during Muon-optimizer training. They who empirically that different operators have different "difficulties", and propose the operator type as a "split" mechanism to allocate the NS iterations. They allocate 3-7 steps. The argument is that by applying enough orthogonalization to the right matrices, improvement is observed in convergence, including downstream tasks.

Critical strengths

I like the empirical study of the NS difficulty, I thin kthis is sound.

I like that the abstract makes very clear what problem you are solving. Clarity is generally pretty good.

Critical weaknesses
  1. This problem is simply not interesting from a performance angle, since overhead of Newton Schultz is tiny. In a well implemented distributed setup, [e.g. https://arxiv.org/abs/2606.27153] the overhead of Muon relative to AdamW is tiny, maybe 1%.

  2. If we are to buy that distributing the NS steps is an important contribution, then a critial baseline is missing: simply apply the maximum number of steps to each matrix (7, in this case). If your claims are accurate, this setup should match of slightly beat the proposed AMO method. If this is worse, then the paper needs a significant re-working to explain why this is so.

Questions for authors
  1. You own work shows that the muon overhead is small (6.13s vs AdamWs 5.96). Why are you so interested in optimising this?
  2. what happens when you uniformly schedule 7 steps everywhere?
4 comments
AuthorsOpenReview ↗

Response to Reviewer qMYL

We thank the reviewer for the feedback and for finding our empirical study of NS difficulty sound. We address the positioning and additional baseline concerns below.

W1/Q1 (why optimize NS if its overhead is small)

We agree that NS accounts for only a small fraction of end-to-end training time. Our objective is therefore to improve model quality at essentially unchanged training cost. A small runtime share does not imply that the NS schedule has little influence on optimization: its steps and coefficients determine the approximate orthogonalized updates applied throughout training. Every Muon variant must already choose this schedule, so the relevant question is how to spend a fixed nominal NS budget well. AMO uses the model's own geometry to guide this choice:

  1. Reallocation. At a fixed total step budget, AMO shifts steps from easy to hard operator types (e.g., 6 steps for attn_q/o and 4 for mlp_gate/down on Qwen3-1.7B, Tab. 2). It outperforms the best tested static per-type schedule on both model families (Sec. 5.2), supporting the value of geometry-informed configuration.
  2. Better models at nearly identical training cost. AMO improves over the strongest baseline by +0.76 points on Llama3.1-1.4B and +0.51 on Qwen3-1.7B, and over Muon by +0.75 at 4B. On Qwen3-1.7B, its locked-phase step time matches Muon (6.13 vs. 6.14 s); including observation overhead, the full-run averages are 6.17 vs. 6.14 s.
  3. A practical quality–budget tradeoff. The reported budget sweep shows monotonically improving downstream scores as the budget increases (Fig. 5a). On Qwen3-0.6B, AMO still exceeds full-budget uniform Muon-PE with a 20% smaller locked-phase NS-step budget (37.53 vs. 37.33). This supports allocating NS effort selectively rather than assuming that every operator requires the same schedule.

Thus, AMO's value lies in improving the quality obtained from an inexpensive but consequential optimizer operation. We have revised the abstract, Sec. 1, and Sec. 4.3 to clarify this objective and distinguish nominal step budgets from measured runtime.

W2/Q2 (uniform 7 steps everywhere)

Thank you for suggesting this important baseline; we have added the comparison.

We train Llama3.1-760M with seven NS steps for every operator type, using PE coefficients recomputed for seven steps at the default lower bound. All other settings are identical to Muon-PE. This baseline uses 49 nominal NS steps across the seven operator types, compared with 35 for default AMO and 42 for AMO at a budget ratio of 1.2.

Uniform-7 Muon-PE achieves 37.44, compared with 37.06 for Uniform-5 Muon-PE, 37.26 for default AMO, and 37.39 for AMO at a budget ratio of 1.2 (full results in Tab. 21 on Page 44). Thus, as the reviewer anticipated, uniformly increasing NS steps can slightly exceed AMO in this setting. We agree that this is an effective alternative when additional NS budget is available.

These results further clarify the scope of our contribution. At the same nominal budget, AMO improves over Uniform-5 Muon-PE by 0.20 points. Increasing the uniform budget by 40% yields a further 0.18-point advantage over default AMO. Meanwhile, AMO at a budget ratio of 1.2 comes within 0.05 points of Uniform-7 while using 14.3% fewer nominal NS steps.

Together with the budget sweep in Sec. 5.1, this comparison supports a quality–budget tradeoff: geometry-informed adaptation improves quality at a fixed nominal budget, while additional uniform steps remain a viable way to improve performance.

Thanks for the reply to my review.

I'm just not convinced this is a useful contribution. You acknowledge that the NS steps are a small part of the overall compute, and you acknowledge that just setting the budget to "max" works best. So the paper could be summarised as "remember to do enough NS steps in Muon".

It appears to me that the "NS step budget" is an artificial construct.

AuthorsOpenReview ↗

Response to Follow-up

Thank you for the follow-up. We would like to clarify what our experiments establish and what AMO contributes.

  1. Small runtime overhead does not imply limited optimization significance. [1] shows that approximation error affects convergence guarantees and interacts with learning-rate and momentum choices, with supporting NanoGPT experiments. In industry-level large-scale practice, DeepSeek-V4 [2] uses a ten-step hybrid NS scheme with different coefficients for its first eight and final two iterations (in their Section 2.4). These findings support studying both NS steps and coefficients, even when their runtime share is small.
  2. The nominal budget serves as an experimental control. Holding the aggregate iteration count fixed tests whether geometry-informed configuration improves quality without simply adding iterations. Under this control, AMO improves over Uniform-5 PE and several baseline methods, including gains of 0.76 points on Llama3.1-1.4B and 0.51 on Qwen3-1.7B. We distinguish this nominal budget from actual FLOPs and end-to-end runtime.
  3. AMO makes orthogonalization configuration informed by measured geometry. The empirical study you found sound identifies systematic differences in orthogonalization difficulty. AMO uses these observations to select operator-specific step counts and coefficients through an observe-then-commit procedure. This provides a concrete method for determining where additional orthogonalization is useful. We acknowledge that Uniform-7 achieves a higher score in the added comparison and is a practical alternative using 1.4x more NS steps. However, this does not explain the improvements AMO obtains without increasing the aggregate iteration count. We therefore believe its contribution should be assessed through both the empirical findings and the demonstrated configuration method.

References

[1] Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Peter Richtárik. Beyond the Ideal: Analyzing the Inexact Muon Update. arXiv:2510.19933, 2025.

[2] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026.

Dear Reviewer qMYL,

We would like to kindly follow up on our rebuttal and ask whether our responses have addressed your concerns. We would greatly appreciate any further feedback or questions you may have.

Thank you for your time and consideration.

Reviewer_bv3pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the Newton–Schulz (NS) orthogonalization in Muon should be allocated across parameter matrices. Through a pilot study on eight Llama3.1 and Qwen3 models, the authors find that the condition number of the NS input is correlated with the resulting orthogonalization error. They also find that this difficulty differs persistently across operator types (e.g., attention query/value versus MLP down-projection), whereas differences across layers are modest and their ordering changes during training. A separate study shows that monotone quality-based data curricula do not outperform random data ordering, which the authors take as motivation for addressing the heterogeneity on the optimizer side.

Based on these findings, the authors propose AMO, an observe-then-commit method. During an early observation window under the standard uniform schedule, AMO measures a geometry signal for each of the seven operator types. It then allocates a fixed total NS step budget across operator types with a greedy algorithm over projected NS error curves, recomputes Polar-Express coefficients for each type, transitions linearly to this schedule, and keeps it fixed for the rest of training. Experiments on Llama3.1 and Qwen3 models from 0.6B to 1.7B parameters (and Llama3.1-4B) show that AMO improves the average accuracy over 12 downstream benchmarks relative to Muon and other Muon variants by about 0.2–0.8 points, in standard, prolonged and continual pretraining, with negligible overhead once the schedule is locked.

Critical strengths
  1. A well-motivated choice of adaptation granularity. The paper does not simply assume that matrices differ; it measures the geometry of Muon's NS inputs across eight models and two families. It shows that the NS-input condition number is predictive of NS error, that operator types are persistently separated, and that layer-wise differences are small and their ordering reverses during training (including after warmup, Appendix G.3). These observations give a clear and empirically grounded reason for adapting at the operator-type level rather than by matrix shape or by depth. I find this careful analysis the most convincing part of the paper.

  2. A broad and carefully documented evaluation. The method is compared with a large set of Muon variants under several training regimes (standard, prolonged up to 10× Chinchilla, and continual pretraining), on two model families and up to 4B parameters. The paper also includes ablations of the main hyperparameters, a budget-control study that separates allocation from additional compute, comparisons with tuned static schedules, and a finer Type × Depth variant. Implementation details and costs are reported in detail. Taken together, this makes the empirical claims easy to examine and gives reasonable confidence that the gains are consistent in the tested settings.

Critical weaknesses
  1. It is not fully clear which part of AMO produces the gains, or whether they come from improved orthogonalization. AMO changes two things at once: the number of NS steps per operator type and the PE coefficients, which are recomputed at a measured ℓtarget\ell_{\text{target}} instead of the conservative ℓ=10−3\ell=10^{-3}. The final schedules (Table 2) differ from the uniform baseline by at most one or two steps, while ℓtarget\ell_{\text{target}} varies by more than an order of magnitude across types. Appendix G.2 varies the step counts with fixed KJ coefficients, and Section 5.2 compares static per-type schedules, which partly addresses this. However, I could not find a comparison that keeps five steps for every type while using the per-type coefficients, so the relative contribution of step reallocation and coefficient re-fitting within AMO remains unclear; the shrinkage ablation with α=0\alpha=0 removes both effects together. In addition, the allocation minimizes an unweighted sum of projected errors from the scalar PE simulation, and the paper does not report whether the measured NS residual of each operator type actually decreases during the locked phase, or how this relates to the loss. As a result, the link between the motivating observation (operator-dependent NS quality) and the reported improvements is not directly established.

  2. The size of the gains is modest relative to the evaluation noise. The main improvements are 0.2–0.8 points in average accuracy over 12 benchmarks, and the differences among the stronger Muon variants are of a similar size. Several of the benchmarks appear to be close to chance level at these model scales; for example, MMLU is around 25%, CommonsenseQA around 20%, MMLU-Pro below 10%, and AGIEval around 17%. Changes on these tasks may mostly reflect noise. The paper does not report pretraining validation loss or perplexity for the main comparisons, which would be a less noisy measure of optimization quality, except for the training-loss curve in Fig. 6. I appreciate the multi-seed experiment on Llama3.1-1.4B, but it would help to see a more direct optimization metric across all main settings.

  3. Some claims seem stronger than what is supported.

    • Section 4.3 and Appendix E state that AMO is in a "distinct complexity regime" and "can reduce actual cost by shifting steps from expensive types to cheaper ones". However, BudgetAllocate minimizes the unweighted projected error under a step-count budget and does not take the per-type cost ∑iCt,i\sum_i C_{t,i} into account. At r=1.0r=1.0 the per-step cost is essentially the same as Muon's, and the exact-SVD observation steps add overhead.
    • The optimality result for BudgetAllocate (Theorem C.1) relies on the discrete convexity of the projected error curves (A3), which is not checked.
    • The data-curriculum study shows that curriculum orderings underperform random ordering. It does not measure whether curricula change the per-matrix heterogeneity, so it is only loosely connected to the motivation for optimizer-side adaptation.

Minor: In Section 5.3, the operator-type AMO score on Llama3.1-1.4B is reported as 39.55 and the Muon score as 38.66, whereas Table 16 reports 39.69 for AMO and 38.81 for Muon-KJ. The numbers in Section 5.3 appear to correspond to different seeds in Table 14 (AMO with seed 2026 and Muon with seed 43); the authors may want to clarify which runs are being compared.

Questions for authors

I would appreciate it if the authors could clarify the points raised in the weaknesses, in particular how much of AMO's improvement comes from reallocating NS steps versus re-fitting the PE coefficients (for example, by keeping five steps for every type while using the per-type ℓtarget\ell_{\text{target}}, which complements Appendix G.2), and whether the measured NS residual of each operator type and the pretraining validation loss improve accordingly.

If these points can be clarified and supported with evidence, I would be glad to reconsider my assessment.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bv3p [part 1]

We thank the reviewer for the careful and constructive review, and for recognizing our analysis of the adaptation granularity and the breadth of our evaluation. In response, we added an offline decomposition of step reallocation vs. coefficient re-fitting with per-type NS residuals, statistical significance tests, held-out validation loss for all optimizers, and a direct check of the assumptions behind Theorem C.1, and we have corrected the overstated claims. Details are below.

W1 (step reallocation vs. coefficient re-fitting; per-type residuals and loss)

We thank the reviewer for this precise question. Existing results partially separate the two components. With the shrinkage coefficient set to 0, AMO reduces to the uniform 5-step PE schedule (37.35 vs. 37.33 for Muon-PE), and App. G.2 varies the step counts under fixed coefficients. To isolate coefficient re-fitting without retraining, we applied three schedules to the same post-lock momentum matrices from each AMO run's final checkpoint: (a) Muon-PE; (b) 5 steps for every type with per-type re-fitted PE coefficients, which is the variant the reviewer suggests; and (c) AMO's locked schedule. The full per-type results for all four models are in the new App. G.3. The main findings are:

  1. Coefficient re-fitting alone does not help. Variant (b) increases the total NS residual by 1–20%. It improves well-conditioned types, but it increases the residual of the most ill-conditioned attention projections by roughly 2–2.6× (from 0.28–0.38 to ≈0.73), because PE polynomials fitted for a much smaller ℓ\ell need more iterations to converge.
  2. Step reallocation makes the re-fitted coefficients pay off. With one extra step, these types reach a residual 41–43% below Muon-PE on all four models. The step is funded by the best-conditioned types. The total residual drops by 31–49% on three models and by 6% on Qwen3-1.7B.

The two components are therefore complementary rather than additive, so a percentage split is not well defined: the gain comes from choosing TT and ℓ\ell jointly, which is what AMO's projected error curves do.

In addition:

  1. Locked phase. Measured on post-lock matrices at the end of training, AMO's residual is lower than Muon-PE's for 23 of the 28 type–model pairs, including every type that receives an extra step. At the end of training, these types also remain by far the most ill-conditioned (ℓmeas≈10−6\ell_{\mathrm{meas}} \approx 10^{-6}, versus ≥10−3\ge 10^{-3} for all other types).
  2. Loss. AMO attains the lowest held-out validation loss among all optimizers in all four settings (Fig. 15).

W2 (gains and evaluation noise)

We address this in four ways.

(1) Statistical significance. We computed the standard error of every benchmark score as p(1−p)/n\sqrt{p(1-p)/n} over its test set and propagated it to the 12-task average. All comparisons below use the unpaired standard error, which is conservative because all methods are scored on exactly the same items. Treating the four main models as independent, AMO improves over Muon-KJ by +0.66 ± 0.23 points (z=2.86z = 2.86, p=0.004p = 0.004) and over Muon-PE by +0.48 ± 0.23 points (z=2.08z = 2.08, p=0.04p = 0.04), and it outperforms both baselines on all four models. Across the 4×124 \times 12 model–benchmark pairs, AMO outperforms Muon-KJ on 38 and underperforms on 9 (sign test, p<10−4p < 10^{-4}). Against Muon-PE the count is 31 versus 14 (p=0.016p = 0.016).

(2) Paired seeds. On Llama3.1-1.4B, AMO improves over Muon in all three seeds (+0.50 on average, std 0.22). It also improves over the best static schedule (Floor-1e-2) in all three seeds, by +0.58 ± 0.08 points (mean ± s.e. over seeds; paired tt-test, p=0.017p = 0.017). The best static schedule is also a per-type PE schedule, but its per-type ℓ\ell and step counts are fixed offline from pilot-model statistics, whereas AMO derives both online from the model's own geometry under a fixed budget. This comparison therefore isolates the benefit of online, geometry-informed adaptation over an offline per-type design.

(3) Held-out validation loss. We evaluated held-out FineWeb-Edu loss on saved checkpoints of all optimizers (Fig. 15 on Page 39). AMO attains the lowest final validation loss among all nine optimizers in all four settings. It improves over Muon-PE by 0.0015–0.0039 and over Muon-KJ by 0.0045–0.0079. For reference, Muon-PE improves over Muon-KJ by 0.0020–0.0040, so AMO's gain over PE is of the same order as PE's own gain over KJ. The ordering is also stable across late-training checkpoints.

(4) Reliable-task subset. Excluding near-chance tasks, AMO leads the strongest baseline by +0.38 to +0.50 on Llama3.1-1.4B and Qwen3-0.6B/1.7B, and leads Muon by +0.66 at 4B. On Llama3.1-760M it ties Muon-PE (50.36 vs. 50.38). Notably, this is also the setting with the smallest validation-loss gap, so the two metrics agree.

AuthorsOpenReview ↗

Response to Reviewer bv3p [part 2]

W3a (cost claims)

Our central cost claim is that AMO matches Muon's cost, and it is supported by direct measurement (Tab. 1). After locking, AMO runs at 6.13 s/step versus 6.14 s/step for Muon, with identical throughput (0.80M tokens/s) and peak memory (61.12 GB). The observation phase adds 20 exact-SVD steps, about 0.45% of total wall-clock time on Qwen3-1.7B (full-run average 6.17 vs. 6.14 s/step). In addition, the locked schedule reduces NS FLOPs by about 7% on Qwen3-1.7B, because the measured hard types (attention projections) are cheaper than the MLP projections.

We agree that BudgetAllocate does not take CtC_t into account, so the phrase "distinct complexity regime" overstated this side effect. We have removed it, and Sec. 4.3 now states the claim exactly as measured: AMO improves quality at cost parity with Muon.

W3b (Assumption A3)

Thank you for raising this. We checked the assumptions of Theorem C.1 directly on all 28 projected error curves (4 models × 7 operator types) used to plan the final AMO schedules, and added the results in Appendix C.2.

  • A2 holds for all 28 curves.
  • A3 does not hold in general. It fails for 13/28 curves, all of them attention projections with ℓtarget≤1.48×10−3\ell_{\mathrm{target}} \le 1.48\times10^{-3} (after shrinkage toward 10−310^{-3}, ℓtarget=0.7 ℓobs+0.3×10−3\ell_{\mathrm{target}} = 0.7\,\ell_{\mathrm{obs}} + 0.3\times10^{-3}); all 15 curves with ℓtarget≥1.58×10−3\ell_{\mathrm{target}} \ge 1.58\times10^{-3} satisfy A3, including every MLP type. The violation is structural, not numerical. For ill-conditioned inputs, the first NS iterations mainly lift the smallest singular values while the error stays close to 1, and only afterwards does the error fall superlinearly. Ei(k)E_i(k) is therefore S-shaped: the marginal gain increases from k=3k=3 to k=5k=5 before it diminishes (minimum second difference −0.218-0.218, always at k=4k=4). Theorem C.1 therefore gives a sufficient condition that does not cover the hardest operator types.
  • BudgetAllocate is still globally optimal in every run. With 7 types and 5 candidate step counts, there are 57=78,1255^7 = 78{,}125 schedules, so we enumerated all of them. For all 4 models the optimum is unique and identical to the output of BudgetAllocate (objective gap exactly 0). The allocation reduces the total projected error ∑iEi(Ti)\sum_i E_i(T_i) by 28–56% relative to uniform 5-step PE.

Because this search agrees with BudgetAllocate in all our runs, all reported results remain unchanged.

W3c (data curriculum experiment)

We would like to clarify the intended role of Sec. 2.1, which our wording did not convey well. The curriculum study is not meant as evidence that curricula fail to reduce per-matrix heterogeneity, nor as a claim that curricula are ineffective in general. It serves as a motivating observation: a natural data-side lever, monotone quality-based ordering, does not reliably improve final performance across 16 configurations. This led us to look on the optimizer side, where Sec. 2.2 then characterizes the heterogeneity directly. We agree that Sec. 2.1 does not measure heterogeneity itself, and we have revised Q1 and the first contribution accordingly, so that Sec. 2.1 is presented as motivation rather than as a test of heterogeneity reduction.

Minor (Section 5.3 numbers)

Thank you for catching this inconsistency. All experiments except the multi-seed study (Tab. 14) use seed 42, including the Type×Depth run. Sec. 5.3 mistakenly quoted Muon and AMO numbers from the multi-seed runs. With same-seed (seed 42) comparisons from Tab. 16, Type×Depth AMO reaches 38.91, slightly above Muon-KJ (38.81) but 0.78 points below operator-type AMO (39.69). The conclusion of Sec. 5.3 is therefore unchanged, and the gap is in fact larger than previously reported. We have corrected the numbers in Sec. 5.3, added the Type×Depth run to Tab. 16, and stated the seeds explicitly in the table captions.

Reviewer_bv3pOpenReview ↗

Response to Authors

Thanks for the detailed response, which addresses most of my concerns. However, my central concern, W1, remains unresolved. The new App. G.3 only shows that AMO reduces the orthogonalization residual. It does not establish that this reduction is the source of the performance gains:

  1. Residual reduction does not track the gains. On Qwen3-1.7B, the residual drops by only 6%, yet the validation-loss improvement is the largest. On Llama3.1-760M, the residual drops by 37%, yet the improvement is the smallest.
  2. The Δ=1\Delta = 1 ablation is hard to reconcile with the proposed mechanism. With Δ=1\Delta = 1, AMO reaches the same locked schedule but loses the entire gain (37.27 vs. 37.33 for Muon-PE). If the benefit came from better orthogonalization in the locked phase, a difference confined to the transition should not remove it. Could the authors show the training-loss curves around the switch for Δ=1\Delta = 1 and Δ=300\Delta = 300?

Therefore, I am inclined to keep my current score unless the authors can provide stronger evidence that the improved orthogonalization is what drives the gains in optimization performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.