- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper argues that a uniform Newton–Schulz schedule in Muon leaves orthogonalization quality uneven across matrices, and that this heterogeneity is organized by operator type. It proposes AMO, which measures a per-type spectral statistic during an observation phase, reallocates a fixed budget of Newton–Schulz steps across the seven operator types using Polar Express coefficients, and then locks the schedule for the rest of training. On Llama3.1 and Qwen3 models up to 4B, it reports gains of 0.2 to 0.8 points in average downstream accuracy over the strongest Muon variants.
Critical strengths
- AMO is a practical, drop-in improvement over strong Muon baselines. It adds no optimizer state, matches the step time of Muon once the schedule is locked, and beats every baseline including Muon-PE in all main comparisons across two model families, several scales and three training regimes. Paired multi-seed runs confirm the gain over Muon.
- The geometry analysis gives the design a sound empirical basis. It shows that conditioning is organized by operator type, varies little across layers and keeps a stable ordering after early training. This matters because it makes operator type a measured choice of adaptation unit rather than a guess, and the finding is useful beyond this method.
Critical weaknesses
- The mechanism behind the gain is not established. The paper attributes the improvement to more uniform orthogonalization, but it never measures the per-type residual after the switch or relates the residual to loss. It also leaves update size uncontrolled. With a fixed output scale, changing the steps and coefficients of an operator type changes the size of its updates, so AMO also acts as a per-type learning-rate change. Two results in the paper fit this reading. Raising the NS budget by 20% improves the downstream average by only about 0.1 points, so more accurate orthogonalization helps little here. Yet reaching the same final schedule in one step rather than over 300 steps removes the entire gain. The paper attributes this to the momentum buffer, but the buffer does not depend on the schedule, which suggests that what matters is how the update itself changes. If the gain comes from update scaling, the right comparison is per-type learning-rate tuning rather than other NS schedules.
- The evaluation is too noisy to support gains this small. The results come from single runs scored on a twelve-task average, with no validation loss. Five of those tasks (MMLU, MMLU-Pro, CommonsenseQA, AGIEval and OpenBookQA) are close to random guessing for models of this size, yet they account for much of AMO’s lead over Muon-PE. Muon-PE is left out of the multi-seed and 4B runs, and AMO’s settings were chosen by sweeping on this same average, so the reported margin is likely optimistic. Since the gains are only a few tenths of a point, these issues make it hard to tell how much of the improvement is real.
Questions for authors
- Does the gain survive when update size is held fixed? Please report per-type update norms and NS residuals after the lock, and rerun AMO with each type’s output rescaled to the norm it has under the uniform schedule. If the gain remains, it supports the orthogonalization account. If it disappears, the gain is better explained by update scaling.
- How large is the gain under a more precise evaluation? Please report held-out validation loss for AMO and Muon-PE in the main comparisons, and add Muon-PE to the paired multi-seed study. Excluding the five tasks at chance, AMO and Muon-PE are tied at 760M, so the smaller margins need this support.
7 comments
Response to Reviewer Zmr5 [part 2]
W2(c) / Q2 (Muon-PE in the multi-seed and 4B comparisons)
We acknowledge that Muon-PE is missing from the multi-seed experiment and the 4B run due to compute constraints. The multi-seed study does include a PE-based static per-type schedule (Best Static), which AMO outperforms in all three seeds (+0.58 ± 0.08, mean ± s.e. over seeds). We will add the Muon-PE runs in the final version if compute permits.
W2(d) (hyperparameter selection and sensitivity)
We thank the reviewer for raising this concern, and would like to clarify that AMO's default hyperparameters were fixed before the ablation study. The subsequent sweep examined sensitivity around those defaults and was not used to select the configuration with the highest evaluation average. For example, achieves a higher score than the default (37.83 vs. 37.77), but we retained the latter for its lower capture cost. The same defaults were used unchanged across all other models and settings (Tab. 3).
Similarly, the learning rate was not tuned separately for AMO. For each model, the base learning rate was computed from the empirical scaling rule of Zhou et al. (2026), using model size and token budget, and shared by AMO and all Muon-style baselines (App. F.1).
The improvement is not confined to the default configuration: eight of the nine non-default adaptive configurations outperform Muon-PE (37.33), with a median score of 37.63. We have revised the caption of Fig. 4.
Thank you for the detailed response and the additional analyses. My main concern, however, is the same as in my review: the improvement is small relative to the evaluation noise.
Your response confirms part of this. After the five near-chance tasks are removed, AMO and Muon-PE are tied on Llama3.1-760M, and the margin over the strongest baseline shrinks elsewhere. The new validation-loss results show the same picture: AMO improves over Muon-PE by 0.0015–0.0039, roughly 0.02–0.05 in perplexity, from single-seed curves without error bars. For a pretraining paper, validation perplexity is a standard and less noisy measure of optimization quality, so I would also suggest reporting it for the main comparisons in the main text rather than only in an appendix figure.
With gains of this size, the paper needs one of two things: (1) a controlled experiment showing that the gain comes from AMO itself, for example AMO with per-type update norms rescaled to those of Muon-PE; or (2) paired multi-seed runs showing that the gain over Muon-PE is stable. Neither is in the revision. I am not asking for large-scale runs. The smallest main setting, Qwen3-0.6B with 12B tokens, would be enough for the discussion period. For either experiment, please report at least the validation perplexity, in addition to the downstream results.
I would appreciate a response to each of these points. If the additional evidence shows that the gain is attributable to AMO or is stable across seeds, I would be glad to raise my score.
Response to Reviewer Zmr5 [part 1]
We thank the reviewer for the careful review and for recognizing AMO's practicality and geometric analysis. We address the mechanism and evaluation concerns below.
W1(a,d) (momentum buffer and schedule transition)
We agree that the momentum buffer accumulates raw stochastic gradients and does not directly depend on the NS schedule. Our intended explanation concerns the orthogonalized update computed from this buffer and applied to the weights: an abrupt schedule switch changes the transformation of the buffer, potentially affecting both the singular-value profile and the update magnitude. We have corrected the wording in Secs. 3.2 and 4.4.
We also agree that the transition ablation alone does not distinguish improved orthogonalization from update scaling. The Δ=1 result (37.27 vs. 37.33 for Muon-PE) shows that an abrupt transition does not retain the observed gain, but does not establish which mechanism accounts for it. We therefore address update magnitude separately below.
W1(b) / Q1 (per-type residuals and update-norm control)
We agree that changing the NS steps and coefficients can change per-type update norms. We applied the uniform Muon-PE schedule and AMO's locked schedule to the same post-lock momentum matrices from each run's final checkpoint, and we report per-type NS residuals and update norms for all four models in the new App. G.3. The main findings are:
This offline analysis shows that the norm changes are small and are a by-product of better orthogonalization, but it cannot exclude a contribution from them. We will include a norm-controlled retraining experiment in the final version.
W1(c) (additional NS accuracy vs. budget reallocation)
We agree that increasing the NS budget beyond the baseline yields only a small downstream improvement in this setting: +0.11 points for 20% more budget. By comparison, AMO improves over Muon-PE by +0.44 points on Qwen3-0.6B while keeping the total NS budget fixed.
These results suggest that allocating NS effort across operator types matters more than uniformly increasing it in this setting. They do not, by themselves, determine whether the gain comes from improved orthogonalization, changes in update magnitude, or both.
W2(a) (evaluation excluding near-chance tasks)
We agree that the near-chance tasks warrant a separate evaluation. After excluding the five identified tasks, AMO and Muon-PE are effectively tied on Llama3.1-760M (50.36 vs. 50.38), which we now state explicitly.
The gains at the other scales remain on this seven-task subset. AMO exceeds the strongest baseline by +0.50 points on Llama3.1-1.4B, +0.39 on Qwen3-0.6B, and +0.38 on Qwen3-1.7B, and exceeds Muon by +0.66 at 4B. In the paired multi-seed study, AMO also exceeds Muon on this seven-task subset in all three seeds (+0.81 / +0.47 / +0.12). Thus, the improvements in these settings are not solely attributable to the five near-chance tasks.
W2(b) / Q2 (held-out validation loss)
We now report held-out validation loss in Fig. 15 (Page 39). AMO achieves the lowest final validation loss among all nine optimizers in each of the four settings, improving over Muon-PE by 0.0015–0.0039. For context, Muon-PE improves over Muon-KJ by 0.0020–0.0040.
The smallest AMO–Muon-PE gap occurs on Llama3.1-760M, consistent with the downstream tie on the seven-task subset. These results provide complementary evidence beyond downstream accuracy.