- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AIMuon augments Muon with the gradient–momentum residual. It transforms momentum and residual separately using approximate polar maps, weights their contributions through estimated nuclear norms, and matches the combined update magnitude to Muon's scale. The paper derives the exact update from AIM's spectral parameter subproblem and proves a stochastic stationarity bound for that idealized iteration. GPT-2-small and GPT-2-medium experiments report lower validation loss and fewer steps to Muon's final loss. The medium-model run reports a 6.6% elapsed-time saving including the extra residual computation.
Critical strengths
-
The construction is coherent and the exact theory is sound. Spectral–nuclear duality yields the two weighted polar directions. Appendix B correctly handles the dependence of the update on stochastic gradient noise. The manuscript clearly states the theoretical scope and explains the effect of practical normalization.
-
The evaluation includes relevant controls and measured training time. Muon and MONA are appropriate comparators. Matched initialization and data order improve fairness, and the ablations examine the residual's placement, weighting, and normalization. Figure 2 reports 8.04 hours versus 8.61 hours for the medium-model target, including the second spectral computation.
Critical weaknesses
-
The main gain needs a robustness check. Appendix C reports one seed per method. Every selected rate is the upper endpoint of a search using about 10% of the full training budget. The loss improvements over Muon are 0.0072 and 0.0029. These results leave uncertainty about training variation and whether nearby higher rates would change the comparison. This is the main reason for weak rejection.
-
The Nesterov baseline needs clarification. The Muon implementation's Nesterov setting is unspecified. The Inside-MSign ablation uses the paper's Nesterov-style combination and reaches 3.2198, compared with AIMuon's 3.2161 and Muon's 3.2233. It receives no independent rate tuning. This closest control is important for establishing the benefit of a separate residual transformation.
Questions for authors
-
Can you report existing higher-rate results and additional paired seeds? A modest rate extension and a few paired repeats on GPT-2-small could establish whether the gain is stable. Consistent loss and target-crossing improvements would address the main obstacle to acceptance.
-
What matrix enters Muon's polar approximation, and is Nesterov enabled? Please identify the implementation and report any independently tuned result for the existing Inside-MSign control. A stable advantage over that comparator, together with the first answer, could move my rating from 2 to 3.
6 comments
W1 & Q1. Broader hyperparameter searches and additional paired seeds
We extended the learning-rate searches and independently repeated hyperparameter selection for each optimizer. We use a sequential two-stage procedure: first select the peak learning rate, then compare momentum coefficients at each optimizer's selected learning rate.
| Setting | Peak learning-rate candidates, in units of | Tokens per candidate in each search stage | Selected peak learning rate |
|---|---|---|---|
| GPT-2-small | 2, 2.5, 3, 3.5, 4, 4.5, 5 | 2.48B | 0.0045 |
| GPT-2-medium | 1, 2, 3, 4, 4.5, 5 | 0.69B | 0.0045 |
| Qwen3-0.6B | 1, 2, 3, 3.5, 4, 4.5 | 1.2B | 0.004 |
In both the learning-rate and momentum searches for the main optimizers, each candidate is trained for 2.48B tokens on GPT-2-small, 0.69B tokens on GPT-2-medium, and 1.2B tokens on Qwen3-0.6B. These are per-candidate budgets in each search stage.
The learning-rate searches initially use for the GPT-2 experiments and for Qwen3-0.6B. The latter choice follows the GPT-2 momentum-search results. Each optimizer is tuned independently, although all methods select the same learning rate within each model setting. The selected learning rates are now interior points of their respective search grids.
Fixing these selected learning rates, we independently search for each optimizer. The selected momentum coefficients are:
| Optimizer | GPT-2-small | GPT-2-medium | Qwen3-0.6B |
|---|---|---|---|
| AIMuon | 0.9 | 0.9 | 0.95 |
| Muon | 0.9 | 0.9 | 0.95 |
| Muon-Nesterov | 0.9 | 0.9 | 0.95 |
| MONA | 0.9 | 0.9 | 0.9 |
AIMuon uses , , and for GPT-2-small, GPT-2-medium, and Qwen3-0.6B, respectively.
Every search candidate starts from scratch using seed 42, with matched initialization, training-data order, and validation data. Each run uses a complete warmup and cosine-decay schedule for its allotted budget. Selection is based on terminal validation loss, and the AdamW fallback momentum coefficients remain unchanged. All updated experiments run on eight NVIDIA H100 GPUs.
Three-seed GPT-2-small results. With the selected hyperparameters fixed, we evaluate each optimizer across seeds 42, 43, and 44:
| Optimizer | Validation loss, seed 42 | Seed 43 | Seed 44 | Mean ± standard deviation |
|---|---|---|---|---|
| Muon | 3.1980 | 3.1974 | 3.1984 | 3.1980 ± 0.0005 |
| Muon-Nesterov | 3.2000 | 3.1997 | 3.1990 | 3.1996 ± 0.0005 |
| MONA | 3.2198 | 3.2215 | 3.2217 | 3.2210 ± 0.0010 |
| AIMuon | 3.1939 | 3.1950 | 3.1946 | 3.1945 ± 0.0005 |
All runs use 2.48B tokens. Within each seed, initialization and training-data order are matched across methods. Seed 42 is also used for hyperparameter selection; seeds 43 and 44 are additional evaluations without further tuning.
AIMuon achieves the lowest validation loss in all three paired runs. Its loss improvement over Muon ranges from approximately 0.0024 to 0.0041. The advantage over the independently tuned Muon-Nesterov baseline also persists in all three seeds.
Larger-model results. We retrained GPT-2-medium for 6.9B tokens and added Qwen3-0.6B pretraining for 12B tokens:
| Optimizer | GPT-2-medium validation loss | Qwen3-0.6B validation loss |
|---|---|---|
| Muon | 3.0255 | 2.7859 |
| Muon-Nesterov | 3.0195 | 2.7830 |
| MONA | 3.0316 | 2.7824 |
| AIMuon | 3.0187 | 2.7819 |
Both settings use seed 42. AIMuon achieves the lowest terminal loss in both comparisons after independent baseline tuning. These results extend the evaluation beyond GPT-2-small.
Steps to baseline performance. We additionally measure the optimization steps AIMuon requires to reach the final validation losses of the independently tuned Muon-Polyak and Muon-Nesterov baselines. AIMuon reaches both targets in fewer steps across all three models:
| Model | Step reduction vs. Muon-Polyak | Step reduction vs. Muon-Nesterov |
|---|---|---|
| GPT-2-small | 8.98% | 10.89% |
| GPT-2-medium | 10.05% | 4.13% |
| Qwen3-0.6B | 6.49% | 3.78% |
For GPT-2-small, we first average validation losses across seeds 42, 43, and 44 at each evaluation step, then interpolate the mean curve to locate AIMuon's first crossing of each baseline's final mean validation loss. GPT-2-medium and Qwen3-0.6B use the seed-42 curves with the same interpolation procedure. The reduction is computed as , where is the baseline's full training length and is AIMuon's interpolated target-reaching step. All crossings use the original full-budget learning-rate schedules.
W2 & Q2. Muon implementation, Nesterov setting, and the Inside-MSign control
Thank you for pointing out the ambiguity in the baseline description. The original Muon baseline uses ordinary EMA momentum, without the Nesterov combination:
The added Muon-Nesterov baseline retains the same momentum state but changes the input to the polar approximation:
Both use the Keller Jordan / modded-nanoGPT-style five-step Newton–Schulz construction, with polynomial coefficients , together with the LLM-style matrix scaling convention. Parameter grouping, the AdamW fallback, data processing, and training schedules are shared. We specify these implementation details explicitly in the revised experimental description.
Inside-MSign definition. This control uses a tunable mixture:
Since
its input equals the Nesterov-style combination when . For general , it represents a more flexible gradient–momentum mixture. Its final magnitude is matched to the momentum-based Muon reference, following the ablation protocol. Thus, equality of the transformation inputs alone does not imply that its complete update is identical to the separate Muon-Nesterov baseline.
Tuning and updated ablations. We independently tune the correction coefficients of Inside-MSign and Linear residual over
Each candidate uses 0.248B tokens and seed 42. Inside-MSign initially selects 0.01, the lower endpoint, so we additionally evaluate 0.005 under the same protocol. The value 0.01 remains better. Linear residual selects 0.025. Each selected configuration is then restarted from scratch and trained for 2.48B tokens.
Muon-Nesterov receives independent learning-rate and momentum tuning. Both residual controls receive independent tuning of and the learning rate, with the momentum coefficient fixed at 0.9. Both Inside-MSign and Linear residual use the learning-rate candidates , with 0.248B tokens per candidate and seed 42. Both select based on terminal validation loss. Together with the coefficient searches above, these searches independently tune both controls under the same candidate ranges and search budgets. The no-residual variant is AIMuon with , using the independently selected Muon learning rate and momentum coefficient. Its retained Frobenius-norm matching introduces numerical differences from the standalone Muon baseline.
| Configuration | Coefficient | Final validation loss |
|---|---|---|
| Full AIMuon | 3.1939 | |
| Fixed residual weight | 3.1944 | |
| Inside-MSign | 3.1949 | |
| No residual | 3.1964 | |
| Linear residual | 3.1984 |
Removing the residual gives a final validation loss of 3.1964, compared with 3.1939 for full AIMuon. Inside-MSign improves upon the no-residual variant, while Linear residual does not. These comparisons support the benefit of independently transforming the residual.
Purpose of magnitude matching. We follow Muon's RMS-matching convention, which aligns its update scale with the typical AdamW update scale (Liu et al., 2025). We then rescale the combined direction to match the RMS magnitude of the corresponding Muon reference at the same momentum state. With the learning rate held fixed, this aligns the update magnitude and helps distinguish the benefit of the update direction from that of a larger step. AIMuon, Inside-MSign, and Linear residual all retain this matching, so their direction constructions are compared under the same magnitude-matching rule.
AIMuon retains a lower validation loss than Inside-MSign after tuning both its correction coefficient and learning rate, supporting the benefit of independently transforming the residual in the evaluated configuration. The fixed-weight variant also captures most of the gain, with adaptive nuclear-norm weighting providing a smaller additional improvement.
The expanded evaluation shows that the final-loss improvement persists after broader learning-rate searches and across three matched GPT-2-small seeds, including against an independently tuned Muon-Nesterov baseline. The independently tuned Inside-MSign comparison provides additional evidence for the separate residual transformation.
Robustness of AIMuon’s Gains and Comparisons with Tuned Baselines
Thank you for the constructive assessment and for recognizing the coherence of the construction and the soundness of the exact analysis. We focused the additional experiments on your two main concerns: the robustness of the gains after broader tuning and repeated runs, and the comparison with Nesterov and the closest inside-transform control.
After extending the learning-rate searches and independently tuning momentum, AIMuon achieves lower final validation loss than Muon and Muon-Nesterov in all three paired GPT-2-small runs. It also reaches their final mean validation losses with 8.98% and 10.89% fewer steps, respectively, as measured from the mean validation-loss curves. The independently tuned Inside-MSign control reaches 3.1949, compared with 3.1939 for AIMuon in the seed-42 comparison.