- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether Muon's pretraining advantage over AdamW survives fine-tuning. It pretrains OLMo-style models from 30M to 600M parameters on DCLM at budgets of 5 to 2,560 tokens per parameter (TPR), about 600 pretraining runs in total. Each base model is fine-tuned on GSM8K, StackMathQA, Alpaca and MusicPile with a wide learning-rate sweep, about 29,000 fine-tuning runs. The two optimizers are compared through a learning–forgetting frontier (fine-tuning loss vs. held-out DCLM loss) summarized by a signed "fine-tuning gap". The main finding is that Muon gives the better frontier at small budgets, but AdamW overtakes it beyond a crossover budget, even though Muon reaches lower pretraining loss at every budget. The crossover TPR falls as models grow; on StackMathQA it drops from 320 at 30M to 10 at 600M. The reversal also appears for loss-optimal base models and under replay or EWC regularization during fine-tuning. Under Gaussian weight perturbations, Muon-pretrained models degrade more than AdamW-pretrained ones, and this gap widens with the pretraining budget. To explain the trend, the authors study a linear associative memory with Gaussian keys and values. A second-order expansion of the loss under perturbation motivates the margin-to-noise ratio (MNR) as a robustness proxy. Over many training steps the MNR favors Muon at small data budgets and gradient descent at large ones, and robustness to perturbation follows the same ordering.
Critical strengths
1. A large, carefully controlled study with a clear and practically important finding. The authors pretrain about 600 models from 30M to 600M parameters at budgets of 5 to 2,560 tokens per parameter, and run about 29,000 fine-tuning runs across four datasets with wide learning-rate sweeps. The main result is consistent across all four datasets. Muon gives the better learning–forgetting frontier at small budgets, but AdamW overtakes it beyond a crossover, even though Muon has lower pretraining loss throughout. That crossover also moves to smaller token–parameter ratios as models grow (320 to 10 TPR on StackMathQA). The result holds in several checks:
- comparing only loss-optimal base models instead of the best-frontier pool;
- splitting by fine-tuning optimizer;
- regularizing fine-tuning with replay or EWC, which shifts the crossover but does not remove it, with one exception at the largest budget.
Muon is increasingly chosen on the basis of pretraining loss, so evidence that this advantage can flip after fine-tuning, and earlier at larger scale, matters to practitioners. This is the main reason I lean toward acceptance.
2. The gap is traced to the pretrained checkpoint rather than the fine-tuning recipe. Fig. 14 shows that Muon-pretrained models forget about the same amount whether they are fine-tuned with AdamW or Muon, and that from 19.2B tokens on the AdamW-pretrained model dominates under either choice. The Gaussian perturbation results point the same way. Across all 36 (model size, budget) cells, AdamW checkpoints tolerate a larger perturbation before losing 0.05, 0.1 or 0.2 nats, by a median factor of 1.4. This ordering holds even when each optimizer gets its most robust checkpoint within 0.02 nats of the best base loss. Placing the effect in the checkpoint itself makes the finding more general than a statement about one fine-tuning setup, and gives a concrete property for follow-up work to study.
Critical weaknesses
1. The theoretical analysis does not explain the AdamW result that the paper is about. The language-model finding is that AdamW overtakes Muon, but the linear-memory analysis compares Muon with gradient descent. When the authors run the same one-step comparison against an AdamW surrogate (Appendix D.3.1), the MNR ratio reaches only 0.93 at , and they state themselves that the one-step reversal is a statement about GD, not about AdamW. The theorem is also weaker than the main text suggests. It holds only for , with an absolute constant that must be at least , while the leading term crosses one at . So the shift from favoring Muon to favoring GD is shown numerically, not proved. The MNR link to robustness is also heuristic: the authors note that under a random perturbation of a trained memory the first-order term vanishes, so is not the quantity that governs degradation. Finally, in the language models, Muon degrades more under Gaussian perturbation at every budget, including the small budgets where it wins after fine-tuning, while in the linear model Muon is the more robust optimizer at small . Section 4 is presented as an explanation of why the gap reverses, so this mismatch weighs on my assessment. As it stands, the paper offers a convincing empirical result with a mechanism that only partly matches it.
2. The size of the effect and the size trend are not established with enough statistical care. Each pretraining configuration appears to use a single seed, and budgets are spaced by factors of two. Near the crossover the fine-tuning gap is very small: about 0.015 nats at 80 TPR in Fig. 3a, and 0.02–0.06 nats beyond that. Seed noise of a similar size could move the reported crossover by one or more grid points. The size trend also rests on few points at the large end. 600M is trained only up to 40 TPR and 300M up to 160 TPR. By Fig. 3b, at 600M and 40 TPR, Muon's pretraining loss advantage has almost disappeared. An earlier crossover at that size may therefore mainly reflect Muon no longer being better at pretraining, rather than an extra robustness cost. All comparisons are also in terms of loss, with no downstream accuracy. Because the headline claim that AdamW overtakes Muon at progressively smaller budgets as models grow is extrapolated toward larger models, these gaps in evidence matter.
Questions for authors
Clarifications and additional results that could change my assessment
-
Seed variation. How many pretraining seeds are used per (size, budget, optimizer) cell? Could you report the seed variation of the fine-tuning gap around the crossover, for example 60M on StackMathQA at 40, 80 and 160 TPR with three seeds, and give an uncertainty range for the crossover TPR? It would also help to know how sensitive is to the frontier window mentioned in Appendix C.
-
Tuning at 300M and 600M. In Table 1, the tuned AdamW learning rate drops from 0.01 at 100M to 0.001–0.0035 at 300M and 600M, and 0.0025 is not on the grid used to build the best-frontier pool ({0.005, …, 0.16}). How is the best frontier constructed at these sizes? Given that Muon's pretraining advantage is nearly zero at 600M and 40 TPR (Fig. 3b), can you separate the two explanations? Plotting the fine-tuning gap against Muon's pretraining loss advantage, rather than against model size alone, would show whether "Muon forgets more" adds anything beyond "Muon's pretraining edge shrinks."
-
AdamW in the linear memory. Does AdamW, or its sign-based surrogate, overtake Muon in MNR or in in the many-step linear-memory experiments over the budgets you ran? If it does not, what connects the linear-model analysis to the AdamW result in language models?
-
Sign of the perturbation result. In language models, Muon's perturbation degradation is larger at every budget, including where Muon wins after fine-tuning. In the linear model, Muon is more robust at small . How do you reconcile these? Relatedly, in Fig. 5 the MNR curves cross near , while the curves cross near . How well does MNR predict robustness beyond the direction of the trend?
-
Downstream metrics. Do the loss-based frontiers translate to task accuracy, for example GSM8K accuracy after fine-tuning, in at least one setting on each side of the crossover?
2 comments
Thank you for the response and the additional experiments. The three-seed results at 60M (crossover between 40 and 80 TPR for every seed), the 600M decomposition (from 20 to 40 TPR, Muon's pretraining-loss advantage stays at 0.017 while the fine-tuning gap and degradation gap keep growing, which suggests that the extra forgetting matters beyond the shrinking pretraining edge), and the GSM8K next-token-accuracy frontiers address my concerns about seed noise, the size trend and downstream metrics reasonably well, although these checks are limited to 60M/600M and teacher-forced accuracy. My concern about the theory remains: the response confirms that AdamW does not overtake Muon in the linear memory over the budgets tested, and the opposite signs of the perturbation result (Muon degrades more at every budget in language models, but is more robust at small γ in the linear model) are not reconciled. I would suggest presenting Section 4 as a GD-vs-Muon analysis that matches the direction of the trend, rather than as an explanation of the AdamW reversal. Overall, I find the empirical contribution convincing and keep my score of 3.
We thank you for acknowledging the value of our scientific contribution. We have added additional experiments to address the concerns.
We run additional experiments to address your concerns—in addition to our original model (which we only run one seed), we pretrain two seeds and subsequently fine-tune on GSM8K, finding that the frontier window is not particularly sensitive to random seed. We find that the standard deviation is relatively small compared to the change in the fine-tuning gap across token budgets, and, in particular, is smaller than the fine-tuning gap at the crossover. Figure 20: Fine-tuning gap across three seeds, 60M
We construct the best-frontier by taking the pretrained models that produce the best model after fine-tuning, which often have higher learning-rates than the tuned lowest-loss model (see prior work) . We also measure both Muon’s pretraining loss advantage and degradation at 600M, finding that in regimes where pretraining-loss advantage is no longer predictive (20-40 TPR), degradation continues to track the fine-tuning gap. Figure 21: 600M fine-tuning gap vs pretrain loss advantage and degradation
We choose to analyze GD rather than AdamW because its dynamics are more tractable theoretically, following prior work that studies simplified gradient-based updates to gain insight into adaptive optimizers (Xie & Li, 2024; Li et al., 2025). In our linear-memory experiments, AdamW does not overtake Muon within the budgets we test. However, the MNR gap continues to close as the data budget increases. We focus on this general trend with token budget rather than the exact crossover point.
We view MNR as capturing the general direction of the shift in optimizer advantage with pretraining budget, rather than predicting the exact fine-tuning crossover. Empirically, we find that the MNR crossover follows the same direction as the fine-tuning crossover, with both decreasing as model size increases. We do not expect the crossover points to match exactly, since the fine-tuning tradeoff also depends on the pretraining loss and the downstream dataset.
We run additional experiments, and address this by showing AdamW can produce better models after fine-tuning as measured by accuracy on real tasks. Because we use small models, we evaluate next-token accuracy averaged over the entire gold response, which provides a much cleaner signal than final-answer accuracy. Therefore, we measure next-token accuracy—the fraction of tokens the model correctly predicts—which captures a direct notion of correctness not reflected in loss alone. We observe the same trend reflected in next-token accuracy: Muon dominates at smaller token budgets, while AdamW dominates at larger budgets. Figure 22: LF frontiers with GSM8K next-token accuracy, 60M