acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

Abstract

Muon has recently emerged as a promising optimizer for pretraining large language models, offering a speedup over AdamW in minimizing pretraining loss. However, it remains unclear whether Muon’s advantage persists after fine-tuning. In this paper, we study the downstream behavior of Muon and AdamW and uncover a surprising trend: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning. Moreover, as model size increases, AdamW overtakes Muon at progressively smaller pretraining token–parameter ratios. We observe the same trend under Gaussian perturbations of the pretrained weights, where Muon-pretrained models degrade increasingly more than AdamW-pretrained models as the data budget grows. We complement our empirical findings with a controlled study of Muon and AdamW in a linear model. In this setting, we show theoretically and empirically that the margin-to-noise ratio, a tractable proxy for robustness motivated by a second-order approximation of the loss, shifts from favoring Muon to favoring gradient descent as the data budget grows, mirroring the fine-tuning trend.

Primary area:
optimization
Keywords:
muon, adamw, optimizer, pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R8QmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether Muon's pretraining advantage over AdamW survives fine-tuning. It pretrains OLMo-style models from 30M to 600M parameters on DCLM at budgets of 5 to 2,560 tokens per parameter (TPR), about 600 pretraining runs in total. Each base model is fine-tuned on GSM8K, StackMathQA, Alpaca and MusicPile with a wide learning-rate sweep, about 29,000 fine-tuning runs. The two optimizers are compared through a learning–forgetting frontier (fine-tuning loss vs. held-out DCLM loss) summarized by a signed "fine-tuning gap". The main finding is that Muon gives the better frontier at small budgets, but AdamW overtakes it beyond a crossover budget, even though Muon reaches lower pretraining loss at every budget. The crossover TPR falls as models grow; on StackMathQA it drops from 320 at 30M to 10 at 600M. The reversal also appears for loss-optimal base models and under replay or EWC regularization during fine-tuning. Under Gaussian weight perturbations, Muon-pretrained models degrade more than AdamW-pretrained ones, and this gap widens with the pretraining budget. To explain the trend, the authors study a linear associative memory with Gaussian keys and values. A second-order expansion of the loss under perturbation motivates the margin-to-noise ratio (MNR) as a robustness proxy. Over many training steps the MNR favors Muon at small data budgets and gradient descent at large ones, and robustness to perturbation follows the same ordering.

Critical strengths

1. A large, carefully controlled study with a clear and practically important finding. The authors pretrain about 600 models from 30M to 600M parameters at budgets of 5 to 2,560 tokens per parameter, and run about 29,000 fine-tuning runs across four datasets with wide learning-rate sweeps. The main result is consistent across all four datasets. Muon gives the better learning–forgetting frontier at small budgets, but AdamW overtakes it beyond a crossover, even though Muon has lower pretraining loss throughout. That crossover also moves to smaller token–parameter ratios as models grow (320 to 10 TPR on StackMathQA). The result holds in several checks:

  • comparing only loss-optimal base models instead of the best-frontier pool;
  • splitting by fine-tuning optimizer;
  • regularizing fine-tuning with replay or EWC, which shifts the crossover but does not remove it, with one exception at the largest budget.

Muon is increasingly chosen on the basis of pretraining loss, so evidence that this advantage can flip after fine-tuning, and earlier at larger scale, matters to practitioners. This is the main reason I lean toward acceptance.

2. The gap is traced to the pretrained checkpoint rather than the fine-tuning recipe. Fig. 14 shows that Muon-pretrained models forget about the same amount whether they are fine-tuned with AdamW or Muon, and that from 19.2B tokens on the AdamW-pretrained model dominates under either choice. The Gaussian perturbation results point the same way. Across all 36 (model size, budget) cells, AdamW checkpoints tolerate a larger perturbation before losing 0.05, 0.1 or 0.2 nats, by a median factor of 1.4. This ordering holds even when each optimizer gets its most robust checkpoint within 0.02 nats of the best base loss. Placing the effect in the checkpoint itself makes the finding more general than a statement about one fine-tuning setup, and gives a concrete property for follow-up work to study.

Critical weaknesses

1. The theoretical analysis does not explain the AdamW result that the paper is about. The language-model finding is that AdamW overtakes Muon, but the linear-memory analysis compares Muon with gradient descent. When the authors run the same one-step comparison against an AdamW surrogate (Appendix D.3.1), the MNR ratio reaches only 0.93 at γ=64\gamma = 64, and they state themselves that the one-step reversal is a statement about GD, not about AdamW. The theorem is also weaker than the main text suggests. It holds only for γ≥16\gamma \ge 16, with an absolute constant C1C_1 that must be at least 19γ13/219\gamma_1^{3/2}, while the leading term crosses one at γ≈5.95\gamma \approx 5.95. So the shift from favoring Muon to favoring GD is shown numerically, not proved. The MNR link to robustness is also heuristic: the authors note that under a random perturbation of a trained memory the first-order term vanishes, so h/κh/\kappa is not the quantity that governs degradation. Finally, in the language models, Muon degrades more under Gaussian perturbation at every budget, including the small budgets where it wins after fine-tuning, while in the linear model Muon is the more robust optimizer at small γ\gamma. Section 4 is presented as an explanation of why the gap reverses, so this mismatch weighs on my assessment. As it stands, the paper offers a convincing empirical result with a mechanism that only partly matches it.

2. The size of the effect and the size trend are not established with enough statistical care. Each pretraining configuration appears to use a single seed, and budgets are spaced by factors of two. Near the crossover the fine-tuning gap is very small: about 0.015 nats at 80 TPR in Fig. 3a, and 0.02–0.06 nats beyond that. Seed noise of a similar size could move the reported crossover by one or more grid points. The size trend also rests on few points at the large end. 600M is trained only up to 40 TPR and 300M up to 160 TPR. By Fig. 3b, at 600M and 40 TPR, Muon's pretraining loss advantage has almost disappeared. An earlier crossover at that size may therefore mainly reflect Muon no longer being better at pretraining, rather than an extra robustness cost. All comparisons are also in terms of loss, with no downstream accuracy. Because the headline claim that AdamW overtakes Muon at progressively smaller budgets as models grow is extrapolated toward larger models, these gaps in evidence matter.

Questions for authors

Clarifications and additional results that could change my assessment

  1. Seed variation. How many pretraining seeds are used per (size, budget, optimizer) cell? Could you report the seed variation of the fine-tuning gap around the crossover, for example 60M on StackMathQA at 40, 80 and 160 TPR with three seeds, and give an uncertainty range for the crossover TPR? It would also help to know how sensitive GFTG_{\mathrm{FT}} is to the frontier window mentioned in Appendix C.

  2. Tuning at 300M and 600M. In Table 1, the tuned AdamW learning rate drops from 0.01 at 100M to 0.001–0.0035 at 300M and 600M, and 0.0025 is not on the grid used to build the best-frontier pool ({0.005, …, 0.16}). How is the best frontier constructed at these sizes? Given that Muon's pretraining advantage is nearly zero at 600M and 40 TPR (Fig. 3b), can you separate the two explanations? Plotting the fine-tuning gap against Muon's pretraining loss advantage, rather than against model size alone, would show whether "Muon forgets more" adds anything beyond "Muon's pretraining edge shrinks."

  3. AdamW in the linear memory. Does AdamW, or its sign-based surrogate, overtake Muon in MNR or in σ50\sigma_{50} in the many-step linear-memory experiments over the budgets you ran? If it does not, what connects the linear-model analysis to the AdamW result in language models?

  4. Sign of the perturbation result. In language models, Muon's perturbation degradation is larger at every budget, including where Muon wins after fine-tuning. In the linear model, Muon is more robust at small γ\gamma. How do you reconcile these? Relatedly, in Fig. 5 the MNR curves cross near γ≈20\gamma \approx 20, while the σ50\sigma_{50} curves cross near γ≈3\gamma \approx 3. How well does MNR predict robustness beyond the direction of the trend?

  5. Downstream metrics. Do the loss-based frontiers translate to task accuracy, for example GSM8K accuracy after fine-tuning, in at least one setting on each side of the crossover?

2 comments

We thank you for acknowledging the value of our scientific contribution. We have added additional experiments to address the concerns.

Q1: Seed variation. How many pretraining seeds are used ... how sensitive is to the frontier window mentioned in Appendix C.

We run additional experiments to address your concerns—in addition to our original model (which we only run one seed), we pretrain two seeds and subsequently fine-tune on GSM8K, finding that the frontier window is not particularly sensitive to random seed. We find that the standard deviation is relatively small compared to the change in the fine-tuning gap across token budgets, and, in particular, is smaller than the fine-tuning gap at the crossover. Figure 20: Fine-tuning gap across three seeds, 60M

TPRSeed 0Seed 1Seed 2MeanStd
20−0.1002−0.1583−0.1328−0.13040.0291
40−0.0363−0.0265+0.0067−0.01870.0225
80+0.0443+0.0668+0.0648+0.05860.0125

Q2: Tuning at 300M and 600M. In Table 1, the tuned AdamW learning rate ... would show whether "Muon forgets more" adds anything beyond "Muon's pretraining edge shrinks."

We construct the best-frontier by taking the pretrained models that produce the best model after fine-tuning, which often have higher learning-rates than the tuned lowest-loss model (see prior work) . We also measure both Muon’s pretraining loss advantage and degradation at 600M, finding that in regimes where pretraining-loss advantage is no longer predictive (20-40 TPR), degradation continues to track the fine-tuning gap. Figure 21: 600M fine-tuning gap vs pretrain loss advantage and degradation

TPR 5TPR 10TPR 20TPR 40
fine-tuning gap-0.0115+0.0144+0.0283+0.0416
Muon pretraining-loss advantage0.0730.0320.0170.017
degradation gap+0.016+0.113+0.080+0.130

Q3: AdamW in the linear memory. Does AdamW, or its sign-based surrogate, overtake Muon ... connects the linear-model analysis to the AdamW result in language models?

We choose to analyze GD rather than AdamW because its dynamics are more tractable theoretically, following prior work that studies simplified gradient-based updates to gain insight into adaptive optimizers (Xie & Li, 2024; Li et al., 2025). In our linear-memory experiments, AdamW does not overtake Muon within the budgets we test. However, the MNR gap continues to close as the data budget increases. We focus on this general trend with token budget rather than the exact crossover point.

Q4: Sign of the perturbation result. In language models, Muon's perturbation degradation ... well does MNR predict robustness beyond the direction of the trend?

We view MNR as capturing the general direction of the shift in optimizer advantage with pretraining budget, rather than predicting the exact fine-tuning crossover. Empirically, we find that the MNR crossover follows the same direction as the fine-tuning crossover, with both decreasing as model size increases. We do not expect the crossover points to match exactly, since the fine-tuning tradeoff also depends on the pretraining loss and the downstream dataset.

Q5: Downstream metrics. Do the loss-based frontiers translate to task accuracy ... each side of the crossover?

We run additional experiments, and address this by showing AdamW can produce better models after fine-tuning as measured by accuracy on real tasks. Because we use small models, we evaluate next-token accuracy averaged over the entire gold response, which provides a much cleaner signal than final-answer accuracy. Therefore, we measure next-token accuracy—the fraction of tokens the model correctly predicts—which captures a direct notion of correctness not reflected in loss alone. We observe the same trend reflected in next-token accuracy: Muon dominates at smaller token budgets, while AdamW dominates at larger budgets. Figure 22: LF frontiers with GSM8K next-token accuracy, 60M

TPR 20TPR 40TPR 80TPR 160
fine-tuning gap-0.0307-0.0089+0.0363+0.0311

Thank you for the response and the additional experiments. The three-seed results at 60M (crossover between 40 and 80 TPR for every seed), the 600M decomposition (from 20 to 40 TPR, Muon's pretraining-loss advantage stays at 0.017 while the fine-tuning gap and degradation gap keep growing, which suggests that the extra forgetting matters beyond the shrinking pretraining edge), and the GSM8K next-token-accuracy frontiers address my concerns about seed noise, the size trend and downstream metrics reasonably well, although these checks are limited to 60M/600M and teacher-forced accuracy. My concern about the theory remains: the response confirms that AdamW does not overtake Muon in the linear memory over the budgets tested, and the opposite signs of the perturbation result (Muon degrades more at every budget in language models, but is more robust at small γ in the linear model) are not reconciled. I would suggest presenting Section 4 as a GD-vs-Muon analysis that matches the direction of the trend, rather than as an explanation of the AdamW reversal. Overall, I find the empirical contribution convincing and keep my score of 3.

Reviewer_MAekOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an extensive empirical study on the behavior of optimizers across the pre-training and post-training (fine-tuning) stages. Through systematic experiments across various data scales, datasets, and learning rates, the authors demonstrate that pre-training validation loss is an unreliable metric for predicting an optimizer's effectiveness during subsequent fine-tuning. The paper also analyzes the learning–forgetting trade-off, highlighting discrepancies in how optimizers (such as AdamW) retain pre-training knowledge while adapting to downstream data.

Critical strengths
  1. The authors provided systematic experiments. The author scanned multiple pre-training/post-training datasets under different data scale settings and different learning rate settings for experiments.
  2. The findings are clear. The paper shows that it is not only based on pre-training loss to judge which training/optimizer setting is more suitable for subsequent fine-tuning.
Critical weaknesses
  1. The paper proposes no modifications to the optimizer, nor does it explain the root cause of the behavioral discrepancy between pre-training and post-training.
  2. The final evaluation relies on next-token loss without reporting benchmark scores. A lower training loss does not necessarily mean to better downstream performance, especially in optimizer-related experiments.
  3. The paper says preservation of performance on the pretraining distribution as a key objective (i.e., Learning–forgetting tradeoff.). What users need is the released post-train model, instead of caring about the knowledge of the pre-train model (or these have been converted into post-train model capabilities). So, it remains unclear whether AdamW’s better learning–forgetting means to a better model. I think the authors should evaluate both downstream task scores and retained general capabilities.
Questions for authors

Could the authors provide deeper insights or analyses (e.g., weight norm drift, gradient variance, or effective rank/curvature changes) into the mechanistic causes behind the behavioral discrepancy between pre-training and fine-tuning (post-training)? What fundamentally changes between these two stages for different optimizers?

2 comments

We thank you for acknowledging the value of our scientific contribution. We have added additional experiments to address the concerns.

The paper proposes no modifications to the optimizer ... discrepancy between pre-training and post-training

We want to clarify that our work has immediate implications for practitioners by showing that AdamW can produce a better final model than Muon after fine-tuning. We therefore propose using AdamW in these settings, even though it appears worse through the lens of pretraining loss! This challenges the conventional wisdom favoring Muon, which has been adopted in models such as DeepSeek-V4, Kimi K2, and GLM-4.5.

The final evaluation relies on next-token loss without reporting benchmark scores. The paper says preservation of performance ... evaluate both downstream task scores and retained general capabilities.

We have added an experiment to address both of these concerns, showing that AdamW can produce better models after fine-tuning as measured by accuracy on real tasks. Because we use small models, we evaluate next-token accuracy averaged over the entire gold response, which provides a much cleaner signal than final-answer accuracy. We fine-tune on GSM8K and report GSM8K next-token accuracy and also average next-token accuracy across 10 common general benchmark tasks (arc_easy, arc_challenge, piqa, hellaswag, winogrande, boolq, sciq, openbookqa, copa, lambada_openai). At a sufficiently large token budget (80+ TPR), AdamW-pretrained models outperform Muon-pretrained models on GSM8K at matched benchmark next-token accuracy, and on the benchmarks at matched GSM8K next-token accuracy.

Fine-tuning: GSM8K answer-token accuracy (%) at avg lm-eval accuracy = 26%

Model40 TPR80 TPR160 TPR320 TPR640 TPR1280 TPR2560 TPR
60M AdamW base47.2759.4560.9259.8362.0263.2664.00
60M Muon base55.1255.9459.5259.7261.0061.8561.44
AdamW − Muon−7.84+3.51+1.40+0.11+1.02+1.41+2.56

Avg general benchmark next-token accuracy (%) at GSM8K answer-token accuracy = 61%

Model40 TPR80 TPR160 TPR320 TPR640 TPR1280 TPR2560 TPR
60M AdamW base22.0925.1125.9625.4026.8227.4827.69
60M Muon base23.7123.7424.9525.0226.0026.4726.14
AdamW − Muon−1.62+1.36+1.01+0.38+0.82+1.01+1.55

Could the authors provide deeper insights or analyses ... between these two stages for different optimizers?

We want to highlight that we do provide a mechanistic explanation, supported by experiments and theory in Section 4 and Appendix B. Empirically, Muon-pretrained models are more sensitive than AdamW-pretrained models to Gaussian weight perturbations, with the gap growing as the pretraining budget increases. Theoretically, in a linear associative memory setting, we also prove that at sufficiently large data budgets, one Muon update gives weaker guarantees of robustness to output perturbations than GD. Since fine-tuning also changes model weights, these results help explain why sensitivity matters for fine-tuning performance beyond pretraining loss.

Thank you for the additional experiments and clarifications.

Although the reported answer-token accuracy measures answer-token acc over the gold response under teacher forcing, rather than final-answer acc from freely generated tokens, it partially addresses my concern about downstream evaluation.

I'll raise my score to 3 accordingly.

Reviewer_fLmKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

the authors claim that, if trained on a sufficient amount of tokens, Muon-pretrained models end up with a worse learning–forgetting trade-off after fine-tuning than AdamW-pretrained ones. the authors also notice that Muon models also degrade more under Gaussian perturbations as the data budget grows

Critical strengths

S1: the authors introduce a "learning-forgetting frontier", a new measure

S2: extensive experiments. the authors train various models, and conduct very long runs, e.g., 60M for 2560 TPR (153.6B tokens)

S3: gaussian perturbations and a linear associative memory are good proxies. in the linear model the margin-to-noise ratio flip from Muon to GD as the budget grows, which confirms the authors findings for LLMs

Critical weaknesses

W1: in Theorem 4.2, the data budget counts stored facts per dimension, and it does not measure how long the model is trained. the authors state that it “plays the role of” TPR but do not test this. the theory also predicts a size-independent crossover in γ\gamma, whereas the main LLM finding is that the crossover TPR shrinks with model size. in my opinion, more intuitive link supported with toy experiments would be a great thing to add

Questions for authors

Q1: as of now, the authors do pretraining with WSD and LR decay down to zero. then you sweep LRs for fine-tuning, record metrics, and plot the learning(forgetting) graph. the recent standard is also not to decay down to zero, but to some moderate 0.1 x maxLR or 0.01 x maxLR, and continue doing SFT from there. i wonder if the authors could launch similar experiment at least at 60M scale, and measure downstream pretrain_loss(sft ckpt), and sft_loss(sft_ckpt) when scaling TPR?

Q2: do you reset the optimizer buffers after pretraining? what is the decay fraction in WSD?

i would appreciate the authors if they could provide some experiments regarding Q1, this could substantially affect my assessment of the paper.

2 comments

We thank you for acknowledging the value of our scientific contribution. We have added additional experiments to address the concerns.

W1: in Theorem 4.2, the data budget counts ... intuitive link supported with toy experiments would be a great thing to add

In the toy model, γ=N/d\gamma =N/d is meant to correspond to the data budget in our language model experiments. We pretrain the language models for a single epoch, so in our setting the TPR is simply the amount of training data. In the toy model, $N/d $ models the amount of data, so studying how the results change with N/dN/d in the toy model corresponds to studying how they change with the TPR in language models.

Q1: as of now, the authors do pretraining ... measure downstream pretrain_loss(sft ckpt), and sft_loss(sft_ckpt) when scaling TPR

We have added an experiment to address your concern, showing that AdamW can produce better models after fine-tuning when pretraining is decayed to 0.1 x maxLR . We find that the original trend when scaling token budget still persists; for smaller token budgets, Muon dominates AdamW, but after sufficient TPR, the trend reverses, and the fine-tuning gap between Muon and AdamW widens. Figure 19: LF frontiers 0.1 x maxLR, 60M

WSD DecayTPR 20TPR 40TPR 80
decay to 0 (reference): fine-tuning gap-0.1002-0.0363+0.0443
decay to 0.1x maxLR: fine-tuning gap-0.1322-0.0127+0.1490

Q2: Do you reset optimizer buffers after pretraining .. what is decay fraction

After pretraining, we reset the optimizer buffers to isolate the effects of pretraining optimization from those of the fine-tuning procedure. For pretraining LR, we decay LR to zero. However, we additionally train models decayed to 0.1xmaxLR to align with more canonical practices, finding that the token budget reversal trend still persists.

Reviewer_fLmKOpenReview ↗

rebuttal acknowledged

i thank the authors for their response, running additional experiments, and apologise for the delay.

my Q1 has been addressed but not the W1 point. theorem 4.2 shows a crossover in γ\gamma with no dependence on model size, whereas the paper’s headline claim is that the crossover TPR shrinks with model size. these claims are not compatible, and a toy experiment varying dd at fixed N/dN/d would settle this discussion.

i also note the exchange with Reviewer R8Qm, where the authors confirm that AdamW does not overtake Muon in the linear memory at any budget tested. together with my W1, i think this makes the case for R8Qm’s suggestion: Section 4 should be presented as a GD vs Muon analysis, not as an explanation of the AdamW's scaling with model size.

given this, i would keep my score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.