acceptodds
Under review as a conference paper at ICLR 2027

Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less

Abstract

Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetuning with the same optimizer as in pretraining achieves a better learning-forgetting tradeoff, i.e., forgetting less while achieving the same or better performance on the new task, than other optimizers and, possibly surprisingly, LoRA, during the supervised finetuning (SFT) stage. We term this phenomenon optimizer-model consistency. To better understand it, through controlled experiments and theoretical analysis, we show that: 1) optimizers can shape the models by having regularization effects on the activations, leading to different landscapes around the pretrained checkpoints; 2) in response to this regularization effect, the weight update in SFT should follow some specific structures to lower forgetting of the knowledge learned in pretraining, which can be obtained by using the same optimizer. Moreover, we specifically compare Muon and AdamW when they are employed throughout the pretraining and SFT stages and find that Muon performs worse when finetuned for reasoning tasks. With a synthetic language modeling experiment, we demonstrate that this can come from Muon's strong tendency towards rote memorization, which may hurt pattern acquisition with a small amount of data, as for SFT.

Primary area:
optimization
Keywords:
optimizers, finetuning, SFT, learning-forgetting tradeoff

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9trNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an interesting finding that optimizers can not be easily switched and there is a cost to switching them. This suggests that different optimizers inject different structures into the network. They also find that Muon memorizes shuffled and corrupted sequences better. Experiments were done on GPT-2, Llama-2-7B and RNJ-1, and the optimizers compared were AdamW and Muon.

Critical strengths

Experiments with the frontier analysis across learning rates and checkpoints is good and yields robust conclusions on how full finetuning can have better learning-forgetting tradeoff than switching the optimizer / using LoRA. In addition, the authors tested both directions of pretraining / finetuning mismatch, and and the larger-model experiments extend the investigation beyond small models.

Critical weaknesses

Appendix B.2 evaluates exact training on examples sampled from the training corpus for both clean and corrupted text. Higher corrupted text performance supports more memorization, but lower clean training text accuracy does not independently establish poorer generalization. The experiment also trains on a repeatedly presented synthetic corpus rather than testing the proposed mechanism during reasoning SFT. Since the controlled math/code comparisons primarily report token-level validation loss, the evidence does not yet connect memorization behavior to downstream reasoning performance.

Questions for authors

What is the correct Llama-2-7B panel for Figure 1, both panels say RNJ-1 (Muon-pretrained) but the caption says "Pareto frontiers for finetuning Llama-2-7B (left) and RNJ-1 (right) on MetaMathQA."

4 comments
AuthorsOpenReview ↗

Reply to Reviewer 9trN

We thank the reviewer for carefully checking the paper and for the insightful and helpful feedback. For ease of reference, we mark additions and clarifications in blue in the revised manuscript.

W1. The connection between synthetic experiments and real SFT

Appendix B.2 evaluates exact training on examples sampled from the training corpus for both clean and corrupted text. Higher corrupted text performance supports more memorization, but lower clean training text accuracy does not independently establish poorer generalization. The experiment also trains on a repeatedly presented synthetic corpus rather than testing the proposed mechanism during reasoning SFT. Since the controlled math/code comparisons primarily report token-level validation loss, the evidence does not yet connect memorization behavior to downstream reasoning performance.

Thank you for this insightful comment. We agree with the reviewer that the synthetic experiment alone is insufficient to explain the observation on reasoning SFT that Muon may not preserve its benefits in pretraining, and that lower clean-training accuracy does not independently establish poorer generalization. Thanks for raising this important gap between the controlled observation and the hypothesized explanation of SFT behavior. Let us clarify the scope of our claim and its relation and gap to real reasoning SFT.

The current results nevertheless provide evidence that Muon has a stronger tendency towards rote memorization in the synthetic setting where patterns and rote memorization are clearly separated. Table 3 and Figure 7 show that Muon fits more corrupted sequences while achieving lower accuracy on clean training sequences than AdamW and SGD. Its corrupted-sequence memorization starts to increase earlier. Together, these observations suggest a tendency to begin rote memorization before patterns have been fully acquired, consistent with the intuition behind our hypothesis, though it may not necessarily imply the standard generalization concept.

We present this experiment as controlled evidence for that intuition. It is related to reasoning SFT in the sense that the data scale in SFT is usually small, and we desire the model to learn the patterns inside the training data instead of purely remembering them, especially for reasoning tasks that may have some new patterns compared to the pretraining data. Therefore, we think this tendency in the synthetic experiment provides some evidence for our hypothesized explanation of the observation.

However, we agree that establishing whether this tendency is a principal cause of the math/code SFT results requires more comprehensive validation. In particular, this may require separating pattern learning from memorization in real SFT tasks and linking them to downstream reasoning performance, which is itself a substantial research problem. This connection remains open and leave it as an important direction for future work.

To conclude, we would like to kindly note that we claimed this tendency only as a possible reason and an evidence for potential explanation of the observed empirical results. We will make this position cleaerer in the paper. Thanks again for raising this concern!

Q1. Llama panel in Figure 1

What is the correct Llama-2-7B panel for Figure 1, both panels say RNJ-1 (Muon-pretrained) but the caption says "Pareto frontiers for finetuning Llama-2-7B (left) and RNJ-1 (right) on MetaMathQA."

Thank you for pointing this issue out, and we apologize for the confusion. An incorrect version of RNJ-1 plot was mistakenly included in the left panel in the place of llama plot. We have replaced it with the correct Llama-2-7B panel, which matches the claims and related results in the paper. It supports our conclusion that full finetuning with AdamW (the same optimizer as pretraining) generally achieves a better learning-forgetting tradeoff than Muon and LoRA on Llama-2-7B.

We have also added Table 4 in Appendix C.1 with key frontier points, GSM8K accuracy ± SE, and 95% confidence intervals. For example, at the same LR=1e-5 and step=1000, AdamW achieves 49.28% GSM8K accuracy versus 41.70% for Muon, at almost identical general accuracy (63.43% versus 63.39%). The corrected figure and added table support the matching-optimizer advantage on Llama-2-7B. The GSM8K intervals quantify test-sample uncertainty; training-seed variation is reported separately for GPT-2 in Appendix C.3, Table 5.

Thanks for fixing Figure 1. I think the Llama result is helpful, AdamW does better on GSM8K at about the same general accuracy.

The shuffled text result is interesting, though I still don’t think we know if it explains the reasoning SFT results. I’m keeping my rating at 3. The optimizer matching result is the main contribution for me.

AuthorsOpenReview ↗

Reply to Reviewer 9trN, thanks!

Thanks for the prompt check of the fixed results and recognition of our contribution for the optimizer matching result. We also appreciate the reviewer's insightful thoughts on the link between SFT and the synthetic experiments. We agree that there still remains a gap between examining our hypothesis that Muon tends to memorize before fully acquiring the patterns in the synthetic experiment, which we did in the paper, and verifying its relation to SFT performance in practice. The latter may require separating patterns and rote memorization in real SFT tasks, which could itself be a substantial and interesting research question that we consider for future directions. We also double-checked our paper to avoid overclaiming that the synthetic experiments already explain the SFT performance.

Thanks again for the careful review and insightful feedback!

Great, thanks!

Reviewer_ZnuoOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the learning-forgetting tradeoff in SFT and names a phenomenon "optimizer-model consistency". Full finetuning with the same optimizer (or optimizer family) as pretraining gives the best Pareto frontier over learning rates and checkpoints. It beats other optimizers and also beats LoRA trained with the matching optimizer.

Critical strengths
  • The GPT-2 pipeline from pretraining to SFT is complete and controlled. The authors pretrain their own AdamW and Muon models. They then finetune each one with AdamW, Muon and SGD on three task types. In all six panels of Figure 2 the matching optimizer has the best frontier.
  • The norm framework links optimizers to model properties and gives testable predictions. In the Aα,β\mathcal{A}_{\alpha,\beta} framework, Muon is A2,2\mathcal{A}_{2,2}, while SignSGD and Adam fall in A1,∞\mathcal{A}_{1,\infty}. Choosing the SFT optimizer by norm family is a more general idea than a plain Adam versus Muon comparison.
  • The memorization experiment is clean, and its ordering is consistent. In Table 3 the corrupted accuracy always follows Muon > AdamW > SGD. On clean data the order reverses . The observation is interesting on its own and could motivate follow-up work on the implicit bias of optimizers.
Critical weaknesses
  • The large-model evidence is weak, and Figure 1 is likely wrong. Figure 1 is the only 7B/8B evidence. Yet both panels carry the title "RNJ-1 (Muon-pretrained)", share the same axis ranges (GSM8K 0.77 to 0.85, general accuracy 0.640 to 0.675) and show very similar background points. In Figure 8 the same Llama-2-7B setting reaches only about 0.35 to 0.60 on GSM8K. So the claim that the conclusion "holds still" on Llama-2-7B ("Results on larger models", Section 2) cannot be checked.
  • The mechanism evidence is only correlational and may contain a definition error. Figures 4 and 5 cover only 4k steps, while full pretraining is about 33k steps. Activation properties of the final checkpoints, which are the ones used for SFT, are not reported. A larger gap is that no quantitative link between activation properties and forgetting is shown. For instance, the proxy in Eq. (2) is never computed on final checkpoints and compared with the measured OWT loss increase.
  • The writing needs a careful pass. Lines 221 to 224 repeat one sentence, GPT-2-small is cited to Ouyang et al. (2022), the Figure 5 caption says "columns ... columns", and the Magicoder dataset in Appendix E differs from the one in the main text.
Questions for authors
  • Is the left panel of Figure 1 really Llama-2-7B? Both panels are titled "RNJ-1", and the GSM8K range (0.77 to 0.85) does not match the Llama results in Figure 8 (about 0.35 to 0.60). Please provide the correct figure and a table with the key frontier points, together with seed variation or GSM8K confidence intervals.
  • For Section 4.1, could pattern learning be measured on held-out clean text (perplexity or exact match on unseen data), with 1 to 2 epochs as in SFT and several learning rates and seeds?
  • Does the Section 4 conclusion still hold when the task is scored by accuracy instead of validation loss, either on larger GPT-2 models or on open Muon- and AdamW-pretrained models of similar size?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer Znuo (Part 1/3)

We thank the reviewer for carefully reading our paper and for the helpful comments. For ease of reference, we mark additions and clarifications in blue in the revised manuscript.

W1 & Q1. Figure 1 is likely wrong

W1: The large-model evidence is weak, and Figure 1 is likely wrong. Figure 1 is the only 7B/8B evidence. Yet both panels carry the title "RNJ-1 (Muon-pretrained)", share the same axis ranges (GSM8K 0.77 to 0.85, general accuracy 0.640 to 0.675) and show very similar background points. In Figure 8 the same Llama-2-7B setting reaches only about 0.35 to 0.60 on GSM8K. So the claim that the conclusion "holds still" on Llama-2-7B ("Results on larger models", Section 2) cannot be checked.

Q1: Is the left panel of Figure 1 really Llama-2-7B? Both panels are titled "RNJ-1", and the GSM8K range (0.77 to 0.85) does not match the Llama results in Figure 8 (about 0.35 to 0.60). Please provide the correct figure and a table with the key frontier points, together with seed variation or GSM8K confidence intervals.

Thank you for carefully checking the figures and identifying this mistake. The left panel was indeed incorrect: an incorrect version of the RNJ-1 plot was mistakenly included in place of the Llama panel. We apologize for the confusion. We have replaced it with the correct Llama-2-7B panel, whose GSM8K scale now agrees with the Llama results in the appendix.

Following your request, we have added Table 4 in Appendix C.1 to report key frontier vertices for every method, including all three LoRA ranks, with GSM8K accuracy ± SE and 95% confidence intervals. We also include the following four checkpoints at the same LR=1e-5 and step=1000 for a direct comparison.

BaseSFT optimizerLRStepGeneral accuracy (%)GSM8K ± SE (%)GSM8K 95% CI (%)
Llama-2-7BAdamW1e-5100063.4349.28 ± 1.38[46.59, 51.98]
Llama-2-7BMuon1e-5100063.3941.70 ± 1.36[39.07, 44.38]
RNJ-1AdamW1e-5100064.8481.96 ± 1.06[79.79, 83.94]
RNJ-1Muon1e-5100066.9083.32 ± 1.03[81.21, 85.24]

General accuracy is the mean of HellaSwag, ARC-Challenge, and WinoGrande accuracy, as defined in the paper. On the AdamW-pretrained Llama-2-7B, using AdamW in full finetuning improves GSM8K accuracy by 7.58 percentage points over Muon at almost identical general accuracy. On the Muon-pretrained RNJ-1, using Muon retains 2.06 percentage points more general accuracy, with similar GSM8K performance (83.32% versus 81.96%). The corrected Llama panel also shows that AdamW full finetuning generally achieves a better learning-forgetting tradeoff than Muon and LoRA, supporting optimizer-model consistency at this larger scale.

All checkpoints use the same 1,319 GSM8K examples. We report the evaluation harness's SE and Wilson 95% confidence intervals for the key frontier points in Table 4. The two-seed GPT-2 comparisons in Appendix C.3, Table 5 additionally report variation across training seeds.

AuthorsOpenReview ↗

Reply to Reviewer Znuo (Part 2/3)

W2. The mechanism evidence

The mechanism evidence is only correlational and may contain a definition error. Figures 4 and 5 cover only 4k steps, while full pretraining is about 33k steps. Activation properties of the final checkpoints, which are the ones used for SFT, are not reported. A larger gap is that no quantitative link between activation properties and forgetting is shown. For instance, the proxy in Eq. (2) is never computed on final checkpoints and compared with the measured OWT loss increase.

Thank you for raising this concern. We have added the two requested measurements: the activation differences persist at the checkpoints used for SFT, and Eq. (2) is strongly correlated with actual OWT loss increases.

We would like to first give a brief clarification of the link developed in Section 3, which is that pretraining shapes the activations xx, while the SFT update ΔW\Delta W changes a linear module's output on these fixed activations by ΔWx\Delta W x. Eq. (2) uses 12E∣ΔWx∣22\frac12 E\\|\Delta W x\\|_2^2 to approximate forgetting. Under the stated assumptions, Theorem 1 shows how this interaction favors matching the optimizer family at the same modeled SFT performance. We think this link is itself clear in logic, and the new measurements as the reviewer suggested provide empirical support for this explanation.

First, activation properties at the actual SFT initialization. Originally, we use the 4k-step runs, since we need to compare several optimizers within our compute budget. We now measure the actual step-31,000 checkpoints used for SFT. We included the results in Table 6 in Appendix C.6, presenting the sparsity metric as in Figure 4, averaged over linear modules (lower means sparser).

Pretraining optimizerInput ratio [95% CI]Output ratio [95% CI]
AdamW0.56947 [0.56878, 0.57021]0.67729 [0.67687, 0.67774]
Muon0.67443 [0.67415, 0.67473]0.75017 [0.74986, 0.75047]

As we can observe from the table, AdamW activations remain sparser at the actual SFT initialization, consistent with the observation in Figures 4–5 beyond the short runs.

Second, computing Eq. (2) and comparing it with actual forgetting. For each SFT checkpoint, we compute Eq. (2) from its actual weight change and the pretrained model's OWT activations, then average over the 48 linear modules. Formally, the resulting module-averaged proxy is defined as Q:=148∑l=14812E∣ΔWlx0,l∣22Q:=\frac{1}{48}\sum_{l=1}^{48}\frac12 E\\|\Delta W_l x_{0,l}\\|_2^2 (Appendix C.6), where ll indexes the modules. Independently, we evaluate the full base and SFT models on the same OWT blocks to measure their loss difference ΔL\Delta L.We compare QQ with ΔL\Delta L across eight MetaMathQA SFT configurations per base: two optimizers and the full learning-rate grid {5×10−5,10−4,3×10−4,5×10−4}\{5\times10^{-5},10^{-4},3\times10^{-4},5\times10^{-4}\}.

New Figure 15 in Appendix C.6 shows a positive relationship between QQ and ΔL\Delta L. Although imperfect to be interpreted as a linear association, this relationship supports Eq. (2) as a meaningful forgetting proxy: larger activation-weighted changes generally correspond to larger actual loss increases. Table 7 reports the correlations between QQ and the actual OWT loss increase ΔL\Delta L, where Spearman measures agreement in ordering, and Pearson measures linear association.

CorrelationAdamW baseMuon base
Spearman (ρ\rho)0.9290.952
Pearson (rr)0.9360.833

The strong correlations provide the requested quantitative check that Eq. (2) is a meaningful proxy for measured forgetting. Eq. (2) explicitly models the interaction between pretrained activations and SFT updates used in our theoretical explanation in Section 3. Calculation and details are in Appendix C.6.

W3. Writing

The writing needs a careful pass. Lines 221 to 224 repeat one sentence, GPT-2-small is cited to Ouyang et al. (2022), the Figure 5 caption says "columns ... columns", and the Magicoder dataset in Appendix E differs from the one in the main text.

Thanks for identifying these errors. We have corrected all four points: removed the repeated sentence, changed the GPT-2-small citation to Radford et al. (2019), changed the second “columns” to “rows” in the Figure 5 caption, and made Appendix E consistently name Magicoder-Evol-Instruct-110K, the dataset used in the experiments. We also corrected its license to Apache-2.0. These corrections are marked in red in the revised manuscript.

AuthorsOpenReview ↗

Reply to Reviewer Znuo (Part 3/3)

Q2. Pattern learning in SFT

For Section 4.1, could pattern learning be measured on held-out clean text (perplexity or exact match on unseen data), with 1 to 2 epochs as in SFT and several learning rates and seeds?

Thank you for this insightful question. We would like to first note that our synthetic experiment on measuring pattern learning is designed to examine optimizers' tendency to begin rote memorization before patterns have been fully acquired. Comparing how the optimizers learn clean and corrupted sequences over training provides controlled evidence for this intuition.

As discussed in Section 4.1, models learn patterns first. With only one or two epochs of training from scratch, we expect all three optimizers to still be primarily learning these patterns. We therefore would not expect the stronger rote memorization seen in the longer experiment to emerge in such short runs. This is consistent with the learning stages described in the paper.

Following the reviewer's suggestion, we also added a clean held-out evaluation after 1, 2, and 5 epochs, which are checkpoints within five-epoch runs, with tuned learning rates and two seeds per optimizer. The table reports perplexity of the held-out clean evaluation (mean ± sample SD across seeds). The details of the experiments are described in Appendix C.7.

Optimizer~1 epoch~2 epochs~5 epochs
AdamW377.5 ± 10.6172.6 ± 4.297.9 ± 2.4
Muon197.4 ± 2.5101.4 ± 0.779.7 ± 0.3
SGD2,287.9 ± 440.11,147.6 ± 112.9526.1 ± 73.8

We can observe that in these short runs, Muon achieves the lowest perplexity on clean text, while corrupted-training exact match at completion lengths 8 and 16 still remains zero throughout these short runs. These results are consistent with the expected early pattern-learning stage; the longer experiment examines the subsequent emergence of stronger rote memorization.

To more closely examine this tendency during SFT, we would need to start from pretrained models and separate learning new patterns from memorizing examples in real SFT tasks. Making this distinction and linking it to downstream reasoning performance is itself a challenging research problem, which we leave as an important direction for future work.

Q3. Section 4 Conclusions on Larger Scale

Does the Section 4 conclusion still hold when the task is scored by accuracy instead of validation loss, either on larger GPT-2 models or on open Muon- and AdamW-pretrained models of similar size?

Thanks for raising the question in extending the observation in Section 4 that Muon's pretraining advantage does not necessarily carry over to SFT on reasoning tasks to a larger scale. However, the major obstacle is we have not found an appropriate pair of AdamW- and Muon-pretrained checkpoints with matched architecture, data, and training budget that supports a fair comparison as we did for GPT-2 models in the paper. Due to the compute limit, it is Doing a comprehensive pretraining of such a pair is beyond our available compute and rebuttal timeframe, so we cannot add this accuracy-based validation now.

But we would also like to note that very recent work provides side supporting evidence at a larger scale. Pretraining Data Scale Reverses the Muon–AdamW Fine-Tuning Gap, study models up to 600M and find that, with sufficient pretraining data, AdamW achieves a better learning-forgetting tradeoff after SFT despite Muon's lower pretraining loss (Section 3, Figures 1–2), including math tasks. Their evaluation also uses held-out loss, but it supports our observation that better pretraining performance with Muon need not translate into better performance after SFT in a larger scale.

Reviewer_rwJsOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies different methods for adapting a general pretrained model to a new task (SFT) and studies the pareto frontier of different fine-tuning methods w.r.t (accuracy on new task ; vs forgetting on general task).

As part of their study, they show that full-finetuning ; and in particular full finetuning with an optimizer consistent with pretraining is has better pareto optimilaty than LORA (which is usually the de-facto method for forgetting less).

They study this by pre-training a GPT2 model from scratch using 2 different optimizers, and comparing pareto-fronteers of SFT methods for each checkpoint. The results show that respectively SFT with muon for the muon-trained GPT2 has the best pareto fronteer, and that adam SFT with adam-trained GPT2 has the best pareto fronteer. They also conduct SFT experiments on larger pretrained models ( Llama-2-7B and RNJ-1) ; with noiser results but still shows the same trend.

They then go into a theoretical analysis of why optimizer consistency is important, and hypothesize that it's due to implicit regularization of activations. The theoretical analysis continues with some explicit approximation of SFT loss and pretraining loss w.r.t parameter update using taylor expansions. Using some assumption, they confirm their initial finding with maths theorems.

Critical strengths
  • clear problem setting (SFT v pretraining pareto tradeoff)
  • great experiments : full pretraining with different optimizers on a small GPT2 model + different SFT settings ; this makes a very clean ablation for (pretraining_setup X sft_setup) + clear conclusions from those experiments. The experiments on larger models (llama and others) well welcome, even if more noisy. Experimental setup is sound, Muon experiments use muon only on 2D matrices which follows SOTA practices.
  • great theoretical basis and careful derivation of the SFT vs pretrain tradeoffs using taylor expansions.
Critical weaknesses
  • some of the figures have noisy data points, the pareto frontier make it seem that the conclusion is very clear but one question remains in terms of how stable the experiments are and why are the points so noisy above the pareto fronteer. It would be interesting to check that the curves are smoother when using a single lr and tracking across training steps; or when using a fixed number of epochs/training steps and gridding lr.
Questions for authors
  • does it matter to initialize the SFT optimizers with gradient moments learned during pre-training or can they be initialized from scratch?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer rwJs

We sincerely thank the reviewer for the insightful comments and recognition of our paper. Many thanks! Below we provide our discussions on the points. For ease of reference, we mark additions and clarifications in blue in the revised manuscript.

W1. Noisy data points

Some of the figures have noisy data points, the pareto frontier make it seem that the conclusion is very clear but one question remains in terms of how stable the experiments are and why are the points so noisy above the pareto fronteer. It would be interesting to check that the curves are smoother when using a single lr and tracking across training steps; or when using a fixed number of epochs/training steps and gridding lr.

Thanks for the helpful suggestion. The scatter partly reflects pooling checkpoints from different learning rates and training stages. We have added Figure 8 in Appendix C.1 with both views as the reviewer suggests: trajectories at fixed LRs and LR sweeps at fixed training steps for RNJ-1. Figure 12 in the paper also already provides fixed-LR trajectories for Llama-2-7B full FT and LoRA.

The curves seem to be smoother in this case, with smaller number of points on the frontier. The matching-optimizer advantage is also visible within individual runs. For example, at LR=1e-5 on RNJ-1, Muon retains 66.72%–67.20% general accuracy across the evaluated steps, compared with 64.54%–66.02% for AdamW. Task accuracy still fluctuates, but the retention advantage persists throughout this trajectory.

For better check of stability of the experiments, we have also added Table 4 in Appendix C.1, reporting key frontier points with GSM8K accuracy ± SE and 95% confidence intervals. The existing two-seed GPT-2 comparisons in Appendix C.3, Table 5 report training-seed variation. These additions make the consistency of the observed trend and its uncertainty clearer to assess.

Q1. Inherit pretraining optimizer states in SFT

Does it matter to initialize the SFT optimizers with gradient moments learned during pre-training or can they be initialized from scratch?

Thanks for this insightful question. The original runs in the paper follows the common way to use fresh states. We now added an experiment on full finetuning with the pretraining optimizer states as initialization on GPT-2 Magicoder setting, following the same settings and full five-LR grid as Figure 2. The conclusion is: initializing from scratch and inheriting pretraining states give almost identical Pareto frontier, with slightly better forgetting. We have documented the initialization and the new experiment in Appendices B.1 and C.4, and the results are added in Figure 10 in Appendix C.4.

For AdamW→AdamW, we inherit both moments and the bias-correction counter. For Muon→Muon, we inherit matrix momentum and the AdamW states for the remaining parameters.

The new Figure 10 in Appendix C.4 presents the resulting frontiers. As one can observe, the frontier of original fresh states and the inherited states almost coincidents. At the final step, task-NLL differences from the original fresh-state runs are below 0.001 across all ten model/LR combinations, with slightly lower OWT loss when inheriting states. A paired comparison at LR=1e-4 is shown below for your reference. Here both losses are lower-is-better.

Matching optimizerStepFresh task / OWT NLLInherited task / OWT NLL
AdamW2001.7084 / 3.05801.7071 / 3.0573
AdamW18001.4849 / 3.16591.4846 / 3.1643
Muon2001.6924 / 3.04721.6929 / 3.0460
Muon18001.4450 / 3.18451.4451 / 3.1834

Therefore, inheriting the pretraining moments gives almost the same performance in learning with a slight improvement in forgetting, and our original conclusions hold still. It is an interesting result actually since inheriting the pretraining moments does not really affect task learning, but slightly help with forgetting. We think this is an insightful potential problem for future study.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Brief Summary of the Rebuttal

Dear AC and Reviewers,

Thank you so much for the careful review and discussion. We are pleased to have had a productive rebuttal, with positive follow-up feedback and additional results addressing the reviewers' concerns. We summarize the main outcomes below.

Reviewer rwJs recognized our controlled experiments and theoretical analysis. Following their insightful suggestions, we added fixed-LR/fixed-step plots and optimizer-state-inheritance experiments, which preserve the matching-optimizer advantage.

Reviewer 9trN confirmed that the corrected Llama result supports optimizer matching and maintained the positive rating. Our follow-up discussion also reached agreement on the synthetic study's scope: it provides evidence of a memorization tendency, while establishing its connection to real reasoning SFT remains unclear and difficult, and would be interesting future work.

Reviewer Znuo has not yet provided follow-up feedback. For the AC's consideration, we summarize how we addressed their concerns:

  • Figure 1 and the large-model results. A wrong version of the RNJ-1 plot was mistakenly included in the Llama panel. We replaced it with the correct figure, consistent with the original experimental results and appendix. At the same LR and step, AdamW achieves 49.28% GSM8K accuracy versus 41.70% for Muon, at nearly identical general accuracy. Table 4 now reports key frontier points with uncertainty estimates, complementing the seed-variation results in Table 5.
  • Activation measurements and the quantitative link to forgetting. The 4k-step sparsity plots were controlled comparisons across optimizers since additional optimizers are included in the experiment, and SFT starts from the final step-31,000 pretrained checkpoints. We clarified these different settings and added activation measurements at the actual SFT initialization, confirming the same sparsity ordering. We also directly computed the Eq. (2) proxy and compared it with measured OWT loss increases across eight SFT configurations for the base model pretrained by each optimizer. The strong correlations and scatter plot provide the requested clear quantitative link (Tables 6–7 & Figure 15).
  • The synthetic study and presentation. We added the requested short-training held-out evaluation with tuned learning rates and two seeds, consistent with the expected early pattern-learning stage, and corrected all listed writing errors. For the larger-scale accuracy question, we explained the need for appropriately matched pretrained checkpoints and discussed complementary larger-scale, loss-based evidence in our reply.

We believe these corrections, additional results, and clarifications provide a substantive response to Reviewer Znuo's concerns and strengthen the evidence supporting our conclusions. We fully understand the reviewers' busy schedules and sincerely appreciate their time and helpful feedback. We hope these additions, together with the contributions already acknowledged in the reviews, will help the AC in the final assessment.

Thank you so much for the time and effort spent reviewing our work and for the careful consideration of our submission.

Authors of Submission 16316

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.