- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an interesting finding that optimizers can not be easily switched and there is a cost to switching them. This suggests that different optimizers inject different structures into the network. They also find that Muon memorizes shuffled and corrupted sequences better. Experiments were done on GPT-2, Llama-2-7B and RNJ-1, and the optimizers compared were AdamW and Muon.
Critical strengths
Experiments with the frontier analysis across learning rates and checkpoints is good and yields robust conclusions on how full finetuning can have better learning-forgetting tradeoff than switching the optimizer / using LoRA. In addition, the authors tested both directions of pretraining / finetuning mismatch, and and the larger-model experiments extend the investigation beyond small models.
Critical weaknesses
Appendix B.2 evaluates exact training on examples sampled from the training corpus for both clean and corrupted text. Higher corrupted text performance supports more memorization, but lower clean training text accuracy does not independently establish poorer generalization. The experiment also trains on a repeatedly presented synthetic corpus rather than testing the proposed mechanism during reasoning SFT. Since the controlled math/code comparisons primarily report token-level validation loss, the evidence does not yet connect memorization behavior to downstream reasoning performance.
Questions for authors
What is the correct Llama-2-7B panel for Figure 1, both panels say RNJ-1 (Muon-pretrained) but the caption says "Pareto frontiers for finetuning Llama-2-7B (left) and RNJ-1 (right) on MetaMathQA."
4 comments
Thanks for fixing Figure 1. I think the Llama result is helpful, AdamW does better on GSM8K at about the same general accuracy.
The shuffled text result is interesting, though I still don’t think we know if it explains the reasoning SFT results. I’m keeping my rating at 3. The optimizer matching result is the main contribution for me.
Reply to Reviewer 9trN, thanks!
Thanks for the prompt check of the fixed results and recognition of our contribution for the optimizer matching result. We also appreciate the reviewer's insightful thoughts on the link between SFT and the synthetic experiments. We agree that there still remains a gap between examining our hypothesis that Muon tends to memorize before fully acquiring the patterns in the synthetic experiment, which we did in the paper, and verifying its relation to SFT performance in practice. The latter may require separating patterns and rote memorization in real SFT tasks, which could itself be a substantial and interesting research question that we consider for future directions. We also double-checked our paper to avoid overclaiming that the synthetic experiments already explain the SFT performance.
Thanks again for the careful review and insightful feedback!
Great, thanks!
Reply to Reviewer 9trN
We thank the reviewer for carefully checking the paper and for the insightful and helpful feedback. For ease of reference, we mark additions and clarifications in blue in the revised manuscript.
W1. The connection between synthetic experiments and real SFT
Thank you for this insightful comment. We agree with the reviewer that the synthetic experiment alone is insufficient to explain the observation on reasoning SFT that Muon may not preserve its benefits in pretraining, and that lower clean-training accuracy does not independently establish poorer generalization. Thanks for raising this important gap between the controlled observation and the hypothesized explanation of SFT behavior. Let us clarify the scope of our claim and its relation and gap to real reasoning SFT.
The current results nevertheless provide evidence that Muon has a stronger tendency towards rote memorization in the synthetic setting where patterns and rote memorization are clearly separated. Table 3 and Figure 7 show that Muon fits more corrupted sequences while achieving lower accuracy on clean training sequences than AdamW and SGD. Its corrupted-sequence memorization starts to increase earlier. Together, these observations suggest a tendency to begin rote memorization before patterns have been fully acquired, consistent with the intuition behind our hypothesis, though it may not necessarily imply the standard generalization concept.
We present this experiment as controlled evidence for that intuition. It is related to reasoning SFT in the sense that the data scale in SFT is usually small, and we desire the model to learn the patterns inside the training data instead of purely remembering them, especially for reasoning tasks that may have some new patterns compared to the pretraining data. Therefore, we think this tendency in the synthetic experiment provides some evidence for our hypothesized explanation of the observation.
However, we agree that establishing whether this tendency is a principal cause of the math/code SFT results requires more comprehensive validation. In particular, this may require separating pattern learning from memorization in real SFT tasks and linking them to downstream reasoning performance, which is itself a substantial research problem. This connection remains open and leave it as an important direction for future work.
To conclude, we would like to kindly note that we claimed this tendency only as a possible reason and an evidence for potential explanation of the observed empirical results. We will make this position cleaerer in the paper. Thanks again for raising this concern!
Q1. Llama panel in Figure 1
Thank you for pointing this issue out, and we apologize for the confusion. An incorrect version of RNJ-1 plot was mistakenly included in the left panel in the place of llama plot. We have replaced it with the correct Llama-2-7B panel, which matches the claims and related results in the paper. It supports our conclusion that full finetuning with AdamW (the same optimizer as pretraining) generally achieves a better learning-forgetting tradeoff than Muon and LoRA on Llama-2-7B.
We have also added Table 4 in Appendix C.1 with key frontier points, GSM8K accuracy ± SE, and 95% confidence intervals. For example, at the same LR=1e-5 and step=1000, AdamW achieves 49.28% GSM8K accuracy versus 41.70% for Muon, at almost identical general accuracy (63.43% versus 63.39%). The corrected figure and added table support the matching-optimizer advantage on Llama-2-7B. The GSM8K intervals quantify test-sample uncertainty; training-seed variation is reported separately for GPT-2 in Appendix C.3, Table 5.