The paper claims that the proposed optimizer wrapper substantially improves Adam/Muon/SOAP in LLM pretraining, but the main pretraining benchmarking setup is not appropriate for such general conclusions.
Specifically, the paper's main experiments follow the benchmarking setup of GaLore [1], which is only appropriate for comparing memory-efficient optimizers and not for deriving such general optimization claims.
GaLore's paper benchmark uses a very small token batchsize of about 0.13M tokens (512 samples x 256 sequence length) and full BF16 precision (i.e., not mixed-precision training - all parameters including the optimizer states remain in BF16). Both are distinctly different from standard, large-scale pretraining settings, where token batch sizes of 1M-16M (with sequence lengths over 8k) and mixed-precision training are the norm [2,3].
Training with small batchsizes reduces the optimizer gap in pretraining [4,5] and (from my experience) keeping the Adam states in BF16 significantly worsens its performance due to accumulating in low precision. This is also apparent from the results of Table 1, where Adam variants with "less" momentum states with such Adafactor or RMSPRop obtain significantly better perplexities than Adam itself (e.g., Adam 1B PPL 16.82, Adafactor 15.00 and RMSPRop 14.69). It is also very strange that Adafactor+Magma (i.e., Adafactor with skipped updates) gives the best overall perplexity at 1B, even better than Muon/SOAP variants.
In addition the current dense benchmark does not use weight decay, which is standard in broader pretraining setups and is needed for regulating the effective update size [2,3,6].
My main point is that the main results of the paper in Table 1 are not reliable for making general conclusions (beyond a very specific setting of memory-efficient optimization) about optimizer performance and do not offer much evidence to support the very unconventional proposed idea of skipping half of the updates. The paper clearly argues about a general benefit of skipping updates in pretraining, for which I find the current experimental evidence very limited.
To support this, I trained the LLaMA 130M model using the GaLore pretraining setup as in the paper but simply changed the precision from full BF16 to mixed (using torch.amp). The results are as follows:
| Method | Validation PPL |
|---|
| Adam (lr=3e-3) | 21.78 |
| Adam (lr=3e-3) +Magma (lr=3e-3) | 22.24 |
| Adam (lr=3e-3) +Magma (lr=5e-3) | 22.03 |
| Adam (lr=3e-3) +Magma (lr=1e-2) | 21.95 |
| Adam (lr=3e-3) +Magma (lr=2e-2) | diverged |
| Adam (lr=3e-3) +Magma (lr=3e-2) | diverged |
The results above show that (i) Adam is severely underperforming with full BF16 (25.42 vs 21.78 with mixed precision) (ii) even with extra tuning of the Magma learning rate, the performance remains worse than just running Adam. I welcome any new results from the authors using mixed precision, along with all necessary hyperparameters needed to reproduce them.
In general, I would suggest the authors to try mixed-precision training (i.e., torch.amp) and to use larger batchsize/sequence length (1M-4M total tokens) to align with standard pretraining settings (beyond memory-efficient specific benchmarking).
Finally, a reproducible result on a competitive benchmark with modern pretraining setup, such as the "modded-nanogpt benchmark, track 3 -optimization" [7] (https://github.com/KellerJordan/modded-nanogpt/tree/master/records/track_3_optimization) would be of great value. If such a result is provided and skipping half of the updates can indeed accelerate the current Muon/Adam optimizer records (e.g., achieve the benchmark's target loss in fewer steps). Given such a result and the necessary hyperparameters for to reproduce it would be a very compeling reason for me to increase my score.
[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. Zhao et al., 2024.
[2] Muon is scalable for llm training. Liu et al., 2025.
[3] Fantastic Pretraining Optimizers and Where to Find Them. Wen et al., 2025.
[4] Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful. Marek et al., 2025.
[5] Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling. Srećković et al., 2025.
[6] Fantastic Pretraining Optimizers and Where to Find Them II:
Hyperball Optimization, Wen et al., 2026.
[7] Modded-NanoGPT Optimization Benchmark. Jordan 2026.
UPDATE (19 November 2026)
The authors uploaded their rebuttal a few minutes before the deadline, making any on-time comment from the reviewers' side effectively impossible. This is very unfortunate, because it defeats the purpose of the discussion period. The updated paper includes substantial changes from the original, making the above choice to upload the rebuttal exactly before the deadline even more problematic. There are several other issues, which I elaborate below.
The new draft introduces a completely new optimizer variant, called "Magma-F", fully absent from the originally submitted paper. In their updated experiments, the originally proposed algorithm, Magma, is now documented to perform worse when added on top of the Muon optimizer both at 350M and 1B scale, with the performance degradation to increase at the 1B scale. This is very problematic as Magma was proposed as an optimizer add-on (i.e., base optimizer + Magma), thus achieving "on par" (as the authors now claim) or worse performance than simply running the base optimizer alone is not useful. There is also no "Magma-F" result provided for the 1B model.
The rebuttal also makes evident a very important confound: the unmasked Adam learning rate for 1d layers and embeddings is tied to the matrix learning rate in the paper's experiments. This is shown from the author's sentence below:
"In all experiments in the paper, Magma uses a single global learning rate for all parameter groups, so that it introduces no additional learning-rate hyperparameter over the base optimizer."
For the 130M experiment for example, this means that in author's the baseline Adam experiment, the 1d and embeddings use LR 5e-3, while in the Adam+Magma experiment the learning rate for the 1d and emebeddings is 3e-2, i.e. 6 times larger. It is important to emphasize again that in both experiments the 1d and embedding weights use the baseline Adam as auxilary optimizer without Magma (the latter being applied only to the hidden matrix parameters), but in the Adam+Magma run the auxilary Adam uses 6 times larger learning rate. This confounds the improvement from adding Magma to the hidden matrix parameters and using a higher learning rate for the 1d and embeddings, the latter being suggested to improve pretraining based from the literature [8,9].
The above confound could be a reason behind the discrepancy in my runs and the authors. In addition, I suspect the author's training code has an evaluation bug. Since the authors are probably using the GaLore training code unchanged, that code has a bug in the evaluation function in line 103 (https://github.com/jiaweizzhao/GaLore/blob/master/torchrun_main.py#L103C1-L103C22), where total_batches is wrongly initialized to 1 instead of 0, giving incorrectly lower loss results and making the evaluation dependent on the number of GPUs used. Nevertheless no code is provided in the submission so it is not possible to check their exact configuration from my side.
Misleading comparison with outdated competitive benchmark records, despite adpoting hyperparameters for their run from recent records:
Even on the requested competitive benchmark (modded-NanoGPT), the authors compare to very outdated records (AdamW (#2) 5625 and Muon (#1) 3600). For both Muon (#36 3250 steps) and Adam (pull request #353 4950 steps), the current records outperform the "plus Magma/Magma-F" supplied results. It is in my opinion very misleading to compare their newly tunned results with the oldest Adam and Muon baseline records, especially since it seems that the authors adopt hyperparameters from more recent, better performing records. For instance, the authors adopted lr=0.035 and wd=0.025 for their Muon+Magma-F runs, which are used in the more recent baseline Muon record number #12 (3325 steps), while they include in the table the old Muon record #1 (3600 steps) with suboptimal lr and wd. A table doing a fair comparison should be as below:
| Method | Steps to reach loss target (lower is better) |
|---|
| AdamW (pull request #353) | 4950 |
| AdamW+Magma | 5375 |
| Muon (record #36) | 3250 |
| Muon+Magma-F | 3375 |
Overall I firmly maintain my recommendation for clear rejection with the highest confidence 4: The authors provided a rebuttal minutes before the deadline, prohibiting any discussion. Further, the newly updated paper shows that the original method "Magma" when added to the base optimizer can indeed result to worse performance when tested on modern pretraining settings, a result contradicting the premise of the original submission. Even the newly proposed method "Magma-F" (which in my opinion should be discarded for this submission as it does not appear anywhere originally) is reported to give only very marginal improvement in some smaller models, which is also unclear if it is due to the learning rate confound. Finally, the competitive benchmarking results compare to very outdated records, and do not show improvement over the current ones, even when using "Magma-F". Importantly, I find very misleading the fact that the authors used hyperparameters from a more recent Muon record to run their method, and yet compared their result with the oldest (and weakest) Muon record available.
[8] Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size. Hayou et al., 2025.
[9] Scaling Exponents Across Parameterizations and Optimizers. Everett et al., 2024.
We thank Reviewer 6y6p for the positive evaluation. Your comments led to a new study that strengthens the paper. The revised manuscript is uploaded, with changes in magenta.
1) Do the benefits of masking persist in substantially larger models?
We could not test beyond 1B within our compute budget, and we now state this as a limitation. Within our range, the gains grow from 60M to 350M and hold at 1B. Under a compute-optimal budget ($C \approx 6ND $, 1× Chinchilla), training compute grows quadratically with model size: one 7B run (6.7B parameters, 134B tokens) needs about 5.4×10^21 FLOPs, roughly 27× a 1B run. Even a minimal tuned comparison (one base optimizer with and without Magma at four learning rates, 8 runs) needs about 4.3×10^22 FLOPs. This will cost roughly 100,000-300,000 USD in commercial cloud compute. Within our range, the trend is favorable: Magma's average gain grows from 3.6% at 60M to 9–10% at 350M–1B (Table 1), and masking itself becomes more beneficial as models grow (see 2). Whether this continues beyond 1B remains to be verified, as the Conclusion now states.
2) Do larger models tolerate higher masking rates?
Random masking itself is slightly costly at small scale (e.g., 60M and 130M), but highly beneficial at large scale (e.g., 1B). We swept the keep probability p at all four scales (C4, base RMSProp), including damping-only Magma (p = 1.0, no masking), with the learning rate tuned over {3e-4, 1e-3, 3e-3, 1e-2} for every (scale, p) pair. Batches are 2–8× larger than in Table 1, with proportionally fewer steps (same token budget). Validation PPL at the best learning rate (bold: best per scale; DIV: diverged at every tested learning rate):
From the above table, we can see that Masking helps more as models grow. Relative to damping only, masking costs 0.1–0.4 PPL at 60M and 0.3–0.8 at 130M, is roughly neutral at 350M (0.10 better to 0.42 worse), and improves every setting at 1B by 0.24–0.60. The best setting moves from no masking (p = 1.0) at 60M–130M to p = 0.4 at 350M–1B. This is consistent with random masking acting as a geometric regularizer, from which larger models with more irregular loss landscapes benefit more.
It also shows that p does not need to be re-tuned per scale: for p \in [0.3,0.7], PPL varies by only 0.24–0.52 at each scale, and the default p=0.5 stays within 0.41 PPL (1.6%) of the best p in this range. In addition, Magma improves on base RMSProp for every p at every scale, with one tie (350M, p = 0.6: 16.93 vs. 16.92); at 1B, base RMSProp diverges at every tested learning rate. Please refer to Appendix D (Table A1) for the setup details and best learning rates.