-
The contribution over prior work is narrow, and the closest prior work is not compared. Treating the embedding table, the hidden layers and the language model head under one norm-based view has been done before, e.g., by the modular norm [1], modular duality [2] and Scion [3]. Scion in particular trains all layers of GPT models with norm-based updates and does not use AdamW at all. The update directions are not new either, as the paper itself acknowledges. The column normalization for the embedding table comes from [2], and the row normalization for the LM head is already an option in Scion and also appears in [4] and [5]. What remains new is the total variation argument for the LM head and a step size that scales with the dual norm of the momentum. Yet the experiments only compare with Ember, another method that treats the two layers specially, and not with Scion. It is therefore unclear what MuonIO brings over existing methods in practice.
-
The experiments also do not isolate the parts that are new. No experiment applies the update to only one of the two layers, so it is unclear whether the gain comes from the embedding update that prior work already proposed, or from the new update for the LM head. The update for the LM head is also never compared with other norm-based choices, such as the sign update that Scion uses for the output layer, or Muon itself. Without these comparisons, the experiments cannot show that the norm chosen by the total variation argument is the right choice.
-
The gains over the tuned Muon baseline are small, between 0.5% and 2.6% in perplexity, and the evidence for them is not strong. MuonIO uses about twice the hidden learning rate of the tuned Muon baseline at every scale. The hidden learning rate is fixed to different values for the two methods while tuning the other learning rates, which does not match the statement in Table 1 that all learning rates are tuned jointly. Part of the gain may therefore come from a better hidden learning rate, and an ablation of the tuned baseline with the same hidden learning rate would clarify it. With Polar Express, where all learning rates are tuned jointly, the new update is slightly worse than AdamW at the smallest scale. Besides, at the largest scale the standard deviation of MuonIO is about as large as its gain.
[1] Large, Tim, et al. “Scalable Optimization in the Modular Norm.” arXiv preprint arXiv:2405.14813 (2024).
[2] Bernstein, Jeremy, and Laker Newhouse. “Modular Duality in Deep Learning.” arXiv preprint arXiv:2410.21265 (2024).
[3] Pethick, Thomas, et al. “Training Deep Learning Models with Norm-Constrained LMOs.” arXiv preprint arXiv:2502.07529 (2025).
[4] Glentis, Athanasios, et al. “Memory-Efficient LLM Pretraining via Minimalist Optimizer Design.” arXiv preprint arXiv:2506.16659 (2025).
[5] Lau, Tim Tsz-Kit, and Weijie Su. “Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers.” arXiv preprint arXiv:2605.18106 (2026).
We thank the reviewer for the comments. We answer the two questions in order. The answer to Q1 also covers the weakness.
W1 / Q1: Comparison with existing column and row normalization updates
We added Scion [1] as a baseline in two variants. We name each variant by the updates it uses for the embedding, the hidden layers and the LM head.
The dimension factors are those of Tables 3 and 4 in the Scion paper. In both Scion variants the step size is fixed by the learning rate and these factors, and it does not depend on how large the momentum is. MuonIO does not use a fixed step. After normalizing the momentum vector of each token, it multiplies the step by d and by the dual norm of the layer's momentum, which is the sum of the norms of the momentum vectors of all tokens (line 5 of Algorithm 1). The step size then changes during training as the momentum changes.
We tuned both variants with the protocol that Appendix D describes for Muon Tuned and MuonIO. We swept the same learning rate grids for the embedding and the LM head, and kept all other settings as in Appendix D.
Final validation perplexity on C4 (lower is better):
We report the mean of the perplexity.
MuonIO reaches 4.2% lower perplexity at 60M and 5.6% lower at 130M than the ColNorm/RowNorm variant. Apart from constants that the tuned learning rates absorb, the I/O update of that variant is the MuonIO update without the dual norm scaling. Compared with the Sign configuration of the Scion paper, MuonIO is 6.2% and 8.8% lower. We will add these results to the revised paper.
Q2: The AdamW FLOP count with weight decay disabled
The count of 13Vd for AdamW in Table 3 is for the general AdamW update [2], which includes decoupled weight decay. Table 4 in Appendix C lists the stages, with N=Vd:
The last stage uses two multiplications, one division and one subtraction per entry. One of the two multiplications is the weight decay term (1−γλ)W, where γ is the learning rate and λ is the weight decay.
When weight decay is disabled, as in our experiments (Section 4.1), this multiplication is not needed. The last stage then costs 3N and AdamW costs 12Vd. Our update costs 7Vd+3V, so the FLOP saving is about 42% in this setting and about 46% when AdamW uses weight decay. The 50% saving in optimizer state does not depend on weight decay.
References
[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning, pp. 49069-49104. PMLR, 2025.
[2] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.