acceptodds
Under review as a conference paper at ICLR 2027

MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads

Abstract

The Muon optimizer derives its update rule for hidden linear layers by solving a local linearization of the loss penalized by the spectral norm, motivated by an RMS-stability argument for dense linear layers. Standard Muon implementations, however, exclude the input (embedding table) and output (language model head) layers from this principled treatment, for which they use AdamW instead. We present MuonIO, a single Muon-style update for both of these layers. For the language model head , we motivate the use of the operator norm, due to the Lipschitz continuity of the softmax output geometry, while for the embedding table , we draw on the operator norm, based on the one-hot input geometry identified by Bernstein & Newhouse (2025). The identity then puts both matrices in the same vocabulary-oriented geometry: MuonIO applies a single normalized-vector rule, which appears as column normalization for and row normalization for . Empirical evaluations demonstrate the effectiveness of our approach, with MuonIO reducing I/O optimizer state memory by 50% and I/O update FLOPs by 46% compared to Muon for 1B LLaMA pretraining on C4, while also improving validation perplexity.

Primary area:
optimization
Keywords:
Muon, norm-aware descent, pretraining, embedding, language model head

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_hWizOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MuonIO, a unified update for embedding tables and language model heads. It derives token-wise normalized updates from embedding geometry and softmax stability. Experiments on C4 at 60M, 130M, and 1B scales report lower validation perplexity than a tuned Muon baseline. The method also halves I/O optimizer state memory compared with AdamW.

Critical strengths

1.The paper provides a clear norm-based justification for treating embeddings and language model heads with a unified update rule.

2.The proposed update is simple, integrates with Muon-style optimizers, and halves I/O optimizer state memory compared with AdamW.

3.The reported C4 results show lower validation perplexity than a tuned Muon baseline at the 60M, 130M, and 1B scales.

Critical weaknesses

The experiments do not directly compare MuonIO with the closely related normalization-based updates discussed in the paper.

Questions for authors

1.Could the authors compare MuonIO with existing embedding column-normalization and output-head row-normalization methods?

2.Could the authors clarify the AdamW FLOP count, given that weight decay is disabled?

1 comment

We thank the reviewer for the comments. We answer the two questions in order. The answer to Q1 also covers the weakness.

W1 / Q1: Comparison with existing column and row normalization updates

We added Scion [1] as a baseline in two variants. We name each variant by the updates it uses for the embedding, the hidden layers and the LM head.

  • Scion (Sign → Spectral → Sign). This is the configuration used in the GPT experiments of the Scion paper. Every entry of the embedding table and of the LM head moves by a fixed amount, with the sign determined by its momentum. On the LM head this amount is divided by the hidden dimension dd.
  • Scion (ColNorm → Spectral → RowNorm). This variant uses column normalization for the embedding table and row normalization for the LM head. The momentum vector of each token is normalized to unit length and then multiplied by a fixed factor, which is d\sqrt{d} for the embedding table and 1/d1/\sqrt{d} for the LM head.

The dimension factors are those of Tables 3 and 4 in the Scion paper. In both Scion variants the step size is fixed by the learning rate and these factors, and it does not depend on how large the momentum is. MuonIO does not use a fixed step. After normalizing the momentum vector of each token, it multiplies the step by d\sqrt{d} and by the dual norm of the layer's momentum, which is the sum of the norms of the momentum vectors of all tokens (line 5 of Algorithm 1). The step size then changes during training as the momentum changes.

We tuned both variants with the protocol that Appendix D describes for Muon Tuned and MuonIO. We swept the same learning rate grids for the embedding and the LM head, and kept all other settings as in Appendix D.

Final validation perplexity on C4 (lower is better):

Configuration60M130M
Plain Muon36.99028.510
Scion (Sign → Spectral → Sign)30.26425.931
Scion (ColNorm → Spectral → RowNorm)29.63025.067
Muon Tuned28.51824.297
MuonIO (ours)28.38323.657

We report the mean of the perplexity.

MuonIO reaches 4.2% lower perplexity at 60M and 5.6% lower at 130M than the ColNorm/RowNorm variant. Apart from constants that the tuned learning rates absorb, the I/O update of that variant is the MuonIO update without the dual norm scaling. Compared with the Sign configuration of the Scion paper, MuonIO is 6.2% and 8.8% lower. We will add these results to the revised paper.

Q2: The AdamW FLOP count with weight decay disabled

The count of 13Vd13Vd for AdamW in Table 3 is for the general AdamW update [2], which includes decoupled weight decay. Table 4 in Appendix C lists the stages, with N=VdN = Vd:

StageFLOPs
First moment3N3N
Second moment4N4N
Corrected denominator2N2N
Decay and adaptive step, W←(1−γλ)W−γ(M⊘D)\mathbf W \leftarrow (1-\gamma\lambda) \mathbf W - \gamma (\mathbf M \oslash \mathbf D)4N4N
Total13N=13Vd13N = 13Vd

The last stage uses two multiplications, one division and one subtraction per entry. One of the two multiplications is the weight decay term (1−γλ)W(1-\gamma\lambda) \mathbf W, where γ\gamma is the learning rate and λ\lambda is the weight decay.

When weight decay is disabled, as in our experiments (Section 4.1), this multiplication is not needed. The last stage then costs 3N3N and AdamW costs 12Vd12Vd. Our update costs 7Vd+3V7Vd + 3V, so the FLOP saving is about 42% in this setting and about 46% when AdamW uses weight decay. The 50% saving in optimizer state does not depend on weight decay.

References

[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning, pp. 49069-49104. PMLR, 2025.

[2] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.

Reviewer_Nbz5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MuonIO, a norm-based update for the embedding and the LM head layer. The motivation is that Muon derives its update for the hidden layers, while the two vocabulary facing layers are still trained with AdamW in practice. For the embedding table, the authors follow prior work and measure the weight change by its largest column norm. For the LM head, they measure the weight change by its largest row norm, which they show that it bounds the change in the predicted distribution. The two norms are transposes of each other, so both layers end up with the same update rule, which normalizes the vector of each token in the vocabulary. The experiments pretrain LLaMA models from 60M to 1B parameters on C4. MuonIO reaches a lower validation perplexity than Muon with separately tuned AdamW on the two IO layers, while halving their optimizer state and reducing the cost of their update.

Critical strengths
  1. The paper addresses a practically relevant gap. Most Muon implementations still train the embedding table and the LM head with AdamW, although these two layers are among the largest matrices of a language model.

  2. The argument for the LM head is clean and intuitive. It bounds how much the predicted distribution can change, and leads to an update that matches the one for the embedding table, rendering a unifed view.

Critical weaknesses
  1. The contribution over prior work is narrow, and the closest prior work is not compared. Treating the embedding table, the hidden layers and the language model head under one norm-based view has been done before, e.g., by the modular norm [1], modular duality [2] and Scion [3]. Scion in particular trains all layers of GPT models with norm-based updates and does not use AdamW at all. The update directions are not new either, as the paper itself acknowledges. The column normalization for the embedding table comes from [2], and the row normalization for the LM head is already an option in Scion and also appears in [4] and [5]. What remains new is the total variation argument for the LM head and a step size that scales with the dual norm of the momentum. Yet the experiments only compare with Ember, another method that treats the two layers specially, and not with Scion. It is therefore unclear what MuonIO brings over existing methods in practice.

  2. The experiments also do not isolate the parts that are new. No experiment applies the update to only one of the two layers, so it is unclear whether the gain comes from the embedding update that prior work already proposed, or from the new update for the LM head. The update for the LM head is also never compared with other norm-based choices, such as the sign update that Scion uses for the output layer, or Muon itself. Without these comparisons, the experiments cannot show that the norm chosen by the total variation argument is the right choice.

  3. The gains over the tuned Muon baseline are small, between 0.5% and 2.6% in perplexity, and the evidence for them is not strong. MuonIO uses about twice the hidden learning rate of the tuned Muon baseline at every scale. The hidden learning rate is fixed to different values for the two methods while tuning the other learning rates, which does not match the statement in Table 1 that all learning rates are tuned jointly. Part of the gain may therefore come from a better hidden learning rate, and an ablation of the tuned baseline with the same hidden learning rate would clarify it. With Polar Express, where all learning rates are tuned jointly, the new update is slightly worse than AdamW at the smallest scale. Besides, at the largest scale the standard deviation of MuonIO is about as large as its gain.

[1] Large, Tim, et al. “Scalable Optimization in the Modular Norm.” arXiv preprint arXiv:2405.14813 (2024).

[2] Bernstein, Jeremy, and Laker Newhouse. “Modular Duality in Deep Learning.” arXiv preprint arXiv:2410.21265 (2024).

[3] Pethick, Thomas, et al. “Training Deep Learning Models with Norm-Constrained LMOs.” arXiv preprint arXiv:2502.07529 (2025).

[4] Glentis, Athanasios, et al. “Memory-Efficient LLM Pretraining via Minimalist Optimizer Design.” arXiv preprint arXiv:2506.16659 (2025).

[5] Lau, Tim Tsz-Kit, and Weijie Su. “Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers.” arXiv preprint arXiv:2605.18106 (2026).

Questions for authors
  1. MuonIO scales the update of the two layers by the dual norm, so its step size changes with the size of the momentum. Muon instead takes a step of fixed size for the hidden layers. Why are the two kinds of layers treated differently, and how does MuonIO perform with a step of fixed size?

  2. Each row of the embedding table is normalized separately. A token that does not appear in the current batch therefore still moves along its old momentum. With AdamW, the step of such a row shrinks quickly, since its first moment decays faster than the square root of its second moment. With MuonIO, the normalization cancels this decay, so the row keeps taking full steps until the norm of its momentum becomes smaller than the constant used to avoid division by zero. How often does this happen in practice, and does it affect rare tokens?

  3. The paper states that the update applies directly to tied embeddings, but all experiments use untied weights. Does the gain also hold when the two layers are tied?

4 comments

We thank the reviewer for the detailed review. We respond to the three weaknesses first and then to the three questions.

W1: Comparison with Scion

We agree with the reviewer's description of what is new. The normalization directions come from prior work. Our additions are the total variation argument for the LM head and the step size that is scaled by the dual norm. To show what these additions bring, we added Scion [1] as a baseline in two variants. We name each variant by the updates it uses for the embedding, the hidden layers and the LM head.

  • Scion (Sign → Spectral → Sign). This is the configuration used in the GPT experiments of the Scion paper. Every entry of the embedding table and of the LM head moves by a fixed amount, with the sign determined by its momentum. On the LM head this amount is divided by the hidden dimension dd.
  • Scion (ColNorm → Spectral → RowNorm). This variant uses column normalization for the embedding table and row normalization for the LM head. The momentum vector of each token is normalized to unit length and then multiplied by a fixed factor, which is d\sqrt{d} for the embedding table and 1/d1/\sqrt{d} for the LM head.

The dimension factors are those of Tables 3 and 4 in the Scion paper. In both Scion variants the step size is fixed by the learning rate and these factors, and it does not depend on how large the momentum is. MuonIO does not use a fixed step. After normalizing the momentum vector of each token, it multiplies the step by d\sqrt{d} and by the dual norm of the layer's momentum, which is the sum of the norms of the momentum vectors of all tokens (line 5 of Algorithm 1). The step size then changes during training as the momentum changes.

We tuned both variants with the protocol that Appendix D describes for Muon Tuned and MuonIO. We swept the same learning rate grids for the embedding and the LM head, and kept all other settings as in Appendix D.

Final validation perplexity on C4 (lower is better):

Configuration60M130M
Plain Muon36.99028.510
Scion (Sign → Spectral → Sign)30.26425.931
Scion (ColNorm → Spectral → RowNorm)29.63025.067
Muon Tuned28.51824.297
MuonIO (ours)28.38323.657

We report the mean of the perplexity.

MuonIO reaches 4.2% lower perplexity at 60M and 5.6% lower at 130M than the ColNorm/RowNorm variant. Apart from constants that the tuned learning rates absorb, the I/O update of that variant is the MuonIO update without the dual norm scaling. Compared with the Sign configuration of the Scion paper, MuonIO is 6.2% and 8.8% lower.

W2: Isolating the two layers and the choice of norm for the LM head

We agree that applying the update to one layer at a time would separate the contribution of the embedding update from that of the LM head update. We could not run these two extra sweeps (embedding only and LM head only) during the rebuttal period because of limited time and compute.

On the choice of norm, the Scion comparison above includes the sign update that Scion uses. Replacing the sign update by column and row normalization lowers perplexity from 30.264 to 29.630 at 60M and from 25.931 to 25.067 at 130M. Adding the dual norm scaling lowers it further, to 28.383 and 23.657.

W3: Size of the gains and the hidden layer learning rate

We would like to respectfully argue that the gains are significant. For comparison, Polar Express Tuned differs from Muon Tuned only in the hidden layer update (Polar Express [2] in place of Muon [3]), and its gain over Muon Tuned is of the same size.

Improvement over Muon Tuned60M130M1B
Polar Express Tuned (hidden layers changed)0.4%0.9%1.1%
MuonIO (I/O layers changed)0.5%2.6%1.1%

At the same time, MuonIO obtains its perplexity improvement while also halving the optimizer state and reducing FLOPs by 46% on the I/O layers. Table 2 of the revised paper now reports all five seeds at 60M and 130M without selection.

The hidden layer learning rates differ for the following reason. A joint sweep over all four learning rates was too expensive for the configurations that use Muon, so we tuned in two stages. We first ran a small sweep for each method and fixed the hidden layer and auxiliary learning rates at the best values found for that method. We then swept the embedding and LM head learning rates as described in Appendix D. The hidden layer learning rates of 0.02 for Muon Tuned and 0.04 for MuonIO at 60M and 130M each came from that method's own initial sweep. The reviewer is right that the caption of Table 1 ("jointly tuned") does not describe this procedure. It describes the Polar Express configurations, which we did tune jointly (Appendix D).

We also ran Muon Tuned with the hidden layer learning rate set to 0.04, the value used for MuonIO, as the reviewer suggested. The results are similar to those with 0.02. We get 28.499 ± 0.029 at 60M and 24.076 ± 0.190 at 130M, compared with 28.518 ± 0.025 and 24.297 ± 0.086 in Table 2. MuonIO is still lower at both scales, at 28.383 ± 0.029 and 23.657 ± 0.033.

Q1: Why the I/O step scales with the momentum, and performance with a fixed step

This is a very good and meaningful question.

Standard Muon [3] does not include the dual norm factor in its step for the hidden layers, and we use standard Muon (or Polar Express [2]) for the hidden layers. Our understanding is that for the spectral norm used by Muon the dual norm is the nuclear norm (equation (2)), which is expensive to compute exactly, for example with an SVD. On the I/O layers the dual norm is the sum of the row norms, which the update already computes. It would be interesting to test whether this factor also helps the hidden layers.

For the fixed step, the new Scion baseline is the test. The Scion (ColNorm → Spectral → RowNorm) variant takes a step of fixed size. It reaches 29.630 at 60M and 25.067 at 130M, compared with 28.383 and 23.657 for MuonIO.

Q2: Stale momentum and rare tokens

We ran an experiment to measure this. During training we counted how often the normalization in line 4 of Algorithm 1 uses ε\varepsilon instead of the norm of the row's momentum. The count is over all rows of the two I/O matrices and all steps.

ModelRow updates using ε\varepsilonRow updates using the row norm
60M9.88%90.12%
130M10.01%89.99%

In about 10% of the row updates the normalization uses ε\varepsilon. In those cases the step is smaller than the full size, because it shrinks in proportion to the norm of the momentum.

Q3: Tied weights

We thank the reviewer for this suggestion. The tied weight setting is an interesting experimental direction. Because time and compute were limited during the rebuttal period, we used our resources first for the experiments above, namely the Scion baselines and the run with the matched hidden layer learning rate. We were not able to add a sweep with tied weights.

References

[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning, pp. 49069-49104. PMLR, 2025.

[2] Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The Polar Express: Optimal matrix sign methods and their application to the Muon algorithm. In The Fourteenth International Conference on Learning Representations, 2026.

[3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.

Thank you for the detailed response and the new experiments. My concerns are mostly addressed, and I will raise my score to 3.

The Scion comparison also raises a new question. Since the dual norm scaling is what makes MuonIO work, it is important to understand why it helps, and why it is applied only to the embedding table and the LM head. The cost of the nuclear norm does not seem to be a strong reason. For the hidden layers, the nuclear norm of the momentum equals its inner product with the orthogonalized momentum. Muon already approximates an orthogonalized momentum in every step via Newton-Schulz, so one extra inner product gives an approximation of the nuclear norm. I hope the authors can test the same scaling on the hidden layers. If it also helps there, it would be a general improvement to Muon and further broadens the impact of this work. If not, the paper should try to explain what makes the embedding table and the LM head different.

Thank you for increasing your score and for raising this insightful point. We will evaluate the adaptive scaling factor for the hidden layers and report our findings in the revised manuscript.

Reviewer_zvyNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MuonIO replaces AdamW on embedding tables and language-model heads with vocabulary-wise normalized momentum. It motivates the output-layer geometry through a tight softmax total-variation bound and reports improved perplexity with reduced I/O optimizer-state memory and arithmetic cost.

Critical strengths

The softmax stability analysis provides an interpretable motivation, and maintaining one momentum buffer halves I/O optimizer-state storage relative to AdamW.

Critical weaknesses
  1. The empirical conclusions are weakened by outcome-based seed selection and confounded comparisons. Table 2 reports only the best three of five seeds at 60M/130M, producing optimistically selected statistics. Table 6 also uses different hidden-layer learning rates for Muon Tuned and MuonIO, so the gains cannot be attributed solely to the I/O update.

  2. The advance over existing normalized I/O updates is insufficiently established. Appendix A acknowledges closely related directions in Scion, SCALE, and symmetry-compatible optimizers, but these are not directly compared under a common training setup. The TV interpretation alone does not demonstrate an algorithmic advantage.

Questions for authors
  1. What are the mean and standard deviation over all five seeds, without selecting runs by validation performance?

  2. Do the gains persist when hidden-layer and auxiliary hyperparameters are held identical? Please also reconcile Table 1’s joint-tuning description with Appendix D’s fixed hidden-layer learning rates.

  3. How does MuonIO compare with existing normalized I/O rules under matched settings, and what benefit comes specifically from its global dual-norm scaling factor?

2 comments

We thank the reviewer for the detailed comments. We answer the three questions in order. Each answer also covers the related weakness.

Q1 / W1: Mean and standard deviation over all five seeds

We agree with this point. We originally planned to run five seeds for every experiment, but the 1B runs turned out to be too expensive and we ran only three seeds at that scale. To keep three seeds in every column, we then reported the best three of the five seeds at 60M and 130M. Table 2 of the revised paper now reports the mean and sample standard deviation over all five seeds at 60M and 130M, without any selection.

Configuration60M130M
Plain Muon (baseline 1)36.990 ± 0.05328.510 ± 0.093
Muon Tuned (baseline 2)28.518 ± 0.02524.297 ± 0.086
Ember28.858 ± 0.04424.076 ± 0.004
MuonIO (ours)28.383 ± 0.02923.657 ± 0.033
Polar Express Tuned28.4133 ± 0.054924.074 ± 0.030
Polar Express IO (ours)28.454 ± 0.03923.657 ± 0.019

The conclusions stay the same. MuonIO improves on Muon Tuned by 0.47% at 60M and 2.64% at 130M, compared with 0.48% and 2.56% in the submitted table. Every MuonIO seed also reaches lower perplexity than every Muon Tuned seed at both scales. The ordering within the Polar Express block has not changed either.

Q2 / W1: Identical hidden layer and auxiliary hyperparameters, and Table 1 versus Appendix D

Yes, the gains persist. For Muon Tuned and MuonIO at 60M and 130M, the auxiliary learning rate (0.02), the AdamW coefficients and the hidden layer momentum are already the same (Table 6). The only difference is the hidden layer learning rate, which is 0.02 for Muon Tuned and 0.04 for MuonIO. We ran Muon Tuned again with the hidden layer learning rate set to 0.04. The results are similar to those with 0.02. We get 28.499 ± 0.029 at 60M and 24.076 ± 0.190 at 130M, compared with 28.518 ± 0.025 and 24.297 ± 0.086 in Table 2. MuonIO is still lower at both scales, at 28.383 ± 0.029 and 23.657 ± 0.033.

We now turn to Table 1 and Appendix D. Plain Muon [1] (baseline 1) and Muon Tuned (baseline 2) use the same optimizer assignment. Muon updates the hidden weight matrices, and AdamW updates the embeddings, LM head, and gains and biases. The two baselines differ in how their learning rates are grouped and tuned. Appendix D describes Plain Muon as a baseline with two learning rates, one for the hidden weights and one AdamW rate shared across the remaining groups. Muon Tuned allows a separate rate for each of the four parameter groups, with the same grouping of learning rates as Ember and MuonIO. At 60M and 130M, the Muon Tuned sweeps searched separate grids for the embedding and LM head rates while holding the hidden weight rate fixed at 0.02.

We will revise the hyperparameter tuning description as:

"For Plain Muon (baseline 1), we perform a two-dimensional grid search over the hidden weight learning rate {0.005, 0.01, 0.02} and the AdamW learning rate {0.0003, 0.0005, 0.001}. In each trial, the embedding, LM head, and gains and biases share the same AdamW learning rate.

For Muon Tuned (baseline 2), MuonIO, and Ember at 60M and 130M, we search separate discrete grids spanning [0.001, 0.3] for the embedding learning rate and [0.0003, 0.5] for the LM head learning rate. During these sweeps, we fix the hidden weight learning rate at 0.02 for Muon Tuned and 0.04 for both MuonIO and Ember."

Q3 / W2: Comparison with existing normalized I/O rules, and the benefit of the dual norm scaling factor

We agree that the total variation bound motivates the choice of norm but does not by itself show an algorithmic advantage. To test this with an experiment, we added Scion [2] as a baseline in two variants. We name each variant by the updates it uses for the embedding, the hidden layers and the LM head.

  • Scion (Sign → Spectral → Sign). This is the configuration used in the GPT experiments of the Scion paper. Every entry of the embedding table and of the LM head moves by a fixed amount, with the sign determined by its momentum. On the LM head this amount is divided by the hidden dimension dd.
  • Scion (ColNorm → Spectral → RowNorm). This variant uses column normalization for the embedding table and row normalization for the LM head. The momentum vector of each token is normalized to unit length and then multiplied by a fixed factor, which is d\sqrt{d} for the embedding table and 1/d1/\sqrt{d} for the LM head.

The dimension factors are those of Tables 3 and 4 in the Scion paper. In both Scion variants the step size is fixed by the learning rate and these factors, and it does not depend on how large the momentum is. MuonIO does not use a fixed step. After normalizing the momentum vector of each token, it multiplies the step by d\sqrt{d} and by the dual norm of the layer's momentum, which is the sum of the norms of the momentum vectors of all tokens (line 5 of Algorithm 1). The step size then changes during training as the momentum changes.

We tuned both variants with the protocol that Appendix D describes for Muon Tuned and MuonIO. We swept the same learning rate grids for the embedding and the LM head, and kept all other settings as in Appendix D.

Final validation perplexity on C4 (lower is better):

Configuration60M130M
Plain Muon36.99028.510
Scion (Sign → Spectral → Sign)30.26425.931
Scion (ColNorm → Spectral → RowNorm)29.63025.067
Muon Tuned28.51824.297
MuonIO (ours)28.38323.657

We report the mean of the perplexity.

Apart from constants that the tuned learning rates absorb, the I/O update of the second Scion variant is the MuonIO update without the dual norm scaling. The gap between these two rows is the benefit of that scaling. MuonIO reaches 4.2% lower perplexity at 60M and 5.6% lower at 130M. Compared with the Sign configuration of the Scion paper, MuonIO is 6.2% and 8.8% lower.

We did not run SCALE [3] or the symmetry-compatible optimizer [4]. SCALE applies its normalization to every weight matrix, including the hidden layers, and uses momentum only on the LM head, so it replaces the optimizer of the whole network. Our experiments keep the hidden layer optimizer fixed and change only the update of the embedding and the LM head. A comparison with SCALE would not isolate the I/O layers. The second Scion variant tests the normalization of each token's vector, which these methods share with MuonIO on the LM head (Appendix A). Their other components are not part of this comparison. We will add the Scion results to the revised paper.

References

[1] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.

[2] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning, pp. 49069-49104. PMLR, 2025.

[3] Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. Memory-efficient LLM pretraining via minimalist optimizer design. In International Conference on Machine Learning, 2026. arXiv:2506.16659.

[4] Tim Tsz-Kit Lau and Weijie Su. Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers. arXiv preprint arXiv:2605.18106, 2026.

Reviewer_fg7zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper addresses the fact that Muon-style optimizers lack a principled update for the input embedding and output head of a language model and fall back to AdamW for them.
  • It proposes MuonIO, a single update for both layers that normalizes the momentum of each vocabulary vector, justified by how much each layer can change the model's input or output.
  • The method achieves lower optimizer memory than AdamW on these layers and lower validation perplexity than tuned Muon baselines when pretraining small and medium language models.
Critical strengths
  1. The derivation is short and clean, and the update is cheap.
Critical weaknesses
  1. The update rule itself is not new, and the methods that already use it are not compared. The paper acknowledges that prior work such as Scion already includes the same normalization, so the new parts are the justification and a step-size scaling, whose effect is not tested.
Questions for authors
  1. Does the step-size scaling improve over Scion-style normalization alone?
2 comments

We thank the reviewer for the comments. The weakness and the question are both about the comparison with Scion and the effect of the step size scaling, so we answer them together.

W1 / Q1: Comparison with Scion and the effect of the step size scaling

The reviewer is correct that the normalization directions already appear in prior work, and the paper says so. What is new is the step size scaling, and the experiment below tests it.

We added Scion [1] as a baseline in two variants. We name each variant by the updates it uses for the embedding, the hidden layers and the LM head.

  • Scion (Sign → Spectral → Sign). This is the configuration used in the GPT experiments of the Scion paper. Every entry of the embedding table and of the LM head moves by a fixed amount, with the sign determined by its momentum. On the LM head this amount is divided by the hidden dimension dd.
  • Scion (ColNorm → Spectral → RowNorm). This variant uses column normalization for the embedding table and row normalization for the LM head. The momentum vector of each token is normalized to unit length and then multiplied by a fixed factor, which is d\sqrt{d} for the embedding table and 1/d1/\sqrt{d} for the LM head.

The dimension factors are those of Tables 3 and 4 in the Scion paper. In both Scion variants the step size is fixed by the learning rate and these factors, and it does not depend on how large the momentum is. MuonIO does not use a fixed step. After normalizing the momentum vector of each token, it multiplies the step by d\sqrt{d} and by the dual norm of the layer's momentum, which is the sum of the norms of the momentum vectors of all tokens (line 5 of Algorithm 1). The step size then changes during training as the momentum changes.

We tuned both variants with the protocol that Appendix D describes for Muon Tuned and MuonIO. We swept the same learning rate grids for the embedding and the LM head, and kept all other settings as in Appendix D.

Final validation perplexity on C4 (lower is better):

Configuration60M130M
Plain Muon36.99028.510
Scion (Sign → Spectral → Sign)30.26425.931
Scion (ColNorm → Spectral → RowNorm)29.63025.067
Muon Tuned28.51824.297
MuonIO (ours)28.38323.657

We report the mean of the perplexity.

So the answer to Q1 is yes. Apart from constants that the tuned learning rates absorb, the I/O update of the second Scion variant is the MuonIO update without the dual norm scaling. MuonIO reaches 4.2% lower perplexity than this variant at 60M and 5.6% lower at 130M. Compared with the Sign configuration of the Scion paper, MuonIO is 6.2% and 8.8% lower. We will add these results to the revised paper.

References

[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning, pp. 49069-49104. PMLR, 2025.

Thank you for adding the comparison. It answers my question, and it shows that the step-size scaling clearly helps.

However, as far as I can tell, the scaling follows the standard steepest descent step of Bernstein & Newhouse (2024), which the paper itself uses in Proposition 2.1, so the novelty still seems limited to me.

I may be missing something here, given that this is not my main area, so for now I will keep my rating and take the other reviewers’ views into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.