- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LionMuon, which takes one Muon step every few iterations and uses cheaper Lion steps in between.
Both updates share one momentum buffer, reducing the average orthogonalization cost and extra optimizer communication.
The authors provide a convergence analysis to study how the switching period affects the trade-off between optimization progress and computational cost.
Experiments on 124M and 355M language models show lower validation loss than Muon and faster time to reach Muon’s loss in the tested four-GPU setting.
Critical strengths
1, Muon provides useful matrix updates but requires additional computation and communication, while Lion updates are much cheaper. Taking occasional Muon steps is a reasonable way to balance update quality and training cost.
2, The method is relatively straightforward and shares one momentum buffer across both update types. It reduces the frequency of expensive orthogonalization without introducing a separate optimizer state or approximating the synchronized gradient.
3, The authors tune learning rates over the full training horizon and repeat the selected 124M configurations with three seeds. They also test hyperparameter transfer to 355M and measure four-GPU costs on PCIe and NVLink.
Critical weaknesses
1, The method mainly combines existing Muon and Lion updates through periodic switching and shared momentum. Although this provides a useful trade-off, the evaluation only covers 124M and 355M language models, with distributed experiments limited to four GPUs on one node. The current evidence does not establish whether these gains hold for larger models, longer pretraining, or multi-node training.
2, Section 3.2 states that embeddings receive LionMuon updates, but the reported 170 MB update payload and orthogonalization FLOPs appear to cover only the internal transformer matrices. This inconsistency affects the reported computation, communication, and optimizer-state savings.
3, Both FineWeb ratios in Table 2 are below one, which makes the simplified trade-off expression improve toward pure Lion. These numbers, therefore, do not explain the claimed optimum at (P=2) or (P=5).
4, The largest model has 355M parameters, and both the 355M and distributed experiments use one seed per configuration. MuonBP is also emulated under data parallelism rather than tested in its intended sharded setting, so the comparisons do not establish the advantage in large-scale distributed training.
Questions for authors
-
Which parameters actually receive LionMuon updates? Section 3.2 states that embeddings are included, but the reported 170 MB update payload matches approximately 84.9M internal transformer parameters in BF16. Including the 38.6M token-embedding parameters would increase this payload to approximately 247 MB. The reported orthogonalization FLOPs also appear to exclude embeddings. Could the authors clarify the parameter routing in each experiment and reconcile the computation, communication, and optimizer-state accounting?
-
How do the measured constants explain the optimal period? Table 2 reports FineWeb trade-off ratios of approximately 0.24 and 0.98. Substituting these values into the simplified expression in Section 4.3 gives a factor that keeps decreasing as P increases, rather than reaching an optimum at (P=2) or (P=5). Could the authors explain which assumptions or omitted terms account for this difference and clarify the claimed agreement between theory and experiments?
-
Could the authors check the mathematical inconsistencies in Equation (6) and Appendix F? Equation (6) appears to reverse the relationship between the entry-wise norm and the nuclear norm, given the definition of in Appendix C. Appendix F shows that the two momentum formulations differ only by a positive scalar, which sign and matrix-sign updates remove, but then introduces a learning-rate conversion. Why is this conversion needed if the update directions are identical?
-
What supports the claim that LionMuon takes the least time to reach Muon’s loss? Table 5 reports the same normalized PCIe time, 0.70, for LionMuon P=5 and AdamW. On NVLink, the difference is only 0.77 versus 0.79. Could the authors distinguish the improvement over Muon from the comparison with AdamW and provide uncertainty estimates for time to the same loss target? Please also clarify that these values are calculated from measured step times and steps-to-target.
-
What exactly do the reported communication volumes measure? The 496 MB gradient figure appears to represent the FP32 tensor payload rather than physical interconnect traffic. Could the authors specify whether the numbers count collective payloads or transferred bytes, and whether they are per GPU or aggregated across GPUs? For P=5, the extra optimizer payload falls by 80%, while the reported total payload falls by approximately 20%; these two savings should be clearly distinguished.
3 comments
Q3. Eq. (6) and Appendix F. Eq. (6) was printed in the wrong direction. It now reads with , which is the direction the analysis uses and matches Appendix C. It also has a separate ratio for the update matrices. The Appendix F conversion was indeed unnecessary, and we removed it. The robustness difference reported there came from our two implementations, not from the momentum form (see our reply to Reviewer yhoU, W3), and we removed that claim as well.
Q4. "Least time". We removed this claim. LionMuon reaches Muon's loss faster than Muon on both interconnects, and is on par with AdamW. Each time is the number of steps to the target multiplied by the measured step time, as you assumed. Time to Muon's loss (3.415), row relative to column, with 95% bootstrap intervals (Appendix G.7 of the revision):
| vs Muon, PCIe | vs Muon, NVLink | vs AdamW, PCIe | vs AdamW, NVLink | |
|---|---|---|---|---|
| AdamW | 0.69 [0.69, 0.70] | 0.78 [0.74, 0.83] | 1 | 1 |
| LionMuon | 0.75 [0.74, 0.75] | 0.78 [0.76, 0.81] | 1.07 [1.06, 1.08] | 0.96 [0.93, 1.07] |
| LionMuon | 0.69 [0.69, 0.70] | 0.76 [0.69, 0.79] | 1.00 [0.99, 1.01] | 0.99 [0.93, 1.04] |
At equal steps LionMuon ends below AdamW ( against ), so against AdamW the gain is in loss. In time it is on par with AdamW on the two nodes of Table 1 (ratios 1.00 and 0.99), and its optimizer state is about two thirds of AdamW's. Table 6 of the revision prints 0.70 and 0.79 where this table has 0.69 and 0.78, because it uses the mean curve and the rounded step-time ratios of Table 1.
Q5. Communication volumes. We apologize for the confusion. The numbers are payloads: the size of the tensors that one GPU hands to the all-reduce calls of one step. They are not bytes on the wire, and they are not summed over GPUs. AdamW, Muon and LionMuon all-reduce their gradients, 496 MB in fp32, as you inferred. Muon also all-reduces its updates, 171 MB in bf16, on every step, and LionMuon does so only on every -th step. So at the optimizer's own payload falls by 80% (171 to 34 MB per step), and the total falls by 20% (667 to 530 MB). The revision defines the quantity and reports both savings separately.
We hope this addresses your concerns, and we are happy to answer any further questions.
Following up on W3 and Q2, we measured the constants of the bound for other periods, to check how they depend on . Each run trains the 124M model for 10,000 steps with the recipe of Appendix C, and only changes. Each value is the mean over the run (one seed per cell), and and use the gradient ratio measured in the same run. The -dependent operations-ratio factor becomes
where and .
| FineWeb | ||||
|---|---|---|---|---|
| LionMuon | 0.77 | 848 | 0.16 | 1.08 |
| LionMuon | 1.42 | 1,535 | 0.44 | 1.37 |
| LionMuon | 2.19 | 3,693 | 0.98 | 1.25 |
| Lion () | 2.24 | 11,347 | 2.31 | 1.00 |
| WikiText-103 | ||||
|---|---|---|---|---|
| LionMuon | 0.86 | 875 | 0.15 | 1.04 |
| LionMuon | 2.45 | 2,265 | 0.46 | 1.11 |
| LionMuon | 6.41 | 6,998 | 1.63 | 1.21 |
| Lion () | 14.23 | 55,565 | 9.61 | 0.99 |
The smoothness constants and grow with the period , and for long periods grows faster. Between our LionMuon () and pure Lion (), the main determining ratio grows by a factor of about 5 on FineWeb and about 21 on WikiText-103. The noise ratio stays between 1.0 and 1.4, and the gradient ratio between 68 and 83.
This is the dependence on to which our reply to W3 refers: our previous bound, which treats the constants as independent of , is optimistic for long periods. In the same runs, pure Lion has the highest loss on both datasets, while and have the lowest. Conditioning on now modulates this behaviour.
We will add these tables to Appendix C with all implementation details.
We thank the reviewer for the careful reading. Your accounting of the parameters and the payload was correct, and it exposed an error in our text.
W1 and W4. Novelty, scale and seeds. Both updates are known. What we contribute is their combination in time, with one shared buffer and a bound that covers the pair under heavy-tailed noise. The closest method, MuonBP (Khaled et al., ICLR 2026), also takes a full Muon step every P-th iteration, but it orthogonalizes blocks in between, so it runs Newton-Schulz on every step. LionMuon's in-between step is a plain Lion step, with no Newton-Schulz and, under replicated data parallelism, no optimizer communication. At equal exposed communication (34 MB per step, P=5, 4 GPUs) LionMuon reaches 3.3757±0.0010 against 3.4114±0.0035 for MuonBP, emulated under data parallelism. Muon, at 171 MB per step, reaches 3.4150±0.0028 (mean ± standard deviation over three seeds).
On scale: the 355M and 4-GPU results now have three seeds, and the gaps to Muon (0.018 and 0.042) are well outside the seed standard deviations (at most 0.003). At 355M the rates are copied from 124M, and one ladder step higher the two methods are level there (see our reply to Reviewer yhoU, Q2). The longest runs are 2.46B tokens at 124M, about 1× Chinchilla, and 8.2B tokens at 355M. Scale beyond 355M is limited by compute (see the general response), and we claim nothing for larger models, multi-node training or MuonBP's tensor-parallel setting.
W2 and Q1. Embedding, head, and the 170 MB. Your accounting is correct, and we apologize for the inconsistency. The sentence in Section 3.2 was wrong. The Muon implementation of the benchmark codebase we build on gives a 2-D parameter the matrix update only if it has fewer than 10,000 rows, and we overlooked this rule when writing the section. The token embedding has 50,304 rows and is tied with the head, so AdamW updates it under every method. The 85.3M parameters that take the matrix update (84.9M in block matrices, 0.4M positional) give 170.66 MB in bf16. We had truncated this to 170, and the revision prints 171. With the embedding it would be 248 MB, close to your 247. The payload and the Newton-Schulz cost in the paper (1.53×1012 FLOPs per step) were computed with the correct routing, so the fix changes neither. The optimizer state is half of AdamW's per updated matrix and about two thirds in total.
W3 and Q2. Measured ratios and the optimal P. To address this weakness, we improved our old trade-off factor ϕ. First, we separated the gradient ratio αG, the update ratio αU and the learning-rate ratio α. This allowed us to obtain a combination of the old smoothness ratio rL:=α2L2L∞ with αU/α. As you mentioned, our new factor ϕ(P) still exhibits concave-like behaviour and has minima at the endpoints, which are either pure Muon or Lion. However, our empirical findings are that LionMuon ends below both Muon and Lion, with a flat optimum between P=2 and P=5. We believe this happens because the smoothness and noise constants, along with the norm ratios, implicitly depend on P, and the formula (7) of the revision holds true only on the small range above 1, where LionMuon's iterates are the most similar to Muon's ones. When the Lion operations dominate, the parameters and the update-matrix landscape might change dramatically, causing large constants, as we noticed while finding the optimal α≈αG, which grows with P.
We retained the narrower statement in the revision that every P has the same rate in ε as Muon and Lion.