acceptodds
Under review as a conference paper at ICLR 2027

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

27 views

Abstract

Pretraining a language model takes enormous compute, and the right optimizer can save a good part of it. Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton–Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every P iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per P steps, and the optimizer state is half of AdamW's. A single-EMA variant, SignMuon, already improves on Muon. We prove complexity bounds under heavy-tailed noise in which the period sets an interpolation between Muon's and Lion's smoothness and noise constants, and which say when LionMuon is faster than both. On 124M and 355M models trained on FineWeb, LionMuon with P=2 and P=5 reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training it reaches Muon's final loss with a third less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient.

Primary area:
optimization
Keywords:
large language model pretraining, optimization, Muon, sign-based optimizers, communication-efficient distributed training, convergence analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qUnoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LionMuon, which takes one Muon step every few iterations and uses cheaper Lion steps in between.

Both updates share one momentum buffer, reducing the average orthogonalization cost and extra optimizer communication.

The authors provide a convergence analysis to study how the switching period affects the trade-off between optimization progress and computational cost.

Experiments on 124M and 355M language models show lower validation loss than Muon and faster time to reach Muon’s loss in the tested four-GPU setting.

Critical strengths

1, Muon provides useful matrix updates but requires additional computation and communication, while Lion updates are much cheaper. Taking occasional Muon steps is a reasonable way to balance update quality and training cost.

2, The method is relatively straightforward and shares one momentum buffer across both update types. It reduces the frequency of expensive orthogonalization without introducing a separate optimizer state or approximating the synchronized gradient.

3, The authors tune learning rates over the full training horizon and repeat the selected 124M configurations with three seeds. They also test hyperparameter transfer to 355M and measure four-GPU costs on PCIe and NVLink.

Critical weaknesses

1, The method mainly combines existing Muon and Lion updates through periodic switching and shared momentum. Although this provides a useful trade-off, the evaluation only covers 124M and 355M language models, with distributed experiments limited to four GPUs on one node. The current evidence does not establish whether these gains hold for larger models, longer pretraining, or multi-node training.

2, Section 3.2 states that embeddings receive LionMuon updates, but the reported 170 MB update payload and orthogonalization FLOPs appear to cover only the internal transformer matrices. This inconsistency affects the reported computation, communication, and optimizer-state savings.

3, Both FineWeb ratios in Table 2 are below one, which makes the simplified trade-off expression improve toward pure Lion. These numbers, therefore, do not explain the claimed optimum at (P=2) or (P=5).

4, The largest model has 355M parameters, and both the 355M and distributed experiments use one seed per configuration. MuonBP is also emulated under data parallelism rather than tested in its intended sharded setting, so the comparisons do not establish the advantage in large-scale distributed training.

Questions for authors
  1. Which parameters actually receive LionMuon updates? Section 3.2 states that embeddings are included, but the reported 170 MB update payload matches approximately 84.9M internal transformer parameters in BF16. Including the 38.6M token-embedding parameters would increase this payload to approximately 247 MB. The reported orthogonalization FLOPs also appear to exclude embeddings. Could the authors clarify the parameter routing in each experiment and reconcile the computation, communication, and optimizer-state accounting?

  2. How do the measured constants explain the optimal period? Table 2 reports FineWeb trade-off ratios of approximately 0.24 and 0.98. Substituting these values into the simplified expression in Section 4.3 gives a factor that keeps decreasing as P increases, rather than reaching an optimum at (P=2) or (P=5). Could the authors explain which assumptions or omitted terms account for this difference and clarify the claimed agreement between theory and experiments?

  3. Could the authors check the mathematical inconsistencies in Equation (6) and Appendix F? Equation (6) appears to reverse the relationship between the entry-wise L1L_1 norm and the nuclear norm, given the definition of α\alpha in Appendix C. Appendix F shows that the two momentum formulations differ only by a positive scalar, which sign and matrix-sign updates remove, but then introduces a learning-rate conversion. Why is this conversion needed if the update directions are identical?

  4. What supports the claim that LionMuon takes the least time to reach Muon’s loss? Table 5 reports the same normalized PCIe time, 0.70, for LionMuon P=5 and AdamW. On NVLink, the difference is only 0.77 versus 0.79. Could the authors distinguish the improvement over Muon from the comparison with AdamW and provide uncertainty estimates for time to the same loss target? Please also clarify that these values are calculated from measured step times and steps-to-target.

  5. What exactly do the reported communication volumes measure? The 496 MB gradient figure appears to represent the FP32 tensor payload rather than physical interconnect traffic. Could the authors specify whether the numbers count collective payloads or transferred bytes, and whether they are per GPU or aggregated across GPUs? For P=5, the extra optimizer payload falls by 80%, while the reported total payload falls by approximately 20%; these two savings should be clearly distinguished.

3 comments

We thank the reviewer for the careful reading. Your accounting of the parameters and the payload was correct, and it exposed an error in our text.

W1 and W4. Novelty, scale and seeds. Both updates are known. What we contribute is their combination in time, with one shared buffer and a bound that covers the pair under heavy-tailed noise. The closest method, MuonBP (Khaled et al., ICLR 2026), also takes a full Muon step every PP-th iteration, but it orthogonalizes blocks in between, so it runs Newton-Schulz on every step. LionMuon's in-between step is a plain Lion step, with no Newton-Schulz and, under replicated data parallelism, no optimizer communication. At equal exposed communication (34 MB per step, P=5P=5, 4 GPUs) LionMuon reaches 3.3757±0.00103.3757 \pm 0.0010 against 3.4114±0.00353.4114 \pm 0.0035 for MuonBP, emulated under data parallelism. Muon, at 171 MB per step, reaches 3.4150±0.00283.4150 \pm 0.0028 (mean ±\pm standard deviation over three seeds).

On scale: the 355M and 4-GPU results now have three seeds, and the gaps to Muon (0.018 and 0.042) are well outside the seed standard deviations (at most 0.003). At 355M the rates are copied from 124M, and one ladder step higher the two methods are level there (see our reply to Reviewer yhoU, Q2). The longest runs are 2.46B tokens at 124M, about 1×1\times Chinchilla, and 8.2B tokens at 355M. Scale beyond 355M is limited by compute (see the general response), and we claim nothing for larger models, multi-node training or MuonBP's tensor-parallel setting.

W2 and Q1. Embedding, head, and the 170 MB. Your accounting is correct, and we apologize for the inconsistency. The sentence in Section 3.2 was wrong. The Muon implementation of the benchmark codebase we build on gives a 2-D parameter the matrix update only if it has fewer than 10,000 rows, and we overlooked this rule when writing the section. The token embedding has 50,304 rows and is tied with the head, so AdamW updates it under every method. The 85.3M parameters that take the matrix update (84.9M in block matrices, 0.4M positional) give 170.66 MB in bf16. We had truncated this to 170, and the revision prints 171. With the embedding it would be 248 MB, close to your 247. The payload and the Newton-Schulz cost in the paper (1.53×10121.53\times10^{12} FLOPs per step) were computed with the correct routing, so the fix changes neither. The optimizer state is half of AdamW's per updated matrix and about two thirds in total.

W3 and Q2. Measured ratios and the optimal PP. To address this weakness, we improved our old trade-off factor ϕ\phi. First, we separated the gradient ratio αG\alpha_G, the update ratio αU\alpha_U and the learning-rate ratio α\alpha. This allowed us to obtain a combination of the old smoothness ratio rL:=L∞α2L2r_L := \frac{L_\infty}{\alpha^2 L_2} with αU/α\alpha_U/\alpha. As you mentioned, our new factor ϕ(P)\phi(P) still exhibits concave-like behaviour and has minima at the endpoints, which are either pure Muon or Lion. However, our empirical findings are that LionMuon ends below both Muon and Lion, with a flat optimum between P=2P=2 and P=5P=5. We believe this happens because the smoothness and noise constants, along with the norm ratios, implicitly depend on PP, and the formula (7) of the revision holds true only on the small range above 1, where LionMuon's iterates are the most similar to Muon's ones. When the Lion operations dominate, the parameters and the update-matrix landscape might change dramatically, causing large constants, as we noticed while finding the optimal α≈αG\alpha \approx \alpha_G, which grows with PP.

We retained the narrower statement in the revision that every PP has the same rate in ε\varepsilon as Muon and Lion.

Q3. Eq. (6) and Appendix F. Eq. (6) was printed in the wrong direction. It now reads ∥∇f∥1≈αG∥∇f∥nuc\lVert\nabla f\rVert_1 \approx \alpha_G \lVert\nabla f\rVert_{\mathrm{nuc}} with 1≤αG≤mn1 \le \alpha_G \le \sqrt{mn}, which is the direction the analysis uses and matches Appendix C. It also has a separate ratio αU\alpha_U for the update matrices. The Appendix F conversion was indeed unnecessary, and we removed it. The robustness difference reported there came from our two implementations, not from the momentum form (see our reply to Reviewer yhoU, W3), and we removed that claim as well.

Q4. "Least time". We removed this claim. LionMuon reaches Muon's loss faster than Muon on both interconnects, and P=5P=5 is on par with AdamW. Each time is the number of steps to the target multiplied by the measured step time, as you assumed. Time to Muon's loss (3.415), row relative to column, with 95% bootstrap intervals (Appendix G.7 of the revision):

vs Muon, PCIevs Muon, NVLinkvs AdamW, PCIevs AdamW, NVLink
AdamW0.69 [0.69, 0.70]0.78 [0.74, 0.83]11
LionMuon P=2P=20.75 [0.74, 0.75]0.78 [0.76, 0.81]1.07 [1.06, 1.08]0.96 [0.93, 1.07]
LionMuon P=5P=50.69 [0.69, 0.70]0.76 [0.69, 0.79]1.00 [0.99, 1.01]0.99 [0.93, 1.04]

At equal steps LionMuon P=5P=5 ends below AdamW (3.3757±0.00103.3757 \pm 0.0010 against 3.4052±0.00163.4052 \pm 0.0016), so against AdamW the gain is in loss. In time it is on par with AdamW on the two nodes of Table 1 (ratios 1.00 and 0.99), and its optimizer state is about two thirds of AdamW's. Table 6 of the revision prints 0.70 and 0.79 where this table has 0.69 and 0.78, because it uses the mean curve and the rounded step-time ratios of Table 1.

Q5. Communication volumes. We apologize for the confusion. The numbers are payloads: the size of the tensors that one GPU hands to the all-reduce calls of one step. They are not bytes on the wire, and they are not summed over GPUs. AdamW, Muon and LionMuon all-reduce their gradients, 496 MB in fp32, as you inferred. Muon also all-reduces its updates, 171 MB in bf16, on every step, and LionMuon does so only on every PP-th step. So at P=5P=5 the optimizer's own payload falls by 80% (171 to 34 MB per step), and the total falls by 20% (667 to 530 MB). The revision defines the quantity and reports both savings separately.

We hope this addresses your concerns, and we are happy to answer any further questions.

Following up on W3 and Q2, we measured the constants of the bound for other periods, to check how they depend on PP. Each run trains the 124M model for 10,000 steps with the recipe of Appendix C, and only PP changes. Each value is the mean over the run (one seed per cell), and rLr_L and rρr_\rho use the gradient ratio measured in the same run. The PP-dependent operations-ratio factor ϕ(P):=N(P)/NMuon\phi(P) := N(P)/N_{\text{Muon}} becomes

ϕ(P)=(1P+(1−1P)1KNS)(αUαL2(P)L2(1)1P+(1−1P)rL(P))(1Pρnuc(P)ρnuc(1)+(1−1P)rρ(P))κκ−1,P>1,\phi(P) = \Bigl(\tfrac1P + \bigl(1-\tfrac1P\bigr)\tfrac{1}{K_\text{NS}}\Bigr)\Bigl(\tfrac{\alpha_U}{\alpha}\frac{L_2(P)}{L_2(1)}\tfrac1P + \bigl(1-\tfrac1P\bigr) r_L(P)\Bigr)\Bigl(\tfrac1P \frac{\rho_{\text{nuc}}(P)}{\rho_{\text{nuc}}(1)} + \bigl(1-\tfrac1P\bigr) r_\rho(P)\Bigr)^{\frac{\kappa}{\kappa-1}}, \quad P > 1,

where rL(P):=L∞(P)α2L2(1)r_L(P) := \frac{L_\infty(P)}{\alpha^2 L_2(1)} and rρ(P):=ρ1(P)αρnuc(1)r_\rho(P) := \frac{\rho_1(P)}{\alpha \rho_{\text{nuc}}(1)}.

FineWebL2(P)L_2(P)L∞(P)L_\infty(P)rL(P)r_L(P)rρ(P)r_\rho(P)
LionMuon P=1P=10.778480.161.08
LionMuon P=2P=21.421,5350.441.37
LionMuon P=5P=52.193,6930.981.25
Lion (P=∞P=\infty)2.2411,3472.311.00
WikiText-103L2(P)L_2(P)L∞(P)L_\infty(P)rL(P)r_L(P)rρ(P)r_\rho(P)
LionMuon P=1P=10.868750.151.04
LionMuon P=2P=22.452,2650.461.11
LionMuon P=5P=56.416,9981.631.21
Lion (P=∞P=\infty)14.2355,5659.610.99

The smoothness constants L2(P)L_2(P) and L∞(P)L_\infty(P) grow with the period PP, and for long periods L∞(P)L_\infty(P) grows faster. Between our LionMuon (P=2P=2) and pure Lion (P=∞P=\infty), the main determining ratio rL(P)r_L(P) grows by a factor of about 5 on FineWeb and about 21 on WikiText-103. The noise ratio rρ(P)r_\rho(P) stays between 1.0 and 1.4, and the gradient ratio α\alpha between 68 and 83.

This is the dependence on PP to which our reply to W3 refers: our previous bound, which treats the constants as independent of PP, is optimistic for long periods. In the same runs, pure Lion has the highest loss on both datasets, while P=2P=2 and P=5P=5 have the lowest. Conditioning on PP now modulates this behaviour.

We will add these tables to Appendix C with all implementation details.

Reviewer_yhoUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SignMuon and LionMuon, which alternate one matrix-sign/Muon update every period P with elementwise-sign updates in between. LionMuon uses Lion's two EMA timescales while retaining one momentum buffer; SignMuon uses a single EMA. The analysis combines spectral- and entrywise-infinity-norm linear minimization oracles under bounded heavy-tailed gradient noise. Empirical evaluation uses 124M GPT-2-style models on FineWeb and WikiText-103, a 355M FineWeb transfer experiment, and four-GPU data-parallel runs. Intermediate periods improve validation loss over the evaluated endpoints, while amortizing Newton–Schulz work and, in the chosen implementation, optimizer-specific communication.

Critical strengths
  1. This is a simple, implementable algorithm that shows competitive performance.
  2. The paper is easy to follow.
  3. The analysis covers both alternating update types under bounded-moment gradient noise.
Critical weaknesses
  1. The practical significance of the reported efficiency gains under an optimized training setup remains unclear. First, with whole-matrix optimizer-state sharding, each owner rank can receive the aggregated gradient, perform NS locally, and synchronize the updated parameters. This avoids redundant NS computation and need not add a separate synchronization round relative to AdamW using the same sharding layout. Second, table 1 reports a 19% per-step time reduction on PCIe for LionMuon P=5, with only 4,096 tokens per GPU per optimizer update. If we increase the batch token, the computation for fwd/bwd goes up while the part for NS remains the same. In my experience with optimized training systems, the optimizer step commonly consumes around or less than 5% of the total iteration runtime in pretraining (depending on model size/context length etc) under proper infra, and even much less in RL due to the long inference time. So I wonder if the authors could provide a runtime breakdown and evaluate larger effective batch sizes/model sizes, including gradient accumulation.
  2. The method introduces extra hyperparameters and the results in Appendix E seem to imply they require joint careful tuning.
  3. Parts of the theoretical interpretation exceed the guarantees actually shown. Appendix B bounds the signed Frank–Wolfe gap over the smaller spectral ball while allowing iterates outside that ball; page 22 explicitly acknowledges that this gap can be negative. An upper bound on a quantity that can already be negative does not certify approximate KKT conditions, so the statement on page 21 that the method preserves fast convergence to Muon inner KKT points is not established by this bound alone. Appendix F also gives an incorrect learning-rate equivalence: under the stated scale-invariant sign/msign updates, proportional HB and EMA momentum buffers produce identical directions and require the same learning rate. The claimed robustness difference therefore needs another explanation.
Questions for authors
  1. In Figure 2, LionMuon P=1 already improves over Muon without alternating update types. How much of the gain comes from the dual-EMA momentum, and does the advantage of alternation persist against a well-tuned Nesterov Muon baseline?
  2. SignMuon P=2 outperforms LionMuon in the 355M experiment, whereas LionMuon performs better in the 124M experiments. Does this reversal persist after retuning the learning rates and repeating the runs with multiple seeds?
  3. Please clarify the optimizer used for embeddings and the output head in each baseline, including whether these weights are tied. Are parameter groups matched across methods? If not, please provide a controlled comparison to isolate the effect of the update rule.
  4. Table 5 shows that AdamW reaches the target loss faster than Muon. In contrast, Jordan et al. (2024, Results, Fig. 2) report better wall-clock efficiency for Muon than AdamW when training a 124M transformer on FineWeb. Although the detailed training configurations differ, the model scale and dataset are closely related. Can the authors explain this reversal, particularly the roles of effective batch size, the learning-rate schedule, Nesterov momentum, and parameter-group configuration?

Jordan et al. (2024). Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/

4 comments

We thank the reviewer for the detailed and technical review. The questions on Nesterov Muon and the runtime breakdown led to new experiments, which we report below.

W1. Efficiency under sharding and larger batches. We agree with both points and have scoped the time claim to replicated optimizer state and its batch (4,096 tokens per GPU) in the abstract, Section 5.3 and the conclusion. Our runs are data parallel (PyTorch DDP): every GPU holds a full copy of the model and of the optimizer state. Muon already runs Newton-Schulz (NS) once per matrix, on one GPU, as you describe, and then all-reduces the updates on every step; LionMuon does so on every PP-th step. With whole-matrix sharded state, an all-gather replaces the update all-reduce, so the communication saving disappears. In any layout, NS still runs only every PP-th step, and LionMuon still reaches a lower loss than Muon. Below, we provide the breakdown (Appendix G.6 of the revision; 124M, 4 GPUs, ms per step on the slowest rank, and the optimizer's share of the step without and with ×16\times 16 gradient accumulation):

optimizer stepNSall-reduceshare, ×1\times 1share, ×16\times 16
AdamW1.3002.6%0.6%
Muon24.46.014.032.7%8.9%
LionMuon P=2P=214.23.07.022.0%5.4%
LionMuon P=5P=58.01.22.813.8%3.1%

The all-reduce and NS make up most of Muon's overhead, and LionMuon divides both by PP. With ×16\times 16 accumulation, Muon's optimizer step is 8.9% of the step and LionMuon P=5P=5 brings it to 3.1%. Without the update all-reduce, as under whole-matrix sharding, Muon's share would be about 4%, in line with your 5%. So at such batches the time saving is small (about 6% of the step for P=5P=5, and about 2% without the all-reduce), and the main benefit of LionMuon is its lower loss at a fraction of Muon's NS cost. The breakdown is instrumented, so we take the step-time ratios from uninstrumented runs on the same node: 0.81 for P=5P=5 and 0.88 for P=2P=2, within the range of Table 1. By FLOPs, NS is about 12% of the forward and backward pass at 124M with our batch and under 1% at 355M with the 524k-token batch.

Published measurements show the same dependence on batch and layout. At very large batches Muon's overhead is small, as you say: 1% to 3% of the forward-backward time in Liu et al. (2025), and in NorMuon (Li et al., ICML 2026) Muon adds 2.8% to the step time of a 5.4B model at 2M tokens per step, or 8.1% without distributed orthogonalization. At smaller per-device batches it is larger: Essential AI (2025) measure Muon at 10% of the step with layer sharding and 43% with replicated computation (2B model, 24k tokens per device). Under tensor parallelism the cost is communication: Khaled et al. (ICLR 2026) report that Muon loses 5% to 10% of throughput against AdamW, and their MuonBP gains 8% over Muon at 8B by running the full orthogonalization only periodically. So running the expensive step only periodically has a precedent in optimized systems.

W2. Hyperparameter complexity. In practice LionMuon needs one learning-rate sweep, not a joint grid. The betas were tuned once and reused in every experiment. The best sign-step rate ηL\eta_L is the same for every PP (10−410^{-4} on FineWeb, about 3×10−43\times10^{-4} on WikiText-103), so it is found once per task. It also carried over to the Qwen3-style model: we kept ηL=10−4\eta_L = 10^{-4}, swept only ηM\eta_M, and LionMuon still had the lowest loss. A check there at P=2P=2 (seed 0) gives 3.458 with a three times smaller ηL\eta_L, 3.458 with ours and 3.463 with a three times larger one. This leaves the Muon rate ηM\eta_M. For P=2P=2, 5 and 20, one ladder step below its best value, at the same ηL\eta_L, costs 0.013 to 0.024 on FineWeb (Muon: 0.068) and 0.024 to 0.037 on WikiText-103 (Muon: 0.046). The step above was usually stopped early by our pruning rule. At 355M we copied every rate from 124M, and LionMuon still ended below Muon.

W3. Theory. We agree with both points and have corrected them. For 1<P<∞1 < P < \infty the iterates can leave the spectral ball, the Muon Frank-Wolfe gap can then be negative, and the Appendix B bound does not certify approximate Muon KKT points; however, for always-positive Lion gaps, minimization still happens. We removed this claim on page 21. Here, we also correct our "still informative" remark from page 22. We propose a new metric that gives KKT guarantees on the iterates inside the Muon Ball, and can be still informative in between the balls.

Indeed, the original minimal gap does not provide useful information once it becomes negative. But we can derive a more subtle bound for the gap averaged over the period PP, similar to the no-weight-decay case. Instead of the minimal Muon gap E[λ⋅GB∥⋅∥2(1/λ)(Wt)]\mathbb{E}[\lambda \cdot \mathcal G_{B_{\Vert\cdot\Vert_2}(1/\lambda)}(W_t)], which can be negative, we bound the following metric (averaged over the whole run, not the minimal one): ηMηL⋅E[λ⋅GB∥⋅∥2(1/λ)(Wi⋅P)]+∑j=1P−1E[λ⋅GB∥⋅∥∞(1/λ)(Wi⋅P+j)]ηMηL+(P−1)\frac{\frac{\eta_M}{\eta_L} \cdot \mathbb{E}[\lambda \cdot \mathcal G_{B_{\Vert\cdot\Vert_2}(1/\lambda)}(W_{i\cdot P})] + \sum_{j=1}^{P-1} \mathbb{E}[\lambda \cdot \mathcal G_{B_{\Vert\cdot\Vert_\infty}(1/\lambda)}(W_{i\cdot P + j})]}{\frac{\eta_M}{\eta_L} + (P-1)} Following the gap estimate (1−λ∥W∥)∥∇f(W)∥∗≤λ⋅GB∥⋅∥(1/λ)(W)≤(1+λ∥W∥)∥∇f(W)∥∗,(1-\lambda\Vert W\Vert) \Vert\nabla f(W)\Vert_\ast \le \lambda \cdot \mathcal G_{B_{\Vert\cdot\Vert}(1/\lambda)}(W) \le (1+\lambda\Vert W\Vert) \Vert\nabla f(W)\Vert_\ast, we can bring both the Lion and Muon gaps to the same order of magnitude using a suitable learning-rate scaling ηMηL\frac{\eta_M}{\eta_L}. The Lion gap always remains non-negative since ∥W∥∞≤1λ\Vert W\Vert_\infty \le \frac{1}{\lambda}, but the Muon gap can become negative for ∥W∥2>1λ\Vert W\Vert_2 > \frac{1}{\lambda}. When LionMuon bounds this metric from above by 00, it actually yields guarantees for two types of iterates near the end of the run. First, for its iterates inside the Muon ball, both gaps are positive and tend to zero, and the method finds KKT points that lie inside this ball. Second, for iterates between the Lion and Muon balls, the Muon gap can be negative, but the positive Lion gaps still have a large impact on the sum, so the bound keeps them small.

In Appendix F, the heavy-ball and EMA buffers differ only by the factor 1−β1-\beta, which sign and matrix-sign updates ignore, so the two give the same iterates at the same learning rate. We stated that the iterates match in the revision. The robustness difference we reported came from the two implementations, not from the momentum form: the early heavy-ball one also used Nesterov lookahead and kept the sign-step rate constant instead of following the schedule. We removed that claim.

Q1. Dual EMA, alternation, and Nesterov Muon. Both lower the loss on FineWeb and on 4 GPUs, where together they give the largest gain. On WikiText-103 the gain comes from the dual EMA, and at 355M with rates copied from 124M alternation alone does best. Best validation loss (mean ±\pm standard deviation over three seeds, Appendix G.5 of the revision):

Muondual EMA only (LionMuon P=1P=1)alternation only (SignMuon P=2P=2)both (LionMuon P=2P=2)
FineWeb 124M3.5271±0.00063.5271 \pm 0.00063.5096±0.00233.5096 \pm 0.00233.5094±0.00173.5094 \pm 0.00173.4984±0.00203.4984 \pm 0.0020
WikiText-1032.8354±0.00202.8354 \pm 0.00202.8255±0.00062.8255 \pm 0.00062.8395±0.00592.8395 \pm 0.00592.8275±0.00122.8275 \pm 0.0012
FineWeb 355M3.0377±0.00193.0377 \pm 0.00193.0239±0.00143.0239 \pm 0.00143.0086±0.00093.0086 \pm 0.00093.0199±0.00033.0199 \pm 0.0003
4 GPUs3.4150±0.00283.4150 \pm 0.00283.3916±0.00103.3916 \pm 0.00103.3935±0.00143.3935 \pm 0.00143.3733±0.00223.3733 \pm 0.0022

Against a Nesterov Muon with tuned momentum, LionMuon P=2P=2 is ahead on FineWeb and level on WikiText-103, at half the NS cost (Section 5.4 and Appendix G.2 of the revision):

Nesterov 0.9Nesterov 0.95Nesterov 0.98Nesterov 0.99LionMuon P=2P=2
FineWeb3.53993.5170±0.00183.5170 \pm 0.00183.50663.5047±0.00143.5047 \pm 0.00143.4988±0.00153.4988 \pm 0.0015
WikiText-1032.84552.8313±0.00332.8313 \pm 0.00332.83672.86512.8302±0.00032.8302 \pm 0.0003

Cells without ±\pm are one seed. Each row was run on one machine, with LionMuon rerun there, so its mean differs slightly from the table above. On FineWeb LionMuon is 0.006 below the best Nesterov Muon, about four seed standard deviations, and on WikiText-103 the two are level. The best momentum lies inside the sweep on both datasets (on FineWeb, momentum 0.995 gives 3.5162; one seed, on another machine). Nesterov Muon with momentum μ\mu is LionMuon with P=1P=1 and (β1,β2)=(μ2,μ)(\beta_1,\beta_2)=(\mu^2,\mu), so its gain over plain Muon (about 0.01 to 0.02 on FineWeb for momentum 0.95 to 0.99) is the same dual-EMA effect that LionMuon already uses. We also swept LionMuon's β1\beta_1: 0.95 and 0.98 give 3.504 and 3.513 against 3.501 at the paper's 0.9 (seed 0 each), so the paper's betas stand. We also ran Nesterov Muon in the two other settings of the paper:

LionMuon P=2P=2Nesterov MuonAdamWMuon
150k steps, batch and schedule of the 4-GPU runs3.38733.3958 (0.99)3.41823.4273
355M, paper's recipe3.0199±0.00033.0199 \pm 0.00033.0234 (0.99)3.0681±0.00943.0681 \pm 0.00943.0377±0.00193.0377 \pm 0.0019

The 150k-step runs (Appendix G.3 of the revision) share one setup: one GPU, one seed each, and another FineWeb sample than the paper's 4-GPU runs. At 355M, Nesterov Muon is one seed on another node, at the better of two momenta (0.95 gives 3.025), so the 355M row is indicative only and is not in the paper.

Q2. SignMuon and LionMuon at 355M. With three seeds the gap persists at the copied rates (SignMuon 3.0086±0.00093.0086 \pm 0.0009, LionMuon 3.0199±0.00033.0199 \pm 0.0003, Muon 3.0377±0.00193.0377 \pm 0.0019). It points to the rate copied from 124M and not to the method: at SignMuon's learning rate, one ladder step up, LionMuon P=2P=2 reaches 3.0021 and 3.0029 (two seeds, on another node). Muon also gains from the higher rate there (3.0034 and 3.0017), so at this rate LionMuon and Muon are level at 355M, with LionMuon running Newton-Schulz only every second step (Appendix G.3 of the revision).

Q3. Embedding, head and parameter groups. We apologize for the confusion: the sentence in Section 3.2 was wrong, and we corrected it. The Muon implementation of the benchmark codebase we build on gives the matrix update only to 2-D parameters with fewer than 10,000 rows, and we overlooked this rule when writing the section. In the code and in every reported number, AdamW updates the token embedding (tied with the output head) and the norm gains under every method, at peak rate 10−310^{-3} with betas (0.8, 0.999). Only the block matrices and the positional embedding take the matrix update, so the groups are the same across Muon, SignMuon, LionMuon, Lion, Signum, Dion and MuonBP. The AdamW baseline is plain AdamW on all parameters (weight decay 0.1 on the block matrices, none on the embeddings and the norm gains). Under the matrix methods the AdamW group has weight decay 0.1. Section 3.2 now has a table of the groups.

Q4. AdamW against Muon. It is mostly step cost at our small batch. AdamW needs 0.93 of Muon's steps to reach Muon's loss, but only 0.69 of its time on PCIe. By the Tm/BTm/B estimate of Jordan et al., Muon's FLOP overhead is 0.7% at their batch and 23% at ours (we count 12%). In loss, Muon leads AdamW at 64k steps (3.5271±0.00063.5271 \pm 0.0006 against 3.5534±0.00963.5534 \pm 0.0096) and at 355M with the speedrun's batch of 524k tokens (3.0377±0.00193.0377 \pm 0.0019 against 3.0681±0.00943.0681 \pm 0.0094). It trails only at 150k steps (3.4150±0.00283.4150 \pm 0.0028 against 3.4052±0.00163.4052 \pm 0.0016 on 4 GPUs). Nesterov momentum closes that gap: at 150k steps Nesterov Muon reaches 3.396 against AdamW's 3.418 (one seed each, same setup), and LionMuon P=2P=2 reaches 3.387 (Q1). The parameter groups are as in Q3, and all methods share the same warmup and cosine schedule.

We hope this addresses your concerns, and we are happy to answer any further questions.

Reviewer_j35fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LionMuon, which alternates one Muon-style spectral update every P iterations with Lion sign updates. The paper develops convergence bounds under bounded heavy-tailed noise and evaluates GPT-2-style models at 124M and 355M parameters, including four-GPU communication and step-time measurements. The reported results favor P=2 or P=5 on validation loss and show lower cost than the tested Muon implementation.

Critical strengths

The idea of alternating spectral and sign descent seems interesting and effective in some cases.

Critical weaknesses

The paper showed P = 2 and P = 5 reached a lower loss than Muon, AdamW, etc. How about the other P values? It would be good to discuss generalization. Also, the paper was tested on GPT-2 style - how about newer, larger models?

Questions for authors

How general are your results and claims, e.g., on P != 2 or 5, and larger models (beyond GPT-2 style)?

1 comment

We thank the reviewer for the feedback and are glad that the idea of alternating spectral and sign descent was found interesting. We address the questions below.

Other values of PP. We added P=3P=3 and P=10P=10 at 124M (Appendix G.1 of the revision), each with its own learning-rate sweep. With ηM\eta_M tuned for each PP, the loss is flat for PP between 2 and 5 and rises beyond that, so the period itself needs no careful tuning. Best validation loss (mean ±\pm standard deviation over three seeds):

FineWebWikiText-103
Muon3.5271±0.00063.5271 \pm 0.00062.8354±0.00202.8354 \pm 0.0020
LionMuon P=1P=13.5096±0.00233.5096 \pm 0.00232.8255±0.00062.8255 \pm 0.0006
LionMuon P=2P=23.4984±0.00203.4984 \pm 0.00202.8275±0.00122.8275 \pm 0.0012
LionMuon P=3P=33.4969±0.00213.4969 \pm 0.00212.8269±0.00442.8269 \pm 0.0044
LionMuon P=5P=53.4983±0.00183.4983 \pm 0.00182.8251±0.00102.8251 \pm 0.0010
LionMuon P=10P=103.5050±0.00323.5050 \pm 0.00322.8386±0.00202.8386 \pm 0.0020
LionMuon P=20P=203.5103±0.00093.5103 \pm 0.00092.8404±0.00142.8404 \pm 0.0014

The rows for P=3P=3 and P=10P=10 are new and were run on other machines than the rest, which moved the loss by at most 0.003 in our controls.

Newer models. We pretrained a Qwen3-style decoder of the same depth and width (116.5M parameters, with RMSNorm, grouped-query attention with QK-norm, RoPE and SwiGLU) on FineWeb, and swept each method's learning rate on it. LionMuon again has the lowest loss, for both P=2P=2 and P=5P=5 (Section 5.4 and Appendix G.4 of the revision). Best validation loss at each method's best rate:

best rateloss
AdamW10−310^{-3}3.4846±0.00203.4846 \pm 0.0020
Muon10−310^{-3}3.4699±0.00043.4699 \pm 0.0004
LionMuon P=2P=2ηM=2×10−3\eta_M = 2\times10^{-3}3.4567±0.00183.4567 \pm 0.0018
LionMuon P=5P=5ηM=3×10−3\eta_M = 3\times10^{-3}3.4590±0.00223.4590 \pm 0.0022

The best rates of AdamW and Muon lie inside their sweeps.

Larger models and generalization. In every setting of the paper, LionMuon with PP from 2 to 5 ends at or below Muon: two datasets, two architectures, 124M and 355M parameters, one and four GPUs, and runs of up to 150k steps. The largest model in the paper is 355M, for the compute reasons given in the general response.

At a larger scale, we have a result from a mixture-of-experts study that we ran on other hardware. The model has 1.03B parameters (280M active per token) and is trained on FineWeb-Edu for 7.3B tokens. Final validation loss, one seed each:

AdamWMuonLionMuon P=5P=5
2.6812.6512.626

The learning rate (1.63×10−31.63\times10^{-3}) follows the scaling law of Tian et al. (ICLR 2026, https://arxiv.org/pdf/2507.17702v2) for mixture-of-experts models and was checked on short runs. LionMuon uses the same rate and weight decay (0.1) as Muon, with the sign-step rate at 1/100 of it. A sign-step rate twice as large gives 2.621, and one half as large gives 2.641. This is one seed on one model, so we do not add it to the claims of the paper.

We hope this addresses your questions, and we are happy to answer any further ones.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful and constructive feedback. We answer two shared points here and then list the changes in the revised PDF. In our replies, a±ba \pm b is the mean and standard deviation over three seeds. Some tables of the paper print the spread (max minus min) in brackets instead.

A tuned Nesterov Muon baseline. Our Muon baseline uses EMA momentum without Nesterov. We have now tuned a Nesterov Muon at 124M: its momentum over 0.9, 0.95, 0.98 and 0.99, and its learning rate at 0.9 and 0.95. Tuning helps Muon, and LionMuon P=2P=2 is still ahead on FineWeb (3.4988±0.00153.4988 \pm 0.0015 against 3.5047±0.00143.5047 \pm 0.0014) and level on WikiText-103 (2.8302±0.00032.8302 \pm 0.0003 against 2.8313±0.00332.8313 \pm 0.0033), while running Newton-Schulz only every second step. Each pair was run on one machine. The result also holds at 150k steps: LionMuon P=2P=2 reaches 3.387, against 3.396 for Nesterov Muon, 3.418 for AdamW and 3.427 for Muon (one seed each, same setup). Details are in our reply to Reviewer yhoU (Q1).

Model scale. The largest model in the paper is 355M. Unfortunately, the compute available to us was mainly a shared node with 8 H200 GPUs used by about 20 researchers, and the experiments of the paper already took about two months on it. One 1B run at 1×1\times Chinchilla takes about 75 H200-hours. Repeating our 355M table at 1B (nine methods, three seeds each) is 27 runs, about 2,000 H200-hours. That is the whole node for ten days, before any learning-rate sweep. We spent our budget on three seeds, tuned baselines, a second architecture and 150k-step training instead. This scale is within the range of recent optimizer papers at ICLR, ICML and NeurIPS. Several evaluate language-model pretraining at 124M only (Defazio et al., NeurIPS 2024; Gruntkowska et al., ICLR 2026; Qi et al., ICML 2026; Riabinin et al., ICML 2026), and others stop at 720M to 774M (Semenov et al., NeurIPS 2026; Amsel et al., ICLR 2026; Crawshaw et al., ICML 2026). Our claims stop at 355M. We report a larger mixture-of-experts result in our reply to Reviewer j35f.

Corrections. We had found most of the errors the reviewers point out in our own audit in October: the parameter-routing sentence in Section 3.2, the direction of Eq. (6), the Appendix F conversion, the "optimal period" claim, the "least time" wording and the single seeds at 355M and on 4 GPUs. With three seeds the ranking is unchanged, except that Lion is now level with Muon on 4 GPUs. The reviews led to further changes. We removed an incorrect claim about an approximate Muon KKT point from Appendix B. Section 4.3 now separates gradients, signed updates, and learning rates ratios and states improved trade-off factor between our LionMuon and Muon and pure Lion. The reported bytes are defined, and the communication saving is stated for replicated optimizer state. We corrected the numerical constants in our proofs while keeping the same rates, and also moved additional (very light) assumptions from the proofs to the main text for explicitness. In Section 5.2, we corrected the tuning advice (keep ηL\eta_L and move ηM\eta_M with PP) and the split of the gain between alternation and the dual EMA, which depends on the dataset.

Revised PDF. We have uploaded a revised PDF ("the revision" below), with new and changed text in blue. It contains every correction named here. We added a Section 5.4 with the Nesterov Muon and Qwen3-style comparisons (a new Table 2), an Appendix G with the further experiments, a parameter-group table in Section 3.2, a three-seed 124M table with a Nesterov Muon row in Appendix E, and three seeds at 355M and on 4 GPUs. Table numbers below are those of the submission. In the revision, Tables 2 to 9 move up by one. We have also added the code and results of the new experiments to the anonymous repository linked in the submission.

AuthorsOpenReview ↗

Kind reminder

With about 30 minutes left in the discussion, we kindly remind the reviewers of our responses and the revised PDF, and we would be grateful if you could take them into account in your final assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.