- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SignMuon, a distributed optimizer for LLM pre-training in which each worker computes a Muon-style polar direction of its local momentum, takes its entrywise sign, and communicates it via int8 AllReduce or packed 1-bit AllGather; the update is the majority vote of the workers' signs. The authors provide conditional stationarity bounds under explicit vote-alignment assumptions, an analysis of the Newton–Schulz solver error, and a cost model of the collective operations. Experiments on GPT-2 Large, GPT-2 XL, and OPT-350M on eight GPUs compare SignMuon with other sign-based optimizers and report lower validation loss under fixed token and wall-clock budgets.
Critical strengths
-
Combining matrix-aware updates with low-bit communication is a relevant direction for large-scale training.
-
The implementation of majority vote via packed AllGather and int8 AllReduce is described in detail, and the communication costs are carefully accounted for.
-
Within the tested set of sign-based optimizers, SignMuon achieves the lowest validation loss in all reported settings.
Critical weaknesses
Critical Weaknesses:
- The convergence guarantees are conditional on assumptions that are vacuous on their own and circular when required to vanish.
Theorem 1 assumes and Theorem 13 assumes , with additive terms , whose origin is not specified. Without restrictions on these terms, both conditions always hold. The guarantees therefore rest entirely on requiring the residuals to vanish, e.g., ; otherwise, as the authors acknowledge, the bound does not vanish.
Note that is not a variance term: the stochastic noise is handled separately through momentum (the term in (37)). Instead, absorbs the signing loss , which is deterministic and present even for a single worker with exact gradients, as well as the polar-solver and vote errors. Since scales linearly with , it is a relative misalignment written as an additive term. Either it is uniformly bounded in relative form, in which case degrades with dimension, or it is not, in which case requiring presupposes the conclusion of the theorem.
Both cases occur in practice. In the only example provided by the authors in Appendix F.4, the alignment constant equals , and for generic gradient matrices it behaves similarly, decreasing as the inverse square root of the matrix dimension, whereas exact Muon has a constant of one. Since this constant enters the bound inversely, the guaranteed rate deteriorates with model size and loses exactly the advantage that the Muon step is supposed to provide. Worse, the alignment can be negative: there are simple small matrices for which the entrywise sign of the polar factor points in an ascent direction, so that even a single worker with exact gradients and an exact polar factor increases a linear objective for any stepsize. In this case no positive alignment constant exists, and the additive residual can vanish only if the gradient itself vanishes. Finally, the experiments use a constant learning rate, fixed momentum, and gradient clipping, for which the bound does not vanish even under the stated assumptions.
The paper provides no convergence guarantee for SignMuon: in the favorable case, the bound degrades with dimension; in the unfavorable case, which does occur, it relies on an assumption equivalent to the claimed result; and in the experimental setting, it does not vanish at all.
- The communication scheme is not a matrix compression, and its properties are not established.
What is communicated is the entrywise sign of the local polar factor, i.e., the same 1-bit coordinate-wise quantization as in signSGD; the authors themselves note that SignMuon and signSGD have the same aggregation cost. The matrix structure exploited by Muon is used only in local preprocessing and is destroyed by the entrywise sign. This is fundamentally different from matrix compression that respects spectral geometry, such as low-rank schemes (PowerSGD; Dion, Ahn et al., 2025). Moreover, the sign operator is a biased, non-contractive compressor, and no error feedback is used, although this is the standard remedy for biased compression (Karimireddy et al., 2019; EF21) and has recently been extended to LMO-based methods such as Muon (Gruntkowska et al., 2025). Instead of a compression lemma, the entire signing error is collected into and then assumed to be small (Weakness 1).
- The theory does not explain the benefit of the Muon step.
Theorem 1 is a generic bound for any sign-based majority vote, expressed through the probabilities of sign errors; the polar step does not enter it. The relative-error condition is close to the success-probability-bound assumption of Safaryan & Richtarik (ICML 2021), which is not cited. As the authors acknowledge, spectral smoothness does not improve the curvature constant for a normalized sign step. Theorems 4–6 are standard cost models of collective operations, identical for any sign method. The algorithmic idea itself, applying an entrywise sign after orthogonalization, is not new either, as the authors note regarding OLion (Wang et al., 2026).
- The experimental comparison is insufficient.
a) The baselines are limited to sign-based methods. Full-precision Muon and AdamW with standard AllReduce are missing, so the cost of compression cannot be assessed. Compression methods with guarantees, e.g., error-feedback signSGD, PowerSGD, and distributed Muon variants, e.g., Dion (Gruntkowska et al., 2025) should also be compared.
b) The claimed communication benefit is not demonstrated. In the weak-scaling experiment (Figure 5), SignMuon has the largest absolute runtime, and the main experiments use a single 8-GPU node, where communication is unlikely to be the bottleneck.
c) The models are heavily undertrained: 20-60M tokens for 0.3-1.5B-parameter models within 30 minutes, with a constant learning rate, no warmup, and no weight decay. The final log-perplexities (about 4–7) are far from a regime where conclusions about pre-training can be drawn.
d) The learning-rate grid spans whole orders of magnitude, and Figure 1 is based on a single seed.
- The proofs are hard to follow, and the assumptions are not stated explicitly (e.g., "Under Assumption 7 and the conditional sampling conditions above" in Theorem 13). The results are not compared with prior bounds for sign-based and Muon-type methods.
Questions for authors
See Weaknesses