acceptodds
Under review as a conference paper at ICLR 2027

SignMuon: Communication-Efficient Distributed Muon for LLM Pretraining

14 views

Abstract

Full-precision gradient exchange can limit the scaling of data-parallel training. We propose SignMuon, a distributed optimizer that combines Muon's matrix update directions with signSGD's majority-vote communication. Each worker approximates the polar factor of its local momentum matrix and communicates only its entrywise signs. Workers aggregate these signs using int8 AllReduce or packed 1-bit AllGather and apply the same update, keeping the polar computation local. We give conditional stationarity bounds for the actual momentum-polar vote and distinguish polar-solver error from the loss introduced by signing. On large language models (LLMs) up to 1.5B parameters trained on eight GPUs, SignMuon achieves lower perplexity at matched token budgets than the tested sign-based optimizers, while also showing favorable log-perplexity trajectories in wall-clock comparisons. We further evaluate optimizer memory usage and distributed communication efficiency in multi-GPU training. Together, these results show that SignMuon combines Muon's local matrix transformation with highly compressed global communication, delivering strong token efficiency and lower log perplexity for distributed LLM pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SignMuon, Muon, distributed optimization, communication-efficient training, gradient compression, sign-based optimization, majority-vote aggregation, matrix-aware optimization, LLM pretraining, data-parallel training, distributed deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EtoHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SignMuon, a distributed optimizer for LLM pre-training in which each worker computes a Muon-style polar direction of its local momentum, takes its entrywise sign, and communicates it via int8 AllReduce or packed 1-bit AllGather; the update is the majority vote of the workers' signs. The authors provide conditional stationarity bounds under explicit vote-alignment assumptions, an analysis of the Newton–Schulz solver error, and a cost model of the collective operations. Experiments on GPT-2 Large, GPT-2 XL, and OPT-350M on eight GPUs compare SignMuon with other sign-based optimizers and report lower validation loss under fixed token and wall-clock budgets.

Critical strengths
  1. Combining matrix-aware updates with low-bit communication is a relevant direction for large-scale training.

  2. The implementation of majority vote via packed AllGather and int8 AllReduce is described in detail, and the communication costs are carefully accounted for.

  3. Within the tested set of sign-based optimizers, SignMuon achieves the lowest validation loss in all reported settings.

Critical weaknesses

Critical Weaknesses:

  1. The convergence guarantees are conditional on assumptions that are vacuous on their own and circular when required to vanish.

Theorem 1 assumes 2Eet≤ρEVt+δt2\mathbb{E}e_t \le \rho\mathbb{E}V_t + \delta_t and Theorem 13 assumes E⟨Mˉt+1,Dt⟩≥κE∣∣Mˉt+1∣∣−εt\mathbb{E} \langle \bar M_{t+1}, D_t\rangle \ge \kappa\mathbb{E}||\bar M_{t+1}|| - \varepsilon_t, with additive terms δt\delta_t, εt\varepsilon_t whose origin is not specified. Without restrictions on these terms, both conditions always hold. The guarantees therefore rest entirely on requiring the residuals to vanish, e.g., εˉT=O(T−1/4)\bar\varepsilon_T=O(T^{-1/4}); otherwise, as the authors acknowledge, the bound does not vanish.

Note that εt\varepsilon_t is not a variance term: the stochastic noise is handled separately through momentum (the term σˉ(1−β)/(1+β)\bar\sigma\sqrt{(1-\beta)/(1+\beta)} in (37)). Instead, εt\varepsilon_t absorbs the signing loss χsign\chi_{\rm sign}, which is deterministic and present even for a single worker with exact gradients, as well as the polar-solver and vote errors. Since χsign\chi_{\rm sign} scales linearly with ∣∣Mˉt+1∣∣||\bar M_{t+1}||, it is a relative misalignment written as an additive term. Either it is uniformly bounded in relative form, in which case κ\kappa degrades with dimension, or it is not, in which case requiring εˉT=O(T−1/4)\bar\varepsilon_T=O(T^{-1/4}) presupposes the conclusion of the theorem.

Both cases occur in practice. In the only example provided by the authors in Appendix F.4, the alignment constant equals 1/d1/\sqrt{d}, and for generic gradient matrices it behaves similarly, decreasing as the inverse square root of the matrix dimension, whereas exact Muon has a constant of one. Since this constant enters the bound inversely, the guaranteed rate deteriorates with model size and loses exactly the advantage that the Muon step is supposed to provide. Worse, the alignment can be negative: there are simple small matrices for which the entrywise sign of the polar factor points in an ascent direction, so that even a single worker with exact gradients and an exact polar factor increases a linear objective for any stepsize. In this case no positive alignment constant exists, and the additive residual can vanish only if the gradient itself vanishes. Finally, the experiments use a constant learning rate, fixed momentum, and gradient clipping, for which the bound does not vanish even under the stated assumptions.

The paper provides no convergence guarantee for SignMuon: in the favorable case, the bound degrades with dimension; in the unfavorable case, which does occur, it relies on an assumption equivalent to the claimed result; and in the experimental setting, it does not vanish at all.

  1. The communication scheme is not a matrix compression, and its properties are not established.

What is communicated is the entrywise sign of the local polar factor, i.e., the same 1-bit coordinate-wise quantization as in signSGD; the authors themselves note that SignMuon and signSGD have the same aggregation cost. The matrix structure exploited by Muon is used only in local preprocessing and is destroyed by the entrywise sign. This is fundamentally different from matrix compression that respects spectral geometry, such as low-rank schemes (PowerSGD; Dion, Ahn et al., 2025). Moreover, the sign operator is a biased, non-contractive compressor, and no error feedback is used, although this is the standard remedy for biased compression (Karimireddy et al., 2019; EF21) and has recently been extended to LMO-based methods such as Muon (Gruntkowska et al., 2025). Instead of a compression lemma, the entire signing error is collected into χsign\chi_{\rm sign} and then assumed to be small (Weakness 1).

  1. The theory does not explain the benefit of the Muon step.

Theorem 1 is a generic bound for any sign-based majority vote, expressed through the probabilities of sign errors; the polar step does not enter it. The relative-error condition is close to the success-probability-bound assumption of Safaryan & Richtarik (ICML 2021), which is not cited. As the authors acknowledge, spectral smoothness does not improve the curvature constant for a normalized sign step. Theorems 4–6 are standard cost models of collective operations, identical for any sign method. The algorithmic idea itself, applying an entrywise sign after orthogonalization, is not new either, as the authors note regarding OLion (Wang et al., 2026).

  1. The experimental comparison is insufficient.

a) The baselines are limited to sign-based methods. Full-precision Muon and AdamW with standard AllReduce are missing, so the cost of compression cannot be assessed. Compression methods with guarantees, e.g., error-feedback signSGD, PowerSGD, and distributed Muon variants, e.g., Dion (Gruntkowska et al., 2025) should also be compared.

b) The claimed communication benefit is not demonstrated. In the weak-scaling experiment (Figure 5), SignMuon has the largest absolute runtime, and the main experiments use a single 8-GPU node, where communication is unlikely to be the bottleneck.

c) The models are heavily undertrained: 20-60M tokens for 0.3-1.5B-parameter models within 30 minutes, with a constant learning rate, no warmup, and no weight decay. The final log-perplexities (about 4–7) are far from a regime where conclusions about pre-training can be drawn.

d) The learning-rate grid spans whole orders of magnitude, and Figure 1 is based on a single seed.

  1. The proofs are hard to follow, and the assumptions are not stated explicitly (e.g., "Under Assumption 7 and the conditional sampling conditions above" in Theorem 13). The results are not compared with prior bounds for sign-based and Muon-type methods.
Questions for authors

See Weaknesses

Reviewer_c6m3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a communication efficient variant of Muon where only the signs are communicated. Since this is done to the polar direction (after which the signs are aggregated) not to the gradient, workers maintain their own local momentum.

Critical strengths

The motivation is strong, the results show large improvement over the given baselines, and the paper is easy to follow.

Critical weaknesses

The paper is missing a major baseline: how does SignMuon compare with communicating local gradient signs, aggregating them by mean or majority vote, and then applying momentum and a polar update locally? This has the same sign-communication budget and thus is the obvious alternative to check against.

Another major issue is that 30-minute runs measure early optimization, not final pretraining quality or downstream performance. Not all settings need to be done for longer, but at least one much longer run should have been done. This is likely the reason for such large differences in results: other optimizers often close most of the gap (to Muon) towards the middle and end of training.

The results show that SignMuon performs the best against wall clock time against other sign based approaches, but does not show results against Muon and AdamW. There might be communication bandwidths at which the full precision version's convergence outweighs the communication benefits, and the paper should show where such a phase would be. Plus, anyway showing the gap to uncompressed is useful for understanding how well the optimizer performs in general.

The theory assumes that the final update points in a useful direction, but this is never established. A good compromise would be to show some empirical support: show that final-vote alignment usually aligns with the true gradient during training. In fact, this type of empirical evidence should be used to motivate the choices like which type of aggregation, why doing signs on polar direction not on the momentum or on the gradient, etc.

All the results are with the all-gather version, which as they mention does not scale well with replicas. Thus there should be some results with the int8 all reduce to show that this is a viable options for the >=16 replica setting.

Questions for authors

Decoupled weight decay is the usual standard choice, any reason why you chose to have weight decay coupled with momentum?

Reviewer_kbbBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes signMuon, an optimizer that reduces the communication message size via 1-bit allgather. Compared to other signed optimizers, signMuon achieves lower GPU memory usage, higher per-step time, but better accuracy.

Critical strengths
  1. The sign extraction, combined with normalization and local orthogonalization mechanism, seems effective as shown in the results.

  2. A clear presentation and easy to understand mechanism but effective in preserving signed Muon's accuracy.

  3. Comprehensive analysis of the compressed optimizer state bit-level accounting. Additionally, the Newton-Schulz bound, signing and voting errors are clearly defined.

Critical weaknesses
  1. The theory puts the key property in the assumption, instead of a rigorous proof. Specifically, Theorems 1 and 13 only valid when assuming the final voted sign aligns with the gradient. In signSGD, this property derived from noise assumptions. However, signMuon cannot make this assumption. This is because even the polar factors are noise-free, the entrywise signs can be opposite to the gradient, by signMuon's design. This assumption should be verified more than a diagonal toy case with κ=1/d\kappa = 1/\sqrt{d}.

  2. Incomplete evaluation. The baseline should also include Muon and AdamW for the "communication-efficient" claim. However, only other signed optimizers are in the baseline.

  3. Evaluation need to change the way to present. Figure 5 presents a longer per-step time of each signed optimizer. For one thing, it should also include the Muon and AdamW results. For another, there is no clear evidence of signMuon over other baselines. An ideal result should show that signMuon has less end-to-end time, a longer per-step time, but fewer steps to similar convergence.

Questions for authors
  1. Can you please measure in real runs, ρ\rho and δt\delta_t from Theorem 1 and κ\kappa and εt\varepsilon_t from Theorem 13, and compare them with SignSGD, SignAdam, and SignLion?
  • Under comparable assumptions, does the analysis give SignMuon any provable advantage over signSGD with majority vote?
  1. How does SignMuon perform compared to Muon, AdamW, and SGD in terms of communication efficiency, convergence, and final log-perplexity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.