acceptodds
Under review as a conference paper at ICLR 2027

RADeMuon: Provable Resource-Aware Quantized Decentralized Muon via Finite-Step Newton–Schulz Orthogonalization

Abstract

Distributed Muon optimizer enables matrix-aware optimization across multiple devices, but its full-precision local training and model exchange impose substantial resource costs, particularly when resource availability changes throughout decentralized training. We propose RADeMuon, a resource-aware quantized framework for decentralized Muon that supports precision configurations varying with available resources. RADeMuon performs local training with quantized parameters, gradients, and momentum states, applies finite-step Newton–Schulz (NS) orthogonalization to the quantized momentum, and directly communicates the resulting quantized parameter for decentralized aggregation. Theoretically, we trace quantization errors from their accumulation in the momentum state to their transformation by finite-step NS and subsequent propagation through decentralized aggregation. This analysis establishes a convergence guarantee whose dominant stationarity term is under vanishing-error conditions. Experiments on image classification and language-model pretraining show that RADeMuon reduces local computation and communication costs by up to 58.34% and 58.67%, respectively, relative to full-precision De-EF21-Muon while maintaining comparable model quality.

Primary area:
optimization
Keywords:
Decentralized learning, Muon, Quantized Training, Newton-Schulz Iteration, Convergence Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YXthOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to reduce the local training and model exchange cost of the distributed Muon optimizer. It propose a resource-aware quantized framework, RADeMuon, to address this challenge. The authors provide the convergence guarantees and experiments results to support their proposed algorithm.

Critical strengths
  1. The proposed algorithms effectively reduces costs by introducing quantization into distributed Muon optimization.
  2. The paper provides theoretical analysis and empirical studies.
Critical weaknesses
  1. The introduction claims that there are three Muon-specific challenges associated with quantization precision. However, challenges (i) and (iii) seem to be common to algorithms with momentum and decentralized network, respectively, instead of being specific to Muon. The authors should further clarify what makes these challenges unique to Muon and where the main technical novelty lies.
  2. Theorem 1 and Corollary 1 contain a term $
6r(1−ϵJ)Δ+1−ΔΔ\frac{6\sqrt{r}}{(1 - \epsilon_J)\Delta} + \frac{1 - \Delta}{\Delta}

(\sigma_1 + \sigma_2)$, which involving Δ\Delta, σ1\sigma_1 and σ2\sigma_2, but not explicitly decrease with TT. Thus, this term cannot be eliminated as TT increases. Although the authors claim that convergence “requires the stochastic and heterogeneity residuals to vanish”, this requirement is not generally necessary in standard distributed learning, which weakens the theoretical results. In addition, Eq. (S.57) simply assume Δ>0\Delta > 0, without establishing the conditions under which this assumption holds. Thus, the statement that O((nKT)−1/4)O((nKT)^{-1/4}) remains questionable. 3. Regarding the experiments, the 8-bit De-EF21-Muon baseline achieve lower communication cost than RADeMuon, although it requires higher computation cost. This seems to contradict the main claim of reducing both computation and communication cost. Furthermore, the paper only reports costs in MB. What about walk-clock time? 4. Table 2 and 3 are difficult to read. I recommend adding an arrow to each metric to indicate whether higher or lower values are better, and using different colors to highlight the best results. 5. The paper provides a relatively limited discussion of existing studies in related work section. I recommend dividing this section into two subsections,(2.1) Distributed Muon, and (2.2) Quantization and compression. For example, [1]-[4] are closely related work, but the relevant literature is notlimiated to these studies. Please provide a more comprehensive related work section. 6. The difference between Adriana (Qiao et al., 2026) and the proposed work is unclear. Does this work directly apply resource-aware quantization to Muon updates, or does it troduce additional methodological contributions?

[1] Junkang Liu, Fanhua Shang, Junchao Zhou, Hongying Liu, Yuanyuan Liu, and Jin Liu. 2025. Fedmuon: Accelerating federated learning with matrix orthogonalization.

[2] Xinwen Zhang and Hongchang Gao. 2025. On provable benefits of muon in federated learning.

[3] Neel Mishra, Kushagara Trivedi, Pawan Kumar. 2026. Signmuon: Communication-efficient distributed muon optimization.

[4] Laurent Condat, Kai Yi, Peter Richtarik. 2022. EF-BV: A unified theory of error feedback and variance reduction mechanisms for biased and unbiased compression in distributed optimization.

Questions for authors

See the weaknesses above.

Reviewer_yoRPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RADeMuon, a decentralized variant of Muon. Each client quantizes its parameters, gradients and momentum with bit widths that can vary per client and per round, orthogonalizes with a finite number of Newton-Schulz steps, and sends the quantized parameters to its neighbors. The authors prove a convergence bound with an O((nKT)^{-1/4}) leading term, assuming a positive "descent margin" and vanishing residual terms. Experiments with 5 clients on TinyImageNet, nanoGPT pretraining and GLUE LoRA report about 58% lower local cost and communication than 32-bit De-EF21-Muon at similar accuracy.

Critical strengths
  • The problem is timely and the empirical picture is useful. Decentralized and low-resource Muon has received little study so far. Across three different tasks, Muon-based methods clearly beat the SGD-based quantized baselines, and quantizing all local states to 8 bits costs very little accuracy.
Critical weaknesses
  • The convergence guarantee does not apply to the paper's own settings. The descent margin in Eq. S.57 requires roughly K*(1-beta) > 2*sqrt(r)*beta, even without any quantization. With beta = 0.95, K = 30 or 50, and r in the hundreds, this fails by a large factor. With the beta chosen in Corollary 1, Delta becomes negative as T grows, which contradicts the stated rate. If I have misread S.57, I would be glad to be corrected.

  • The bound does not vanish even when Delta > 0. It keeps a constant term proportional to sqrt(r) times the noise and heterogeneity levels. The parameter-quantization terms also do not decay at a fixed bit width. So the O((nKT)^{-1/4}) rate only holds for deterministic, homogeneous, full-precision training, which is weaker than existing Muon analyses.

  • The efficiency claims rely on a weak baseline. The 58% savings are measured against 32-bit De-EF21-Muon. But 8-bit De-EF21-Muon matches or beats RADeMuon in accuracy with clearly less communication, for example 124 vs 204 MB on TinyImageNet. In addition, the "resource-aware" schedule is a fixed 32/16/8 split for all clients on a fixed topology, and no error bars are reported.

Questions for authors

Clarifications:

  • What is the value of Delta from S.57 for your experimental settings? Is it ever positive?
  • With the beta from Corollary 1, does Delta stay positive as T grows? If not, for what range of T does the corollary hold?
  • Can the constant noise term be removed, as in existing Muon analyses with momentum?
  • Lemma 3 assumes delta_0 < 1 uniformly over all steps. Is this realistic for ill-conditioned quantized momentum? How do the experiments, which use the practical Muon polynomial, relate to the Taylor polynomial analyzed here?
  • What exactly does "Comp. (MB)" measure?

Suggestions:

  • Add RADeMuon at a fixed 8 bits, and De-EF21-Muon with 8-bit local states, as baselines.
  • Test clients with different and changing budgets, and time-varying topologies, since these motivate the paper.
  • Report means and standard deviations over several seeds.
Reviewer_cPQAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors designed a decentralized version of Muon (they named it RADeMuon) in which clients train locally and average their parameters with neighbors, and in which parameters, gradients and momentum are all kept at low precision. Their aim is to reduce memory and communication costs while keeping the quality of Muon, and it is important that the precision can change for each client and each round to follow the resources available, and is supplied from outside the method. Their main contribution is a convergence guarantee that holds for any such choice of precision and accounts for the approximate orthogonalization Muon uses in practice. They also provide experiments on image classification and language-model pretraining showing quality close to a full-precision Muon baseline with less memory and communication, although the experimental setting does not fully match the setting the theory and motivation describe.

Critical strengths
  • The theory analyzes Muon with a finite number of Newton-Schulz steps, which is how Muon is run in practice, and most Muon theory assumes exact orthogonalization, so this brings the theory closer to practice which is a valubale contribution.
  • The assumptions are mostly standard and reasonable, for example smoothness, unbiased gradients with bounded noise, bounded differences in data across clients, and a standard assumption on how clients communicate
  • The experiments cover different architectures and task types, specifically image classification, language-model pretraining, and fine-tuning.
  • The language model is trained to a full, standard token budget, including the end of the learning-rate schedule, so the results are not limited to early training
Critical weaknesses
  • The motivation is that clients whose resources differ and change during training but this is barely tested emprirically, as all clients always use the same precision, and precision follows one fixed schedule chosen manually, not the resources a client has. The framework also does not choose the precision itself, it is supplied from outside.
  • All clients are simulated on one machine, so there is no real network, and the communication savings are counted in bytes and never tested where communication thus would be the real limit. The authors don't seem to claim time savings, but to me this leaves open whether the savings matter in practice.
  • There are no repeated runs and no error bars, and the paper does not acknowledge this, and the main claim is that quality stays comparable to full precision, but the differences are small enough that they could be noise, so the claim cannot be judged
  • The size of the savings follows from the precision schedule the authors picked, and it does not come from the method or the theory, and the theory does not say how far precision can be lowered before quality gets hurt
  • A simpler baseline, the existing Muon method at a fixed low precision, matches full-precision quality and sends less data than RADeMuon (tables 2 and 3). RADeMuon’s advantage over it is only in local memory, so the communication benefit appears to hold only against full precision
  • The analysis of finite-step Newton-Schulz is a good step toward matching the theory to practice, but its empirical support feels weak, and the only evidence is that more steps help, which is expected anyway.
  • The paper doesn't say which Newton-Schulz polynomial or degree the experiments use, so it is unclear whether the theory describes the runs
Questions for authors
  • With the momentum and local steps used in the experiments, the descent margin appears to be negative even without quantization, and with the corollary’s choice of momentum it decreases as training gets longer. Could the authors state the regime where it is positive, and clarify whether the stated rate holds? If this is not clarified, it would seem to me that the main theorem may not apply to the settings used in the experiments, and the stated rate doesn't follow
  • The theorem relies on two conditions that seem important but I can only find them in the appendix, that the starting matrix for Newton-Schulz is never too far from orthogonal, and that the descent margin is positive, do these hold for the settings used in the experiments?
  • Which Newton-Schulz polynomial and degree do the experiments use?
  • Given that fixed low precision already matches full precision, why start training at 32 bits?
  • The network assumption is standard for gossip, and it includes the fully connected case, so how relevant is the gossip setting for large-scale training, where all-reduce is the norm?
  • In the fine-tuning results, Muon methods no longer beat the SGD-type methods, why do the authors think this happens?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.