acceptodds
Under review as a conference paper at ICLR 2027

AdaSC-Muon: Adaptive Subspace Compression for Communication-Efficient Muon Training

Abstract

Muon has emerged as a promising optimizer for large-scale neural network training, but communication and orthogonalization costs can limit its efficiency in distributed settings. EF21-Muon by Gruntkowska et al. reduces communication by compressing momentum differences. Its best-performing compressor still requires workers to transmit local singular bases at every iteration, so the worker-to-server communication cost scales with the original matrix dimensions. We introduce adaptive subspace-compressed Muon (AdaSC-Muon), which uses shared subspaces for both momentum compression and Muon computation. The key insight is that aggregating projected momentum before orthogonalization enables a global update to be computed directly in a reduced matrix space. Workers reuse the shared bases and quantize projected momentum differences with error feedback. Periodic full-momentum corrections update the subspaces as momentum changes during training. We establish convergence guarantees and communication complexity bounds for AdaSC-Muon. Language model pretraining experiments with NanoGPT on FineWeb show that, compared with EF21-Muon, AdaSC-Muon and its diagonal variant reduce worker-to-server communication cost by up to 47.8% and reach the target validation loss in up to 53.8% less wall-clock time.

Primary area:
optimization
Keywords:
Muon, compression, error feedback, distributed training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bzETOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper targets the communication and orthogonalization costs of the Muon optimizer in synchronous distributed training. It builds on EF21-Muon, whose RankK compressor transmits new singular factors every iteration. The proposed method keeps shared left and right rank-r bases from a global momentum approximation. Workers project momentum onto these bases and quantize differences with error feedback. Every T iterations compressed full-momentum differences refresh the bases; a diagonal option transmits r entries. Experiments pretrain NanoGPT-124M on FineWeb10B (four B200 GPUs) against EF21-Muon with RankK 15% plus Natural compression under fixed tokens, communication, time, and target loss, reporting up to 47.8% lower worker-to-server communication and up to 53.8% less wall-clock time. Code is released.

Critical strengths

S1. Well-written paper.

S2. Strong theoretical results.

S3. Good experimental evaluation on modern hardware. Shows substantial gains, especially in terms of wall-clock execution time.

Critical weaknesses

W1. The gains are unattributed: all runs use one four-GPU B200 node where the counted traffic (about 15 MiB/s; Table 1, Appendix D.2) is far below intra-node bandwidth, so communication is not shown to be a bottleneck.

Questions for authors

Q1. Explain W1.

1 comment

Dear Reviewer bzET,

We are writing to follow up on the rebuttal that we posted in response to your review. We have also uploaded a revised version of the paper with all changes highlighted in blue. We welcome any further discussion and would value your feedback on our rebuttal and the revised paper.

Many thanks for your time and your helpful comments.

Sincerely,

The Authors of Submission 44076

Reviewer_L7tSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaSC-Muon, a communication-efficient variant of Muon that combines shared subspace compression, error feedback, and periodic momentum corrections. It provides convergence analysis and communication complexity results, and evaluates the method on NanoGPT pretraining. The experiments report communication savings and wall-clock speedups over EF21-Muon, although the evaluation is limited in scale.

Critical strengths

S1: The method combines subspace compression with error feedback in a reasonable way to reduce communication overhead in Muon training.

S2: The paper provides theoretical convergence analysis and experimental results showing promising communication savings and wall-clock speedups.

Critical weaknesses

W1: The experimental evaluation is limited to NanoGPT-124M on a single node with four GPUs. It remains unclear whether the method scales effectively to larger models and multi-node training, which limits the evidence for its practical significance.

W2: The source of the reported wall-clock speedups is insufficiently analyzed, particularly the trade-off between communication savings and the overhead of orthogonalization and compression. The applicability of the theoretical compressor assumptions to the practical low-bit implementation also needs clarification.

W3: The experimental comparison is not sufficiently comprehensive to establish the advantages of AdaSC-Muon over alternative communication-compression strategies. More comparisons under consistent training configurations would help isolate the benefits of the proposed subspace compression and momentum correction components.

Questions for authors

Q1:At 5B tokens, communication time decreases by 24.6%, while wall-clock time decreases by 52.9%. Could the authors provide a breakdown of the runtime savings to explain this gap?

Q2:Could the authors compare AdaSC-Muon with additional baselines, including FedSubMuon, TSR-Adam, and uncompressed Muon, and evaluate larger models or multi-node settings? Experiments across multiple random seeds would also help establish the reliability of the reported improvements.

Q3:What communication compression ratio does AdaSC-Muon achieve in practice, and how much does it reduce communication time? Do compression and decompression introduce significant overhead?

Q5:Could the authors provide ablation studies to quantify the individual contributions of shared subspace compression, error feedback, and periodic momentum corrections to the overall performance?

1 comment

Dear Reviewer L7tS,

We would like to kindly remind you that we have posted our rebuttal, in which we respond to each of your concerns. In addition to the rebuttal, we have uploaded a revised PDF with all changes shown in blue. We would be grateful for your feedback and we are happy to discuss any further questions.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 44076

Reviewer_kkxrOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

They propose extending EF21-Muon by compressing momentum in a shared subspace that is refreshed periodically, so workers send small matrices and the server orthogonalizes a smaller one, and call this AdaSC-Muon. They argue this lowers communication cost and training time in multi-GPU training, but they only test NanoGPT on FineWeb, on one node with one seed

Critical strengths
  • The problem is important, I think communication is a real cost in large-scale distributed training, so the motivation is good
  • The key idea is simple, the point of shared bases, averaging and projection commute, so the server’s update is computed from the projected global momentum and that is sound.
  • The approach could be practically useful if the benefits hold where communication is the bottleneck. I am just not sure if this is the case, see weaknesses bellow.
  • The theory uses mostly standard assumptions, and I did not find errors.
  • The related work discussion is broad and states how the method differs from most of the closely related methods.
  • The plots are clear and easy to read.
Critical weaknesses
  • It is only one model size, dataset, node, and seed. It is hard to know whether the results generalize
  • The method is motivated by settings where communication is the bottleneck, such as many nodes or slow links, but it is only tested on a single node with fast links, so the importance of the problem supports the motivation more than the results seem to.
  • I think the wall-clock speedup is not explained well as both methods need about the same number of steps, so the gain comes from cheaper steps, and on a single fast node there is no clear reason to believe it comes from communication. It may come from the baseline’s per-step SVD for example. As reported, it reads as a communication benefit tho.
  • The rank is close to full so the savings in bytes seem to come mostly from sending values at lower precision than the baseline, so the contribution of the subspace itself is unclear.
  • The only baseline is EF21-Muon and without uncompressed Muon, it isn’t possible to see how much loss compression costs, and the subspace methods discussed in related work are not compared
  • the learning rate was searched for AdaSC-Muon but taken from a reference value for EF21-Muon, so I am not even sure they are directly comparable.
  • In their theory they assume exact orthogonalization and a momentum value set from unknown constants, but the runs use Newton-Schulz and a fixed momentum, where the bound doesn’t converge to zero. The rate is the same order as the baseline with worse constants, so I am not convinced that the theory explain the empirical gains
  • The contraction assumption is likely satisfied only with a very small guaranteed fraction for their quantizer, which makes the bound very loose, and this is not discussed or checked.
  • The communication complexity result is only theoretical, and the trade-off it describes between rank and refresh period is not empirically shown
  • This is minor, but the writing is also very dense, rrsults paragraphs repeat the numbers in the tables instead of interpreting them, and the diagonal option is poorly explained
Questions for authors
  • Was the reference learning rate for EF21-Muon tuned in this same setup?
  • Why a rank of 90%?
  • How would the method work with sharded weights and more workers, given that the server receives every message and broadcasts the bases?
  • What rounding does the quantizer use?
  • Is there a reason to expect the theory to describe the runs, given fixed momentum and approximate orthogonalization?
3 comments

Dear Reviewer kkxr,

This is a gentle reminder regarding our rebuttal, in which we have carefully addressed your comments and concerns. We have also uploaded a revised version of the paper with all changes highlighted in blue. We would greatly appreciate it if you could review our rebuttal and the revised paper when convenient, and we welcome any further discussion.

Thank you very much for your time and consideration.

Best regards,

The Authors of Submission 44076

I appreciate the added experiments, especially on this timeline.

On strenghts, I do believe that the larger model with both methods tuned the same way, the additional seeds, and the uncompressed Muon reference make the comparison with EF21-Muon more convincing, and the new results are consistent with the original claims. Also I think the per-step time breakdown explains where the wall-clock savings come from which make the udnerstanding of the method better. I also appreciate the stated rounding scheme, the measured contraction during training, the rank and refresh-period ablations, the clearer explanation of the diagonal option, and the more explicit statement of where the theory and the runs differ. Based on this, I am raising my score.

I don't expect new experiments at this stage, of course, but a few concerns remain, and I would appreciate the authors’ thoughts on them if time allows.

  1. the paper is framed around communication efficiency but all runs are on a single node with fast links (where communication isn't the bottleneck) and the slower-bandwidth results are calculated estimates only. I assume compute budget is the constraint here, but given the premise of the paper this remains an important gap, and I would suggest at least stating it clearly in the paper as a limitation.

  2. from the rebuttal counting traffic in both directions (worker to server and server to worker), the method as run seems to send more data in total than EF21-Muon, and with the proposed broadcast change only somewhat less. Could the authors report total traffic in both directions, or correct me if I am misreading this?

  3. EF21-Muon is the only compressed baseline, so every claim is relative to it. I understand the design differences from the other subspace methods, and I would suggest stating this scope clearly in the claims.

  4. The measured contraction and the time breakdown cover only the first few hundred steps of training. Could the authors state this in the paper and say whether they can reasonably expect, ideally with some emprirical evidence, the results to hold later in training?

For future work, I'd also suggest evaluating on another dataset, yet multiple models and seeds at this stage is reasonable.

Thank you so much for the detailed suggestions!! We answer each remaining point below and have uploaded a revised paper again with the changes marked in blue.

Single node. We agree and now state this as a limitation in Section 5, with evaluation on multiple nodes as future work. Our estimates at lower bandwidths add the communication cost divided by the bandwidth to the time measured on a single node (Section 4.2 of our paper). AdaSC-Muon computes less per iteration and, with the broadcast described below, sends less in both directions (Table R15). Its estimated time to the target validation loss is therefore shorter at any bandwidth.

Communication in both directions. As in EF21-Muon, the server broadcasts the m×nm\times n update in FP32 at every iteration, and each worker adds it to its copy of the model. Our experiments also broadcast the bases QLQ_L and QRQ_R in FP32 every TT iterations. Our paper counts only worker-to-server communication because EF21-Muon fixes the server-side compressor to the identity and assumes that broadcasting is free (Sections 3.3 and 4.1 of our paper).

We note that broadcasting the m×nm\times n update is not necessary. In Algorithm 1, the server broadcasts the bases at each basis update and Xk+1=Xk+ηQLkWk(QRk)⊤X^{k+1}=X^k+\eta Q_L^kW^k(Q_R^k)^\top at every iteration. Since every worker holds XkX^k and the bases, the server can broadcast only the r×rr\times r projected LMO direction WkW^k, as noted in our response to Concern 9. Each worker then computes QLkWk(QRk)⊤Q_L^kW^k(Q_R^k)^\top itself and obtains exactly the same model Xk+1X^{k+1} as in our experiments. The training is therefore identical, and all validation losses and worker-to-server communication costs in our paper stay the same. Algorithm 1 itself does not change, so Theorem 1 also applies without change. The server computes the same two matrix products in our experiments. They now run on all workers in parallel, so the time per iteration does not increase.

Table R15 reports the communication in both directions. With the broadcast of WkW^k, AdaSC-Muon sends 32.1% less than EF21-Muon from the workers to the server and 14.5% less from the server to the workers, or 16.3% less in total. Appendix D.2 now includes this comparison.

Table R15. Communication per iteration in MiB for the hidden weight matrices of NanoGPT-124M.

MethodBroadcastWorker to serverServer to workerTotal
EF21-Muonm×nm\times n update36.4324.0360.4
AdaSC-MuonWkW^k and bases24.7276.9301.6

Scope of the comparison. We agree and have made the scope explicit in the paper. Section 4.1 now states that EF21-Muon is our only compressed baseline and gives the reasons from our response to Concern 4. Section 1 now also names EF21-Muon in the claim on the wall-clock time. The abstract and Sections 1 and 5 therefore name EF21-Muon in every comparison with a baseline. Uncompressed Muon serves only as a reference (Section 4.2 and Appendix E.5 of our paper).

Later iterations. As stated in Appendices E.6 and E.8 of our paper, the breakdown of the wall-clock time and the measured contraction parameters come from the first 400 iterations. Section 3.2 now also states this for the measured contraction parameters.

We note that the time per iteration does not increase later in training. The last row of Table R4 gives the average time per iteration over the whole 5B-token runs, 3.29 seconds for EF21-Muon and 1.50 seconds for AdaSC-Muon. Both are slightly below the totals over the first 400 iterations, 3.350 and 1.557 seconds, which include the overhead of the per-component timing. The 17-hour run of AdaSC-Muon in Table 2 of our paper, with 41,961 iterations, also averages 1.46 seconds per iteration.

For the contraction, Assumption 4 holds with the worst-case parameters for every input, so Theorem 1 applies at every iteration. The measured values only show how loose this worst case is. Appendices E.6 and E.8 now include these points.

Other datasets. We use FineWeb because EF21-Muon reports its results on it (our response to Concern 1). Section 5 now lists evaluation on other datasets as future work.

We hope that these answers and revisions resolve your remaining concerns, and we would be grateful if you could take them into account in your evaluation. If you have any additional questions, please let us know. We look forward to your reply.

Comments on the paper

1 comment

Dear Reviewers,

We would like to kindly remind you once again of our rebuttal and the revised version of our paper. Our rebuttal responds to each of your concerns, and the revised paper highlights all changes in blue. We would greatly appreciate it if you could review them before the end of the discussion period. We are happy to answer any further questions and welcome any discussion.

Thank you very much for your time and consideration.

Best regards,

The Authors of Submission 44076

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.