acceptodds
Under review as a conference paper at ICLR 2027

MuonMix: A Mixed-LMO Block-Periodic Algorithm for Efficient Distributed Muon Training

Abstract

The Muon optimizer accelerates language model pretraining by applying Newton-Schulz orthogonalization to hidden weight matrices, but scaling it to distributed training requires gathering gradient shards across devices at every step. MuonBP reduces this cost through block-periodic orthogonalization, but applying the spectral-norm linear minimization oracle (LMO) to local shards can introduce a partition-dependent approximation error. Since max-norm LMO updates do not suffer from this error, we propose MuonMix, a mixed-LMO block-periodic algorithm for distributed Muon training that assigns each hidden weight matrix to either the spectral-norm LMO or the max-norm LMO under a shared synchronization period , based on a per-layer selection heuristic motivated by a layerwise convergence upper bound. Layers assigned to the max-norm LMO require no gather or scatter operations and introduce no block approximation error. We evaluate the proposed algorithm on language model pretraining at 160M, 960M, and 1.2B parameters on FineWeb, and show that it improves throughput by 3.0%, 4.8%, and 6.5% over MuonBP and achieves faster early convergence across all model sizes, and reaches target validation losses up to 7.1% faster in wall-clock time.

Primary area:
optimization
Keywords:
Muon, linear minimization oracle, distributed training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_21PUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a mixed-optimizer strategy, assigning hidden weight matrices to either MuonBP's spectral-norm linear minimization oracle (LMO) or an element-wise max-norm LMO. The former retains periodic full-matrix orthogonalization, while the latter avoids Newton-Schulz computation and optimizer-specific gather/scatter operations. A heuristic determines each matrix's optimizer using estimated descent surrogates. Evaluations include a 160M-model trained on eight B200 GPUs, and virtual-sharding of 4GPUs to simulate 128 and 256 GPU parallelism for a 960M and 1.2B model respectively. The 160M experiments show a throughput increase of 3.0% above MuonBP. The simulated experiments show a throughput increase of 4% for 128GPUs, and 6.5% for 256 GPUs.

Critical strengths

Assigning certain matrices to an element-wise optimizer removes orthogonalization computation and optimizer-specific communication, which is seen in the evaluation to lead to increased throughput.

Critical weaknesses
  1. In Appendix A.1, it is mentioned that training comparisons do not establish runtime savings if profiling is included, but it is not discussed how well the selected layer assignments transfer across different training runs. It would be helpful to see individual training runs and what the variance in both throughput and loss is across runs.
  2. Results are insufficient to confidently establish a general selection principle, since all model sizes are tested using one Modded-NanoGPT architecture.
Questions for authors
  1. How transferable are selected layer assignments to training runs with different hyperparameters? Can you report the variability across individual runs and how they compare against a standalone AdamW/Signum implementations?
  2. How transferable are selected layer assignments to different architectures? How does this static assignment compare to profiling the new architecture directly, and can you draw a conclusion relating to when it is feasible to keep the same assignment and when re-profiling is necessary?
1 comment

Dear Reviewer 21PU,

We are writing to follow up on the rebuttal that we posted in response to your review. A revised version of the paper is now also available, with all changes highlighted in blue. We would value your feedback on our rebuttal and the revised paper at your convenience. We are happy to discuss any further questions.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 36827

Reviewer_2kErOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MuonMix, which combines MuonBP and Signum across weight matrices to reduce distributed training costs. A profiling score selects which matrices use each optimizer. Experiments on three language model sizes show faster early convergence and modest improvements in training speed.

Critical strengths
  1. A practical idea with a clear motivation. Elementwise updates on selected matrices avoid both orthogonalization costs and errors caused by applying Muon separately to shards.
  2. Language model experiments across three sizes, with additional AdamW tests and layer-selection ablations. Evidence supporting the choice of query and key matrices for elementwise updates.
Critical weaknesses
  1. An overly strong assumption behind the theory. Assumption 1 bounds each layer’s gradient change using only that layer’s weight change, for arbitrary full-model states. This forces its gradient to remain unchanged when other layers change, excluding normal interactions between network layers. A revised assumption and proof needed.
  2. No demonstrated net saving after profiling. Appendix A.1 reports about 9,390 seconds for just one profiling configuration, compared with roughly 80 seconds saved during the 160M training run. Different GPU counts, but still a substantial upfront cost. No measured break-even point when reusing the assignment.
  3. Limited validation of distributed speedups. The 128- and 256-GPU results use virtual shards on four GPUs, with estimated wall-clock times. The estimates also omit communication for global orthogonalization, a central motivation of the method. Actual multi-node measurements needed.
  4. Limited support for the selection score beyond the tested architecture. Comparing descent bounds does not guarantee better actual progress, and the estimated smoothness values include gradient noise and changes in other layers. The ablations support selecting query and key matrices, but leave unclear whether costly profiling offers an advantage over simply fixing that assignment.
Questions for authors

N/A

1 comment

Dear Reviewer 2kEr,

We would like to kindly remind you that we have posted our rebuttal, in which we respond to each of your concerns. In addition to the rebuttal, we have uploaded a revised PDF with all changes shown in blue. We would be grateful for your feedback and we are happy to discuss any further questions.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 36827

Reviewer_8Z8FOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MuonMix, a distributed optimizer that combines spectral-norm and max-norm linear minimization oracles within MuonBP’s block-periodic framework. Selected hidden weight matrices use element-wise Signum updates, avoiding their Newton–Schulz computation and optimizer-specific gather/scatter operations. A layer-selection heuristic motivated by descent bounds assigns query and key projections to Signum. Experiments include measured 160M training on eight GPUs and virtual-sharding experiments for 960M and 1.2B models, with performance-model estimates for the corresponding larger clusters.

Critical strengths
  1. The proposed mechanism is clear and practically implementable. Elementwise Signum commutes with shard concatenation, so selected matrices avoid optimizer-specific gathering and orthogonalization while producing the same direction as full-matrix Signum. This provides a concrete motivation for combining update geometries.

  2. The paper provides useful experimental support beyond the main comparison. In the measured 160M setting, MuonMix improves throughput by approximately 3% over MuonBP and reduces time to selected validation-loss targets. Appendix A.4 also evaluates substitutions across attention and MLP matrix types, supporting the Q/K assignment in the tested architecture. Experiments with AdamW provide an additional practical extension.

Critical weaknesses
  1. Assumption 1 appears to exclude cross-layer coupling. For arbitrary full parameter states, the assumption requires
\leq L_{\ell}\left\Vert x^{\ell}-y^{\ell} \right\Vert_{\ell}.$$ Consequently, keeping one layer's parameters fixed forces its gradient to remain unchanged even when other layers change. This is substantially stronger than conventional block smoothness with other blocks fixed and does not generally describe a coupled Transformer objective. The assumption is used to control gradient drift in Appendix B and derive the joint-update descent inequality in Eq. B.18. If only the current layer is intended to vary, those steps require additional treatment of cross-layer effects. This limits the theory's applicability, without necessarily invalidating the theorem under its literal assumptions. 2. **The large-scale efficiency claims rely on performance estimates rather than measured distributed execution.** Actual end-to-end measurements cover the 160M model on one eight-GPU node. Virtual sharding can investigate optimization behavior under finer partitions, but cannot reproduce the execution characteristics of a real 128/256-GPU cluster. Larger-model throughput and time-to-target results share the same estimated step times. The model adds computation and communication without overlap, excludes global orthogonalization communication, and transfers efficiency factors from other GPU configurations. These choices need empirical calibration before the estimated gains can substantiate deployment-level scalability claims. 3. **The complete procedure has not demonstrated a net efficiency benefit after layer selection.** Appendix A.1 reports approximately 9,390.5 seconds for the Q-projection profiling configuration, whereas the reported 160M training saves approximately 80 seconds over 5,000 steps. These runs use different GPU counts, so their wall times cannot directly determine amortization. Nevertheless, the discrepancy makes profiling cost central to the claimed efficiency. The paper does not establish the total profiling GPU-hours, the conditions under which assignments can be reused, or the break-even workload. Since all reported model sizes ultimately use Q/K assignments, the benefit of costly profiling over a fixed Q/K rule also remains unclear. 4. **The theoretical motivation does not establish the practical superiority of the layer-selection rule.** Comparing descent lower bounds does not order actual decreases, and cancellation of other layers’ contributions requires a fixed reference trajectory. Switching optimizers changes that trajectory. Furthermore, the empirical secant ratios are not certified smoothness bounds, as Appendix C.4 acknowledges. Theorem 1 also assumes exact LMOs, whereas the implementation uses NS5 without an accompanying approximation-error analysis. Finally, a partition-dependent worst-case bound does not establish that actual optimization error increases with device count. These gaps leave the selection rule as a theoretically motivated heuristic whose comparative value requires broader validation.
Questions for authors
  1. Does Assumption 1 allow other layers to change simultaneously? If so, how can it accommodate cross-layer coupling? If it is intended as conventional block smoothness, how should the momentum-drift and joint-descent arguments be modified?

  2. Can the performance model be calibrated against measured configurations on the available hardware? How do predicted gains change when accounting for communication overlap and global orthogonalization communication?

  3. What is the complete profiling cost in GPU-hours, under what conditions can assignments be reused, and when does the procedure break even? Does profiling improve upon a fixed Q/K assignment across different architectures or partitioning configurations?

  4. What guarantees survive the use of empirical smoothness proxies and NS5 updates? Can the authors distinguish the formal fixed-trajectory comparison from evidence that the score predicts actual benefits after multiple matrices are reassigned?

1 comment

Dear Reviewer 8Z8F,

This is a gentle reminder regarding our rebuttal. We have carefully addressed your comments and concerns in our response. We have also uploaded a revised version of the paper with all changes marked in blue. We would greatly appreciate it if you could review our rebuttal and the revised paper when convenient. We welcome any further discussion and would be glad to address any remaining concerns.

Thank you very much for your time and consideration.

Best regards,

The Authors of Submission 36827

Comments on the paper

1 comment

Dear Reviewer,

We would like to gently remind you once more that we have submitted our rebuttal and a revised version of the paper. All changes in the revised paper are highlighted in blue. While the discussion period is still open, we would greatly appreciate it if you could let us know whether our responses address your concerns. Any further questions or comments are very welcome.

Thank you very much for your time and consideration.

Best regards,

The Authors of Submission 36827

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.