acceptodds
Withdrawn from ICLR 2027

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

18 views

Abstract

The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM pretraining. Recent work introduces row-wise normalization into Muon to balance update magnitudes and improve pretraining performance. However, row-wise normalization alone cannot accommodate different imbalance patterns in update matrices. In this paper, we propose an improved Muon optimizer, called matrix-equilibrating Muon (), for LLM pretraining. balances both row and column magnitudes through normalization that can be automatically tailored to different imbalance patterns without manual intervention. Moreover, eliminates the need to store AdamW's second-moment estimates, reducing optimizer-state memory usage. Empirical results demonstrate that achieves better convergence performance than AdamW, Muon, and other baselines in LLM pretraining.

Primary area:
optimization
Keywords:
LLM pretraining, optimizer, Muon

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hzyZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

the authors propose MeqMuon, a variant of the Muon optimizer that, after the Newton–Schulz procedure, normalizes either the rows or the columns, choosing whichever has the larger coefficient of variation (CV). for embeds and the lm head, it replaces the Adamw update with a two-sided normalization of the unorthogonalized momentum, and for 1D parameters it uses rms-normalized momentum. thus, they remove all Adamw second-moment state. the authors show that MeqMuon is superior over Muon and AdamW on C4 pretraining of different models (Llama, Qwen, SmolLM)

Critical strengths

S1: he row/column CV analysis shows that orthogonalization equalizes norms only on the smaller dim of the matrix and leaves the larger one imbalanced. this motivates the method well

S1: since they can remove one buffer of AdamW, they get a real memory saving

Critical weaknesses

W1: MeqMuon changes the update scale of embeds, the lm head and 1D parameters, yet weight decay is fixed at 0.1 for all methods and LRs are not reported. without model norm and update norm plots, it is impossible to tell whether the ppl gains come from the normalization itself or from a different effective LR / equilibrium norm

W2: all runs use the chinchilla horizon which is too short for thorough comparison. one can easily pick very large LR and outperform all other methods. it would be cool to include at least 5-8 x Chinchilla tokens runs

W3: Pethick et al., 2025, arXiv:2502.07529 uses column- and row-norm updates for embeds and the lm head without Adam buffers. SWAN (Ma et al., 2024, arXiv:2412.13148) combines row normalization with whitening. SinkGD (Scetbon et al., 2025, arXiv:2502.06742) alternates row and column normalization. norm-controlling optimizers: HyperBall (Wen et al., 2026) and MuonMD (Hägele et al., 2026, arXiv:2606.25971) control weight and update norms directly and are natural baseline. none of these are cited or compared against

Questions for authors

Q1: what LRs were used for each method and model, and how were they tuned?

Q2: could the authors please plot weight norms and update rms over training for Muon, NorMuon and MeqMuon, and compare against at least one norm-controlling baseline (MuonMD or HyperBall)?

Q3: could the authors compare against Scion’s embed/head normalization rules and against SinkGD?

addressing main points regarding comparisons with missed baseline would enhance the score significantly

Reviewer_792nOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MeqMuon addresses row/column RMS imbalances in Muon by adaptively choosing row or column normalization for hidden layers via coefficient-of-variation comparisons. For embeddings and output heads, it replaces AdamW with two-sided (row-then-column) momentum normalization and applies global-RMS scaling to 1D weights. By removing second-moment buffers entirely, it lowers optimizer memory while achieving superior validation perplexity across Llama, SmolLM2, and Qwen2 models (up to 0.5B) pretrained on C4.

Critical strengths

The paper's motivation is solid and goes beyond the usual practice of normalizing rows.

Critical weaknesses
  1. The reported gains over NorMuon are marginal (e.g., 15.98 to 15.94 on Llama-350M, 17.20 to 17.16 on SmolLM2-360M in Table 4) and difficult to evaluate without multi-seed error bars. Crucially, the paper lacks explicit details on hyperparameter tuning—such as search grids and peak learning rates—making it unclear whether these narrow margins represent genuine algorithmic progress or slight tuning discrepancies. Additionally, while the reduction in optimizer-state memory is documented, the paper omits peak training memory and end-to-end wall-clock speedups, leaving the broader efficiency claims incomplete.

  2. The necessity of dynamic direction selection remains unverified. Table 7 conflates normalization with direction selection, while Table 6 reveals that most modules exhibit nearly static directional preferences (e.g., rectangular MLP matrices consistently select the same side). To justify whether runtime per-step CV computation is truly necessary, the authors should ablate against: (1) a static rule based strictly on matrix shape/type, (2) purely row-only and column-only baselines, and (3) an RMS-matched control. Furthermore, because square-matrix imbalances stem from finite Newton–Schulz steps, evaluating a higher-accuracy polar factor would help decouple truncation error from the proposed correction. Ultimately, reducing CV is insufficient without demonstrating that this specific dynamic mechanism drives optimization performance.

Questions for authors

NA

Reviewer_rS6fOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MeqMuon augments Muon with CV-based row/column normalization and replaces AdamW on the remaining parameters with normalized momentum. The paper reports lower validation perplexity and reduced optimizer-state memory across several small language models.

Critical strengths

The method is simple to implement, removes second-moment buffers, and is evaluated across three model families.

Critical weaknesses
  1. The central contribution is insufficiently validated. Table 7 ablates parameter groups but does not establish whether adaptive CV-based selection outperforms fixed row-wise, column-wise, or module-specific normalization. Table 6 shows strongly predictable selection patterns, leaving the necessity of per-step adaptation unclear.

  2. The experimental evidence does not establish a reliable advantage. Peak learning rates and their selection procedure are not reported, preventing assessment of baseline tuning. Several gains are small, such as 22.81 versus 22.80 PPL on SmolLM2-135M, yet no repeated-run variability is provided.

Questions for authors
  1. What peak learning rates and tuning budgets were used for each optimizer and model?

  2. Do the small improvements over Muon and NorMuon persist across multiple seeds?

  3. Does CV-based adaptation outperform fixed row/column and module-specific rules when the remaining optimizer components are held identical?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.