- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
the authors propose MeqMuon, a variant of the Muon optimizer that, after the Newton–Schulz procedure, normalizes either the rows or the columns, choosing whichever has the larger coefficient of variation (CV). for embeds and the lm head, it replaces the Adamw update with a two-sided normalization of the unorthogonalized momentum, and for 1D parameters it uses rms-normalized momentum. thus, they remove all Adamw second-moment state. the authors show that MeqMuon is superior over Muon and AdamW on C4 pretraining of different models (Llama, Qwen, SmolLM)
Critical strengths
S1: he row/column CV analysis shows that orthogonalization equalizes norms only on the smaller dim of the matrix and leaves the larger one imbalanced. this motivates the method well
S1: since they can remove one buffer of AdamW, they get a real memory saving
Critical weaknesses
W1: MeqMuon changes the update scale of embeds, the lm head and 1D parameters, yet weight decay is fixed at 0.1 for all methods and LRs are not reported. without model norm and update norm plots, it is impossible to tell whether the ppl gains come from the normalization itself or from a different effective LR / equilibrium norm
W2: all runs use the chinchilla horizon which is too short for thorough comparison. one can easily pick very large LR and outperform all other methods. it would be cool to include at least 5-8 x Chinchilla tokens runs
W3: Pethick et al., 2025, arXiv:2502.07529 uses column- and row-norm updates for embeds and the lm head without Adam buffers. SWAN (Ma et al., 2024, arXiv:2412.13148) combines row normalization with whitening. SinkGD (Scetbon et al., 2025, arXiv:2502.06742) alternates row and column normalization. norm-controlling optimizers: HyperBall (Wen et al., 2026) and MuonMD (Hägele et al., 2026, arXiv:2606.25971) control weight and update norms directly and are natural baseline. none of these are cited or compared against
Questions for authors
Q1: what LRs were used for each method and model, and how were they tuned?
Q2: could the authors please plot weight norms and update rms over training for Muon, NorMuon and MeqMuon, and compare against at least one norm-controlling baseline (MuonMD or HyperBall)?
Q3: could the authors compare against Scion’s embed/head normalization rules and against SinkGD?
addressing main points regarding comparisons with missed baseline would enhance the score significantly