- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates what Muon gains from the matrix representation of momentum beyond vector normalization. Muse varies the representation used for the spectral update while keeping parameters and momentum in their original coordinates. Experiments on 130M/600M language models show that ordered reshaping can preserve Muon's advantage over nSGDM, whereas fixed-shape entry shuffling can remove it. The theory decomposes single-step progress into nuclear support, alignment, and rank cost, and establishes stationarity guarantees with explicit NS5 constants. Under heterogeneous row-block curvature, the analysis gives tighter nuclear-stationarity bounds for exact-polar Muon than for nSGDM, and conditionally for NS5.
Critical strengths
-
S1. A useful separation of shape and entry organization. Ordered reshaping retains Muon's advantage over Vector, while fixed-shape shuffling removes it. The main contrast persists with fixed auxiliary AdamW at 600M, and exact-polar controls show that the effect is not merely an NS5 artifact. This is an interesting empirical finding.
-
S2. A concrete curvature-sensitive analysis. The paper connects spectral row-energy constraints to averaged rather than maximum curvature costs, allows cross-group coupling, and derives explicit constants for the prescribed NS5 polynomial. These results provide a sufficient theoretical mechanism for improved bound coefficients.
Critical weaknesses
-
W1. The theoretical comparison depends on the stationarity measure and proof relaxation. Proposition 3.4 compares nuclear-stationarity upper bounds after relaxing nSGDM's Frobenius first-order term. Retaining that term gives a Frobenius-stationarity bound with coefficient for nSGDM, while converting the polar bound gives , with for . Although the stated nuclear-norm comparison is valid, its ordering depends on the stationarity measure and the relaxations used. This limits how strongly it can explain Muon's empirical advantage without a sharper, effective-rank-aware comparison.
-
W2. The proposed mechanism is not specific enough to explain Muon's gains. The row-energy and alignment argument also yields the bound for rescaled RMNP [1] under common trajectory-wise curvature bounds. In the quadratic experiments, RMNP achieves lower selected final loss than both Native variants for every tested , but Muse does not include RMNP in its language-model comparisons. While Hessian block heterogeneity has been documented previously [2], the language-model probes here do not measure curvature along competing update directions or verify the trajectory-wise assumptions used in the theory. Moreover, Appendix K reports higher first-order action for Skinny at intermediate checkpoints despite its worse final validation loss. The evidence therefore does not isolate spectral coupling from row-wise normalization or establish which mechanism explains the language-model gains.
-
W3. Representation ambiguity and uneven tuning weaken the finer shape comparisons. At 600M, each Skinny variant has only one candidate in the three-seed selection grid and is absent from the fixed-AdamW comparison. The unspecified divisor tie-break can make Skinny-75% and Skinny-50% identical, consistent with their identical diagnostics in Table 12. The precise transposition/flattening order is also unclear. These issues weaken the finer shape comparisons, although the Native/Square–Vector contrast remains convincing.
Questions for authors
-
Q1 (W1). Can Proposition 3.4 retain nSGDM's Frobenius first-order term and incorporate effective rank to obtain a more informative comparison? In which nontrivial rank regime does the resulting analysis predict an advantage for Native, and does that prediction remain informative when stationarity is measured in the Frobenius norm?
-
Q2 (W2). Does the analysis also apply to rescaled RMNP [1], and can independently tuned RMNP match Muon's language-model gains? At shared LLM checkpoints, could the authors extend Figure 14's normalized first-order action analysis to shuffled Muon and RMNP, and compare directional curvature , squared Hessian-action magnitude , and native-row energy allocation across attention and MLP blocks? Can these measurements distinguish row-wise normalization from additional spectral effects?
-
Q3 (W3). What are the exact per-module representations and the Skinny divisor tie-break? Are Skinny-75% and Skinny-50% identical at 600M? For distinct variants, does the ordering persist after matrix-rate tuning with fixed auxiliary AdamW? Please also clarify run reuse across Tables 6–8.
Required Corrections
-
Square asymptotics (§3.2, lines 225–229; Remark F.1, lines 1385–1388). The Square-to-Vector support comparison requires , which does not follow from choosing the largest divisor below . The constant additionally requires an asymptotically square representation.
-
Representation definition (§2.1, lines 129–140). Specify the deterministic Skinny tie-break and the precise transposition/flattening order. Identify any labels implementing identical transformations.
-
Learning-rate list (§I.3, lines 1794–1800). Add to the 130M fixed-shuffle candidates, consistent with §I.6 and Figure 10.
-
Typographical error (Figure 3, lines 419–423). Change “Replicated eight-neuron MLP Zhang et al.” to “Replication of the eight-neuron MLP setup of Zhang et al. (2024, Appendix D.3).”
References
[1] S. Deng et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306:24191–24239, 2026.
[2] Y. Zhang et al. Why Transformers Need Adam: A Hessian Perspective. Advances in Neural Information Processing Systems (NeurIPS), 2024.
8 comments
W2 / Q2: RMNP’s shared energy mechanism and 600M/shared-state results
A2: For momentum, theory-consistent RMNP uses , zero for zero rows. Then , , and .
Under Proposition 3.4's common curvature assumptions, RMNP has exact polar's -type coefficient (Appendix E.5, lines 1350–1362). They control row-group update energy by row-wise normalization and matrix spectral transformation, respectively.
600M training. RMNP uses the original configuration without Nesterov or weight decay, independently sweeping coupled base learning rates 0.002–0.01 and selecting 0.006/0.006.
| Method | Validation loss (mean ± SD, 3 seeds) |
|---|---|
| Native Muon | 2.6684±0.0014 |
| RMNP | 2.7110±0.0073 |
| Vector / nSGDM | 2.7491±0.0020 |
RMNP selects on one seed; Native/Vector use three-seed mean terminal loss; final statistics include selection seeds (Appendix L, Table 15, lines 2635–2653). RMNP clearly outperforms nSGDM; Native Muon has the lowest validation loss.
Shared-state diagnostics. Fixing parameters, momentum and validation prefixes of 512 input tokens each, we map updates to original coordinates, normalize to unit Frobenius norm and measure signed , and , where , is the prefix Hessian and its token weight.
Layer 16, step 9600, eight shared 512-token validation prefixes:
| Module | Update direction | |||
|---|---|---|---|---|
| Q | Native / Square | −6.754 | 0.000874 | 0.000921 |
| Q | Fixed Shuffle | −215.785 | 0.7038 | 0.7120 |
| Q | RMNP | −79.078 | 0.3264 | 0.3357 |
| Q | Vector | −253.322 | 1 | 1 |
| MLP down | Native | 0.295 | 0.1224 | 0.1469 |
| MLP down | Square | −1.159 | 0.1122 | 0.1346 |
| MLP down | Fixed Shuffle | −2.116 | 0.9442 | 0.9446 |
| MLP down | RMNP | −1.908 | 0.9481 | 0.9538 |
| MLP down | Vector | −1.870 | 1 | 1 |
These per-module unit-norm results are separate from the joint full-model diagnostics with module learning-rate multipliers (Appendix K, Fig. 15).
For Q, RMNP substantially reduces relative to Vector, while Native/Square responses are about three orders of magnitude lower than Vector's; in MLP-down, RMNP is closer to Vector. Shared row-energy control can improve local curvature responses; Muon's matrix spectral transformation produces directional differences that row-wise normalization cannot reproduce.
- Energy exchange: Across Q/K at all 32 layers and four checkpoints, we exchange Native/Vector attention-head update energies, fixing within-group directions. Row-group energy differences contribute a median 17.5% (Q) and 19.1% (K) to their directional-curvature gap (Appendix M.1, lines 2742–2746).
- Curvature relocation: For 36 states each in Q and K, we fix the Hessian spectrum, update directions, and first-order action , changing only attention-head curvature-block placement. Median falls 6.680→2.204 (Q) and 1.086→0.249 (K); random relocation yields 4.377 and 0.586, respectively (Appendix M.1, lines 2750–2760).
These results distinguish row-group energy allocation, curvature correspondence, and update directions, providing direct model evidence for Muse's curvature–energy mechanism and its link to spectral transformations.
Dear Reviewer MpFC,
Thank you again for your thoughtful review. We have provided a detailed response and additional experimental evidence, including new RMNP comparisons and further analyses of the proposed mechanism.
We would appreciate any further feedback you may have and would be happy to clarify any remaining questions or concerns.
Thank you for your time and consideration.
We thank the reviewer for the constructive comments. The revisions in the manuscript are marked in blue. We have added an effective-rank comparison, RMNP training results, and shared-state diagnostics, and clarified representation mappings and learning-rate selection.
W1 / Q1: Can retaining the Frobenius first-order term yield an effective-rank-dependent comparison?
A1: In Appendix E.5 (lines 1329–1349), we retain nSGDM's Frobenius first-order decrease term and introduce effective rank to compare the two methods under the same Frobenius stationarity measure.
Let BK=2(1+lnK)/K1/4 and reff(G)=∣∣G∣∣∗2/∣∣G∣∣F2. Under the common assumptions of Proposition 3.4, after separately optimizing the base learning rates, nSGDM satisfies mint≤KE∣∣GtnSGDM∣∣F≤BKRC3.
For exact-polar Native, suppose its trajectory at the optimized learning rate satisfies E∣∣GtP∣∣∗≥r0E∣∣GtP∣∣F (1≤t≤K), where r0∈[1,p]. Then
\le B_K\sqrt{\frac{pRC_2}{r_0}}.$$ A sufficient condition for the trajectory condition above is $r_{\mathrm{eff}}(G_t^P)\ge r_0$ at every step (almost surely for nonzero gradients). Therefore, a sufficient condition for exact-polar Native to have a tighter upper bound under Frobenius stationarity is $r_0>pC_2/C_3$. For $p>1$ and heterogeneous row-group curvature, $1<pC_2/C_3<p$. Therefore, in the nonempty high-effective-rank interval $(pC_2/C_3,p]$, exact-polar Native's theoretical advantage still holds under the Frobenius measure. This result specifies the effective-rank condition required to extend the nuclear-stationarity comparison in the original Proposition 3.4 to Frobenius stationarity. The original comparison condition for NS5, $C_1<C_3$, remains unchanged (Proposition 3.4, lines 322–341). --- **W3 / Q3**: Exact reshape mappings, Skinny tuning, and run reuse across tables **A3:** **Representation mappings.** For $d=mn$, Square selects the largest valid divisor $p$ no greater than $\sqrt d$; Skinny uses $\lfloor\theta p+1/2\rfloor$ as its target and selects the closest valid divisor no greater than $p$, choosing the smaller divisor in the event of a tie. Here, $\theta\in\{1/8,1/4,1/2,3/4\}$ (section 2.1, lines 133–146). The actual mappings for the 600M model are as follows (Appendix I.1, lines 1713–1719): |Module|Native|Square|Skinny-75% / 50%|Skinny-25%| |---|---|---|---|---| |Q/K/V/O|$1024\times1024$|$1024\times1024$|$512\times2048$|$256\times4096$| |Gate/Up|$4096\times1024$|$2048\times2048$|$1024\times4096$|$512\times8192$| |Down|$1024\times4096$|$2048\times2048$|$1024\times4096$|$512\times8192$| Vector is $1\times d$. Native preserves the original matrix layout, transposing as needed for the NS5 computation and restoring it before returning; the other ordered representations follow the original logical row-major order for flatten → reshape → NS5 → inverse reshape. Parameters and momentum are always stored in original coordinates. Skinny-75% and Skinny-50% have exactly the same shapes and index mappings for all 224 matrices in the 600M model, and the same learning rate of 0.001 is selected. Therefore, their small difference in validation loss does not represent an independent representation effect. **Learning-rate selection.** In Fig. 9, for each 600M Skinny representation under native-dimension scaling, 0.001 has three seeds, while 0.002, 0.005, and 0.01 each have two seeds. Table 9 selects the learning rate using the mean terminal loss over three seeds; therefore, the three-seed Skinny comparison selects 0.001, while the remaining candidates serve as an additional learning-rate sweep (Fig. 9, lines 2091–2097). Under the fixed-auxiliary-AdamW protocol, the different 130M Skinny variants retain the same ordering (Table 7). For 600M, Table 7 covers Native, Square, Vector, and Shuffle, yielding the same core ordering as the coupled-rate protocol: Native/Square outperform Vector, and Shuffle further reduces performance. The 600M Skinny comparison is provided by the coupled-rate sweep. **Run reuse across tables.** In the revised manuscript, Tables 8/9 expand the per-seed results of Tables 1/2, respectively, using the same training trajectories; Table 7 uses an experimental protocol with fixed auxiliary AdamW and independently selected matrix learning rates. Thus, the results under the two learning-rate protocols and their per-seed breakdowns have a clear correspondence, and the main Native/Square–Vector and Shuffle comparisons remain consistent across both protocols.