acceptodds
Under review as a conference paper at ICLR 2027

Muse: Representation Geometry of Muon Beyond Normalized Momentum

Abstract

Muon applies a spectral transformation to matrix momentum, while normalized SGD with momentum (nSGDM) normalizes its vectorized form. What does the matrix representation contribute beyond vector normalization? We introduce Muse to investigate this question by varying the matrix representation used to compute the update. In 130M and 600M language models, order-preserving reshaping retains lower validation loss than nSGDM. However, fixed-shape entry shuffling removes Muon's advantage over nSGDM in a controlled row-block problem and 600M language-model training. To illustrate how representation affects optimization, we first show that single-step parameter-error progress depends on both nuclear support and update alignment. We then analyze convergence under heterogeneous row-block curvature. Muon's spectral constraint limits the update energy in each row, yielding bounds that depend on average row curvature, whereas the corresponding nSGDM bounds depend on the maximum. We establish stationarity guarantees for exact polar and fixed-iteration Newton–Schulz updates. Under heterogeneous native row-block curvature, the optimized bound is tighter for exact polar than for nSGDM; we also identify conditions under which this advantage extends to the five-step update.

Primary area:
optimization
Keywords:
Muon, Matrix Representation, Polar Optimization, Nuclear Support, Language Model Pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MpFCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates what Muon gains from the matrix representation of momentum beyond vector normalization. Muse varies the representation used for the spectral update while keeping parameters and momentum in their original coordinates. Experiments on 130M/600M language models show that ordered reshaping can preserve Muon's advantage over nSGDM, whereas fixed-shape entry shuffling can remove it. The theory decomposes single-step progress into nuclear support, alignment, and rank cost, and establishes O~(K−1/4)\widetilde O(K^{-1/4}) stationarity guarantees with explicit NS5 constants. Under heterogeneous row-block curvature, the analysis gives tighter nuclear-stationarity bounds for exact-polar Muon than for nSGDM, and conditionally for NS5.

Critical strengths
  • S1. A useful separation of shape and entry organization. Ordered reshaping retains Muon's advantage over Vector, while fixed-shape shuffling removes it. The main contrast persists with fixed auxiliary AdamW at 600M, and exact-polar controls show that the effect is not merely an NS5 artifact. This is an interesting empirical finding.

  • S2. A concrete curvature-sensitive analysis. The paper connects spectral row-energy constraints to averaged rather than maximum curvature costs, allows cross-group coupling, and derives explicit constants for the prescribed NS5 polynomial. These results provide a sufficient theoretical mechanism for improved bound coefficients.

Critical weaknesses
  • W1. The theoretical comparison depends on the stationarity measure and proof relaxation. Proposition 3.4 compares nuclear-stationarity upper bounds after relaxing nSGDM's Frobenius first-order term. Retaining that term gives a Frobenius-stationarity bound with coefficient RC3\sqrt{RC_3} for nSGDM, while converting the polar bound gives pRC2\sqrt{pRC_2}, with pC2>C3pC_2>C_3 for p>1p>1. Although the stated nuclear-norm comparison is valid, its ordering depends on the stationarity measure and the relaxations used. This limits how strongly it can explain Muon's empirical advantage without a sharper, effective-rank-aware comparison.

  • W2. The proposed mechanism is not specific enough to explain Muon's gains. The row-energy and alignment argument also yields the C2C_2 bound for rescaled RMNP [1] under common trajectory-wise curvature bounds. In the quadratic experiments, RMNP achieves lower selected final loss than both Native variants for every tested ρ≥10\rho\geq10, but Muse does not include RMNP in its language-model comparisons. While Hessian block heterogeneity has been documented previously [2], the language-model probes here do not measure curvature along competing update directions or verify the trajectory-wise assumptions used in the theory. Moreover, Appendix K reports higher first-order action for Skinny at intermediate checkpoints despite its worse final validation loss. The evidence therefore does not isolate spectral coupling from row-wise normalization or establish which mechanism explains the language-model gains.

  • W3. Representation ambiguity and uneven tuning weaken the finer shape comparisons. At 600M, each Skinny variant has only one candidate in the three-seed selection grid and is absent from the fixed-AdamW comparison. The unspecified divisor tie-break can make Skinny-75% and Skinny-50% identical, consistent with their identical diagnostics in Table 12. The precise transposition/flattening order is also unclear. These issues weaken the finer shape comparisons, although the Native/Square–Vector contrast remains convincing.

Questions for authors
  • Q1 (W1). Can Proposition 3.4 retain nSGDM's Frobenius first-order term and incorporate effective rank to obtain a more informative comparison? In which nontrivial rank regime does the resulting analysis predict an advantage for Native, and does that prediction remain informative when stationarity is measured in the Frobenius norm?

  • Q2 (W2). Does the C2C_2 analysis also apply to rescaled RMNP [1], and can independently tuned RMNP match Muon's language-model gains? At shared LLM checkpoints, could the authors extend Figure 14's normalized first-order action analysis to shuffled Muon and RMNP, and compare directional curvature ⟨D,HD⟩F/∣∣D∣∣F2\langle D,HD\rangle_F/||D||_F^2, squared Hessian-action magnitude ∣∣HD∣∣F2/∣∣D∣∣F2||HD||_F^2/||D||_F^2, and native-row energy allocation across attention and MLP blocks? Can these measurements distinguish row-wise normalization from additional spectral effects?

  • Q3 (W3). What are the exact per-module representations and the Skinny divisor tie-break? Are Skinny-75% and Skinny-50% identical at 600M? For distinct variants, does the ordering persist after matrix-rate tuning with fixed auxiliary AdamW? Please also clarify run reuse across Tables 6–8.

Required Corrections

  • Square asymptotics (§3.2, lines 225–229; Remark F.1, lines 1385–1388). The Θ(d1/4)\Theta(d^{1/4}) Square-to-Vector support comparison requires pSquare=Θ(d)p_{\mathrm{Square}}=\Theta(\sqrt d), which does not follow from choosing the largest divisor below d\sqrt d. The constant β(1)\beta(1) additionally requires an asymptotically square representation.

  • Representation definition (§2.1, lines 129–140). Specify the deterministic Skinny tie-break and the precise transposition/flattening order. Identify any labels implementing identical transformations.

  • Learning-rate list (§I.3, lines 1794–1800). Add 0.00050.0005 to the 130M fixed-shuffle candidates, consistent with §I.6 and Figure 10.

  • Typographical error (Figure 3, lines 419–423). Change “Replicated eight-neuron MLP Zhang et al.” to “Replication of the eight-neuron MLP setup of Zhang et al. (2024, Appendix D.3).”

References

[1] S. Deng et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306:24191–24239, 2026.

[2] Y. Zhang et al. Why Transformers Need Adam: A Hessian Perspective. Advances in Neural Information Processing Systems (NeurIPS), 2024.

8 comments

We thank the reviewer for the constructive comments. The revisions in the manuscript are marked in blue. We have added an effective-rank comparison, RMNP training results, and shared-state diagnostics, and clarified representation mappings and learning-rate selection.

W1 / Q1: Can retaining the Frobenius first-order term yield an effective-rank-dependent comparison?

A1: In Appendix E.5 (lines 1329–1349), we retain nSGDM's Frobenius first-order decrease term and introduce effective rank to compare the two methods under the same Frobenius stationarity measure.

Let BK=2(1+ln⁡K)/K1/4B_K=2(1+\ln K)/K^{1/4} and reff(G)=∣∣G∣∣∗2/∣∣G∣∣F2r_{\mathrm{eff}}(G)=||G||_*^2/||G||_F^2. Under the common assumptions of Proposition 3.4, after separately optimizing the base learning rates, nSGDM satisfies min⁡t≤KE∣∣GtnSGDM∣∣F≤BKRC3\min_{t\le K}\mathbb E||G_t^{\mathrm{nSGDM}}||_F\le B_K\sqrt{RC_3}.

For exact-polar Native, suppose its trajectory at the optimized learning rate satisfies E∣∣GtP∣∣∗≥r0 E∣∣GtP∣∣F\mathbb E||G_t^P||_*\ge\sqrt{r_0}\,\mathbb E||G_t^P||_F (1≤t≤K1\le t\le K), where r0∈[1,p]r_0\in[1,p]. Then

\le B_K\sqrt{\frac{pRC_2}{r_0}}.$$ A sufficient condition for the trajectory condition above is $r_{\mathrm{eff}}(G_t^P)\ge r_0$ at every step (almost surely for nonzero gradients). Therefore, a sufficient condition for exact-polar Native to have a tighter upper bound under Frobenius stationarity is $r_0>pC_2/C_3$. For $p>1$ and heterogeneous row-group curvature, $1<pC_2/C_3<p$. Therefore, in the nonempty high-effective-rank interval $(pC_2/C_3,p]$, exact-polar Native's theoretical advantage still holds under the Frobenius measure. This result specifies the effective-rank condition required to extend the nuclear-stationarity comparison in the original Proposition 3.4 to Frobenius stationarity. The original comparison condition for NS5, $C_1<C_3$, remains unchanged (Proposition 3.4, lines 322–341). --- **W3 / Q3**: Exact reshape mappings, Skinny tuning, and run reuse across tables **A3:** **Representation mappings.** For $d=mn$, Square selects the largest valid divisor $p$ no greater than $\sqrt d$; Skinny uses $\lfloor\theta p+1/2\rfloor$ as its target and selects the closest valid divisor no greater than $p$, choosing the smaller divisor in the event of a tie. Here, $\theta\in\{1/8,1/4,1/2,3/4\}$ (section 2.1, lines 133–146). The actual mappings for the 600M model are as follows (Appendix I.1, lines 1713–1719): |Module|Native|Square|Skinny-75% / 50%|Skinny-25%| |---|---|---|---|---| |Q/K/V/O|$1024\times1024$|$1024\times1024$|$512\times2048$|$256\times4096$| |Gate/Up|$4096\times1024$|$2048\times2048$|$1024\times4096$|$512\times8192$| |Down|$1024\times4096$|$2048\times2048$|$1024\times4096$|$512\times8192$| Vector is $1\times d$. Native preserves the original matrix layout, transposing as needed for the NS5 computation and restoring it before returning; the other ordered representations follow the original logical row-major order for flatten → reshape → NS5 → inverse reshape. Parameters and momentum are always stored in original coordinates. Skinny-75% and Skinny-50% have exactly the same shapes and index mappings for all 224 matrices in the 600M model, and the same learning rate of 0.001 is selected. Therefore, their small difference in validation loss does not represent an independent representation effect. **Learning-rate selection.** In Fig. 9, for each 600M Skinny representation under native-dimension scaling, 0.001 has three seeds, while 0.002, 0.005, and 0.01 each have two seeds. Table 9 selects the learning rate using the mean terminal loss over three seeds; therefore, the three-seed Skinny comparison selects 0.001, while the remaining candidates serve as an additional learning-rate sweep (Fig. 9, lines 2091–2097). Under the fixed-auxiliary-AdamW protocol, the different 130M Skinny variants retain the same ordering (Table 7). For 600M, Table 7 covers Native, Square, Vector, and Shuffle, yielding the same core ordering as the coupled-rate protocol: Native/Square outperform Vector, and Shuffle further reduces performance. The 600M Skinny comparison is provided by the coupled-rate sweep. **Run reuse across tables.** In the revised manuscript, Tables 8/9 expand the per-seed results of Tables 1/2, respectively, using the same training trajectories; Table 7 uses an experimental protocol with fixed auxiliary AdamW and independently selected matrix learning rates. Thus, the results under the two learning-rate protocols and their per-seed breakdowns have a clear correspondence, and the main Native/Square–Vector and Shuffle comparisons remain consistent across both protocols.

W2 / Q2: RMNP’s shared energy mechanism and 600M/shared-state results

A2: For p×qp\times q momentum, theory-consistent RMNP uses Ui,:=Mi,:/(p ∣∣Mi,:∣∣2)U_{i,:}=M_{i,:}/(\sqrt p\,||M_{i,:}||_2), zero for zero rows. Then ∣∣UIj,:∣∣F2≤∣Ij∣/p||U_{I_j,:}||_F^2\le |I_j|/p, ∣∣U∣∣F≤1||U||_F\le1, and ⟨G,U⟩F≥∣∣G∣∣∗/p−2∣∣M−G∣∣F\langle G,U\rangle_F\ge||G||_*/\sqrt p-2||M-G||_F.

Under Proposition 3.4's common curvature assumptions, RMNP has exact polar's C2C_2-type coefficient (Appendix E.5, lines 1350–1362). They control row-group update energy by row-wise normalization and matrix spectral transformation, respectively.

600M training. RMNP uses the original configuration without Nesterov or weight decay, independently sweeping coupled base learning rates 0.002–0.01 and selecting 0.006/0.006.

MethodValidation loss (mean ± SD, 3 seeds)
Native Muon2.6684±0.0014
RMNP2.7110±0.0073
Vector / nSGDM2.7491±0.0020

RMNP selects on one seed; Native/Vector use three-seed mean terminal loss; final statistics include selection seeds (Appendix L, Table 15, lines 2635–2653). RMNP clearly outperforms nSGDM; Native Muon has the lowest validation loss.

Shared-state diagnostics. Fixing parameters, momentum and validation prefixes of 512 input tokens each, we map updates to original coordinates, normalize to unit Frobenius norm and measure signed a=⟨g,u⟩Fa=\langle g,u\rangle_F, q=⟨u,Hu⟩Fq=\langle u,Hu\rangle_F and h=∑bwb∣∣Hbu∣∣F2h=\sum_b w_b||H_bu||_F^2, where H=∑bwbHbH=\sum_b w_bH_b, HbH_b is the prefix Hessian and wbw_b its token weight.

Layer 16, step 9600, eight shared 512-token validation prefixes:

ModuleUpdate direction105a10^5aq/qVectorq/q_{\mathrm{Vector}}h/hVectorh/h_{\mathrm{Vector}}
QNative / Square−6.7540.0008740.000921
QFixed Shuffle−215.7850.70380.7120
QRMNP−79.0780.32640.3357
QVector−253.32211
MLP downNative0.2950.12240.1469
MLP downSquare−1.1590.11220.1346
MLP downFixed Shuffle−2.1160.94420.9446
MLP downRMNP−1.9080.94810.9538
MLP downVector−1.87011

These per-module unit-norm results are separate from the joint full-model diagnostics with module learning-rate multipliers (Appendix K, Fig. 15).

For Q, RMNP substantially reduces q,hq,h relative to Vector, while Native/Square responses are about three orders of magnitude lower than Vector's; in MLP-down, RMNP is closer to Vector. Shared row-energy control can improve local curvature responses; Muon's matrix spectral transformation produces directional differences that row-wise normalization cannot reproduce.

  • Energy exchange: Across Q/K at all 32 layers and four checkpoints, we exchange Native/Vector attention-head update energies, fixing within-group directions. Row-group energy differences contribute a median 17.5% (Q) and 19.1% (K) to their directional-curvature gap (Appendix M.1, lines 2742–2746).
  • Curvature relocation: For 36 states each in Q and K, we fix the Hessian spectrum, update directions, and first-order action aa, changing only attention-head curvature-block placement. Median 103(qVector−qNative)10^3(q_{\mathrm{Vector}}-q_{\mathrm{Native}}) falls 6.680→2.204 (Q) and 1.086→0.249 (K); random relocation yields 4.377 and 0.586, respectively (Appendix M.1, lines 2750–2760).

These results distinguish row-group energy allocation, curvature correspondence, and update directions, providing direct model evidence for Muse's curvature–energy mechanism and its link to spectral transformations.

Dear Reviewer MpFC,

Thank you again for your thoughtful review. We have provided a detailed response and additional experimental evidence, including new RMNP comparisons and further analyses of the proposed mechanism.

We would appreciate any further feedback you may have and would be happy to clarify any remaining questions or concerns.

Thank you for your time and consideration.

Reviewer_ZtySOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how the matrix representation used to compute Muon's update affects optimization. Muse reshapes or permutes momentum before applying the spectral map and returns the direction to the original parameter coordinates. A one-row representation gives the direction of per-matrix normalized momentum. The analysis separates nuclear support, update alignment, and finite-step costs, proves stationarity bounds for exact polar and fixed-iteration Newton–Schulz updates, and derives a curvature-dependent comparison using row groups. Teacher–student experiments, controlled quadratics, and 130M/600M language-model runs show that ordered reshaping can retain lower loss than Vector and that fixed-shape shuffling can remove this advantage.

Critical strengths
  1. The representation interventions are informative and well controlled. Parameters and momentum stay in their original coordinates, and the Vector endpoint gives a closely matched baseline. Table 6 fixes auxiliary AdamW and tunes matrix rates separately, preserving the main Native/Square–Vector and ordered–shuffle differences. Fixed shuffling and the exact-polar toy control strengthen the evidence that entry organization matters.

  2. The analysis supports the experimental findings. The tuned NS5 constants, momentum-tracking argument, and row-group curvature comparison are sound under their assumptions. The scalar-compensation experiments address update-length effects, and Appendix L includes coupled groups, row-normalized momentum, and fresh starts. These results show how limits on row-group update energy can affect optimization under the stated curvature conditions.

Critical weaknesses
  1. The connection between the curvature mechanism and language-model performance remains partial. Appendix M measures aggregated Hessian responses for six Q/K matrices at one checkpoint on a 128-token sequence. Proposition 3.4 requires operator-norm bounds and uniform control along update segments. The Native–Square change at 600M occurs in MLP matrices, which these probes leave unexamined. Appendix K also reports that local first-order action and a nominal one-step loss change can favor representations with worse terminal loss. The intervention is convincing. The authors should explain more directly how much of the training gap their curvature measurements account for. This is the main reason for weak acceptance.

  2. Fine method rankings depend on evaluation and selection details. Reported means include rate-selection seeds, the coupled 600M Skinny comparison uses one eligible rate per variant, and 130M evaluation uses one batch per checkpoint. Its token count and reuse should be specified. These details limit precision for nearby rankings. The independently tuned Table 6 results provide substantial support for the broad representation effect.

Questions for authors
  1. How do the response measurements in Appendix M relate to the operator-norm quantities in Assumption 3.4? Please state which part of the language-model explanation they establish. If existing results permit it, a compact directional-curvature comparison in a reshaped MLP block would strengthen the connection. This additional diagnostic is optional for acceptance. A convincing explanation of the observed training effect could raise my rating to clear acceptance.

  2. Please specify the number of validation sequences or tokens, evaluation-batch reuse across methods and seeds, and the seeds used for rate selection and final reporting. Is the broad ordering stable under any larger common evaluation already available? A reversal of the main ordering would materially affect my recommendation.

5 comments

We thank the reviewer for the constructive comments. The revisions in the manuscript are marked in blue. We have added diagnostics of curvature structure and update directions in a real model and further clarified the training, learning-rate selection, and evaluation protocols.

W1 / Q1: Curvature measurements, Assumption 3.4, the LLM gap, and MLP reshaping

A1: We added Hessian structure measurements and shared-state directional diagnostics at four checkpoints of the 600M model (2400/4800/7200/9600) to examine Assumption 3.4's row-group curvature structure and how representation-dependent energy allocation affects local directional curvature.

Row-group curvature heterogeneity and coupling. We partition the Hessian according to native row groups, estimate the local block operator norms Lj≈∣∣Hjj∣∣2L_j\approx||H_{jj}||_2 and cross-group coupling ε≈∣∣H−blkdiag⁡(Hjj)∣∣2\varepsilon\approx||H-\operatorname{blkdiag}(H_{jj})||_2, and compute

\qquad h_2=\frac{(\max_jL_j)^2}{\sum_j(|I_j|/p)L_j^2}.$$ Q/K groups contain the 64 rows of an attention head; other modules use native single-row groups. Q and K each cover 32 layers and 4 checkpoints (128 states each); single-row structure measurements for MLP-down cover 36 states. Results for the seven module types follow: |Module|$h_2$|$\rho_{\mathrm{off}}$| |---|---|---| |Q|5.64|0.090| |K|5.53|0.078| |V|7.16|8.629| |O|1.57|10.697| |MLP-gate|26.07|1.139| |MLP-up|224.27|2.497| |MLP-down|2.01|68.591| All seven module types exhibit row-group curvature heterogeneity, but the strength of cross-group coupling differs substantially. Gate and Up exhibit the most pronounced curvature heterogeneity, Q/K have the weakest cross-group coupling, and MLP-down has the strongest cross-row coupling. These results correspond to the heterogeneous row-group curvature and nonzero cross-group coupling structure considered in Assumption 3.4. Because Q/K have the weakest cross-group coupling, we further focus on these two modules to examine the effect of row-group energy allocation on local curvature cost. **Representation dependence and curvature effects.** Fig. 5 and Figs. 23–27 show that, in Q/K, V, Gate, and Up, Native, Square, and Skinny generally have more uniform native row-group update-energy allocation than Vector (nSGDM) and Fixed Shuffle, whereas the differences are relatively small in Output and Down. Thus, representation-induced energy allocation changes occur across attention and MLP matrices, not only in Q/K. To further quantify the effect of this change, we compare Native and Vector under the same parameters, momentum, and Hessian, map the update directions back to original coordinates, and normalize them to unit Frobenius norm. We then fix the within-group unit update directions and use a symmetric decomposition based on exchanging row-group energies to calculate the contribution of energy allocation to the difference in local directional curvature $q=\langle D,HD\rangle_F$. Across the 128 shared states for each module, the median contributions of energy allocation to the Native–Vector difference in local $q$ are 17.5% for Q, 19.1% for K, and 3.3% for MLP-down. These percentages quantify the source of the local directional-curvature difference at fixed states, not the final validation-loss gap. Together, these results support the curvature-dependent row-energy mechanism and reveal its module dependence. The shared-state directional diagnostics cover 32 layers × 7 matrix types × 4 checkpoints, for a total of 896 physical matrix states. Fig. 5 shows the Q/K results for the four checkpoints in layer 16, Fig. 22 shows Q/K Hessian responses across layers, and Figs. 23–27 extend the analysis to all layers and other modules. The Hessian is the token-weighted average over common prefixes of 512 input tokens each, $H=\sum_b w_bH_b$. These measurements provide local structural and mechanistic evidence at fixed states; the common curvature bounds required by Assumption 3.4 still involve the Hessians along the update segments of each method (section 4.5, lines 477–500; Appendix M.1, lines 2732–2749). **Reshaped MLP matrix.** For MLP-down, we compare the local Hessian responses of Native and Square at the same states. Among 128 shared states, 101/128 satisfy $q_{\mathrm{Square}}<q_{\mathrm{Native}}$, with a median per-state ratio of 0.900; 109/128 satisfy $h_{\mathrm{Square}}<h_{\mathrm{Native}}$, with a median ratio of 0.921, where $h=\sum_b w_b||H_bu||_F^2$. These results show that Square has a lower local curvature response in most measured states. However, the local second-order response must be considered together with first-order action and the actual step size to assess the single-step optimization benefit, and therefore cannot by itself determine the final training-loss ranking.

W2 / Q2: Sharing of validation samples, learning-rate selection protocol, and additional evaluation

A2: We further clarify the validation data, learning-rate selection, and statistical reporting of the final results.

Validation data and reuse. Each 130M evaluation uses one batch containing 65,536 validation input tokens; each 600M evaluation contains 10,158,080 validation input tokens. Both use a sequence length of 4096. All 600M methods and training seeds use the same fixed C4 validation samples, which are reused for evaluation at different checkpoints.

Learning-rate selection and result reporting. All experiments select learning rates using terminal validation loss and report the mean ± SD over three independent training seeds (seeds 1/2/3). Specifically, for the 130M experiments and 600M Shuffle (coupled-rate), the learning rate is selected using seed 1's terminal loss, followed by repeated training with two additional seeds; for 600M Ordered (coupled-rate), the learning rate is selected using the mean terminal loss over the three seeds. For the 600M fixed-auxiliary experiments (Table 7), we fix the AdamW learning rate and select only the matrix learning rate using seed 1's terminal loss, followed by repeated training with two additional seeds. All final statistics include the seeds used for learning-rate selection.

Re-evaluation on a larger common validation set. To test whether the main method ranking is stable, we conducted an additional offline evaluation of the selected checkpoints for all eight methods in Table 9, each with three training seeds. The new validation set uses a different fixed set of C4 samples, consisting of 4096 sequences of length 4096, for a total of 16,773,120 valid scored tokens. All methods and seeds use exactly the same validation data.

MethodOriginal evaluation (Table 9)Larger common validation set
Native2.6684±0.00142.6659±0.0013
Square2.6745±0.00132.6772±0.0015
Skinny-75%2.6856±0.00122.6838±0.0011
Skinny-50%2.6857±0.00132.6839±0.0012
Skinny-25%2.7111±0.00262.7145±0.0023
Vector / nSGDM2.7491±0.00202.7468±0.0022
Fixed Shuffle2.8911±0.03052.8973±0.0302
Stepwise Shuffle2.8882±0.03012.8941±0.0300

The ordering of the mean losses for all eight methods on the larger common validation set is identical to that in Table 9, with no reversals. Native remains the best, followed by Square and the Skinny representations; Vector / nSGDM and the two Shuffle methods have successively higher validation losses. Thus, after expanding the common validation set, Muse's main experimental conclusion that representation changes affect Muon's optimization performance remains stable.

Dear Reviewer ZtyS,

Thank you again for your thoughtful review. We have provided a detailed response with additional experimental results and clarifications regarding the evaluation protocol and the proposed mechanism.

We would appreciate any further feedback you may have and would be happy to clarify any remaining questions or concerns.

Thank you for your time and consideration.

Reviewer_wEVqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Muon’s dependence on momentum representation. Order-preserving reshaping retains its advantage over normalized momentum, while random entry shuffling can remove it. The authors connect these findings to update alignment and heterogeneous row-block curvature through theory and controlled experiments.

Critical strengths
  1. Reshaping and shuffling experiments provide useful evidence about which aspects of matrix representation matter.
  2. The curvature analysis offers a conditional explanation for Muon’s advantage, supported by controlled experiments.
Critical weaknesses
  1. Practical benefits are unclear: no representation consistently outperforms Native Muon, and no training-efficiency improvement is established.
  2. The language-model experiments do not isolate the proposed curvature mechanism from accompanying changes in momentum spectra and update alignment.
Questions for authors
  1. What actionable representation-selection rule follows from the results?
  2. Can targeted permutations isolate curvature correspondence from spectral and alignment effects? 3) Can measured curvature predict when reshaping will outperform Native Muon?
4 comments

We thank the reviewer for the constructive comments. The revisions in the manuscript are marked in blue. We have further clarified the practical implications of matrix representation and examined the curvature-dependent row-energy mechanism through additional experiments.

W1 / Q1: Practical value of the study and an actionable principle for representation selection

A1: Muse aims to explain why Muon's advantage over normalized momentum (nSGDM) depends on matrix representation. We use ordered reshaping to change matrix shape and entry shuffling to change entry organization at a fixed shape. The experiments show that ordered reshaping can preserve Muon's advantage over Vector, whereas entry shuffling can cause this advantage to disappear. This indicates that the effect of a representation depends not only on matrix shape but also on how the row-group energy constraints imposed by the representation correspond to the actual curvature structure.

Accordingly, we propose a conditional principle for representation selection: when native row groups exhibit pronounced curvature heterogeneity and weak cross-group coupling, priority should be given to preserving their boundaries. In the 600M model, with the auxiliary AdamW rate fixed and the matrix learning rate selected independently, Native, Square, and Vector achieve validation losses of 2.6683±0.0015, 2.6747±0.0012, and 2.7950±0.0050, respectively, while fixed-shape Shuffle performs worse (Table 7). Our response to W2/Q2 below further examines the curvature heterogeneity and curvature–row-energy correspondence underlying this principle through measurements of Q/K curvature structure and controlled permutation experiments.

This mechanism also helps us understand other update methods. For example, RMNP[1] limits update energy in native row groups through row-wise normalization. Appendix E.5 (lines 1350–1362) shows that, under the same curvature assumptions, appropriately scaled RMNP can obtain the same C2C_2-type upper bound as exact-polar Muon. In the 600M coupled-rate experiment, RMNP achieves a validation loss of 2.7110±0.0073, better than nSGDM's 2.7491±0.0020, but still worse than Native Muon's 2.6684±0.0014 (Table 15). These results are consistent with the theoretical analysis that row-group energy constraints can help reduce heterogeneous curvature costs, and they also provide a reference for understanding and designing row-normalized update methods.

[1] S. Deng et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306:24191–24239, 2026.

W2 / Q2: Can curvature correspondence be distinguished from changes in the momentum spectrum and update alignment?

A2: We first measured the curvature structure of native row groups in the Q/K matrices of the 600M model, covering 32 layers and 4 checkpoints, with 128 states for each projection. The median cross-group coupling ratios ρoff\rho_{\mathrm{off}} for Q and K are 0.090 and 0.078, respectively, and the curvature heterogeneity measures h2h_2 are 5.64 and 5.53, respectively. These results indicate that Q/K exhibit pronounced row-group curvature heterogeneity and relatively weak cross-group coupling, consistent with the structural conditions considered in the theory.

Building on this, we added two types of shared-state interventions: one examines the role of row-group energy allocation, and the other examines the independent contribution of curvature correspondence while keeping the updates fixed.

First, fix the matrix shape and exchange row-group energies. At steps 2400 and 9600 for Q/K in layer 16, we compared Ordered and Global Shuffle under four matrix shapes, using three permutation seeds per shape, for a total of 48 matched pairs. In all 48/48 comparisons, Global Shuffle increased the share of update energy in the four highest-curvature native attention heads by 11.21–41.53 percentage points. We further fixed the within-group unit update directions, exchanged only the row-group energies, and symmetrically decomposed the Global–Ordered difference in directional curvature. The median contribution of the energy component was 21.4% (Appendix M.1, lines 2746–2749).

Second, fix the update direction and rearrange Hessian curvature. To further distinguish the effects of curvature correspondence from those of the momentum spectrum and update alignment, we permuted the curvature coordinates of complete attention heads in a local quadratic model of the measured Hessian: HP=P⊤HP,qP=u⊤HPu.H_P=P^\top HP,\qquad q_P=u^\top H_Pu. Here, uu is the update direction in the original parameter coordinates, normalized to unit Frobenius norm, and HH is the token-weighted principal Hessian estimated on common evaluation prefixes of 512 input tokens each. For each representation, this intervention preserves the singular-value spectrum of the momentum, the update direction and length, and a=⟨g,u⟩Fa=\langle g,u\rangle_F, while also preserving the eigenvalues of the Hessian.

We used a separate 512-token calibration prefix to select four disjoint pairs of attention heads for targeted swaps and used random controls with the same number of swaps. We then evaluated them on seven other common 512-token prefixes. Q and K each cover 9 layers × 4 checkpoints, for a total of 36 states, with the following results:

ModuleOriginal correspondenceTargeted curvature relocationRandom relocation (32 repetitions)
Q6.6802.2044.377
K1.0860.2490.586

The table reports 103Δq10^3\Delta q, where Δq=qVector−qNative\Delta q=q_{\mathrm{Vector}}-q_{\mathrm{Native}}. Each column gives the median of the signed differences across states. For the random control, the 32 permutation results are first averaged within each state, followed by the median across states.

The results show that, while the momentum spectrum and update alignment are held fixed, changing only the correspondence between curvature and the fixed updates can substantially narrow the difference in second-order curvature cost between Native and Vector. Together with the Q/K curvature structure measurements and energy-exchange results above, this provides further evidence for the role of the curvature-dependent row-energy mechanism in real language models (Appendix M.1, lines 2750–2760).

Q3: Can measured curvature predict when reshaping will outperform Native Muon?

A3: Measured curvature can help assess the local second-order costs of different representations at a fixed state, but predicting final training outcomes also requires consideration of first-order action and the actual update length.

Consider the Native–Square reshape of MLP-down. Among 128 shared states, 101 satisfy qSquare<qNativeq_{\mathrm{Square}}<q_{\mathrm{Native}}, and the median of the per-state ratios is 0.900. However, in full 600M training, Native still achieves a slightly lower final validation loss than Square, indicating that lower local directional curvature does not necessarily lead to a better final training outcome.

Specifically, the local objective change is approximately

\qquad a=\langle g,u\rangle_F,\quad q=u^\top Hu,$$ where $s$ is the actual update length. Therefore, row-group curvature structure provides conditions for representation selection, while $a$, $q$, and the actual step size jointly determine the first-order benefit and second-order cost at a fixed state; the final benefit still needs to be validated through multi-step training. This result is also consistent with the manuscript's analysis of spectral support, update alignment, and finite step sizes in section 3.1–3.2.

Dear Reviewer wEVq,

Thank you again for your thoughtful review. We have provided a detailed response and additional experimental evidence addressing your concerns, including new curvature measurements and controlled analyses of the proposed mechanism.

We would appreciate any further feedback you may have and would be happy to clarify any remaining questions or concerns.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.