- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors study the question why Muon achieves excelent performance in neural network optimization. The main argument for this phenomenon made in this paper is that Muon's implicit preconditioner shares a similar row-block-diagonally-dominant structure as the neural network Hessian at initialization. The authors provide a theoretical justification for this observation for 4 model matrix optimization problems: symmetric matrix factorization (Theorem 1), deep linear networks (Theorem 2), 2-layer relu networks (Theorem 3), and single-layer linear attention (Theorem 4). They also provide a practical justification: simulation of theoretical results, and experiments with initialization of large language models.
Critical strengths
The main strength is a somewhat interesting theoretical analysis used in establishing Theorems 1-4, which characterizes the expectation over random initializations for several neural network models.
The connection of the results to row-wise normalization could also be useful. There is a DASGO algorithm An et al. (2025), which does similar normalization and reports good performance in the early stages of training, which is relevant to the initialization-focused analysis of this paper. The difference is that DASGO performs column-wise adaptive scaling instead of row-wise normalization. Hence, it would be interesting to see if the results of these papers could be extended to the matrix .
Critical weaknesses
There are several weaknesses:
-
The fact that is approximately diagonal does not mean that is approximately diagonal, where the expectation is taken over random initialization. However, Muon's preconditioner depends on the particular realization , which may not be approximately diagonal. To make the argument of this paper work, we would need to establish some concentration results.
-
Muon's update has an equivalent formulation using right preconditioning, which is automatically block-diagonal by design. Hence, it is unclear, why do we even need the theory established in this paper. To justify the need for these results, we need to argue that left preconditioning interpretation is more relevant compared to right preconditioning interpretation. However, I could not find such arguments in this paper.
-
The authors intended reasoning would establish that the full left preconditioner is approximately diagonal, not merely block-dagonal, provided that necessary concentration results hold. Such an explanation would rather fit to row-normalization methods, rather than orthogonalized methods (see also critical strenghts).
-
The overal argument does not justify the Muon's superior performance. Even if the preconditioner is structurally similar to Hessian, it does not mean that it captures Hessian's curvature and improves conditioning of the problem.
Questions for authors
n/a
3 comments
Part II (Response to W4)
W4. On justifying Muon's performance. The core question of this work is not whether Muon improves the conditioning of the problem or captures the Hessian's curvature; this is beyond the scope of the paper. Our motivation comes from [2, 3] and [1]. The former established, empirically and theoretically, the row-block diagonally dominant structure of the neural-network Hessian, and the latter verified empirically that Muon's preconditioner has the same block structure for LLMs. Both [1] and the recent work [4] point out that why the preconditioner of neural networks has this block diagonally dominant structure remains an important open question. Answering this question theoretically is the central contribution and motivation of this paper.
References
[1] Deng, S., et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. 2026.
[2] Zhang, Y., et al. Why Transformers Need Adam: A Hessian Perspective. NeurIPS 2024.
[3] Dong, Z., Zhang, Y., Luo, Z.-Q., Yao, J., Sun, R. Towards Quantifying the Hessian Structure of Neural Networks. 2025.
[4] Ma, J., et al. Preconditioning Benefits of Spectral Orthogonalization in Muon. arXiv:2601.13474, 2026.
We sincerely thank you for the time and effort devoted to reviewing our paper. If our responses have addressed your concerns, we would be grateful if you would consider re-evaluating the score. Please let us know if you have any follow-up questions.
Dear Reviewer 8RaT,
As today is the last day of the discussion period, we would like to kindly follow up on our rebuttal.
We want to make sure that we have clarified and addressed all of your concerns, and we would be happy to answer any further questions you may have. Thank you again for your time and feedback.
Part I (Response to W1 W2 W3)
We thank the reviewer for the time and effort spent on our paper. In response to the comments, we have added supplementary theoretical results and new experiments to the revised manuscript; the additional experimental results and the corresponding code are available at https://anonymous.4open.science/r/orthadapt-iclr-rebuttal-8107/. We address each point below.
W1. Single draw versus expectation. Block diagonal dominance of Muon's preconditioner for a single initialization has already been observed empirically along language-model training trajectories by [1], which motivated this work. The orders we derive for E[GG⊤] can be understood as the typical behavior of GG⊤ in a statistical sense, and the scaling with the hidden dimension explains why this observed dominance arises. Also, in the theoretical study of Hessian heterogeneity in [3], the analysis of the Hessian structure is likewise carried out in expectation over the initialization.
Concentration and single-draw simulations. To address the concern directly, we provide a concentration result for matrix factorization in Appendix B.6 of the revised manuscript: Proposition 1 bounds the deviation of a single draw GG⊤ from E[GG⊤] with high probability, and Theorem 5 shows that, with probability at least 1−η, the diagonal-to-off-diagonal ratio of a single initialization grows at least like k with the hidden dimension. We also added numerical simulations for all four setups in which the ratio is computed for a single initialization (the ratio is computed for each single initialization and then averaged over 10,000 initializations), reported in Figures 2-4 in the linked repository and further described in Appendix H of the revised manuscript.
W2. Left versus right preconditioning.
There is a misunderstanding here: Muon's orthogonalization can be written as a linear map on the vectorized gradient in two ways, depending on how G∈Rm×n is vectorized.
With row stacking, vecr(G)=(G1,:,…,Gm,:)⊤,
vecr(Orth(G))=((GG⊤)−1/2⊗In)vecr(G).
With column stacking, vecc(G)=(G:,1,…,G:,n)⊤,
vecc(Orth(G))=(In⊗(GG⊤)−1/2)vecc(G).
These are the same linear map, written in two coordinate systems. The block diagonal dominance of the Hessian described in [2, 3] is indexed by rows of the weight matrix: the parameters are partitioned by output neurons, and dominance holds in this row-block structure. Under the row-stacked vectorization, the preconditioner (GG⊤)−1/2⊗In is block diagonally dominant in this same structure only when GG⊤ is diagonally dominant, which does not hold automatically. Writing A=(GG⊤)−1/2,
A⊗In=A11InA21In⋮Am1InA12InA22In⋮Am2In⋯⋯⋱⋯A1mInA2mIn⋮AmmIn,
so the off-diagonal blocks vanish exactly when the off-diagonal entries of A do. The real question is therefore why GG⊤ is diagonally dominant specifically for neural networks. [1] verified empirically that Muon's preconditioner is block diagonally dominant for LLMs, and that the dominance becomes stronger as the model scale increases. The contribution of this work is the theoretical explanation of this observation: we prove that the diagonal dominance of GG⊤ arises from over-parameterization of the weight matrices, at explicit rates in the hidden dimensions.
W3. On "approximately diagonal" versus "block diagonal". This comment rests on the same misunderstanding as the previous one. In Section 1 (paragraph "Preconditioner structure versus Hessian structure", Eq. 2) we show that the orthogonalization D=Orth(V)=(VV⊤)−1/2V of a matrix V is equivalent to a linear transformation of that matrix,
vec(D)=((VV⊤)−1/2⊗In)vec(V),HMuon=(VV⊤)1/2⊗In,
whose matrix is block structured by the rows of V. "Diagonally dominant GG⊤" therefore means that HMuon is block diagonally dominant in the row-block structure of the Hessian. This is precisely the structure that [1] observed empirically for LLMs: along training trajectories, Muon's preconditioner, like the Hessian, is block diagonally dominant. Our theory explains why this block diagonally dominant structure arises.