acceptodds
Under review as a conference paper at ICLR 2027

How Does Orthogonalization Adapt to the Neural Network Hessian Structure? A Gradient Self Outer-Product Analysis at Initialization

16 views

Abstract

Muon orthogonalizes a weight matrix's momentum before each step, and on neural networks this simple preconditioner beats AdamW in a wide range of tasks. Existing analyses explain Muon within general optimization frameworks or on a few specific models, and many of them view orthogonalization as a preconditioner. Since the Hessian is the optimal preconditioner under a quadratic approximation of the loss, these works leave a basic question open: what makes orthogonalization suited to the Hessian of a neural network? Prior work shows that the layer-wise Hessian of a neural network is row-block diagonally dominant, and Muon's implicit preconditioner, the Kronecker product with the momentum, has the same row-block structure exactly when is diagonally dominant. Diagonal dominance of has been observed in language models, but why a neural network produces it has not been explained. This work provides a quantitative asymptotic theory for neural networks at initialization, and points out the intuition that the overparameterization of the weight matrices drives the diagonal dominance. We analyze , which equals at initialization, in four canonical models: symmetric matrix factorization, deep linear networks, two-layer ReLU networks, and single-layer linear attention. In each case the diagonal entries outgrow the off-diagonal ones at explicit rates as the hidden dimensions grow; for the query, key and value weights of attention the off-diagonal vanishes altogether. Simulations of the four models and experiments on language models at initialization confirm the predicted scaling. The code for all our simulations and language-model experiments is available at https://anonymous.4open.science/r/orthogonalization-hessian-CB2C/.

Primary area:
optimization
Keywords:
Muon, Hessian, Asymptotic Theory for NN, Precondition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8RaTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors study the question why Muon achieves excelent performance in neural network optimization. The main argument for this phenomenon made in this paper is that Muon's implicit preconditioner shares a similar row-block-diagonally-dominant structure as the neural network Hessian at initialization. The authors provide a theoretical justification for this observation for 4 model matrix optimization problems: symmetric matrix factorization (Theorem 1), deep linear networks (Theorem 2), 2-layer relu networks (Theorem 3), and single-layer linear attention (Theorem 4). They also provide a practical justification: simulation of theoretical results, and experiments with initialization of large language models.

Critical strengths

The main strength is a somewhat interesting theoretical analysis used in establishing Theorems 1-4, which characterizes the expectation E[GG⊤]\mathbb{E}[GG^\top] over random initializations for several neural network models.

The connection of the results to row-wise normalization could also be useful. There is a DASGO algorithm An et al. (2025), which does similar normalization and reports good performance in the early stages of training, which is relevant to the initialization-focused analysis of this paper. The difference is that DASGO performs column-wise adaptive scaling instead of row-wise normalization. Hence, it would be interesting to see if the results of these papers could be extended to the matrix E[G⊤G]\mathbb{E}[G^\top G].

Critical weaknesses

There are several weaknesses:

  • The fact that E[GG⊤]\mathbb{E}[GG^\top] is approximately diagonal does not mean that GG⊤GG^\top is approximately diagonal, where the expectation is taken over random initialization. However, Muon's preconditioner depends on the particular realization GG⊤GG^\top, which may not be approximately diagonal. To make the argument of this paper work, we would need to establish some concentration results.

  • Muon's update has an equivalent formulation using right preconditioning, which is automatically block-diagonal by design. Hence, it is unclear, why do we even need the theory established in this paper. To justify the need for these results, we need to argue that left preconditioning interpretation is more relevant compared to right preconditioning interpretation. However, I could not find such arguments in this paper.

  • The authors intended reasoning would establish that the full left preconditioner is approximately diagonal, not merely block-dagonal, provided that necessary concentration results hold. Such an explanation would rather fit to row-normalization methods, rather than orthogonalized methods (see also critical strenghts).

  • The overal argument does not justify the Muon's superior performance. Even if the preconditioner is structurally similar to Hessian, it does not mean that it captures Hessian's curvature and improves conditioning of the problem.

Questions for authors

n/a

3 comments
AuthorsOpenReview ↗

Part I (Response to W1 W2 W3)

We thank the reviewer for the time and effort spent on our paper. In response to the comments, we have added supplementary theoretical results and new experiments to the revised manuscript; the additional experimental results and the corresponding code are available at https://anonymous.4open.science/r/orthadapt-iclr-rebuttal-8107/. We address each point below.

W1. Single draw versus expectation. Block diagonal dominance of Muon's preconditioner for a single initialization has already been observed empirically along language-model training trajectories by [1], which motivated this work. The orders we derive for E[GG⊤]\mathbb{E}[GG^\top] can be understood as the typical behavior of GG⊤GG^\top in a statistical sense, and the scaling with the hidden dimension explains why this observed dominance arises. Also, in the theoretical study of Hessian heterogeneity in [3], the analysis of the Hessian structure is likewise carried out in expectation over the initialization.

Concentration and single-draw simulations. To address the concern directly, we provide a concentration result for matrix factorization in Appendix B.6 of the revised manuscript: Proposition 1 bounds the deviation of a single draw GG⊤GG^\top from E[GG⊤]\mathbb{E}[GG^\top] with high probability, and Theorem 5 shows that, with probability at least 1−η1-\eta, the diagonal-to-off-diagonal ratio of a single initialization grows at least like k\sqrt{k} with the hidden dimension. We also added numerical simulations for all four setups in which the ratio is computed for a single initialization (the ratio is computed for each single initialization and then averaged over 10,000 initializations), reported in Figures 2-4 in the linked repository and further described in Appendix H of the revised manuscript.

W2. Left versus right preconditioning.

There is a misunderstanding here: Muon's orthogonalization can be written as a linear map on the vectorized gradient in two ways, depending on how G∈Rm×nG \in \mathbb{R}^{m\times n} is vectorized.

With row stacking, vecr(G)=(G1,:,…,Gm,:)⊤\mathrm{vec_r}(G) = (G_{1,:}, \dots, G_{m,:})^\top,

vecr(Orth(G))=((GG⊤)−1/2⊗In)vecr(G).\mathrm{vec_r}\big(\mathrm{Orth}(G)\big) = \big((GG^\top)^{-1/2} \otimes I_n\big) \mathrm{vec_r}(G).

With column stacking, vecc(G)=(G:,1,…,G:,n)⊤\mathrm{vec_c}(G) = (G_{:,1}, \dots, G_{:,n})^\top,

vecc(Orth(G))=(In⊗(GG⊤)−1/2)vecc(G).\mathrm{vec_c}\big(\mathrm{Orth}(G)\big) = \big(I_n \otimes (GG^\top)^{-1/2}\big) \mathrm{vec_c}(G).

These are the same linear map, written in two coordinate systems. The block diagonal dominance of the Hessian described in [2, 3] is indexed by rows of the weight matrix: the parameters are partitioned by output neurons, and dominance holds in this row-block structure. Under the row-stacked vectorization, the preconditioner (GG⊤)−1/2⊗In(GG^\top)^{-1/2}\otimes I_n is block diagonally dominant in this same structure only when GG⊤GG^\top is diagonally dominant, which does not hold automatically. Writing A=(GG⊤)−1/2A = (GG^\top)^{-1/2},

A⊗In=(A11InA12In⋯A1mInA21InA22In⋯A2mIn⋮⋮⋱⋮Am1InAm2In⋯AmmIn),A \otimes I_n = \begin{pmatrix} A_{11} I_n & A_{12} I_n & \cdots & A_{1m} I_n \\\\ A_{21} I_n & A_{22} I_n & \cdots & A_{2m} I_n \\\\ \vdots & \vdots & \ddots & \vdots \\\\ A_{m1} I_n & A_{m2} I_n & \cdots & A_{mm} I_n \end{pmatrix},

so the off-diagonal blocks vanish exactly when the off-diagonal entries of AA do. The real question is therefore why GG⊤GG^\top is diagonally dominant specifically for neural networks. [1] verified empirically that Muon's preconditioner is block diagonally dominant for LLMs, and that the dominance becomes stronger as the model scale increases. The contribution of this work is the theoretical explanation of this observation: we prove that the diagonal dominance of GG⊤GG^\top arises from over-parameterization of the weight matrices, at explicit rates in the hidden dimensions.

W3. On "approximately diagonal" versus "block diagonal". This comment rests on the same misunderstanding as the previous one. In Section 1 (paragraph "Preconditioner structure versus Hessian structure", Eq. 2) we show that the orthogonalization D=Orth(V)=(VV⊤)−1/2VD=\mathrm{Orth}(V)=(VV^\top)^{-1/2}V of a matrix VV is equivalent to a linear transformation of that matrix,

vec(D)=((VV⊤)−1/2⊗In)vec(V),HMuon=(VV⊤)1/2⊗In,\mathrm{vec}(D)=\big((VV^\top)^{-1/2}\otimes I_n\big) \mathrm{vec}(V),\qquad H_{\mathrm{Muon}}=(VV^\top)^{1/2}\otimes I_n,

whose matrix is block structured by the rows of VV. "Diagonally dominant GG⊤GG^\top" therefore means that HMuonH_{\mathrm{Muon}} is block diagonally dominant in the row-block structure of the Hessian. This is precisely the structure that [1] observed empirically for LLMs: along training trajectories, Muon's preconditioner, like the Hessian, is block diagonally dominant. Our theory explains why this block diagonally dominant structure arises.

AuthorsOpenReview ↗

Part II (Response to W4)

W4. On justifying Muon's performance. The core question of this work is not whether Muon improves the conditioning of the problem or captures the Hessian's curvature; this is beyond the scope of the paper. Our motivation comes from [2, 3] and [1]. The former established, empirically and theoretically, the row-block diagonally dominant structure of the neural-network Hessian, and the latter verified empirically that Muon's preconditioner has the same block structure for LLMs. Both [1] and the recent work [4] point out that why the preconditioner of neural networks has this block diagonally dominant structure remains an important open question. Answering this question theoretically is the central contribution and motivation of this paper.

References

[1] Deng, S., et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. 2026.

[2] Zhang, Y., et al. Why Transformers Need Adam: A Hessian Perspective. NeurIPS 2024.

[3] Dong, Z., Zhang, Y., Luo, Z.-Q., Yao, J., Sun, R. Towards Quantifying the Hessian Structure of Neural Networks. 2025.

[4] Ma, J., et al. Preconditioning Benefits of Spectral Orthogonalization in Muon. arXiv:2601.13474, 2026.

We sincerely thank you for the time and effort devoted to reviewing our paper. If our responses have addressed your concerns, we would be grateful if you would consider re-evaluating the score. Please let us know if you have any follow-up questions.

Dear Reviewer 8RaT,

As today is the last day of the discussion period, we would like to kindly follow up on our rebuttal.

We want to make sure that we have clarified and addressed all of your concerns, and we would be happy to answer any further questions you may have. Thank you again for your time and feedback.

Reviewer_6SVeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to explain why Muon works so well for training neural networks. Muon reshapes each weight update, and the authors argue that this reshaping takes on the same structure as the network’s curvature (the Hessian).

To explain why that happens, Paper studies four simple models at the moment of initialization: matrix factorization, deep linear networks, two-layer ReLU networks, and single-layer linear attention. In each one, they show mathematically that the diagonal grows faster than everything else as the network gets wider, so wider networks make Muon behave more like simple per-neuron normalization.

Critical strengths
  1. Paper for the most part is easily readable with provides a solid experimental basis using 4 different configuration to show/explain diagonally dominant behavior

  2. Provides a Monte Carlo based simulations of the four models, plus width sweeps of Qwen3.5-0.8B, Granite-4.0-1B and Llama-3.2-1B at random initialization, showing the ratio R̂ (mean diagonal over mean off-diagonal) rising with width.

  3. Exact, careful closed forms across four standard models.

Critical weaknesses

Nothing Specific, Please refer to Questions sections

Questions for authors
  1. Direct Hessian comparison. Can you compute the layer-wise Hessian in at least one of your models (e.g. matrix factorization or the deep linear network) and show that diag(P)^{1/2} actually tracks the Hessian’s diagonal-block scale? Also, which model, layer and method produced the Hessian in Figure 1?

  2. Realistic initialization. Under standard fan-in initialization (Var(W_ij) = 1/fan-in), what are the orders in Table 1 for all four models? For matrix factorization, I get E[GGᵀ] → (I − M⋆)², so the diagonal and off-diagonal are both Θ(1) and R does not grow with k. Remark 6 in Appendix E.6 (“Discussion”) says the same for attention W_O. If so, how does the claim that “over-parameterization drives diagonal dominance” still hold?"

3 comments
AuthorsOpenReview ↗

Part I (Response to Q1 Q2)

We thank the reviewer for the time and effort spent on our paper. In response to the comments, we have added supplementary theoretical results and new experiments to the revised manuscript; the additional experimental results and the corresponding code are available at https://anonymous.4open.science/r/orthadapt-iclr-rebuttal-8107/. We address each point below.

Q1.

Thank you for the suggestion. We provide the theory for the row-block structure of the layer-wise Hessian in Theorem 6 of Appendix B.7 (matrix factorization) and Theorem 7 of Appendix C.6 (deep linear network), and simulation visualizations of the Hessian and GG⊤GG^\top for all four setups in Figures 8-13 in the linked repository, further described in Appendix H of the revised manuscript. The Hessian shows row-block diagonal dominance similar to the preconditioner across all models. Figure 1 is a conceptual view, there are many similar figure in [1] . We have made the necessary modifications to the statement in the revised version of the article.

Reference:

[1] Dong, Z., Zhang, Y., Luo, Z.-Q., Yao, J., Sun, R. Towards Quantifying the Hessian Structure of Neural Networks. 2025.

Q2.

We thank the reviewer for this question. The reviewer's computation is correct: under fan-in initialization the ratio for matrix factorization is Θ(1)\Theta(1). To further address the concern, we clarify below that constant-variance initialization is a practical setting, widely used in LLM pretraining, and we provide theory and numerical evidence to support that in the three neural-network setups, diagonal dominance survives under fan-in initialization for every weight matrix except the output layer.

On the constant-std initialization. Initializing weights with a constant standard deviation is standard practice in LLM pretraining, used by GPT-1 [1], Megatron-LM [2], OPT [3], Cerebras-GPT [4], DeepSeek-V2 [5], DeepSeek-V3 [6], OLMo 2 [7] and OLMoE [8]. For instance, OLMo 2 [7, Section 3.2] initializes every parameter from N(0,0.022)\mathcal{N}(0, 0.02^2) and reports that this scheme is more stable than the depth-scaled initialization of OLMo-0424 (their Figure 4); DeepSeek-V3 [6, Section 4.2] initializes all parameters with standard deviation 0.0060.006; and OLMoE [8, Section 4.2.2] notes that an std of 0.020.02 "has been commonly used as popularized by Shoeybi et al." These references are now cited in Section 3.2 of the revised manuscript.

Fan-in initialization results. We include the matrix-factorization setup because [9] analyzes Muon on it and lists the connection between Muon's preconditioner and the Hessian as future work. The diagonal dominance result for matrix factorization is affected by fan-in initialization, but as shown below, the main results for the other three neural network setups are not. The table gives the order of E[(GG⊤)aa]/E[(GG⊤)ab]\mathbb{E}[(GG^\top)_{aa}]/\mathbb{E}[(GG^\top)_{ab}] as the hidden dimension grows, with input/output dimensions and the target fixed; "∞\infty" means the off-diagonal entries are exactly zero.

SettingWeightN(0,1)\mathcal{N}(0,1) init (paper)Fan-in init (Var=1/fan-in\mathrm{Var}=1/\text{fan-in})
Matrix factorizationWWΘ(k)\Theta(k)Θ(1)\Theta(1)
Deep linear networkhidden layers W1,…,WL−1W_1,\dots,W_{L-1}∞\infty∞\infty
Deep linear networkoutput layer WLW_LΘ(∏jdj)\Theta(\prod_j d_j)Θ(1)\Theta(1)
Two-layer ReLUW1W_1Θ(d1)\Theta(d_1)Θ(d1)\Theta(d_1)
Two-layer ReLUW2W_2Θ(d1)\Theta(d_1)Θ(1)\Theta(1)
Linear attentionWQ,WK,WVW_Q,W_K,W_V∞\infty∞\infty
Linear attentionWOW_OΘ(dvd03)\Theta(d_v d_0^3)Θ(1)\Theta(1)

On the reviewer's underlying concern that the dominance is an artifact of the variance: the diagonal entry ∥ga∥2\Vert g_a\Vert^2 is the self inner product of a gradient row and the off-diagonal entry ⟨ga,gb⟩\langle g_a, g_b\rangle is the correlation between two rows, so diagonal dominance says that gradient rows are weakly correlated relative to their own norm. This is determined by how the forward and backward passes of the architecture compose the gradient. The variance rescales the gradient and sets the absolute order of the diagonal, so it is a sufficient but not a necessary condition.

Why fan-in initialization only affects the output layer. The gradient of a weight matrix WW is G=(backward error)×(forward input)⊤G = (\text{backward error}) \times (\text{forward input})^\top, and GG⊤GG^\top compares rows of GG, i.e., different output units of WW. Whether these rows are correlated depends on where WW sits in the network.

The detailed clarification continues in the next part of our response.

AuthorsOpenReview ↗

Part II (Response to Q2)

  • Hidden matrices. For a matrix whose error signal passes through downstream random weights, row aa of GG is driven by column aa of the next weight matrix WnextW_{\text{next}}. Different rows are thus driven by independent Gaussian columns of WnextW_{\text{next}}. The cross term E[GaGb⊤]\mathbb{E}[G_a G_b^\top] for a≠ba \neq b then contains the factor E[(Wnext)⋅a⊤(Wnext)⋅b]=0\mathbb{E}[(W_{\text{next}})_{\cdot a}^\top (W_{\text{next}})_{\cdot b}] = 0 by independence and zero mean, so the off-diagonal entries of E[GG⊤]\mathbb{E}[GG^\top] vanish regardless of the variance (deep linear hidden layers, attention Q,K,VQ,K,V). In the ReLU first layer the nonlinearity leaves a nonzero cross term, but it is produced by the same downstream weights as the diagonal term, so a change of initialization variance rescales both by the same factor and leaves the ratio at Θ(d1)\Theta(d_1). In short, for hidden matrices the ratio is determined by independence and symmetry in the surrounding weights, not by the initialization scale.

  • Output matrices. For a matrix that directly produces the output (WLW_L in the deep linear network, W2W_2 in the ReLU network, WOW_O in linear attention), there is no WnextW_{\text{next}}: row aa of GG is driven by row aa of the residual f(x)−Φxf(x) - \Phi x, where f(x)f(x) is the network output at initialization and Φx\Phi x is the target. The residual splits into a model part f(x)f(x), which is random and whose rows are driven by independent rows of the output weight, and a target part Φx\Phi x, which is deterministic and shared by all rows. The cross term E[GaGb⊤]\mathbb{E}[G_a G_b^\top] for a≠ba \neq b therefore vanishes for the model part, as in the hidden case, but not for the target part, which contributes (ΦΦ⊤)ab≠0(\Phi\Phi^\top)_{ab} \neq 0. The closed form is then E[GG⊤]=cI+c′ΦΦ⊤\mathbb{E}[GG^\top] = c I + c' \Phi\Phi^\top with the diagonal governed by the model term cc and the off-diagonal governed by the target term c′c' (Theorems 2-4 and Theorem 11 in Appendix F). Unlike the hidden case, these two terms are of different degrees in the weights, so a change of initialization variance rescales them differently. Under constant-variance initialization the output f(x)f(x) grows with the hidden dimension, cc dominates c′c', and the ratio diverges; under fan-in initialization the output is kept at unit scale by design, cc and c′c' are of the same order, and the ratio is Θ(1)\Theta(1), exactly as the reviewer computed. In short, for output matrices the ratio is determined by the scale of the initial output relative to the target, which is exactly what the initialization sets.

In summary, under fan-in initialization the hidden-layer matrices remain diagonally dominant, since the dominance arises from over-parameterization through the forward structure of the network. The behavior of the output layer does not represent a failure of this mechanism.

The fan-in results for the three neural-network models are stated and proved in Appendix F of the revised manuscript (Theorems 8 to 11, Table 2), with a pointer in Section 3.2, and all entries are verified numerically and illustrated in Figures 5-7 in the linked repository, with their respective setups described in Appendices H.2-H.4 of the revised manuscript.

References

[1] Radford, A., et al. Improving Language Understanding by Generative Pre-Training. OpenAI Technical Report, 2018.

[2] Shoeybi, M., et al. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053, 2019.

[3] Zhang, S., et al. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068, 2022.

[4] Dey, N., et al. Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster. arXiv:2304.03208, 2023.

[5] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024.

[6] DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024.

[7] Team OLMo. 2 OLMo 2 Furious. arXiv:2501.00656, 2024.

[8] Muennighoff, N., et al. OLMoE: Open Mixture-of-Experts Language Models. arXiv:2409.02060, 2024.

[9] Ma, J., Huang, Y., Chi, Y., Chen, Y. Preconditioning Benefits of Spectral Orthogonalization in Muon. arXiv:2601.13474, 2026.

We sincerely thank you for the time and effort devoted to reviewing our paper. If our responses have addressed your concerns, we would be grateful if you would consider re-evaluating the score. Please let us know if you have any follow-up questions.

Dear Reviewer 6SVe,

As today is the last day of the discussion period, we would like to kindly follow up on our rebuttal.

We want to make sure that we have clarified and addressed all of your concerns, and we would be happy to answer any further questions you may have. Thank you again for your time and feedback.

Reviewer_Ph3JOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the workings of Muon by showing how its preconditioner matches the structure of the Hessian of a neural network. Muon uses a layerwise preconditioner, and it shares the Hessian’s row-block diagonal structure only when the covariance of the momentum is diagonal. The paper's hunch is that overparameterization makes the momentum covariance diagonally dominant, by showing that, under width scaling, the diagonal entries grow faster than the off-diagonal ones. All the work has been done at initialization when the momentum covariance equals the gradient covariance on 4 toy models: symmetric matrix factorization, deep linear networks, two-layer ReLU networks and single-layer linear attention; where they derive the closed-form expression for the expected gradient covariance of the 4 toy models and supplement with Monte Carlo simulations of the four models that reproduce the predicted scaling rates, plus experiments on randomly initialized attention heads of Qwen3.5, Granite-4.0 and Llama-3.2 showing the diagonal-dominance ratio grows with model width.

Critical strengths

The paper turns the empirical observation that gradient outer products are diagonally dominant in large models (Deng et al., 2026) into a precise question about the structure of Muon’s implicit preconditioner, and answers it with explicit scaling rates in the four toy models at initialization. The derivations give exact closed-form expressions for E[GG⊤]\mathbb{E}[GG^\top], combining Wishart moment identities, Wick/Isserlis contractions, Stein’s lemma with the Cho–Saul arc-cosine kernel for the ReLU network, and sign- and permutation-symmetry arguments for the exact zeros. The closed-form expressions and the resulting diagonal and off-diagonal scaling rates are mathematically correct under the stated assumptions.

Shenyang Deng, Zhuoli Ouyang, Tianyu Pang, Zihang Liu, Ruochen Jin, Shuhua Yu, and Yaoqing Yang. RMNP: Row-momentum normalized preconditioning for scalable matrix-based optimization. arXiv preprint arXiv:2603.20527, 2026.

Critical weaknesses

W1. The headline rates come from the unit-variance initialization, not from overparameterization All four theorems assume i.i.d. N(0,1)\mathcal{N}(0,1) weights with no 1/fan-in1/\mathrm{fan\text{-}in} scaling (and the released code as well). Under this initialization, the network's output grows polynomially with width (e.g. WW⊤≈kIWW^\top \approx kI in matrix factorization), so the gradient is dominated by target-independent noise whose expected Gram is close to isotropic. The dominance therefore reflects the initialization scale rather than overparameterization. Under standard fan-in initialization the ratio no longer grows with the hidden dimension. In the LLM experiments they use the Hugging Face default settings (a fixed standard deviation) that does not shrink with width, so the effective initialization scale grows as width\sqrt{\mathrm{width}} and confounds the trend in Figure 6, which in any case shows only about a 5×5\times increase over a 64×64\times range of widths, which I won't claim as a cubic growth in model width (see Section 4.4, page 6, last line).

W2. Muon uses a single draw, but the paper only analyzes averages over initializations At the first step Muon orthogonalizes the gradient of the one initialization it was given, so the relevant object is G0G0⊤G_0 G_0^\top for a single weight draw. In contrast, the paper studies E[GG⊤]\mathbb{E}[GG^\top] averaged over initializations (Section 3, page 4). A diagonal expectation does not imply a diagonal realization, and since orthogonalization is nonlinear, the claim that it reduces to row normalization (Appendix F, page 35) does not follow. In particular, the exact zeros for the hidden linear layers and WQ,WK,WVW_Q, W_K, W_V are a symmetry identity that holds for any loss, data and sign-invariant initialization, as Remark 3 (page 35) itself acknowledges, so there is no link between these zeros and overparameterization.

Questions for authors

Q1. How do Theorems 1–4 change under standard fan-in initialization (variance 1/fan-in1/\mathrm{fan\text{-}in})? Which of the scaling rates in Table 1 still grow with the hidden dimension?

Q2. The LLM width sweep keeps the Hugging Face initialization std fixed as width grows (apply_variant in run_llm_attention_sweeps.py, lines 705–718); if the std is instead scaled as 1/width1/\sqrt{\mathrm{width}}, does R^\widehat{R} in Figure 6 still increase with width?

Q3. For matrix factorization (Theorem 1) and the exact-zero cases (hidden linear layers and WQ,WK,WVW_Q, W_K, W_V), can you report the dominance ratio of G0G0⊤G_0 G_0^\top for a single initialization, or bound its deviation from E[GG⊤]\mathbb{E}[GG^\top]? For the LLMs, can you report it for a single minibatch gradient instead of averaged per-sequence outer products?

5 comments
AuthorsOpenReview ↗

Part I (Response to W1)

We thank the reviewer for the time and effort spent on our paper. In response to the comments, we have added supplementary theoretical results to the revised manuscript and new experiments; the additional experimental results and the corresponding code are available at https://anonymous.4open.science/r/orthadapt-iclr-rebuttal-8107/. We address each point below.

W1

We thank the reviewer for this question. To further address the concern, below we first clarify that constant-variance initialization is a practical setting, widely used in LLM pretraining, and we provide theory and numerical evidence to support that in the three neural-network setups, diagonal dominance still holds under fan-in initialization for every weight matrix except the output layer.

On the constant-std initialization. Initializing weights with a constant standard deviation is standard practice in LLM pretraining, used by GPT-1 [1], Megatron-LM [2], OPT [3], Cerebras-GPT [4], DeepSeek-V2 [5], DeepSeek-V3 [6], OLMo 2 [7] and OLMoE [8]. For instance, OLMo 2 [7, Section 3.2] initializes every parameter from N(0,0.022)\mathcal{N}(0, 0.02^2) and reports that this scheme is more stable than the depth-scaled initialization of OLMo-0424 (their Figure 4); DeepSeek-V3 [6, Section 4.2] initializes all parameters with standard deviation 0.0060.006; and OLMoE [8, Section 4.2.2] notes that an std of 0.020.02 "has been commonly used as popularized by Shoeybi et al." These references are now cited in Section 3.2 of the revised manuscript.

Fan-in initialization results. We include the matrix-factorization setup because [9] analyzes Muon on it and lists the connection between Muon's preconditioner and the Hessian as future work. The diagonal dominance result for matrix factorization is affected by fan-in initialization, but as shown below, the main results for the other three neural network setups are not. The table gives the order of E[(GG⊤)aa]/E[(GG⊤)ab]\mathbb{E}[(GG^{\top})_{aa}]/\mathbb{E}[(GG^{\top})_{ab}] as the hidden dimension grows, with input/output dimensions and the target fixed; "∞\infty" means the off-diagonal entries are exactly zero.

SettingWeightN(0,1)\mathcal{N}(0,1) init (paper)Fan-in init (Var=1/fan-in\mathrm{Var}=1/\text{fan-in})
Matrix factorizationWWΘ(k)\Theta(k)Θ(1)\Theta(1)
Deep linear networkhidden layers W1,…,WL−1W_1,\dots,W_{L-1}∞\infty∞\infty
Deep linear networkoutput layer WLW_LΘ(∏jdj)\Theta(\prod_j d_j)Θ(1)\Theta(1)
Two-layer ReLUW1W_1Θ(d1)\Theta(d_1)Θ(d1)\Theta(d_1)
Two-layer ReLUW2W_2Θ(d1)\Theta(d_1)Θ(1)\Theta(1)
Linear attentionWQ,WK,WVW_Q,W_K,W_V∞\infty∞\infty
Linear attentionWOW_OΘ(dvd03)\Theta(d_v d_0^3)Θ(1)\Theta(1)

The fan-in results for all four setups are stated and proved in Appendix F of the revised manuscript (Theorems 8 to 11, Table 2), with a pointer in Section 3.2, and all entries are verified numerically and illustrated in Figures 5-7 in the linked repository, with their respective setups described in Appendix H.2-H.4 of the revised manuscript.

References

[1] Radford, A., et al. Improving Language Understanding by Generative Pre-Training. OpenAI Technical Report, 2018.

[2] Shoeybi, M., et al. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053, 2019.

[3] Zhang, S., et al. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068, 2022.

[4] Dey, N., et al. Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster. arXiv:2304.03208, 2023.

[5] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024.

[6] DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024.

[7] Team OLMo. 2 OLMo 2 Furious. arXiv:2501.00656, 2024.

[8] Muennighoff, N., et al. OLMoE: Open Mixture-of-Experts Language Models. arXiv:2409.02060, 2024.

[9] Ma, J., Huang, Y., Chi, Y., Chen, Y. Preconditioning Benefits of Spectral Orthogonalization in Muon. 2026.

AuthorsOpenReview ↗

Part II (Response to W1)

On the reviewer's underlying concern that the dominance is an artifact of the variance: the diagonal entry ∥ga∥2\Vert g_a\Vert^2 is the self inner product of a gradient row and the off-diagonal entry ⟨ga,gb⟩\langle g_a, g_b\rangle is the correlation between two rows, so diagonal dominance says that gradient rows are weakly correlated relative to their own norm. This is determined by how the forward and backward passes of the architecture compose the gradient. The variance rescales the gradient and sets the absolute order of the diagonal, so it is a sufficient but not a necessary condition.

Why fan-in initialization only affects the output layer. The gradient of a weight matrix WW is G=(backward error)×(forward input)⊤G = (\text{backward error}) \times (\text{forward input})^\top, and GG⊤GG^\top compares rows of GG, i.e., different output units of WW. Whether these rows are correlated depends on where WW sits in the network.

  • Hidden matrices. For a matrix whose error signal passes through downstream random weights, row aa of GG is driven by column aa of the next weight matrix WnextW_{\text{next}}. Different rows are thus driven by independent Gaussian columns of WnextW_{\text{next}}. The cross term E[GaGb⊤]\mathbb{E}[G_a G_b^\top] for a≠ba \neq b then contains the factor E[(Wnext)⋅a⊤(Wnext)⋅b]=0\mathbb{E}[(W_{\text{next}})_{\cdot a}^\top (W_{\text{next}})_{\cdot b}] = 0 by independence and zero mean, so the off-diagonal entries of E[GG⊤]\mathbb{E}[GG^\top] vanish regardless of the variance (deep linear hidden layers, attention Q,K,VQ,K,V). In the ReLU first layer the nonlinearity leaves a nonzero cross term, but it is produced by the same downstream weights as the diagonal term, so a change of initialization variance rescales both by the same factor and leaves the ratio at Θ(d1)\Theta(d_1). In short, for hidden matrices the ratio is determined by independence and symmetry in the surrounding weights, not by the initialization scale.

  • Output matrices. For a matrix that directly produces the output (WLW_L in the deep linear network, W2W_2 in the ReLU network, WOW_O in linear attention), there is no WnextW_{\text{next}}: row aa of GG is driven by row aa of the residual f(x)−Φxf(x) - \Phi x, where f(x)f(x) is the network output at initialization and Φx\Phi x is the target. The residual splits into a model part f(x)f(x), which is random and whose rows are driven by independent rows of the output weight, and a target part Φx\Phi x, which is deterministic and shared by all rows. The cross term E[GaGb⊤]\mathbb{E}[G_a G_b^\top] for a≠ba \neq b therefore vanishes for the model part, as in the hidden case, but not for the target part, which contributes (ΦΦ⊤)ab≠0(\Phi\Phi^\top)_{ab} \neq 0. The closed form is then E[GG⊤]=cI+c′ΦΦ⊤\mathbb{E}[GG^\top] = c I + c' \Phi\Phi^\top with the diagonal governed by the model term cc and the off-diagonal governed by the target term c′c' (Theorems 2-4 and Theorem 11 in Appendix F). Unlike the hidden case, these two terms are of different degrees in the weights, so a change of initialization variance rescales them differently. Under constant-variance initialization the output f(x)f(x) grows with the hidden dimension, cc dominates c′c', and the ratio diverges; under fan-in initialization the output is kept at unit scale by design, cc and c′c' are of the same order, and the ratio is Θ(1)\Theta(1). In short, for output matrices the ratio is determined by the scale of the initial output relative to the target, which is exactly what the initialization sets.

In summary, under fan-in initialization, the hidden-layer matrices remain diagonally dominant, since the dominance arises from over-parameterization through the forward structure of the network. The behavior of the output layer does not represent a failure of this mechanism.

For Figure 6. Our theory for attention is derived for a specific, representative model, which provides the intuition that increasing the hidden dimension is positively correlated with the strength of diagonal dominance. Practical models differ from this representative model, and nonlinearities and more complex architectural designs introduce additional factors that affect the exact growth order.

AuthorsOpenReview ↗

Part III (Response to W2,Q1,Q2,Q3)

W2

Single draw versus expectation. Block diagonal dominance of Muon's preconditioner for a single initialization has already been observed empirically along language-model training trajectories by [1], which motivated this work. The orders we derive for E[GG⊤]\mathbb{E}[GG^\top] can be understood as the typical behavior of GG⊤GG^\top in a statistical sense, and the scaling with the hidden dimension explains why this observed dominance arises. Also, in the theoretical study of Hessian heterogeneity in [2], the analysis of the Hessian structure is likewise carried out in expectation over the initialization.

Concentration and single-draw simulations. To address the concern directly, we provide a concentration result for matrix factorization in Appendix B.6 of the revised manuscript: Proposition 1 bounds the deviation of a single draw GG⊤GG^\top from E[GG⊤]\mathbb{E}[GG^\top] with high probability, and Theorem 5 shows that, with probability at least 1−η1-\eta, the diagonal-to-off-diagonal ratio of a single initialization grows at least like k\sqrt{k} with the hidden dimension. We also added numerical simulations for all four setups in which the ratio is computed for a single initialization (the diagonal ratio is calculated based on a single initialization), reported in Figures 2-4 in the linked repository, and further described in Appendix H of the revised manuscript.

Why does diagonality in expectation justify reducing nonlinear orthogonalization to row normalization. Block diagonal dominance of Muon's preconditioner for a single initialization has already been observed empirically along language-model training trajectories by [1], which motivated our work. The discussion in our appendix only explores the connection to [1] in an asymptotic-approximation sense, where diagonal dominance of GG⊤GG^\top relates orthogonalization to row normalization at the level of the preconditioner.

On the exact zeros and overparameterization. The reviewer is correct that the exact zeros for the hidden linear layers and for WQ,WK,WVW_Q, W_K, W_V are a symmetry identity, as Remark 3 states. Diagonal dominance, however, is a comparison between two quantities: the correlation between different gradient rows (off-diagonal) and the self inner product of a row (diagonal). In these settings, the architecture makes the row-to-row correlation vanish in expectation, while the self inner product is the second moment of a gradient row, which grows with the hidden dimension. Overparameterization can act on both the row correlation and the row self inner product; in these settings the symmetry removes the former in expectation and overparameterization acts through the latter. The dominance is therefore a statement about the dimensions of the network rather than about symmetry alone, and the single-draw simulations above identify which dimension controls the realized ratio.

References

[1] Deng, S., et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. 2026.

[2] Dong, Z., Zhang, Y., Luo, Z.-Q., Yao, J., Sun, R. Towards Quantifying the Hessian Structure of Neural Networks. 2025.

Q1. See the table in our response to W1. The same results are included in the revised manuscript as Table 2 in Appendix F, with the corresponding statements and proofs for deep linear networks, two-layer ReLU networks, and attention given in Theorems 8-11 of Appendix F. Figures 5-7 in the linked repository provide the simulation results under fan-in initialization, and are described further in Appendices H.2-H.4 of the revised manuscript.

Q2. Thank you for your suggestion. We repeated the experiment varying the model width, with the initialization standard deviation scaled as 1/width1/\sqrt{\text{width}}. As shown in Figure 1 of the linked repository and described in Appendix I.2 of the revised manuscript, R^\widehat R generally increases with model width for all four attention projections across Qwen, Granite, and Llama, indicating that the observed trend persists under this initialization scheme.

Q3. We provide a probabilistic bound on the deviation of a single draw GG⊤GG^\top from E[GG⊤]\mathbb{E}[GG^\top] for matrix factorization in Theorem 5 of Appendix B.6, and the single-initialization dominance ratios for all setups in the numerical simulations described in our response to W2 (Figures 2-4 in the linked repository). Further, we report the LLM results for a single minibatch gradient, as suggested, in Figure 14 in the linked repository and in Appendix I.2 of the revised manuscript.

We sincerely thank you for the time and effort devoted to reviewing our paper. If our responses have addressed your concerns, we would be grateful if you would consider re-evaluating the score. Please let us know if you have any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.