acceptodds
Under review as a conference paper at ICLR 2027

Toward a More Favorable Curvature Structure: A Basis Perspective on Matrix Optimizers

Abstract

Matrix optimizers have shown promising performance in large language model training. Although prior explanations have linked their effectiveness to curvature, the underlying mechanisms remain incompletely understood. To better understand these mechanisms, we view optimizer updates as basis selection followed by a diagonally preconditioned update in the induced basis. We find that bases induced by matrix optimizers can make curvature structure more favorable for diagonal preconditioning, partly explaining their effectiveness. Empirically, we examine Gauss–Newton curvature during training and find that (i) matrix-optimizer-induced bases generally yield greater diagonal concentration of curvature in hidden parameter blocks than the ambient basis used by optimizers such as Adam; (ii) in the basis induced by each optimizer, the scaling used by its diagonal preconditioner closely corresponds to the diagonal curvature; and (iii) optimizers with higher diagonal concentration of curvature in their induced bases generally achieve better training performance. We further support these findings with an initial theoretical analysis of quadratic problems and neural networks, identifying conditions under which bases induced by matrix optimizers yield more favorable curvature structure and examining how this structure can contribute to improved optimization performance.

Primary area:
optimization
Keywords:
large language model, optimization, curvature

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2X2QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper states that each optimizer can be rotated into an orthogonal basis and then apply the diagonal preconditioner in that basis. This idea includes SGD-based, Adam-like, SOAP, and Muon methods, as well as Gauss-Newton methods. They support their claims with theoretical and experimental results.

Critical strengths
  1. The paper develops good theoretical results through proofs.
  2. The paper is quite easy to follow.
Critical weaknesses
  1. Lack of reference, e.g., Adam and Lion
  2. What is this "≽" notation? Need to clarify.
  3. Figure 2: what are the x-axis and y-axis? (You should identify them for every plot)
  4. Actually, I didn't understand the relationship between Obs. 1 and Theorem 3. In obs 1, you mentioned "D^{-1} Diag(H^{Q}{GN}^{1/2}", while in Theorem 3 you explained a tighter bound: each optimizer’s D^{−1} Diag(H^{Q}{GN}
  5. It seems Theorem 2 just considers a weak condition (Haar basis). Actually, I didn't get!
Questions for authors
  1. The lack of Novelty. The idea is not well motivated. The diagonal curvature benefits of Adam were revealed earlier, and the rotate-then-diagonal precondition idea is well established in SOAP, KFAC, ... So the novelty is limited.
  2. The experiments are small. I wonder if it is possible to consider larger models, e.g., Llama 1.3B (I know the limited time, but I wonder if you have any ideas).
7 comments
AuthorsOpenReview ↗

Response (Part Ⅰ)

We sincerely thank the reviewer for the careful review and constructive feedback.

W1: “Lack of reference, e.g., Adam and Lion”

We would like to kindly clarify that both references were included in the original submission (Adam cited in [Section 1, line 035], and Lion in [Section 4.1, line 245]).


W2 & W3: “What is this "⪰\succeq" notation? Need to clarify.”; “Figure 2: what are the x-axis and y-axis? (You should identify them for every plot)”

Thank you very much for pointing out these notation issues. The notation ⪰\succeq denotes the positive semidefinite (PSD) ordering. In every panel of Figure 2, the x- and y-axes show the rescaled D−1D^{-1} and Diag⁡(HGNQ)1/2\operatorname{Diag}(H_{\mathrm{GN}}^{Q})^{1/2}, respectively, and play symmetric roles in the comparison. We will add the missing notion and axis labels in the revised version.


W4: “Actually, I didn’t understand the relationship between Obs. 1 and Theorem 3. In obs 1, you mentioned D−1Diag⁡(HGNQ)1/2D^{-1}\operatorname{Diag}(H_{\mathrm{GN}}^{Q})^{1/2}, while in Theorem 3 you explained a tighter bound: each optimizer’s D−1Diag⁡(HGNQ)D^{-1}\operatorname{Diag}(H_{\mathrm{GN}}^{Q})”

The analysis in Theorem 3 accommodates diagonal scaling with different exponents, including the inverse-square-root scaling observed in Observation 1. We will revise the paper accordingly to avoid confusion and present an updated Theorem 3 explicitly treating D∝Diag⁡(HGNQ)−αD\propto\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{-\alpha} for any α∈(0,1]\alpha\in(0,1], along with an expanded discussion to explain the result and clarify its connection to Observation 1.

Updated Theorem 3 [GN model descent under diagonal concentration and scaling]. Assume HGN≻0H_{\mathrm{GN}}\succ0 and g≠0g\ne0. With the original definitions,

\ge e^{\mathcal J(Q,D)}, \quad\text{where}\quad \mathcal J(Q,D) =\underbrace{\log\frac{\det H_{\mathrm{GN}}} {\prod_i(H_{\mathrm{GN}}^Q)_{ii}}}_{\text{diagonal concentration}} +\underbrace{d\log\frac{\left(\prod_iD_{ii}(H_{\mathrm{GN}}^Q)_{ii}\right)^{1/d}} {\frac1d\sum_iD_{ii}(H_{\mathrm{GN}}^Q)_{ii}}}_{\text{diagonal scaling alignment}} .$$ In particular, let $Q_1,Q_2\in\mathrm{O}(d)$ and suppose the diagonal vectors of $H_{\mathrm{GN}}^{Q_1}$ and $H_{\mathrm{GN}}^{Q_2}$ are comparable by majorization. For $D_k\propto\operatorname{Diag}(H_{\mathrm{GN}}^{Q_k})^{-\alpha}$, $k=1,2$, with $\alpha\in(0,1]$, it holds that $$\rho_{\mathrm{diag}}(H_{\mathrm{GN}}^{Q_1}) \ge \rho_{\mathrm{diag}}(H_{\mathrm{GN}}^{Q_2}) \quad\Longrightarrow\quad e^{\mathcal J(Q_1,D_1)}\ge e^{\mathcal J(Q_2,D_2)}.$$ **Expanded discussion of updated Theorem 3.** This theoretical result links the preconditioned GN-model decrease to both GN diagonal concentration in the basis $Q$ and the diagonal preconditioner $D$ applied within it. Specifically, a basis with strong GN diagonal concentration, paired with a preconditioner that effectively exploits its diagonal curvature, can guarantee a substantial fraction of the optimal GN-model decrease. As both terms in $\mathcal J(Q,D)$ approach zero, the guaranteed decrease approaches this optimum. Also, to focus more directly on the role of basis choice in training performance, and **motivated by the finding in Observation 1 (Section 4.1)** that different optimizers’ diagonal preconditioners generally align closely with GN diagonal curvature, we further establish a cross-basis comparison of the guaranteed preconditioned GN-model decrease in terms of the GN diagonal Frobenius norm ratio under the common scaling rule $D\propto\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{-\alpha}$, with fixed $\alpha\in(0,1]$. The additional majorization condition is also empirically supported for the ambient and momentum-SVD bases, as observed in [[New Results: Figure 10(b) in the revised paper]](https://anonymous.4open.science/r/iclr27-rebuttal-figs-B025/step002500_gn_majorization_captioned.pdf). These results collectively offer a theoretical explanation for why stronger GN diagonal concentration favors diagonal preconditioners that effectively exploit GN diagonal curvature, consistent with the training performance observed in Figure 5. The proof is given in [Appendix E.4 in the revised paper]. **Final remark: relationship between Observation 1 and updated Thm. 3.** The updated Thm. 3 **covers** the alignment pattern observed in Observation 1 and extends the analysis to a **broader** range of scaling exponents: cross-basis comparison of guaranteed preconditioned GN-model decrease in terms of the GN diagonal Frobenius norm ratio under $D \propto \\,\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{-\alpha}$, for **any** $\alpha\in(0,1]$. The alignment examined in Observation 1 corresponds to the **special case** $\alpha=\tfrac12$ in the updated Thm. 3, where $D\propto\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{-1/2}$.
AuthorsOpenReview ↗

Response (Part Ⅱ)

W5: “It seems Theorem 2 just considers a weak condition (Haar basis).”

Thanks for the question. Let us clarify: Theorem 2 indeed focuses on the gap between the GN diagonal Frobenius norm ratio in the gradient SVD basis and its expectation under Haar-random rotations. For comparison with the ambient basis, we also derive separate conditions ensuring higher ratios in bases induced by different matrix optimizers (Appendix F).

Together, these two results provide complementary theoretical support for our empirical observations and analysis.

Comparison with Haar-random bases in Theorem 2.

II We use the expected GN diagonal Frobenius norm ratio under random rotation as a baseline for average diagonal concentration across orthogonal bases. This comparison highlights the unusually strong diagonal concentration observed in the gradient SVD basis. (II) The analysis also helps explain the weak GN diagonal concentration observed after random rotation in Figure 5.

Comparison with the ambient basis in Appendix F.

II These results support our observation that GN diagonal concentration is substantially stronger in bases induced by matrix optimizers than in the ambient basis. (II) The sufficient conditions for this improvement offer insight into cases where matrix optimizers may outperform methods that apply diagonal preconditioning in the ambient basis, such as standard Adam.


Q1: “The lack of Novelty. The idea is not well motivated. The diagonal curvature benefits of Adam were revealed earlier, and the rotate-then-diagonal precondition idea is well established in SOAP, KFAC, ... So the novelty is limited.”

Thank you for the constructive feedback. Since Adam already exploits diagonal curvature in ambient coordinates, a fundamental question arises: why do matrix optimizers achieve further performance gains in large model training? Our work is inspired by prior studies on Adam’s diagonal curvature benefits and coordinate-transformation methods such as KFAC and SOAP.

To address this question, we develop an explanatory chain connecting optimizer-induced bases, Gauss–Newton (GN) curvature representation, within-basis scaling, and optimization effects. While these two existing lines of work provide valuable conceptual starting points, they do not by themselves establish the specific explanatory chain studied here. We establish and support these connections through the following two new contributions.

1. New Gauss–Newton Curvature Structure Across Bases

Across coordinate systems, diagonal GN concentration exhibits an overall empirical ordering of Random < Ambient < Matrix-Optimizer-Induced: Matrix-optimizer-induced bases such as Muon’s momentum SVD basis and SOAP’s gradient-Gram eigenbasis concentrate substantially more GN curvature on the diagonal than the ambient basis. This finding is non-trivial: coordinate systems derived from momentum or gradient statistics do not automatically guarantee stronger GN diagonal concentration.

  • Empirical evidence. In Pythia-70M, the momentum SVD basis raises the GN diagonal Frobenius norm ratio from approximately 11–18% in the ambient basis to 59–84% across the examined training checkpoints (Figure 3). Figures 4–5 further show this stronger concentration across most individual hidden blocks, different models, and actual optimizer training trajectories.

  • Theoretical support: Theorem 2 compares an appropriate gradient SVD basis against the Haar-random expectation, while Appendix F provides sufficient conditions for momentum SVD and population SOAP bases to strictly improve over the ambient basis.

2. A New Theoretical Explanation for Matrix-Optimizer Gains

The choice of basis determines which Gauss–Newton curvature information enters the diagonal representation, while coordinate scaling governs how that curvature is exploited. Our theoretical analysis shows how a richer diagonal representation with appropriate coordinate scaling can support more effective preconditioning. Together, basis representation and within-basis scaling help explain why matrix optimizers achieve gains over ambient diagonal preconditioning.

  • Theoretical connection: Theorem 3 links local GN model decrease to diagonal concentration and diagonal scaling alignment, explaining the joint role of curvature representation and within-basis scaling.

  • Empirical support: Across optimizer-induced bases, inverse optimizer scaling closely corresponds to square-root diagonal GN curvature during training (Figure 2, Table 2). This correspondence provides evidence that these optimizers actively exploit diagonal curvature within their operating coordinates.

Our new findings and analyses explain matrix optimizers’ advantage over Adam, a phenomenon not fully explained by prior insights on diagonal curvature and rotated preconditioning.

AuthorsOpenReview ↗

Response (Part Ⅲ)

Q2: “The experiments are small. I wonder if it is possible to consider larger models, e.g., Llama 1.3B (I know the limited time, but I wonder if you have any ideas).”

Thanks for the suggestion and your understanding.

Additional 1B results. We extend the GN curvature analysis in Figure 7 from Pythia-410M to Pythia-1B at steps 32k, 64k, and 128k of its 143k-step pretraining run.

For each weight matrix type, we report:

=\frac{\rho_{\mathrm{diag}}(H_{\mathrm{GN}}^Q)} {\rho_{\mathrm{diag}}(H_{\mathrm{GN}})}, \quad \text{where } Q \text{ is the momentum SVD basis.}$$ **Table: SVD-to-Ambient Diagonal-Norm Ratio through training.** | Matrix Type | Step 32k | Step 64k | Step 128k | | --- | --- | --- | --- | | Q | 2.65 | 2.76 | 2.27 | | K | 2.00 | 2.30 | 1.43 | | V | 1.76 | 1.55 | 1.67 | | O | 4.16 | 3.10 | 2.12 | | MLP FC | 1.16 | 1.25 | 1.50 | | MLP Proj | 5.92 | 5.94 | 6.64 | The pooled SVD-to-ambient diagonal-norm ratios of **$1.16$–$6.64$** support greater GN diagonal concentration in the momentum SVD basis at the 1B scale. We will include the experiment results in the revised paper [[New Results: Figure 8 in the revised paper]](https://anonymous.4open.science/r/iclr27-rebuttal-figs-B025/figure8-pythia-1b.pdf). **Difficulty in extending the remaining experiments to 1B.** Our goal is to explain matrix optimizer performance through detailed curvature analysis, which is already computationally demanding at our existing model sizes. For example, extending the coordinate-wise analysis in Figure 2 and Table 2 via direct exact evaluation of the full GN diagonal for a 1B-parameter model would require: - **$10^9$ GN–vector products** per basis and checkpoint; - each applying a **$10^9\times10^9$ GN operator to a $10^9$-dimensional vector**. Given this workload, the additional training required, and the limited rebuttal time, we prioritized the diagonal-concentration extension above. Thanks again for the great questions. We are happy to further discuss any other potential follow-up questions.
Reviewer_UFBeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies matrix optimizers by decomposing their updates into basis selection and diagonal preconditioning. It reports greater Gauss–Newton (GN) diagonal concentration in optimizer-induced bases, particularly the momentum SVD basis, than in the ambient basis. Through theoretical analysis and numerical experiments, it examines how optimizer scaling corresponds to diagonal curvature and how diagonal concentration relates to training performance.

Critical strengths
  1. A useful unifying perspective The basis-selection perspective provides a useful framework for relating Adam, Muon, and SOAP. The distinct observations for hidden blocks, embeddings, and the LM head add detail beyond an aggregate comparison.

  2. An informative theoretical decomposition Theorem 3 explicitly separates diagonal scaling mismatch from residual off-diagonal curvature, identifying concrete quantities that help characterize when a favorable basis can translate into improved optimization.

Critical weaknesses
  1. My main concern is that the paper studies matrix optimizers primarily from an optimization perspective, while their potential generalization benefits remain insufficiently understood. For example, Figure 5 relates diagonal concentration to optimizer performance using validation loss. However, lower validation loss at a fixed training budget may reflect faster optimization, better generalization, or both. The local GN analysis does not distinguish these effects. Comparing validation loss at matched training loss would help assess whether the observed advantage extends beyond optimization efficiency and clarify the scope of the proposed explanation.

  2. The presentation could also be improved. For example, Theorem 3 has a detailed technical statement but relatively little accompanying explanation, making its main insights and relevance to the broader argument difficult to be captured. Simplifying the statement or expanding the discussion of its intuition and implications might help readers understand the result and its role in the paper.

Questions for authors

Please see the weaknesses part.

At the current stage, I consider the paper borderline for acceptance at a top-tier ML conference, with a slight lean toward acceptance. I remain open to revising my assessment based on the authors’ response and subsequent discussion.

3 comments
AuthorsOpenReview ↗

Response (Part Ⅰ)

We are grateful for the careful review of our paper and the great questions. Please find our respectful reply below.

W1: “My main concern is that the paper studies matrix optimizers primarily from an optimization perspective, while their potential generalization benefits remain insufficiently understood.”; “Comparing validation loss at matched training loss would help assess whether the observed advantage extends beyond optimization efficiency and clarify the scope of the proposed explanation.”

Thank you for raising this concern. We agree that validation loss at a fixed training budget cannot distinguish optimization efficiency from potential generalization differences. Following your suggestion, we compared training and validation loss on nanoGPT 124M and Llama 130M in [New Experiments: Figure 11 in the revised paper].

(1) Faster training-loss reduction. To make training loss comparable across methods and checkpoints, we evaluated each checkpoint on the same fixed set of previously seen training examples. As shown in the left panels, SOAP, Muon, and KL-SOAP reach a given training loss in substantially fewer steps than Adam, whereas Randomly Rotated Adam requires more steps.

(2) Similar validation loss at matched training loss. Plotting validation loss against training loss from the same checkpoints shows that all five methods’ curves nearly overlap within their common training-loss range (right panels). At three targets within this range, validation losses obtained by linear interpolation between adjacent checkpoints differ from Adam’s by less than 0.0050.005 in absolute value for both models. These results support interpreting the fixed-budget validation advantages in Figure 5 primarily in terms of optimization progress, consistent with the optimization focus of our local Gauss–Newton analysis.

AuthorsOpenReview ↗

Response (Part Ⅱ)

W2: “The presentation could also be improved; Simplifying the statement or expanding the discussion of its intuition and implications might help readers understand the result and its role in the paper.”

Thanks for the valuable suggestion. We will revise the paper accordingly to improve the presentation of Theorem 3. In the revised version, we will provide an updated Theorem 3 with deeper insight and a clearer statement, along with an expanded discussion.

Updated Theorem 3 [GN model descent under diagonal concentration and scaling] . Assume HGN≻0H_{\mathrm{GN}}\succ0 and g≠0g\ne0. With the original definitions,

\ge e^{\mathcal J(Q,D)}, \quad\text{where}\quad \mathcal J(Q,D) =\underbrace{\log\frac{\det H_{\mathrm{GN}}} {\prod_i(H_{\mathrm{GN}}^Q)_{ii}}}_{\text{diagonal concentration}} +\underbrace{d\log\frac{\left(\prod_iD_{ii}(H_{\mathrm{GN}}^Q)_{ii}\right)^{1/d}} {\frac1d\sum_iD_{ii}(H_{\mathrm{GN}}^Q)_{ii}}}_{\text{diagonal scaling alignment}} .$$ In particular, let $Q_1,Q_2\in\mathrm{O}(d)$ and suppose the diagonal vectors of $H_{\mathrm{GN}}^{Q_1}$ and $H_{\mathrm{GN}}^{Q_2}$ are comparable by majorization. For $D_k\propto\operatorname{Diag}(H_{\mathrm{GN}}^{Q_k})^{-\alpha}$, $k=1,2$, with $\alpha\in(0,1]$, it holds that $$\rho_{\mathrm{diag}}(H_{\mathrm{GN}}^{Q_1}) \ge \rho_{\mathrm{diag}}(H_{\mathrm{GN}}^{Q_2}) \quad\Longrightarrow\quad e^{\mathcal J(Q_1,D_1)}\ge e^{\mathcal J(Q_2,D_2)}.$$ **Expanded discussion of updated Theorem 3.** This theoretical result links the preconditioned GN-model decrease to both GN diagonal concentration in the basis $Q$ and the diagonal preconditioner $D$ applied within it. Specifically, a basis with strong GN diagonal concentration, paired with a preconditioner that effectively exploits its diagonal curvature, can guarantee a substantial fraction of the optimal GN-model decrease. As both terms in $\mathcal J(Q,D)$ approach zero, the guaranteed decrease approaches this optimum. Also, to focus more directly on the role of basis choice in training performance, and motivated by the observation in Section 4.1 that different optimizers’ diagonal preconditioners generally align closely with GN diagonal curvature, we further establish a cross-basis comparison of the guaranteed preconditioned GN-model decrease in terms of the GN diagonal Frobenius norm ratio under the common scaling rule $D\propto\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{-\alpha}$, with fixed $\alpha\in(0,1]$. The additional majorization condition is also empirically supported for the ambient and momentum-SVD bases, as observed in [[New Results: Firgure 10(b) in the revised paper]](https://anonymous.4open.science/r/iclr27-rebuttal-figs-B025/step002500_gn_majorization_captioned.pdf). These results collectively offer a theoretical explanation for why stronger GN diagonal concentration favors diagonal preconditioners that effectively exploit GN diagonal curvature, consistent with the training performance observed in Figure 5. The proof is given in [Appendix E.4 in the revised paper]. Thanks again for your valuable feedback. We warmly welcome any further questions and discussion.
AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer,

Thanks once again for the time and effort you have invested in reviewing our paper. We greatly appreciate your insightful comments and suggestions. We have replied to your concerns and questions in our responses above. Please feel free to review them at your convenience.

We would like to briefly summarize below the main improvements based on your kind suggestions:

  • New experiments comparing validation loss at matched training loss (please see Part I of our response for details).

    [New Experiments: Figure 11 in the revised paper]

  • Updated Theorem 3 with a clearer statement and expanded discussion (please see Part II of our response for details).

We hope these additional experiments and improvements address your concerns. We would greatly appreciate your feedback and warmly welcome further discussion.

Reviewer_uaiKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper views matrix optimizers as basis selection followed by diagonal preconditioning. It studies how these bases affect the diagonal concentration of Gauss–Newton (GN) curvature and uses experiments and theory to connect this structure with training performance.

Critical strengths
  1. The paper studies an important topic. Matrix optimizers that use rotations, such as SOAP, ARO, and Muon, are still not well understood. The basis perspective is interesting.

  2. The paper presents interesting empirical findings. In particular, GN curvature has stronger diagonal concentration in the SVD basis than in the original basis. This pattern appears across different layers and parameter blocks.

  3. Section 4.3 makes a useful attempt to connect diagonal concentration with training performance and explain the performance differences between optimizers.

Critical weaknesses
  1. The novelty of Contribution 1 is unclear. Diagonal and block-diagonal curvature structures have been studied before. For example, [1, 2] report a block-diagonal Hessian structure and relate it to network architecture. The authors should explain more clearly what their GN analysis across bases adds beyond these findings and how it connects to prior work.

  2. Contribution 3 does not clearly explain how the empirical findings account for the behavior of actual optimizers. The results remain mostly descriptive and provide limited insight into why these optimizers work or how to design better ones. For example, AdamW shows stronger alignment in Table 2 than Muon, yet performs worse in Figure 5. This gap needs a clearer explanation. The comparison also focuses on Muon and SOAP, leaving out related methods such as KL-SOAP and ARO.

  3. The experiments are limited to small models. The largest model analyzed has 410M parameters, while the largest model used in the training comparisons has 130M parameters. Increasing model width may change the curvature spectrum and lead to larger condition numbers. Without results on larger models, it is unclear whether the same observations still hold.

[1] Y. Zhang et al. Adam-mini: Use Fewer Learning Rates To Gain More. ICML 2026

[2] Y. Zhang et al. Why Transformers Need Adam: A Hessian Perspective. Neurips 2024

Questions for authors
  1. The cosine similarities in Table 2 are computed over different numbers of nonzero coordinates for different optimizers. Are these scores directly comparable? How does the number of coordinates affect the comparison, especially the higher similarity reported for AdamW?

  2. Table 2 shows that AdamW has stronger alignment between D−1D^{-1} and Diag⁡(HGNQ)1/2\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{1/2} than Muon. Why does AdamW still perform worse in Figure 5? Can the authors explain more clearly how basis choice and diagonal scaling jointly account for this result?

  3. If basis choice is the main reason for Muon's advantage, how do the differences in diagonal concentration between Muon and SOAP relate to their relatively small performance gap in Figure 5? What role does the diagonal update rule play?

  4. Why does randomly rotated AdamW perform worse than standard AdamW? What makes the identity basis better than a random orthogonal basis?

  5. Could the authors provide results on larger models, for example by analyzing checkpoints from a model with around 1B parameters, to check whether the same observations hold?

9 comments
AuthorsOpenReview ↗

Response (Part Ⅰ)

We sincerely thank the reviewer for the insightful comments and constructive feedback. We respectfully respond to each point below.

W1: “The novelty of Contribution 1 is unclear. The authors should explain more clearly what their GN analysis across bases adds beyond existing findings such as those in [1, 2] and how it connects to prior work.”

Thank you for the valuable suggestion. We would like to provide further clarification.

Prior work and our contribution. Cited ambient-coordinate studies establish near-block-diagonal structure, block heterogeneity, and diagonal concentration in neural network Hessians [1–3, 5, 6], linking them to architecture and Adam’s advantage over SGD [2]. These ambient-coordinate curvature findings alone do not explain the performance advantages of matrix optimizers. Our contribution is a new finding that matrix-optimizer-induced bases further strengthen GN diagonal concentration beyond the ambient basis, together with a theoretical explanation of how this structure contributes to their performance advantages, as detailed below.

GN diagonal concentration across bases. We find that GN diagonal concentration increases from random bases to the ambient basis and further to matrix-optimizer-induced bases, yielding the empirical ordering: Random < Ambient < Matrix-Optimizer-Induced. We detail both comparisons and their empirical and theoretical support below.

(a) Random < Ambient: diagonal concentration in GN curvature.

While the cited studies examine Hessian structure, we show that GN curvature itself also exhibits diagonal concentration in the ambient basis. Our measurements along training trajectories show roughly 2×2\times–4×4\times higher concentration than under random rotations (Figure 5). These results extend prior observations of Hessian diagonal concentration to the GN curvature surrogate.

(b) Ambient < Matrix-Optimizer-Induced: uncovering GN structure beyond ambient coordinates.

We find stronger GN diagonal concentration in the bases used by matrix optimizers than in the ambient basis. This holds for both Muon’s momentum SVD basis and the eigenbasis of SOAP’s accumulated gradient Gram matrices. Our empirical and theoretical results support this finding, which goes beyond the curvature structure examined in the cited ambient-coordinate studies.

  • Empirical evidence. In Pythia-70M, the momentum SVD basis raises the GN diagonal Frobenius norm ratio from approximately 11–18% in the ambient basis to 59–84% across the examined training checkpoints (Figure 3). Figures 4–5 further show this stronger concentration across most individual hidden blocks, different models, and actual optimizer training trajectories.

  • Theoretical support. Theoretically, we establish a quantitative comparison with the Haar-random expectation for a suitable gradient SVD basis (Theorem 2), and sufficient conditions for momentum SVD and population SOAP bases to improve strictly over the ambient basis (Appendix F).

Importance of our newly observed GN structure. In the submission, Theorem 3 connects GN diagonal concentration in an optimizer-induced basis to the effectiveness of diagonal preconditioning, together with within-basis scaling. Combined with our cross-basis findings, this connection helps explain how the stronger GN diagonal concentration in matrix-optimizer-induced bases contributes to their performance advantages. This establishes the importance of our newly identified structure: it provides a curvature-based explanation for why diagonal preconditioning can be more effective in matrix-optimizer-induced bases.

References.

[1] Y. Zhang et al. Adam-mini: Use Fewer Learning Rates To Gain More. ICLR 2025.

[2] Y. Zhang et al. Why Transformers Need Adam: A Hessian Perspective. Neurips 2024

[3] Z. Dong et al. Towards Quantifying the Hessian Structure of Neural Networks. arXiv:2505.02809, 2025.

[4] S. Deng et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. ICML 2026.

[5] J. Kohler et al. Adaptive Norms for Deep Learning with Regularized Newton Methods. arXiv:1905.09201v4, 2020.

[6] R. Podorozhny. Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View. arXiv:2608.22145, 2026.

AuthorsOpenReview ↗

Response (Part Ⅱ)

W2 & Q2: “Contribution 3 does not clearly explain how the empirical findings account for optimizer behavior or inform optimizer design.”; “In particular, why does AdamW perform worse than Muon in Figure 5 despite showing stronger alignment between D−1D^{-1} and Diag⁡(HGNQ)1/2\operatorname{Diag}(H_{\mathrm{GN}}^{Q})^{1/2} in Table 2? How do basis choice and diagonal scaling jointly explain this result?”; “The comparison also focuses on Muon and SOAP, leaving out related methods such as KL-SOAP and ARO.”

2-1. Further interpretation of our empirical findings.

Thank you for the comment. Below, we will explain each observation in more detail and clarify the underlying connections between them.

First, Table 2 (together with Figure 2) shows that each optimizer’s diagonal preconditioner aligns closely with GN diagonal curvature in its own basis. This alignment indicates that the scaling captures diagonal curvature information, an important prerequisite of effective diagonal preconditioning. Yet the alignment scores can NOT directly reflect the effectiveness of diagonal preconditioners. We illustrate this with a conditioning bound and an empirical comparison.

  • In the field of optimization, a standard way to assess a diagonal preconditioner DD is to examine the condition number of DHGNQDH_{\mathrm{GN}}^Q. For HGNQ≻0H_{\mathrm{GN}}^Q\succ0 and a positive diagonal DD:
\ge \frac{\max_i D_{ii}(H_{\mathrm{GN}}^Q)_{ii}} {\min_i D_{ii}(H_{\mathrm{GN}}^Q)_{ii}}.$$ As shown, misalignment leads to a large condition number, limiting the efficacy of diagonal preconditioning. However, better alignment may only improve this lower bound, without guaranteeing a smaller condition number. - Comparison between Adam and Lion in the ambient basis provides direct evidence: both achieve strong alignment ($\gtrsim 0.8$ shown in Table 2). Although Adam scores substantially higher, the two optimizers generally perform similarly in practice. **Second,** when we view different optimizers as applying the diagonal preconditioners discussed above to effectively exploit GN diagonal curvature in their respective bases, **basis choice plays a central role in training performance**. The experiments in Figure 5 show: - Both training performance and GN diagonal concentration vary substantially across different basis types, yet their rankings are closely aligned: **stronger GN diagonal concentration generally corresponds to better training performance**. - Across models (NanoGPT and Llama), the relative ordering of basis types remains **consistent** in both training performance and GN diagonal concentration. In particular, matrix optimizers that use bases built on the gradient SVD, such as Muon and SOAP, share the **top** tier in both respects. This echoes their recurring performance advantage in practice. Together, these findings explain why Muon outperforms AdamW despite AdamW’s higher alignment score in Table 2. They also suggest a potential principle for optimizer design: find bases that yield stronger GN diagonal concentration at low computational cost and apply existing diagonal preconditioners in them. **2-2 Additional comparison.** Given the limited rebuttal time, we prioritized KL-SOAP, whose covariance-based basis construction is more closely aligned with the curvature perspective studied in our work. We extended Figure 5 to include KL-SOAP on nanoGPT 124M and Llama 130M under the original experimental settings. See [[Updated Figure 5 including KL-SOAP]](https://anonymous.4open.science/r/iclr27-rebuttal-figs-B025/figure5-five-methods.pdf). Across both nanoGPT and Llama, KL-SOAP exhibits a similar pattern to SOAP and Muon: its induced basis yields substantially higher GN diagonal concentration than the ambient basis, while its final validation loss is comparable to theirs. These observations are consistent with our interpretation that optimizer-induced bases expose more GN curvature to diagonal preconditioning, and extend the empirical support for this perspective to KL-SOAP. The plot below presents the extended comparison at matched training steps and matched validation loss.
AuthorsOpenReview ↗

Response (Part Ⅲ)

W3 & Q5: “The experiments are limited to small models. The largest model analyzed has 410M parameters, while the largest model used in the training comparisons has 130M parameters.”; “Could the authors provide results on larger models, for example by analyzing checkpoints from a model with around 1B parameters, to check whether the same observations hold?”

Thank you for the constructive suggestion. We extend the GN curvature analysis of Pythia-410M in Figure 7 to Pythia-1B, evaluating checkpoints at steps 32k, 64k, and 128k of its 143k-step pretraining run. In every tested checkpoint, the momentum SVD basis concentrates substantially more GN curvature onto the diagonal than the ambient basis. For each weight matrix type, the table reports the ratio of GN diagonal concentration in the momentum SVD basis to that in the ambient basis:

=\frac{\rho_{\mathrm{diag}}(H_{\mathrm{GN}}^Q)} {\rho_{\mathrm{diag}}(H_{\mathrm{GN}})}, \quad \text{where } Q \text{ is the momentum SVD basis.}$$ **Table: SVD-to-Ambient Diagonal-Norm Ratio through training.** | Matrix Type | Step 32k | Step 64k | Step 128k | | --- | --- | --- | --- | | Q | 2.65 | 2.76 | 2.27 | | K | 2.00 | 2.30 | 1.43 | | V | 1.76 | 1.55 | 1.67 | | O | 4.16 | 3.10 | 2.12 | | MLP FC | 1.16 | 1.25 | 1.50 | | MLP Proj | 5.92 | 5.94 | 6.64 | Across all six weight matrix types and three checkpoints, the pooled SVD-to-ambient diagonal-norm ratios range from $1.16$ to $6.64$, extending our observation of greater GN diagonal concentration in the momentum SVD basis to Pythia-1B. We will include the experiment results in the revised paper [[Figure 8 in the revised paper]](https://anonymous.4open.science/r/iclr27-rebuttal-figs-B025/figure8-pythia-1b.pdf). --------- **Q1:** *“The cosine similarities in Table 2 are computed over different numbers of nonzero coordinates for different optimizers.”* *“Are these scores directly comparable, and how does the number of coordinates affect the comparison, especially AdamW’s higher similarity?”* Thanks for the great question. We would like to first clarify why we compute cosine similarities over each optimizer’s nonzero coordinates in Table 2: **preconditioner values at zero coordinates do not affect the update**. Specifically, under our rotate-then-precondition formulation in Eq. (1), $$w^{+}=w-\eta QDQ^{\top}g,$$ if $(Q^{\top}g)_i=0$, then $D_{ii}(Q^{\top}g)_i=0$ for any $D_{ii}$, so this diagonal entry does not affect the update. We also appreciate and fully understand the reviewer’s concern about the comparability of cosine similarities across optimizers with different numbers of nonzero coordinates, and therefore conduct the following new experiments based on Table 2. **New experiments: Cosine similarity under matched nonzero coordinate counts.** We randomly subsample nonzero ambient coordinates to match the nonzero momentum-SVD coordinate count (500 independent trials), then compute cosine similarities for Adam and Lion on these subsets to eliminate coordinate-count effects. **Table: Full vs. Subsampled results (Mean, Min, Max, and Central 95% range).** | Checkpoint | Method | Full | Mean | Min | Max | Central 95% range | | --- | --- | --- | --- | --- | --- | --- | | 25% steps | Adam | 0.9649 | 0.9649 | 0.9449 | 0.9720 | \[0.9537, 0.9696\] | | 25% steps | Lion | 0.7873 | 0.7878 | 0.7584 | 0.8048 | \[0.7742, 0.7999\] | | 50% steps | Adam | 0.9675 | 0.9673 | 0.9287 | 0.9748 | \[0.9509, 0.9728\] | | 50% steps | Lion | 0.7893 | 0.7896 | 0.7632 | 0.8051 | \[0.7790, 0.7990\] | | 75% steps | Adam | 0.9788 | 0.9790 | 0.9623 | 0.9820 | \[0.9729, 0.9815\] | | 75% steps | Lion | 0.7883 | 0.7890 | 0.7462 | 0.8040 | \[0.7771, 0.7995\] | The new results show that the scores can reflect average similarity across each optimizer’s nonzero coordinates and are **virtually unaffected by differences in coordinate counts**. ----- **Q3:** *“If basis choice is the main reason for Muon’s advantage, how do the differences in diagonal concentration between Muon and SOAP relate to their relatively small performance gap in Figure 5? What role does the diagonal update rule play?”* Thanks for this interesting question. We would like to highlight that, across modded-nanoGPT and Llama, **both** the performance gap and the difference in GN diagonal concentration between Muon and SOAP are small relative to their gains over the other methods: they differ by at most $5\\%$ in GN ratio (roughly one quarter of their gap with AdamW) and by around $2\\%$ in performance near the end of training. The close correspondence between performance and GN diagonal concentration in the induced bases reinforces our finding that basis choice plays a central role in the performance gains of matrix optimizers such as Muon and SOAP. It also supports our explanation of the role of the diagonal update rule (see the response to W2 & Q2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes and some further clarifications

Dear reviewers and ACs,

We sincerely thank all reviewers and ACs for their time and effort in reviewing our paper and providing valuable feedback. We have carefully addressed each concern and revised the paper accordingly. Revisions made during the rebuttal discussion are highlighted in red for ease of review. We summarize the key changes below.

1. New experiment in Section 4.4 (page 10). Following W3 & Q5 by uaiK and Q2 by 2X2Q, we extended our GN curvature analysis to Pythia-1B. Specifically, the momentum-SVD basis exhibits substantially stronger GN diagonal concentration than the ambient basis across all six weight matrix types and three checkpoints, further substantiating our findings at a larger model scale.

2. Updated experiment in Section 4.3 (page 8). Following W2 by uaiK, we added KL-SOAP to the comparison of GN diagonal concentration and training performance on modded-nanoGPT and Llama. KL-SOAP joins Muon and SOAP in exhibiting stronger GN diagonal concentration and better training performance, extending our findings to a broader range of matrix optimizers.

3. New experiment in Appendix G.1 (page 29). Following W1 by UFBe, we added training-loss curves and compared validation losses at matched training loss on modded-nanoGPT and Llama to distinguish optimization advantages from potential generalization benefits. Muon, SOAP, and KL-SOAP reach a given training loss in fewer steps than Adam but achieve nearly identical validation losses at matched training loss, supporting our explanation of their performance gains from an optimization standpoint.

4. Updated discussion in Section 4.3 (pages 8–9). To better connect our theoretical analysis with the empirical findings, we updated Theorem 3 and expanded its accompanying discussion. We hope these revisions provide a clearer theoretical explanation of why GN diagonal concentration favors diagonal preconditioning, the central finding of Section 4.3.

We are grateful for the reviewers' insightful comments and believe these revisions improve the clarity and overall quality of our paper. We hope the new results above and additional discussion address their concerns. Thank you again for your careful review.


Further clarifications on our contribution. Our main motivation is to explain the performance advantages of matrix optimizers over diagonally preconditioned methods such as Adam by analyzing GN curvature structure. Building on the rebuttal discussion, we would like to further highlight and summarize our key findings beyond prior work and their significance.

  • New GN curvature structure in optimizer-induced bases beyond the ambient basis. We extend prior analyses of curvature structure in the ambient basis to GN curvature across optimizer-induced bases, revealing substantial differences in diagonal concentration. In particular, matrix-optimizer-induced bases exhibit stronger concentration than the ambient basis across models, hidden parameter blocks, and training stages. This finding is nontrivial: bases derived from gradient or momentum statistics do not automatically yield stronger GN diagonal concentration. Theorem 2 and Appendix F provide theoretical support for these cross-basis structural differences.

  • Explanation of matrix optimizers' performance gains from a rotated GN curvature perspective. Combining the above structural findings with the observation that each optimizer's diagonal preconditioner effectively exploits GN diagonal curvature in its own basis, we highlight the central role of basis choice: stronger GN diagonal concentration accompanies better training performance across models (Figure 5). Our theoretical analysis supports this connection by linking GN diagonal concentration and within-basis scaling to preconditioned GN conditioning and local model decrease. Together, these findings provide a curvature-based explanation for the shared performance advantages of matrix optimizers over diagonally preconditioned methods such as Adam.

We believe our findings can also offer valuable insights to the optimization community and inspire future optimizer design. Finally, we sincerely thank all reviewers and ACs once again for their time and constructive feedback on our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.