- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper states that each optimizer can be rotated into an orthogonal basis and then apply the diagonal preconditioner in that basis. This idea includes SGD-based, Adam-like, SOAP, and Muon methods, as well as Gauss-Newton methods. They support their claims with theoretical and experimental results.
Critical strengths
- The paper develops good theoretical results through proofs.
- The paper is quite easy to follow.
Critical weaknesses
- Lack of reference, e.g., Adam and Lion
- What is this "≽" notation? Need to clarify.
- Figure 2: what are the x-axis and y-axis? (You should identify them for every plot)
- Actually, I didn't understand the relationship between Obs. 1 and Theorem 3. In obs 1, you mentioned "D^{-1} Diag(H^{Q}{GN}^{1/2}", while in Theorem 3 you explained a tighter bound: each optimizer’s D^{−1} Diag(H^{Q}{GN}
- It seems Theorem 2 just considers a weak condition (Haar basis). Actually, I didn't get!
Questions for authors
- The lack of Novelty. The idea is not well motivated. The diagonal curvature benefits of Adam were revealed earlier, and the rotate-then-diagonal precondition idea is well established in SOAP, KFAC, ... So the novelty is limited.
- The experiments are small. I wonder if it is possible to consider larger models, e.g., Llama 1.3B (I know the limited time, but I wonder if you have any ideas).
7 comments
Response (Part Ⅱ)
W5: “It seems Theorem 2 just considers a weak condition (Haar basis).”
Thanks for the question. Let us clarify: Theorem 2 indeed focuses on the gap between the GN diagonal Frobenius norm ratio in the gradient SVD basis and its expectation under Haar-random rotations. For comparison with the ambient basis, we also derive separate conditions ensuring higher ratios in bases induced by different matrix optimizers (Appendix F).
Together, these two results provide complementary theoretical support for our empirical observations and analysis.
Comparison with Haar-random bases in Theorem 2.
We use the expected GN diagonal Frobenius norm ratio under random rotation as a baseline for average diagonal concentration across orthogonal bases. This comparison highlights the unusually strong diagonal concentration observed in the gradient SVD basis. (II) The analysis also helps explain the weak GN diagonal concentration observed after random rotation in Figure 5.
Comparison with the ambient basis in Appendix F.
These results support our observation that GN diagonal concentration is substantially stronger in bases induced by matrix optimizers than in the ambient basis. (II) The sufficient conditions for this improvement offer insight into cases where matrix optimizers may outperform methods that apply diagonal preconditioning in the ambient basis, such as standard Adam.
Q1: “The lack of Novelty. The idea is not well motivated. The diagonal curvature benefits of Adam were revealed earlier, and the rotate-then-diagonal precondition idea is well established in SOAP, KFAC, ... So the novelty is limited.”
Thank you for the constructive feedback. Since Adam already exploits diagonal curvature in ambient coordinates, a fundamental question arises: why do matrix optimizers achieve further performance gains in large model training? Our work is inspired by prior studies on Adam’s diagonal curvature benefits and coordinate-transformation methods such as KFAC and SOAP.
To address this question, we develop an explanatory chain connecting optimizer-induced bases, Gauss–Newton (GN) curvature representation, within-basis scaling, and optimization effects. While these two existing lines of work provide valuable conceptual starting points, they do not by themselves establish the specific explanatory chain studied here. We establish and support these connections through the following two new contributions.
1. New Gauss–Newton Curvature Structure Across Bases
Across coordinate systems, diagonal GN concentration exhibits an overall empirical ordering of Random < Ambient < Matrix-Optimizer-Induced: Matrix-optimizer-induced bases such as Muon’s momentum SVD basis and SOAP’s gradient-Gram eigenbasis concentrate substantially more GN curvature on the diagonal than the ambient basis. This finding is non-trivial: coordinate systems derived from momentum or gradient statistics do not automatically guarantee stronger GN diagonal concentration.
-
Empirical evidence. In Pythia-70M, the momentum SVD basis raises the GN diagonal Frobenius norm ratio from approximately 11–18% in the ambient basis to 59–84% across the examined training checkpoints (Figure 3). Figures 4–5 further show this stronger concentration across most individual hidden blocks, different models, and actual optimizer training trajectories.
-
Theoretical support: Theorem 2 compares an appropriate gradient SVD basis against the Haar-random expectation, while Appendix F provides sufficient conditions for momentum SVD and population SOAP bases to strictly improve over the ambient basis.
2. A New Theoretical Explanation for Matrix-Optimizer Gains
The choice of basis determines which Gauss–Newton curvature information enters the diagonal representation, while coordinate scaling governs how that curvature is exploited. Our theoretical analysis shows how a richer diagonal representation with appropriate coordinate scaling can support more effective preconditioning. Together, basis representation and within-basis scaling help explain why matrix optimizers achieve gains over ambient diagonal preconditioning.
-
Theoretical connection: Theorem 3 links local GN model decrease to diagonal concentration and diagonal scaling alignment, explaining the joint role of curvature representation and within-basis scaling.
-
Empirical support: Across optimizer-induced bases, inverse optimizer scaling closely corresponds to square-root diagonal GN curvature during training (Figure 2, Table 2). This correspondence provides evidence that these optimizers actively exploit diagonal curvature within their operating coordinates.
Our new findings and analyses explain matrix optimizers’ advantage over Adam, a phenomenon not fully explained by prior insights on diagonal curvature and rotated preconditioning.
Response (Part Ⅲ)
Q2: “The experiments are small. I wonder if it is possible to consider larger models, e.g., Llama 1.3B (I know the limited time, but I wonder if you have any ideas).”
Thanks for the suggestion and your understanding.
Additional 1B results. We extend the GN curvature analysis in Figure 7 from Pythia-410M to Pythia-1B at steps 32k, 64k, and 128k of its 143k-step pretraining run.
For each weight matrix type, we report:
=\frac{\rho_{\mathrm{diag}}(H_{\mathrm{GN}}^Q)} {\rho_{\mathrm{diag}}(H_{\mathrm{GN}})}, \quad \text{where } Q \text{ is the momentum SVD basis.}$$ **Table: SVD-to-Ambient Diagonal-Norm Ratio through training.** | Matrix Type | Step 32k | Step 64k | Step 128k | | --- | --- | --- | --- | | Q | 2.65 | 2.76 | 2.27 | | K | 2.00 | 2.30 | 1.43 | | V | 1.76 | 1.55 | 1.67 | | O | 4.16 | 3.10 | 2.12 | | MLP FC | 1.16 | 1.25 | 1.50 | | MLP Proj | 5.92 | 5.94 | 6.64 | The pooled SVD-to-ambient diagonal-norm ratios of **$1.16$–$6.64$** support greater GN diagonal concentration in the momentum SVD basis at the 1B scale. We will include the experiment results in the revised paper [[New Results: Figure 8 in the revised paper]](https://anonymous.4open.science/r/iclr27-rebuttal-figs-B025/figure8-pythia-1b.pdf). **Difficulty in extending the remaining experiments to 1B.** Our goal is to explain matrix optimizer performance through detailed curvature analysis, which is already computationally demanding at our existing model sizes. For example, extending the coordinate-wise analysis in Figure 2 and Table 2 via direct exact evaluation of the full GN diagonal for a 1B-parameter model would require: - **$10^9$ GN–vector products** per basis and checkpoint; - each applying a **$10^9\times10^9$ GN operator to a $10^9$-dimensional vector**. Given this workload, the additional training required, and the limited rebuttal time, we prioritized the diagonal-concentration extension above. Thanks again for the great questions. We are happy to further discuss any other potential follow-up questions.
Response (Part Ⅰ)
We sincerely thank the reviewer for the careful review and constructive feedback.
W1: “Lack of reference, e.g., Adam and Lion”
We would like to kindly clarify that both references were included in the original submission (Adam cited in [Section 1, line 035], and Lion in [Section 4.1, line 245]).
W2 & W3: “What is this "⪰" notation? Need to clarify.”; “Figure 2: what are the x-axis and y-axis? (You should identify them for every plot)”
Thank you very much for pointing out these notation issues. The notation ⪰ denotes the positive semidefinite (PSD) ordering. In every panel of Figure 2, the x- and y-axes show the rescaled D−1 and Diag(HGNQ)1/2, respectively, and play symmetric roles in the comparison. We will add the missing notion and axis labels in the revised version.
W4: “Actually, I didn’t understand the relationship between Obs. 1 and Theorem 3. In obs 1, you mentioned D−1Diag(HGNQ)1/2, while in Theorem 3 you explained a tighter bound: each optimizer’s D−1Diag(HGNQ)”
The analysis in Theorem 3 accommodates diagonal scaling with different exponents, including the inverse-square-root scaling observed in Observation 1. We will revise the paper accordingly to avoid confusion and present an updated Theorem 3 explicitly treating D∝Diag(HGNQ)−α for any α∈(0,1], along with an expanded discussion to explain the result and clarify its connection to Observation 1.
Updated Theorem 3 [GN model descent under diagonal concentration and scaling]. Assume HGN≻0 and g=0. With the original definitions,
\ge e^{\mathcal J(Q,D)}, \quad\text{where}\quad \mathcal J(Q,D) =\underbrace{\log\frac{\det H_{\mathrm{GN}}} {\prod_i(H_{\mathrm{GN}}^Q)_{ii}}}_{\text{diagonal concentration}} +\underbrace{d\log\frac{\left(\prod_iD_{ii}(H_{\mathrm{GN}}^Q)_{ii}\right)^{1/d}} {\frac1d\sum_iD_{ii}(H_{\mathrm{GN}}^Q)_{ii}}}_{\text{diagonal scaling alignment}} .$$ In particular, let $Q_1,Q_2\in\mathrm{O}(d)$ and suppose the diagonal vectors of $H_{\mathrm{GN}}^{Q_1}$ and $H_{\mathrm{GN}}^{Q_2}$ are comparable by majorization. For $D_k\propto\operatorname{Diag}(H_{\mathrm{GN}}^{Q_k})^{-\alpha}$, $k=1,2$, with $\alpha\in(0,1]$, it holds that $$\rho_{\mathrm{diag}}(H_{\mathrm{GN}}^{Q_1}) \ge \rho_{\mathrm{diag}}(H_{\mathrm{GN}}^{Q_2}) \quad\Longrightarrow\quad e^{\mathcal J(Q_1,D_1)}\ge e^{\mathcal J(Q_2,D_2)}.$$ **Expanded discussion of updated Theorem 3.** This theoretical result links the preconditioned GN-model decrease to both GN diagonal concentration in the basis $Q$ and the diagonal preconditioner $D$ applied within it. Specifically, a basis with strong GN diagonal concentration, paired with a preconditioner that effectively exploits its diagonal curvature, can guarantee a substantial fraction of the optimal GN-model decrease. As both terms in $\mathcal J(Q,D)$ approach zero, the guaranteed decrease approaches this optimum. Also, to focus more directly on the role of basis choice in training performance, and **motivated by the finding in Observation 1 (Section 4.1)** that different optimizers’ diagonal preconditioners generally align closely with GN diagonal curvature, we further establish a cross-basis comparison of the guaranteed preconditioned GN-model decrease in terms of the GN diagonal Frobenius norm ratio under the common scaling rule $D\propto\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{-\alpha}$, with fixed $\alpha\in(0,1]$. The additional majorization condition is also empirically supported for the ambient and momentum-SVD bases, as observed in [[New Results: Figure 10(b) in the revised paper]](https://anonymous.4open.science/r/iclr27-rebuttal-figs-B025/step002500_gn_majorization_captioned.pdf). These results collectively offer a theoretical explanation for why stronger GN diagonal concentration favors diagonal preconditioners that effectively exploit GN diagonal curvature, consistent with the training performance observed in Figure 5. The proof is given in [Appendix E.4 in the revised paper]. **Final remark: relationship between Observation 1 and updated Thm. 3.** The updated Thm. 3 **covers** the alignment pattern observed in Observation 1 and extends the analysis to a **broader** range of scaling exponents: cross-basis comparison of guaranteed preconditioned GN-model decrease in terms of the GN diagonal Frobenius norm ratio under $D \propto \\,\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{-\alpha}$, for **any** $\alpha\in(0,1]$. The alignment examined in Observation 1 corresponds to the **special case** $\alpha=\tfrac12$ in the updated Thm. 3, where $D\propto\operatorname{Diag}(H_{\mathrm{GN}}^Q)^{-1/2}$.