acceptodds
Under review as a conference paper at ICLR 2027

From Data Imbalance to Optimizer Differences: A Hessian Perspective

Abstract

Prior work has used data imbalance to explain performance differences between optimizers, but frequency statistics alone are insufficient to characterize the optimization difficulties faced by different optimizers. In this work, we connect data imbalance to the blockwise Hessian structure that shapes optimizer behavior. Specifically, we characterize this structure through two properties—block-scale heterogeneity and within-block anisotropy—and find that optimizers respond differently to them: Adam cancels block-scale factors but does not necessarily alleviate within-block anisotropy, whereas Muon can somewhat mitigate the optimization difficulties arising from both properties. To understand how these two properties arise, we study a two-layer linear bigram model. Under mean-squared error (MSE), the Hessians depend on the data distribution only through input frequencies: input imbalance induces block-scale heterogeneity in embedding training, while shaping within-block anisotropy in output-head training. Under cross-entropy (CE), output frequencies also play a role in shaping these properties. Together, these results show that data imbalance does not act alone: it interacts with the choice of trainable parameters and loss function to shape these two Hessian properties, thereby influencing the relative performance of different optimizers.

Primary area:
optimization
Keywords:
Optimization, Data imbalance, Hessian structure

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YMbyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper connects data imbalance to the Hessian structure to study optimizer differences on a simple two-layer linear bigram model. The paper splits input and output imbalance and shows that it introduces different properties in the block statistics of the Hessian.

Critical strengths
  • Nice small experiment setup to test concrete hypotheses to isolate the effects
  • Pretty well written paper and pleasant to read
Critical weaknesses

Weaknesses

  • Equations 3 + 4 give identical Hessians whether the outputs are balanced or imbalanced (l. 164f.). Regardless, Figure 4 (a) and (c) and Figure 4 (b) and (d) show that Muon's final loss differ by three orders of magnitude and Adam's by two. These effects are of similar size to the ones the paper attributes to the Hessian structure and I believe this needs some explanations.
  • In realistic settings we would likely almost always have imbalanced inputs and outputs which gets little to no discussion in the presented paper and as such may be hard to find practical applications.

Minor Comments

  • Figure 2 (b) / (c) has some overlap of the legend with the lines, which can be easily fixed.
  • The diagonal-block spectra in e.g. Figure 2/3 (d) are too small to actually read
  • l. 418: missing period after "Section 5.1"
  • l. 428: "homogeneous This", missing period likely
Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Response to weakness 1

Thank you for highlighting these differences. In the balanced-input MSE experiments, output imbalance increases Muon's final optimization gap in both embedding and head training. For Adam, the increase is substantial in head training, while the embedding-training gaps remain comparable across the two output distributions. Output imbalance changes the regression target and hence the optimization objective and its optimum, even though the raw Hessian remains unchanged when the input distribution and frozen layer are fixed.

Our analysis relates optimizer differences to the Hessian block structure induced by the data distribution, model, and loss function. In the input balanced MSE setting, the Hessian exhibits relatively homogeneous curvature both across blocks and within block. In this regime, the heterogeneity-based analysis does not yield a definite optimizer ranking. To investigate the gap in these settings, we complement the structural analysis by examining how each optimizer’s actual displacement aligns with the direction toward that optimum in the geometry induced by the Hessian.

For quadratic MSE, let H\mathcal H denote the full Hessian with respect to the trainable parameters. The gradient and parameter error satisfy

dt=θt−θ⋆,gt=Hdt,−H−1gt=−dt.d_t=\theta_t-\theta^\star,\qquad g_t=\mathcal H d_t,\qquad -\mathcal H^{-1}g_t=-d_t.

Since output imbalance shifts the optimum, we examine how each optimizer’s actual displacement aligns with the direction toward that optimum in the geometry induced by the Hessian. We quantify this alignment using the Hessian-metric cosine between the observed displacement Δt\Delta_t and the Newton direction −dt-d_t. We take checkpoints every 20 steps and compute

=\frac{-\Delta_t^\top\mathcal H d_t} {\sqrt{\Delta_t^\top\mathcal H\Delta_t}\\,\sqrt{d_t^\top\mathcal H d_t}}.$$ Here, $\Delta_t$ denotes the displacement between successive checkpoints, and a cosine closer to 1 indicates better alignment with the direction toward the corresponding closed-form optimum. In the experiments, averaging over steps 40–140, Muon’s alignment is 0.92 versus 0.47 for balanced versus imbalanced outputs in embedding training, and 0.90 versus 0.54 in head training. Adam shows a similar distinction in head training (0.78 versus 0.29), while its embedding-training alignment remains approximately unchanged; GD’s mean alignment exceeds 0.99 in all four settings. The lower alignment under output imbalance for Muon in both settings and Adam in head training is consistent with their larger residual optimization gaps. These diagnostics are reported in Appendix C, Figure 5.
AuthorsOpenReview ↗

Response to weakness 2

Thank you for pointing out the need to discuss simultaneously imbalanced inputs and outputs and to clarify the practical relevance of our settings.

First, we have added new experiments and layerwise Hessian analyses for joint training of the linear bigram model and deeper nonlinear MLPs on synthetic data (Section 6 and Appendices E.4–E.5). Both sets of experiments cover all four input/output marginal combinations, including the both-imbalanced case. We have also added analyses of the both-imbalanced frozen-layer CE case (Appendices D.1–D.2). These additional analyses further support our geometric interpretation of optimizer differences.

Second, practical training settings need not have both imbalanced inputs and imbalanced outputs. For example, image-classification benchmarks such as CIFAR-100 have balanced output classes, while class counts alone do not determine the statistics of input images or learned features. This motivates our separate analysis of input and output distributions.

We have added experiments on a class-balanced 2,500-image CIFAR-100 subset in Section 6 and Appendix E.6. Final training CE follows Muon < Adam < SGD. Even with balanced outputs, the raw layerwise Hessian blocks exhibit both block-scale heterogeneity and within-block anisotropy, with different degrees of heterogeneity across layers.

We compare the positive spectra of raw and preconditioned Hessian blocks. For both Adam and Muon, we can write the update direction in vectorized parameter coordinates as

dt=Pt−1gt,d_t=P_t^{-1}g_t, where gtg_t is the current layer gradient and Pt−1P_t^{-1} is the corresponding symmetric positive preconditioning operator, with scalar step-size factors omitted. Holding PtP_t fixed at each checkpoint, we define Hb,trawH_{b,t}^{\mathrm{raw}} and Hb,tpreH_{b,t}^{\mathrm{pre}} as the bb-th diagonal blocks of the full layer Hessian Ht\mathcal H_t and Pt−1/2HtPt−1/2P_t^{-1/2}\mathcal H_tP_t^{-1/2}, respectively, under the specified row or column partition. Detailed definitions of the preconditioned Hessians for Adam and Muon are provided in Appendix E.3.

To summarize the results, we use MscaleM_{\mathrm{scale}} to quantify block-scale heterogeneity and MwithinM_{\mathrm{within}} to quantify within-block anisotropy. Let λb,k\lambda_{b,k} denote block b's retained positive eigenvalues in descending order. We compute the metrics separately for raw and preconditioned spectra:

M_{\mathrm{scale}} &=\operatorname{Std}_{b} \left(\operatorname{Mean}_{k}\log_{10}\lambda_{b,k}\right),\\\\ M_{\mathrm{within}} &=\operatorname*{median}_{b} \left(\operatorname{Std}_{k}\log_{10}\lambda_{b,k}\right). \end{aligned}$$ $\operatorname{Mean}$ and $\operatorname{Std}$ denote arithmetic mean and population standard deviation. Smaller $M_{\mathrm{scale}}$ indicates less block-scale heterogeneity, while smaller $M_{\mathrm{within}}$ indicates less within-block anisotropy. The metrics are defined in Section 6, with full definitions and spectral-support conventions in Appendix E.3. The final training follows Muon < Adam < SGD. At the same checkpoint, Adam reduces block-scale heterogeneity $M_{\mathrm{scale}}$ in all three layers, while Muon reduces within-block anisotropy $M_{\mathrm{within}}$ in all three: | **Layer** | **Adam: $M_{\mathrm{scale}}$, raw → preconditioned** | **Muon: $M_{\mathrm{within}}$, raw → preconditioned** | | --- | --- | --- | | Input (row blocks) | 0.1350 → **0.1347** | 1.0880 → **0.9459** | | Hidden | 0.7618 → **0.6316** | 1.0944 → **0.9894** | | Output | 0.4095 → **0.3676** | 1.0335 → **1.0223** | Within-block anisotropy is pronounced in this setting, and the observed optimizer ranking aligns with our geometric interpretation. These measurements provide direct geometric analysis of neural network training on real images, in a setting where output classes are balanced but input and feature statistics need not be. They demonstrate the practical value of studying how different input and output distributions shape layerwise geometry.
AuthorsOpenReview ↗

Response to minor comments

Thank you for these suggestions. We have repositioned the legends in Figure 2(b) and (c) to avoid overlap with the curves, and reduced excess whitespace and adjusted the layout of the diagonal-block spectra in Figures 2 and 3 to improve readability. We have also corrected the two missing periods.

Reviewer_YMbyOpenReview ↗

Response to Rebuttal

My concerns have been addressed and I'll raise my score to 3.

Reviewer_6THDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how input/output frequency imbalance affects GD, Adam, and Muon in a two-layer linear bigram model. It derives MSE Hessian structures for training the embedding or output head, analyzes block-scale heterogeneity and within-block anisotropy, and extends the analysis to cross-entropy using a block-diagonal Hessian approximation. Controlled experiments compare balanced and Zipf-imbalanced marginals.

Critical strengths
  1. The paper is generally well-written and clearly structured, making the problem formulation, main ideas, and experimental findings relatively easy to follow.
  2. The paper provides a reasonably thorough theoretical analysis of optimizer behavior under data imbalance, using Hessian structures and spectral properties to explain the differences among GD, Adam, and Muon.
Critical weaknesses
  1. The paper motivates its study by arguing that frequency statistics alone cannot characterize optimizer-specific optimization difficulties. However, prior work has also investigated optimizer differences through Hessian geometry and curvature structures. The paper does not sufficiently clarify what limitations remain in these existing analyses or what fundamentally new insights its proposed framework provides beyond them.

  2. The experiments do not report results across multiple random initializations or provide sufficient sensitivity analyses of learning-rate choices and data-generation settings. Moreover, the evaluation is limited to a two-layer linear bigram model with one layer frozen during training, leaving it unclear whether the proposed explanations generalize to jointly trained nonlinear networks or deeper architectures.

Questions for authors
  1. Prior work has already examined optimizer differences through Hessian block heterogeneity and curvature properties. Could the authors clarify what specific gaps remain in these studies and how the proposed distinction between block-scale heterogeneity and within-block anisotropy provides insights beyond existing Hessian-based analyses?

  2. Could the authors clarify the robustness of the observed optimizer rankings across different random initializations and learning-rate choices? Additionally, do the conclusions remain consistent when the input–output coupling structure is varied while keeping the marginal distributions fixed?

  3. Have the authors tested the proposed explanations on jointly trained nonlinear networks or deeper architectures? If not, how should the scope of the conclusions be limited given that the experiments are conducted on a two-layer linear bigram model with one layer frozen?

5 comments
AuthorsOpenReview ↗

Response to Question 1

Thank you for these comments. We have revised the Introduction to clarify how our framework connects prior Hessian-based analyses and what additional insights it provides.

Prior analyses link Adam’s advantages to diagonally dominant Hessians [1], blockwise curvature differences [2], and gradient–curvature scale correlations [3]. Within-block ill-conditioning may nevertheless remain after diagonal adaptive scaling, motivating a comparison with matrix-wise updates such as Muon. Existing work explains Muon’s advantage through directional curvature [4], providing a complementary perspective, but its theoretical framework does not directly accommodate Adam’s coordinatewise normalization.

Our framework brings these perspectives together by comparing GD, Adam, and Muon within a common quadratic model that separates block-scale heterogeneity from within-block anisotropy. The resulting one-step loss-contraction bounds clarify their different responses: GD is affected by both sources of ill-conditioning; Adam cancels explicit block-scale factors but need not alleviate within-block anisotropy; and the analyzed Muon update may mitigate either source.

We further identify how data imbalance gives rise to these different geometric difficulties. Under MSE, the same input imbalance induces block-scale heterogeneity in embedding training but shapes within-block anisotropy in output-head training. This connects the optimizer comparison to the underlying learning problem and explains why the same data imbalance can favor different optimizers across training settings.

For a detailed discussion of related work, please see Appendix A.

References

[1] Rudrajit Das, Naman Agarwal, Sujay Sanghavi, and Inderjit S. Dhillon. Towards Quantifying the Preconditioning Effect of Adam. arXiv:2402.07114, 2024.

[2] Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why Transformers Need Adam: A Hessian Perspective. Advances in Neural Information Processing Systems, 37:131786–131823, 2024.

[3] Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models. Advances in Neural Information Processing Systems, 37:30106–30148, 2024.

[4] Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann, and Zhuoran Yang. Why Muon Outperforms Adam: A Curvature Perspective. arXiv:2606.04662, 2026.

AuthorsOpenReview ↗

Response to Question 2

We agree that the original presentation did not adequately document variability and hyperparameter sensitivity. We have therefore expanded the evaluation to three initialization seeds (42, 43, 44), a denser learning rate grid 1,1.5,2,3,5,7,8×10k\\{1, 1.5, 2, 3, 5, 7, 8\\}\times10^k, and three coupling strengths (ρ=0.3,0.5,0.8\rho=0.3, 0.5, 0.8) while preserving the prescribed marginals.

The expanded population experiments cover frozen-layer MSE (training either layer), frozen-layer CE (training the output layer), and joint two-layer CE, with uniform or Zipf input/output marginals and Zipf exponents ζ∈1,1.5,2\zeta\in\\{1,1.5,2\\}. All statistics below compare GD, Adam, and Muon by training loss at 200 updates and exclude duplicate uniform/uniform settings.

Random initializations

The complete optimizer ranking agrees across all three seeds in 115/120 distinct settings (95.8%); only 5/120 settings (4.2%) show a ranking change.

Learning rate choices

Relative to the original 1,2,5×10k\\{1, 2, 5\\}\times10^k grid, the denser grid changes the complete ranking in only 10/350 matched comparisons (2.9%) and the first-place optimizer in 7/350 (2.0%). Thus, 97.1% of complete rankings and 98.0% of first-place identities are preserved.

Coupling at fixed marginals

With the prescribed input and output marginals fixed, varying ρ\rho changes the ranking based on mean loss across the three seeds in only 1/40 settings (2.5%), preserving 97.5% of rankings. Comparing ρ\rho separately within each seed gives 8/120 ranking changes (6.7%).

Together, these results support the robustness of the optimizer rankings and our main conclusions across the tested random initializations, learning-rate grids, and fixed-marginal coupling strengths. These robustness analyses are reported in Appendix F.

AuthorsOpenReview ↗

Response to Question 3

Thank you for raising this important question about the scope of our geometric explanations. We have added experiments that jointly train the linear bigram model on synthetic data and deeper nonlinear MLPs on synthetic data and CIFAR-100. These analyses have been added to Section 6 and Appendix E.

We compare the positive spectra of raw and preconditioned Hessian blocks. For both Adam and Muon, we can write the update direction in vectorized parameter coordinates as

dt=Pt−1gt,d_t=P_t^{-1}g_t, where gtg_t is the current layer gradient and Pt−1P_t^{-1} is the corresponding symmetric positive preconditioning operator. Holding PtP_t fixed at each checkpoint, we define Hb,trawH_{b,t}^{\mathrm{raw}} and Hb,tpreH_{b,t}^{\mathrm{pre}} as the bb-th diagonal blocks of the full layer Hessian Ht\mathcal H_t and Pt−1/2HtPt−1/2P_t^{-1/2}\mathcal H_tP_t^{-1/2}, respectively, under the specified row or column partition. Detailed definitions of the preconditioned Hessians for Adam and Muon are provided in Appendix E.3.

To summarize the results, we use MscaleM_{\mathrm{scale}} to quantify block-scale heterogeneity and MwithinM_{\mathrm{within}} to quantify within-block anisotropy. Let λb,k\lambda_{b,k} denote block b's retained positive eigenvalues in descending order. We compute the metrics separately for raw and preconditioned spectra:

M_{\mathrm{scale}} &=\operatorname{Std}_{b} \left(\operatorname{Mean}_{k}\log_{10}\lambda_{b,k}\right),\\\\ M_{\mathrm{within}} &=\operatorname*{median}_{b} \left(\operatorname{Std}_{k}\log_{10}\lambda_{b,k}\right). \end{aligned}$$ $\operatorname{Mean}$ and $\operatorname{Std}$ denote arithmetic mean and population standard deviation. Smaller $M_{\mathrm{scale}}$ indicates less block-scale heterogeneity, while smaller $M_{\mathrm{within}}$ indicates less within-block anisotropy. **Joint training.** We jointly train the embedding and head of the linear bigram model under CE across all four input/output marginal combinations. The result below is at update 200 with $\zeta=2$ and $\rho=0.8$: | **Input / output marginals** | **Final CE (ascending)** | **Adam $M_{\mathrm{scale}}$: raw → preconditioned** | **Muon $M_{\mathrm{within}}$: raw → preconditioned** | | --- | --- | --- | --- | | Balanced / balanced | Muon < GD < Adam | Embedding: 2.728e−5 → 0.0022; Head: 2.581e−4 → 0.0020 | Embedding: 0.1316 → **0.0958**; Head: 0.1246 → **0.0924** | | Balanced / imbalanced | Adam < Muon < GD | Embedding: 0.4739 → 0.5018; Head: 2.2813 → **1.0903** | Embedding: 1.1903 → **0.8312**; Head: 0.6689 → **0.6653** | | Imbalanced / balanced | Adam ≈ Muon < GD | Embedding: 2.8149 → **1.8397**; Head: 0.0129 → 0.6200 | Embedding: 0.5195 → 0.5359; Head: 0.9299 → **0.5470** | | Imbalanced / imbalanced | Adam < Muon < GD | Embedding: 2.0349 → **0.4681**; Head: 1.5369 → **0.3354** | Embedding: 0.4183 → **0.4021**; Head: 1.0404 → **1.0060** | Bold marks a reduction from the same optimizer's raw value. With balanced inputs and outputs, Muon achieves the lowest CE, consistent with its reduction of $M_{\mathrm{within}}$ in both layers; $M_{\mathrm{scale}}$ is already negligible, leaving little block-scale heterogeneity for Adam to remove. With output imbalance alone, Adam performs best and substantially reduces output-layer $M_{\mathrm{scale}}$; Muon also outperforms GD while reducing $M_{\mathrm{within}}$, especially in the input layer. With input imbalance alone, Adam and Muon attain similar CE while reducing input-layer $M_{\mathrm{scale}}$ and output-layer $M_{\mathrm{within}}$, respectively. When both marginals are imbalanced, both outperform GD, with Adam reducing $M_{\mathrm{scale}}$ and Muon reducing $M_{\mathrm{within}}$ in both layers. These results extend our layerwise geometric explanation of optimizer behavior to joint training (Appendix E.4).
Reviewer_oQLQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the block-wise Hessian structure of training a 2-layer linear network on a bi-gram task with MSE loss and try to analyze the relative performance of Muon, Adam and GD on this task with imbalanced input distribution . The paper shows that Adam can potentially address the block-scale heterogeneity but couldn't handle the within-block anisotropy whereas Muon can potentially address both. They also consider a potential generalization to cross entropy loss, where the output distribution also shape the hessian

Critical strengths
  1. The paper performs a detailed analysis on the 2-layer network setting and the paper is overall clear to read.

  2. The problem studied is generally interesting and the authors' synthetic experiment shows some insight on the comparison between optimizers.

Critical weaknesses
  1. The reviewer found it very hard to connect the Adam theorem (Theorem 2) to what the authors claimed as the Adam theorem discussed the preconditioned Hessian instead of the original hessian, which makes it unclear to the reviewer whether this actually show what the authors claimed as what is in P is not discussed at all.

  2. All three theorems talked about optimizers with one step and a specific learning rate. On the case of GD it seems plausible that this learning rate is optimal but it is unclear to the reviewer whether the same optimality holds for Adam and Muon.

  3. There is no discussion or empirical measurement in any neural network training (except reference to Adam-mini paper), which makes it hard to reconcile this work with real network training.

  4. In the case of CE loss, the benefit of Muon is very apparent when both input and output distribution is fully balanced, and the anistrophy argument does not explain that.

Questions for authors

The reviewers would appreciate authors' comments on the critical weaknesses listed.

5 comments
AuthorsOpenReview ↗

Response to weakness 1

Thank you for this comment. Adam’s normalization cancels the block-scale factor because the same factor appears in both the gradient and the square root of its second moment. The remaining normalizer, P, retains information about the shared curvature and the error history. Theorem 2 uses the resulting effective curvature to bound the decrease of the original loss.

This distinction explains the connection to our observations. In embedding training, input frequencies rescale the Hessian blocks, so Adam directly cancels these frequency-induced scales. In head training, the blocks already have equal scales, while input imbalance affects their shared internal curvature, whose conditioning need not improve under diagonal normalization. This is consistent with Adam’s advantage over GD in embedding training and their similar performance in head training.

We have clarified the meaning of P, the relationship between the original and effective Hessians, and this interpretation in Section 4.2, with the detailed cancellation derivation in Appendix H.2. The revisions are marked in blue.

AuthorsOpenReview ↗

Response to weakness 2

Thank you for raising this question. We have clarified the learning-rate choices as follows.

For GD and Adam, the revised theorems use

\qquad \eta_{\mathrm{Adam},t}=\frac{2}{\Lambda_{M,t}+\mu_{M,t}},$$ where the GD spectral extrema are defined in Appendix H.1 and the spectral extrema of Adam’s effective blocks are defined in Theorem 2. These choices minimize the worst-case spectral contraction bounds associated with the Hessian and Adam’s current effective blocks, respectively. Our original $1/\Lambda$ choices are standard in quadratic gradient-descent analysis and its preconditioned counterpart. The refinement changes the contraction factor from $$\left(1-\frac1\kappa\right)^2 \quad\text{to}\quad \left(1-\frac{2}{\kappa+1}\right)^2.$$ Since $1/\kappa\leq 2/(\kappa+1) < 2/\kappa$, the inverse-condition-number terms inside the square differ by less than a factor of two. Thus, the improvement concerns constants; the conditioning dependence and our structural conclusions remain unchanged. For **Muon**, we retain $$\eta=\frac1L,\qquad L=\min\left\\{\lambda_{\max}(S)\operatorname{tr}(H), \lambda_{\max}(H)\operatorname{tr}(S)\right\\}.$$ This choice minimizes the quadratic upper bound on the one-step loss change derived using the polar factor. The derivations are provided in Appendix H. We have updated Theorems 1–2 and added a remark at the end of Section 4.3 explaining these choices and the constant comparison.
AuthorsOpenReview ↗

Response to weakness 3

Thank you for pointing out the need for measurements in neural network training. We have now run additional experiments on jointly trained three-layer nonlinear MLPs, using both controlled synthetic distributions and CIFAR-100. These experiments measure the concentration of Hessian energy in diagonal blocks, as well as block-scale heterogeneity and within-block anisotropy before and after preconditioning. These analyses have been added to Section 6 of the main text and Appendix E.

First, within each layer, we group the weights associated with each neuron into one parameter block, motivated by the neuron-level Hessian structure observed in Adam-mini (Zhang et al., 2025, Fig. 3). We then measure the fraction of Hessian Frobenius energy concentrated in diagonal blocks, defined as

Dblock=∑b=17∥Hbb∥F2∑b,c=17∥Hbc∥F2.\mathsf D_{\mathrm{block}} = \frac{\sum_{b=1}^{7}\lVert H_{bb}\rVert_F^2}{\sum_{b,c=1}^{7}\lVert H_{bc}\rVert_F^2}.

The numerator sums diagonal-block energies, while the denominator includes all blocks of the analyzed raw Hessian submatrix. Across all 45 terminal measurements on the synthetic CE and CIFAR-100 experiments, this fraction exceeds the uniform-energy reference of 1/71/7, with 4.65–7.00× enrichment. These results provide empirical support for our blockwise analysis (Appendix E.1).

Second, we compare the positive spectra of raw and preconditioned Hessian blocks. For both Adam and Muon, we can write the update direction in vectorized parameter coordinates as

dt=Pt−1gt,d_t=P_t^{-1}g_t, where gtg_t is the current layer gradient and Pt−1P_t^{-1} is the corresponding symmetric positive preconditioning operator, with scalar step-size factors omitted. Holding PtP_t fixed at each checkpoint, we define Hb,trawH_{b,t}^{\mathrm{raw}} and Hb,tpreH_{b,t}^{\mathrm{pre}} as the bb-th diagonal blocks of the full layer Hessian Ht\mathcal H_t and Pt−1/2HtPt−1/2P_t^{-1/2}\mathcal H_tP_t^{-1/2}, respectively, under the specified row or column partition. Detailed definitions of the preconditioned Hessians for Adam and Muon are provided in Appendix E.3.

To summarize the results, we use MscaleM_{\mathrm{scale}} to quantify block-scale heterogeneity and MwithinM_{\mathrm{within}} to quantify within-block anisotropy. Let λb,k\lambda_{b,k} denote block b's retained positive eigenvalues in descending order. We compute the metrics separately for raw and preconditioned spectra:

M_{\mathrm{scale}} &=\operatorname{Std}_{b} \left(\operatorname{Mean}_{k}\log_{10}\lambda_{b,k}\right),\\\\ M_{\mathrm{within}} &=\operatorname*{median}_{b} \left(\operatorname{Std}_{k}\log_{10}\lambda_{b,k}\right). \end{aligned}$$ $\operatorname{Mean}$ and $\operatorname{Std}$ denote arithmetic mean and population standard deviation. Smaller $M_{\mathrm{scale}}$ indicates less block-scale heterogeneity, while smaller $M_{\mathrm{within}}$ indicates less within-block anisotropy. **Results.** On CIFAR-100 at epoch 200, final training CE follows Muon < Adam < SGD. At the same checkpoint, Adam reduces block-scale heterogeneity $M_{\mathrm{scale}}$ in all three layers, while Muon reduces within-block anisotropy $M_{\mathrm{within}}$ in all three: | **Layer** | **Adam: $M_{\mathrm{scale}}$, raw → preconditioned** | **Muon: $M_{\mathrm{within}}$, raw → preconditioned** | | --- | --- | --- | | Input (row blocks) | 0.1350 → **0.1347** | 1.0880 → **0.9459** | | Hidden | 0.7618 → **0.6316** | 1.0944 → **0.9894** | | Output | 0.4095 → **0.3676** | 1.0335 → **1.0223** | Within-block anisotropy is pronounced in this setting, and the observed optimizer ranking aligns with our geometric interpretation. These experiments provide direct curvature measurements and a layerwise analysis of optimizer behavior during joint nonlinear neural network training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.