- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper connects data imbalance to the Hessian structure to study optimizer differences on a simple two-layer linear bigram model. The paper splits input and output imbalance and shows that it introduces different properties in the block statistics of the Hessian.
Critical strengths
- Nice small experiment setup to test concrete hypotheses to isolate the effects
- Pretty well written paper and pleasant to read
Critical weaknesses
Weaknesses
- Equations 3 + 4 give identical Hessians whether the outputs are balanced or imbalanced (l. 164f.). Regardless, Figure 4 (a) and (c) and Figure 4 (b) and (d) show that Muon's final loss differ by three orders of magnitude and Adam's by two. These effects are of similar size to the ones the paper attributes to the Hessian structure and I believe this needs some explanations.
- In realistic settings we would likely almost always have imbalanced inputs and outputs which gets little to no discussion in the presented paper and as such may be hard to find practical applications.
Minor Comments
- Figure 2 (b) / (c) has some overlap of the legend with the lines, which can be easily fixed.
- The diagonal-block spectra in e.g. Figure 2/3 (d) are too small to actually read
- l. 418: missing period after "Section 5.1"
- l. 428: "homogeneous This", missing period likely
Questions for authors
N/A
4 comments
Response to weakness 2
Thank you for pointing out the need to discuss simultaneously imbalanced inputs and outputs and to clarify the practical relevance of our settings.
First, we have added new experiments and layerwise Hessian analyses for joint training of the linear bigram model and deeper nonlinear MLPs on synthetic data (Section 6 and Appendices E.4–E.5). Both sets of experiments cover all four input/output marginal combinations, including the both-imbalanced case. We have also added analyses of the both-imbalanced frozen-layer CE case (Appendices D.1–D.2). These additional analyses further support our geometric interpretation of optimizer differences.
Second, practical training settings need not have both imbalanced inputs and imbalanced outputs. For example, image-classification benchmarks such as CIFAR-100 have balanced output classes, while class counts alone do not determine the statistics of input images or learned features. This motivates our separate analysis of input and output distributions.
We have added experiments on a class-balanced 2,500-image CIFAR-100 subset in Section 6 and Appendix E.6. Final training CE follows Muon < Adam < SGD. Even with balanced outputs, the raw layerwise Hessian blocks exhibit both block-scale heterogeneity and within-block anisotropy, with different degrees of heterogeneity across layers.
We compare the positive spectra of raw and preconditioned Hessian blocks. For both Adam and Muon, we can write the update direction in vectorized parameter coordinates as
where is the current layer gradient and is the corresponding symmetric positive preconditioning operator, with scalar step-size factors omitted. Holding fixed at each checkpoint, we define and as the -th diagonal blocks of the full layer Hessian and , respectively, under the specified row or column partition. Detailed definitions of the preconditioned Hessians for Adam and Muon are provided in Appendix E.3.
To summarize the results, we use to quantify block-scale heterogeneity and to quantify within-block anisotropy. Let denote block b's retained positive eigenvalues in descending order. We compute the metrics separately for raw and preconditioned spectra:
M_{\mathrm{scale}} &=\operatorname{Std}_{b} \left(\operatorname{Mean}_{k}\log_{10}\lambda_{b,k}\right),\\\\ M_{\mathrm{within}} &=\operatorname*{median}_{b} \left(\operatorname{Std}_{k}\log_{10}\lambda_{b,k}\right). \end{aligned}$$ $\operatorname{Mean}$ and $\operatorname{Std}$ denote arithmetic mean and population standard deviation. Smaller $M_{\mathrm{scale}}$ indicates less block-scale heterogeneity, while smaller $M_{\mathrm{within}}$ indicates less within-block anisotropy. The metrics are defined in Section 6, with full definitions and spectral-support conventions in Appendix E.3. The final training follows Muon < Adam < SGD. At the same checkpoint, Adam reduces block-scale heterogeneity $M_{\mathrm{scale}}$ in all three layers, while Muon reduces within-block anisotropy $M_{\mathrm{within}}$ in all three: | **Layer** | **Adam: $M_{\mathrm{scale}}$, raw → preconditioned** | **Muon: $M_{\mathrm{within}}$, raw → preconditioned** | | --- | --- | --- | | Input (row blocks) | 0.1350 → **0.1347** | 1.0880 → **0.9459** | | Hidden | 0.7618 → **0.6316** | 1.0944 → **0.9894** | | Output | 0.4095 → **0.3676** | 1.0335 → **1.0223** | Within-block anisotropy is pronounced in this setting, and the observed optimizer ranking aligns with our geometric interpretation. These measurements provide direct geometric analysis of neural network training on real images, in a setting where output classes are balanced but input and feature statistics need not be. They demonstrate the practical value of studying how different input and output distributions shape layerwise geometry.Response to minor comments
Thank you for these suggestions. We have repositioned the legends in Figure 2(b) and (c) to avoid overlap with the curves, and reduced excess whitespace and adjusted the layout of the diagonal-block spectra in Figures 2 and 3 to improve readability. We have also corrected the two missing periods.
Response to Rebuttal
My concerns have been addressed and I'll raise my score to 3.
Response to weakness 1
Thank you for highlighting these differences. In the balanced-input MSE experiments, output imbalance increases Muon's final optimization gap in both embedding and head training. For Adam, the increase is substantial in head training, while the embedding-training gaps remain comparable across the two output distributions. Output imbalance changes the regression target and hence the optimization objective and its optimum, even though the raw Hessian remains unchanged when the input distribution and frozen layer are fixed.
Our analysis relates optimizer differences to the Hessian block structure induced by the data distribution, model, and loss function. In the input balanced MSE setting, the Hessian exhibits relatively homogeneous curvature both across blocks and within block. In this regime, the heterogeneity-based analysis does not yield a definite optimizer ranking. To investigate the gap in these settings, we complement the structural analysis by examining how each optimizer’s actual displacement aligns with the direction toward that optimum in the geometry induced by the Hessian.
For quadratic MSE, let H denote the full Hessian with respect to the trainable parameters. The gradient and parameter error satisfy
dt=θt−θ⋆,gt=Hdt,−H−1gt=−dt.
Since output imbalance shifts the optimum, we examine how each optimizer’s actual displacement aligns with the direction toward that optimum in the geometry induced by the Hessian. We quantify this alignment using the Hessian-metric cosine between the observed displacement Δt and the Newton direction −dt. We take checkpoints every 20 steps and compute
=\frac{-\Delta_t^\top\mathcal H d_t} {\sqrt{\Delta_t^\top\mathcal H\Delta_t}\\,\sqrt{d_t^\top\mathcal H d_t}}.$$ Here, $\Delta_t$ denotes the displacement between successive checkpoints, and a cosine closer to 1 indicates better alignment with the direction toward the corresponding closed-form optimum. In the experiments, averaging over steps 40–140, Muon’s alignment is 0.92 versus 0.47 for balanced versus imbalanced outputs in embedding training, and 0.90 versus 0.54 in head training. Adam shows a similar distinction in head training (0.78 versus 0.29), while its embedding-training alignment remains approximately unchanged; GD’s mean alignment exceeds 0.99 in all four settings. The lower alignment under output imbalance for Muon in both settings and Adam in head training is consistent with their larger residual optimization gaps. These diagnostics are reported in Appendix C, Figure 5.