- The novelty is limited and some key related work is missing. For example, pre-/post-activation splits and their effect H′=B⊤HB on the Hessian from sec. 3-4 are more or less [1,2], and the additions on top of these results (e.g., making the diagonal K explicit, or calculating the pseudodeterminant) from sec. 5 are straightforward.
- I feel that in general, analyzing the effects of Hessian across widths is somewhat more subtle than done here, and one needs to be careful about which aspects are actually effects of width vs. parameterization-dependent (i.e., how initialization and LRs are scaled with width). If this is not properly controlled for, results drawn from comparing aspects of the Hessian across width might be confounded by, e.g., different effective LRs, and it seems to me that this paper conflates it at a few points. For example, the experiments appear to use the maximal update parameterization (muP, see app. M.2), but in that case one should be careful about which normalization/preconditioner to apply to the Hessian before drawing conclusions from it. Let the parameter space be Θ=Rp and use layer-wise LRs D=Diag(d)∈Rp×p. Then, the Hessian whose eigenvalues govern stability, convergence, etc. are those of D1/2HD1/2, not the raw H. Under muP, for example, if we take its form without forward multipliers (see, e.g., [5]), then trainable/effective coordinates coincide with the ones in this paper, i.e., init stds (din−1/2,n−1/2,n−1) and SGD LRs (ηn,η,η/n) for input/hidden/output weights, respectively. But then, scaling width from n0 to n=kn0 multiplies the layerwise LRs by (k,1,k−1). If we compare this to K in Thm. 4.3 for an even split of every hidden neuron, we notice that the muP LR scaling gives exactly Dn=Dn0K−1, i.e. an additional K−1 factor to the preconditioner if we want to look at the Hessian in optimizer coordinates relative to the narrow network. In other words, at a zero-loss minimum the Hessian in optimizer coordinates Dn1/2H′Dn1/2 of the expansion has the exact same nonzero spectrum as that of the narrow network. For other uneven splits, this means that moving along the orbit only ever increases curvature in the optimizer coordinates. For the results in the paper, this means that in the metric that is actually relevant for optimization dynamics, the theory in sec. 4/5 would not predict any systematic conditioning benefit from increasing width. In other words, one could also read the theory on preconditioning as a statement about the fact that layerwise LRs matter.
- It seems to me that the experiments don’t measure the correct metric either, as they report characteristics of the spectrum of the raw Hessian across width, while, as it seems, scaling width via muP. In fact, under muP it has been observed empirically that top eigenvalues of the properly normalized Hessian D1/2HD1/2 (or n⋅H in the mean-field representative) are in fact largely width-consistent, even at larger scales than tested here and not under function-preserving expansion but for independently trained networks of different widths [3]. However, the raw Hessian should have some blocks that scale as O(n−1) and some as O(1) (depending on the muP representative), which would be entirely consistent with sharpness, condition, and AM-GM ratio decreasing with width and then saturating at some point. Hence, as reported at the moment, the experiments do not distinguish the paper’s proposed explanation from simply parameterization artifacts. Additionally, for networks trained with Adam the relevant quantity would also be the properly preconditioned Hessian (see, e.g., [4]).
- About the condition number (as, e.g., reported in fig. 4), I am wondering how stable and relevant this quantity is in general, and if it being smaller actually has any impact on optimization. Namely, Hessian spectra often have a lot of eigenvalues very close to zero, so choosing the smallest positive one should be numerically unstable, but the associated directions have little influence on the loss anyway so a condition number involving them should say little about optimization. The appendix cleared up for me that fig. 4 just looks at the top 100 eigenvalues (which is a somewhat arbitrary choice, also since the number of parameters grows as n2). Also condition and AM/GM ratio are quite close to 1 i.e. the top 100 eigenvalues are very similar so I wonder how much these numbers really tell us about the geometry of the minima (and whether this maybe has something to do with the synthetic task). Perhaps just reporting the spectra themselves would be more informative.
[1] Fukumizu, Yamaguchi, Mototake, Tanaka. Semi-flat minima and saddle points by embedding neural networks to overparameterization. NeurIPS 2019.
[2] Zhang, Zhang, Luo, Xu. Embedding principle of loss landscape of deep neural networks. NeurIPS 2021.
[3] Noci, Meterez, Hofmann, Orvieto. Super consistency of neural network landscapes and learning rate transfer. NeurIPS 2024.
[4] Cohen et al. Adaptive gradient methods at the edge of stability. arXiv:2207.14484, 2022.
[5] Yang, Hu, Babuschkin, Sidor, Liu, Farhi, Ryder, Pachocki, Chen, Gao. Tensor Programs V: Tuning large neural networks via zero-shot hyperparameter transfer. NeurIPS 2021.
We thank the reviewer for their time in reviewing our work.
This is true. We didn’t state the assumption in the main text due to space constraints but will clarify that it holds under this assumption with reference to Appendix.
Yes, sorry. Thanks for catching this. This was a typo in Section 5.1. It was meant to say the geometric mean “decreases in post-activation splits with biases” and “Volume grows by a factor √m”. The conditioning statement that follows is for the case |S| = |D|. With a bias the conditioning can still improve, but the trace has to decrease by slightly more, by a factor of m^{-1/p}.
We apologize for the late reply but are happy to answer any questions that might remain.