acceptodds
Under review as a conference paper at ICLR 2027

Symmetry and Optimization in Overparameterized Networks

Abstract

Overparameterization is central to the success of deep learning, yet the mechanisms by which it improves optimization remain incompletely understood. In hopes of better understanding how the loss landscape transforms as we add neurons, we study neural networks of different width that represent the same function. Building on prior work, we characterize function-preserving transformations across network width in terms of families of matrices. These width symmetries allow for the study of a hierarchy of solutions across neural network width. We then use this construction to derive consequences for loss landscape geometry and optimization. First, we prove that these symmetries act as a form of diagonal preconditioning on the Hessian, enabling the existence of better-conditioned minima within each class of functionally identical solutions. Second, we find that overparameterization increases the probability mass of global minima near typical initializations, making these favourable solutions more reachable. These results offer a potential link between loss landscape geometry and simplicity bias. Empirically, we observe wider networks have lower top eigenvalues, smaller condition numbers and faster convergence, supporting our analysis. Together, these results describe width growth as a geometric transformation of the loss landscape.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Weight-Space Symmetry, Overparameterization, Optimization, Hessian Spectrum, Loss Landscape

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NwhrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors studies how function preserving width expansions change neural network loss landscapes. They represents neuron splitting through expansion and contraction matrices and derives a Hessian transformation at zero loss minima. The paper also derives associated Hessian trace and pseudo determinant formulas and proposes a lower bound on the probability of initializing near global minima after adding a neuron. The experiments examine explicit expansions, teacher student learning and direct data fitting across several architectures.

Critical strengths
  1. Comparing parametrizations that implement the same function helps separate the geometric effects of widening from chanches in representational capacity.
  2. The authors shows useful spectral identities under proportional splitting.
  3. The explicit expansion experiments compare predicted and measured Hessians and spectra. The additional checks with sigmoid and ReLU, strengthen this part of the paper.
Critical weaknesses
  1. If I understand correctly, Theorem 6.1 omits the initialization support assumption used in Appendix L and the assumption that minimal width minima consist only of permutation copies.
Questions for authors
  1. If you take an even split ∣D∣=∣S∣+1|D|=|S|+1, Theorem 4.4 gives pdet(H′)=pdet(H)/mpdet(H')=pdet(H)/m, that implies that the geometric mean of the positive eigenvalues decreases. Is not contrary to Section 5.1 statement that increases for post-activation splitting?
1 comment

We thank the reviewer for their time in reviewing our work.

Theorem 6.1 omits the initialization support assumption used in Appendix L

This is true. We didn’t state the assumption in the main text due to space constraints but will clarify that it holds under this assumption with reference to Appendix.

Is theorem 4.4 contrary to section 5.1?

Yes, sorry. Thanks for catching this. This was a typo in Section 5.1. It was meant to say the geometric mean “decreases in post-activation splits with biases” and “Volume grows by a factor √m”. The conditioning statement that follows is for the case |S| = |D|. With a bias the conditioning can still improve, but the trace has to decrease by slightly more, by a factor of m^{-1/p}.

We apologize for the late reply but are happy to answer any questions that might remain.

Reviewer_M6YHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the effect of width on the loss landscape via function-preserving transformations of networks of smaller width to larger width, by adding/splitting neurons. As the Hessian varies over the orbit of function-preserving widenings, the authors argue that, in a teacher-student setup, widening can create functionally identical minima that are better conditioned, and show that such overparameterization increases the probability of initialization being close to a minimum. Empirically, the authors report lower sharpness, condition number and faster convergence for wider networks, and conclude with a conjectured link to simplicity bias.

Critical strengths
  • The paper is nicely written and generally easy to follow, the introduced concepts visualized and explained with examples, and from what I can tell the math is correct.
  • The discussion (sec. 8.1, appendix A) is honest about the limitation that the results characterize the minima on the orbit but not which representative is actually reached by the optimizer.
Critical weaknesses
  • The novelty is limited and some key related work is missing. For example, pre-/post-activation splits and their effect H′=B⊤HBH' = B^\top HB on the Hessian from sec. 3-4 are more or less [1,2], and the additions on top of these results (e.g., making the diagonal KK explicit, or calculating the pseudodeterminant) from sec. 5 are straightforward.
  • I feel that in general, analyzing the effects of Hessian across widths is somewhat more subtle than done here, and one needs to be careful about which aspects are actually effects of width vs. parameterization-dependent (i.e., how initialization and LRs are scaled with width). If this is not properly controlled for, results drawn from comparing aspects of the Hessian across width might be confounded by, e.g., different effective LRs, and it seems to me that this paper conflates it at a few points. For example, the experiments appear to use the maximal update parameterization (muP, see app. M.2), but in that case one should be careful about which normalization/preconditioner to apply to the Hessian before drawing conclusions from it. Let the parameter space be Θ=Rp\Theta = \mathbb{R}^p and use layer-wise LRs D=Diag(d)∈Rp×pD = \mathrm{Diag}(d) \in \mathbb{R}^{p \times p}. Then, the Hessian whose eigenvalues govern stability, convergence, etc. are those of D1/2HD1/2D^{1/2}HD^{1/2}, not the raw HH. Under muP, for example, if we take its form without forward multipliers (see, e.g., [5]), then trainable/effective coordinates coincide with the ones in this paper, i.e., init stds (din−1/2,n−1/2,n−1)(d_{in}^{-1/2}, n^{-1/2}, n^{-1}) and SGD LRs (ηn,η,η/n)(\eta n, \eta, \eta/n) for input/hidden/output weights, respectively. But then, scaling width from n0n_0 to n=kn0n=kn_0 multiplies the layerwise LRs by (k,1,k−1)(k, 1, k^{-1}). If we compare this to KK in Thm. 4.3 for an even split of every hidden neuron, we notice that the muP LR scaling gives exactly Dn=Dn0K−1D_n = D_{n_0}K^{-1}, i.e. an additional K−1K^{-1} factor to the preconditioner if we want to look at the Hessian in optimizer coordinates relative to the narrow network. In other words, at a zero-loss minimum the Hessian in optimizer coordinates Dn1/2H′Dn1/2D_n^{1/2}H'D_n^{1/2} of the expansion has the exact same nonzero spectrum as that of the narrow network. For other uneven splits, this means that moving along the orbit only ever increases curvature in the optimizer coordinates. For the results in the paper, this means that in the metric that is actually relevant for optimization dynamics, the theory in sec. 4/5 would not predict any systematic conditioning benefit from increasing width. In other words, one could also read the theory on preconditioning as a statement about the fact that layerwise LRs matter.
  • It seems to me that the experiments don’t measure the correct metric either, as they report characteristics of the spectrum of the raw Hessian across width, while, as it seems, scaling width via muP. In fact, under muP it has been observed empirically that top eigenvalues of the properly normalized Hessian D1/2HD1/2D^{1/2}HD^{1/2} (or n⋅Hn \cdot H in the mean-field representative) are in fact largely width-consistent, even at larger scales than tested here and not under function-preserving expansion but for independently trained networks of different widths [3]. However, the raw Hessian should have some blocks that scale as O(n−1)O(n^{-1}) and some as O(1)O(1) (depending on the muP representative), which would be entirely consistent with sharpness, condition, and AM-GM ratio decreasing with width and then saturating at some point. Hence, as reported at the moment, the experiments do not distinguish the paper’s proposed explanation from simply parameterization artifacts. Additionally, for networks trained with Adam the relevant quantity would also be the properly preconditioned Hessian (see, e.g., [4]).
  • About the condition number (as, e.g., reported in fig. 4), I am wondering how stable and relevant this quantity is in general, and if it being smaller actually has any impact on optimization. Namely, Hessian spectra often have a lot of eigenvalues very close to zero, so choosing the smallest positive one should be numerically unstable, but the associated directions have little influence on the loss anyway so a condition number involving them should say little about optimization. The appendix cleared up for me that fig. 4 just looks at the top 100 eigenvalues (which is a somewhat arbitrary choice, also since the number of parameters grows as n2n^2). Also condition and AM/GM ratio are quite close to 1 i.e. the top 100 eigenvalues are very similar so I wonder how much these numbers really tell us about the geometry of the minima (and whether this maybe has something to do with the synthetic task). Perhaps just reporting the spectra themselves would be more informative.

[1] Fukumizu, Yamaguchi, Mototake, Tanaka. Semi-flat minima and saddle points by embedding neural networks to overparameterization. NeurIPS 2019.

[2] Zhang, Zhang, Luo, Xu. Embedding principle of loss landscape of deep neural networks. NeurIPS 2021.

[3] Noci, Meterez, Hofmann, Orvieto. Super consistency of neural network landscapes and learning rate transfer. NeurIPS 2024.

[4] Cohen et al. Adaptive gradient methods at the edge of stability. arXiv:2207.14484, 2022.

[5] Yang, Hu, Babuschkin, Sidor, Liu, Farhi, Ryder, Pachocki, Chen, Gao. Tensor Programs V: Tuning large neural networks via zero-shot hyperparameter transfer. NeurIPS 2021.

Questions for authors
  • How do sec.4/5 relate to [1] and [2], and what do you see as the main additional contribution relative to them?
  • In which coordinates do you compute the Hessian in the experiments (and which parameterizations do you use), and can you report the spectral quantities in optimizer/scaling coordinates (and the properly preconditioned Hessian for Adam)?
  • Have you checked whether trained minima of wide networks are actually expansions of the narrower ones, or close to one?
  • As directions along the orbit do not give a gradient, what is supposed to move the optimizer towards these better-conditioned representatives?
  • L. 100-103: universal approximation is only up to some ε\varepsilon error, so the student would retain nonzero residual loss. Have you thought about how the results would extend to such a setting?

Minor:

  • L. 32-33 (”A major line of theory explains this in the infinite limit via linearization”): What does “this” refer to? If it is the fact that “overparameterized neural networks consistently outperform their smaller counterparts”, I don’t see how the NTK literature supports this? In the NTK limit, GD simply reduces to a kernel regression with a fixed kernel and that literature shows convergence to a global minimum once width is sufficiently large. If anything, this lazy limiting regime is known to underperform proper feature learning, so I don’t think “wider is better” follows from that line of research.
  • Only a suggestion, but I feel that fig. 2 is placed quite prominently in the main body for just sanity-checking the math, so the space might be better used for something that tests a prediction.
2 comments

We thank the reviewer for their time and the detailed review. Their comments were very helpful in improving our work.

Relation to [1] and [2]

Thank you for pointing these out. They are definitely relevant. We will cite and discuss both. [1] and [2] construct the same kind of narrow-in-wide network embeddings as we do and show that critical points of a narrow network remain critical in the wider one. The gradient scaling under a split appears in the proof of Theorem 1 in [2], and Lemma 4 of [1] gives the Hessian at the embedded point in a reparameterized basis. These works focus on whether the embedded point is a minimum or a saddle and on the number of flat directions.

In addition, we show that the nonzero eigenvalues of the expanded Hessian are the eigenvalues of HK for an explicit diagonal K for splitting, and give an expression for K for general symmetries. From this we derive how the trace, pseudo-determinant and average-case conditioning change under a split. We also bound how the probability of initializing near a global minimum changes when a neuron is added (Section 6), which neither paper addresses. Finally, we add some empirical evidence supporting the importance of the embedding.

muP and optimizer coordinates

This is true when every hidden neuron is split evenly with the same m. In that case the muP learning rate scaling is exactly K^{-1} and the spectrum in optimizer coordinates is unchanged. This is a nice connection to [3] and we will add it as a remark.

Our results are stated for the Hessian in parameter coordinates, which is the relevant one for an optimizer with a single global learning rate. The cancellation also does not hold in general. muP rescales a whole layer by one factor, while K has a separate entry for each neuron that depends on its number of copies. If neuron i has m_i copies and the layer average is m̄, a factor of m̄/m_i remains on its incoming weights and m_i/m̄ on its outgoing weights, so curvature can go down as well as up. Trained students are in this regime, since their clusters have different sizes (Figure 3).

Experiments report the raw Hessian under muP

We compute the Hessian in raw parameter coordinates. The MLP experiments used muP through the mup library (base width 8), which uses an output multiplier and per-layer learning rates. It was mainly used to avoid re-tuning the learning rate at each width. The CNN and Transformer experiments used standard parameterization (Table 5) and show the same trends.

Following the reviewer's suggestion, we recomputed the spectra in optimizer coordinates, P^{1/2} H P^{1/2}, where P is each parameter's learning rate divided by the base rate. The table shows depth-1 ReLU students trained with SGD, averaged over 5 students and 3 seeds. The condition number is the ratio of the first to the D-th eigenvalue, where D is the output dimension.

Teacher widthStudent widthTop eig. (raw)Top eig. (optimizer)Cond. (raw)Cond. (optimizer)
8160.400.571.161.25
8320.330.581.091.26
8640.300.671.051.24
81280.270.571.021.16
16320.180.371.131.33
16640.150.361.061.27
161280.140.351.031.24
162560.130.351.011.21

In optimizer coordinates the top eigenvalue is roughly constant with width, as the reviewer predicted and consistent with [3]. The condition number decreases with width in both coordinates, more modestly in optimizer coordinates. We have not computed the Adam-preconditioned Hessian. We will add these results and state which coordinates each quantity is reported in.

Top 100 eigenvalues and values close to 1

We agree that the smallest positive eigenvalue is unstable, which is why we use the top k. In the teacher-student setting the number of nonzero eigenvalues at an expansion is set by the teacher (Theorem 4.3), so we keep k fixed as width grows. The values close to 1 are specific to the synthetic MLP task. The range is much larger on California Housing and CIFAR-10 (Figure 5) and for the CNN and Transformer students (Figure 15).

Are trained minima expansions of narrower ones?

In the small setting they are. For the activation used in Section 7.1 all global minima are on the teacher's orbit (Simsek et al., 2021), and Figures 3 and 12 show the predicted clustering. For sigmoid, some runs converge to an expansion of the teacher and others reach global minima off the orbit (Appendix M.4).

For the MLP student-teacher experiments we checked the depth-1 ReLU students trained with Adam. For each student neuron we compute the cosine similarity between its incoming weights and those of the nearest teacher neuron. "Copies" is the share of the network on neurons with cosine above 0.99, where each neuron is weighted by the product of its incoming and outgoing weight norms. "Merged loss" is the loss after merging each group of copies back into a single neuron, which gives a network of the teacher's width. Losses are relative to the variance of the teacher's output.

Teacher widthStudent widthCopiesStudent lossMerged loss
163295%0.26%0.32%
166493%0.21%0.26%
1612883%0.38%0.95%
1625678%0.38%0.50%
81676%0.22%17%
83278%0.17%5.0%
86466%0.20%9.3%
812848%0.29%6.7%

With teacher width 16 the students are close to an expansion. Most of the network consists of copies of teacher neurons, and merging them gives a network of the teacher's width with nearly the same loss. The students are closer to an expansion for the wider teacher than for the narrower one. We will add these results to the appendix.

This is consistent with prior work. Martinelli et al. (2024) train several overparameterized students on the same teacher and recover the teacher's weights from the weight vectors the students have in common. They do this for deep and convolutional networks as well. [2] also finds that wide networks are often attracted to critical points embedded from narrower ones.

What moves the optimizer along the orbit?

The loss is the same everywhere on the orbit, so the gradient cannot tell the representatives apart. We don't show which one training reaches (Section 8.1), but there are a few reasons to expect the flatter ones.

The first is stability. Gradient descent with learning rate η cannot converge to a minimum whose top eigenvalue is above 2/η (Cohen et al., 2021), so the sharp representatives on an orbit are ruled out. The second is noise. SGD with label noise drifts along the set of minima towards lower trace (Blanc et al., 2020; Li et al., 2022), and by Theorem 5.1 the even split has the lowest trace. The third is volume. Representatives with larger local volume take up more of parameter space, so a random initialization is more likely to start near them (Sections 4.4 and 6).

We will make this more explicit in Section 8.1.

Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process. In Conference on Learning Theory, PMLR 125:483-513, 2020.

Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? A mathematical framework. In International Conference on Learning Representations, 2022.

Universal approximation is only up to some error… Have you thought about how the results would extend to such a setting?

This is an interesting point. Several of the results do extend.

A split preserves the function and the loss everywhere, and the gradient relation in Theorem 4.2 also holds everywhere, so a critical point of the narrow network is still a critical point of the wide one. For the Hessian, H' = B^T H B holds exactly for the Gauss-Newton part (Appendix G). The remaining term is proportional to the residual and vanishes at zero loss.

In practice this term is small. Tables 2 and 3 show it is under 10% of the Hessian norm at the losses we reach and changes the condition number by less than 1%. We can reword L. 100-103 to make this clearer.

L. 32-33 (”A major line of theory explains this in the infinite limit via linearization”)

We only meant the NTK results support that the optimization is easy in this limit and is essentially convex from initialization to the minimum. We agree the sentence can be read differently and will reword it.

We apologize for the late reply but are happy to answer any questions that might remain.

Reviewer_u1b5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the loss landscape changes as neurons are added by studying neural networks of different widths that represent the same function. It characterizes function-preserving transformations across width through families of matrices. It derives that width symmetries act as a form of diagonal preconditioning on the Hessian, enabling better-conditioned minima. It also finds that overparameterization increases the probability of global minima near typical initializations, making these solutions more reachable. Validation includes width sweeps on MLPs, CNNs, and Transformers, showing that wider networks have lower top eigenvalues, smaller condition numbers, and faster convergence.

Critical strengths
  • The paper presents a useful connection between network width symmetries and curvature. It shows that splitting reshapes the Hessian's curvature in a way equivalent to applying a diagonal preconditioner to the original Hessian. This explains how wider networks can contain better-conditioned minima representing the same function.

  • Experiments support the theoretical predictions. The small network experiments closely match the predicted Hessian spectra and speudo-determinants. Broader experiments on MLPs, CNNs, and Transformers show that conditioning number and convergence improve with width, consistent with the effects predicted by the theory.

Critical weaknesses
  • The conditioning covers only part of the Hessian spectrum. Most experiments use the largest 100 eigenvalues or 50 for transformers (p35, Table 5). However, the full condition number also depends on the smallest positive eigenvalue which these experiments may exclude, weakening the broader empirical claim.
Questions for authors
  • Are the reported condition numbers and AM/GM ratios calculated using only the largest k eigenvalues? If yes, do the improvements with width also hold when the full Hessian spectrum is measured?

  • Do any existing experiments directly connect faster convergence to the proposed neuron splitting mechanism? For instance, have you compared convergence near solutions that represent the same function but have different Hessian conditioning due to splitting? This comparison would help connect the theoretical results to the observed training behavior.

1 comment

We thank the reviewer for their time in reviewing our work and for their supportive recommendation.

Condition numbers, AM/GM with top 100 eigenvalues

In practice, the spectra of neural networks have a large number of near-0 eigenvalues, making estimates of the condition number numerically unstable and highly sensitive. As a result, it is common practice in empirical work to treat eigenvalues below a small threshold as zero (as we do in this work) or to compute quantities for the top-K eigenvalues (Sagun et al., 2017; Yao et al., 2020).

Computing these statistics with the full spectrum gives values that are very sensitive to numerical noise and does not give any robust measurement. We will state it more clearly in the main text.

We apologize for the late reply but are happy to answer any further questions that might remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.