acceptodds
Under review as a conference paper at ICLR 2027

Same Spectral Bulk, Different Stable Steps: State Memory in Wide Neural Networks

Abstract

A common approach to understanding neural network training is to describe directional sensitivities through eigenvalue distributions and use this description to assess stable learning rates. This seems natural: the distribution records how many directions are relatively flat and how many are sensitive. We show that even this overall description can be insufficient to determine local training stability. We construct wide neural networks with the same limiting eigenvalue distribution and all fixed-order normalized spectral moments, yet the same learning rate is stable for one and unstable for the other. Stability can be constrained by a single exceptional direction whose contribution to the distribution vanishes as width grows. Our analysis shows that matching aggregate statistics does not match the critical directions: the network computation can selectively amplify a few directions that determine stability while leaving the limiting distribution unchanged. We characterize when these directions emerge and when they restrict the learning rate in Gaussian networks with two nonlinear layers, for a single input near zero training error with all weights trained. We further give explicit conditions under which a trainable output layer preserves the difference or averages it away. Experiments confirm that networks with nearly identical spectral bulk can admit different stable learning rates, and that a few forward-pass statistics capturing the exceptional directions improve prediction of this difference. These findings can help guide neural network training by identifying the information needed beyond aggregate spectral descriptions.

Primary area:
learning theory
Keywords:
Deep Learning Theory, Neural Network Spectra, Random Matrix Theory, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_2ZMNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper constructs wide neural networks with the same limiting eigenvalue distribution, yet the same learning rate is stable for one and unstable for the other and validates this phenomenon experimentally. This is because the stable learning rate depends on the largest eigenvalue rather than the limiting eigenvalue distribution. The authors also give conditions under which the output layer preserves or the difference between the largest eigenvalue and the boundary of the limiting spectrum.

Critical strengths

This paper constructs wide neural networks with the same limiting eigenvalue distribution, yet the same learning rate is stable for one and unstable for the other, which raises an important research question that which part of the weights spectra is useful to understand neural network training.

Critical weaknesses
  1. Unclear novelty and significance. The main message of the paper appears to be that two neural networks can have the same limiting eigenvalue distribution but different stable learning rates. However, this is a direct consequence of the fact that the stability threshold depends on the largest eigenvalue rather than the limiting spectral distribution. It is already well known that a few outlier eigenvalues can have a significant impact on optimization. Therefore, I find the main contribution of this work unclear. The paper does not sufficiently explain what new insights it provides beyond existing results on spiked random matrices and spectral outliers.

  2. Poor presentation and unnatural problem formulation. Section 3 is particularly difficult to follow. The authors introduce a large number of variables, matrices, and technical definitions without clearly explaining their roles. This makes it difficult to understand the main results. The asymptotic limits in these theorems are not clearly defined. Moreover, The theoretical models are not introduced in a way that makes their connection to neural networks immediately clear. For example, Section 3.1 starts with a gated Gaussian random matrix. I believe it would be much more natural to first specify the neural network architecture, activation functions, trainable parameters, and loss function, and then derive the corresponding matrices.

  3. Limited connection to actual neural network training. Much of the theoretical analysis concerns the spectra and outliers of certain random matrices, while the connection to training is mainly established through the stability condition determined by the largest NTK eigenvalue. It is unclear how these results inform the training dynamics of neural networks in more realistic learning problems.

Questions for authors
  1. Since the importance of spectral outliers for stability is already well understood, what is the main novel contribution of this paper?

  2. Can the authors substantially simplify and reorganize Section 3? In particular, can the models be formulated directly as neural networks, with the corresponding Jacobians and NTKs derived afterward?

  3. How relevant are the theoretical results to actual neural network training? Can the authors provide evidence that the proposed mechanism affects training in a nontrivial supervised learning setting?

Reviewer_9V1WOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines whether the limiting spectral distribution of a wide neural network is sufficient to determine its local training stability. It constructs Gaussian networks with identical limiting empirical spectral distributions and normalized moments but different largest eigenvalues and critical learning rates. The analysis attributes this difference to correlations between derivative gates and preactivations. The authors derive outlier conditions for two nonlinear layers and characterize when an additional trainable Gaussian output layer preserves or suppresses the effect. Numerical experiments compare the predicted eigenvalues and local stability thresholds with finite-width measurements and gradient-descent trajectories near interpolation.

Critical strengths

The paper derives explicit conditions under which networks with the same limiting spectral bulk have different largest eigenvalues and local stability thresholds. It determines when a trainable Gaussian output layer preserves or suppresses this spectral difference.

Critical weaknesses

The proof asserts that the boundedness of the derivative-gate range KℓK_\ell makes it compact and subsequently identifies this range with the support of the corresponding Gaussian pushforward distribution. Neither conclusion follows as stated. For example, the smooth activation ϕ(x)=tanh⁡(x)\phi(x)=\tanh(x) has derivative ϕ′(x)=sech⁡2(x)\phi'(x)=\operatorname{sech}^2(x), whose range is (0,1](0,1]. This set is bounded but not compact. The corresponding pushforward distribution under a nondegenerate Gaussian has support [0,1][0,1], not (0,1](0,1]. The argument therefore contains a genuine mathematical error. Lemma F.2 is used in establishing strong convergence of the gate algebra, which in turn supports the reference spectral analysis. The authors need to correct this step and verify that the subsequent convergence arguments remain valid under the corrected formulation. Taking the closure of the gate range appears to be a possible repair, so the error alone does not establish that the main spectral results are false. The stability comparison is derived at an interpolating solution for a single input, using the local parameter NTK. The construction demonstrates that equal limiting spectral distributions need not determine the same local stability threshold. It does not establish the corresponding effect during general multi-sample nonlinear training. The experiments do not bridge this gap. Consequently, the mathematical construction supports a narrower conclusion than the broader learning-rate implications suggested by the paper.

Questions for authors
  1. In Appendix F.2, how is the assertion that the bounded derivative-gate range KℓK_\ell is compact justified? For smooth activations such as tanh⁡\tanh, the range need not be closed. Does replacing KℓK_\ell with its closure suffice to establish every strong-convergence statement subsequently used?
  2. Does the correction to Lemma F.2 require any changes to the assumptions or proofs of the spectral-support results that depend on the strong convergence of the gate algebra?
  3. Can the authors establish that the predicted separation in stable learning rates persists for nontrivial multi-sample training beyond the local interpolation setting?
Reviewer_AdUzOpenReview ↗
Rating
2(−1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores the limitations of NTK’s eigenvalue distribution for training stability. This is done by constructing wide neural networks with the same eigenvalue distributions but different stable learning rate ranges. This is because even a single outlier eigenvalue defines the maximal stable learning rate, and the matching aggregate statistics do not match these critical directions. The paper characterizes these directions in Gaussian networks with two nonlinear layers for a single input around zero training loss. The experimental section demonstrates these characterizations on finite-width networks and shows that forward pass statistics can improve their predictions.

Critical strengths
  • The overall message is clear and intuitive: the distribution cannot determine learning rate stability. Showing simple and concrete examples is helpful for intuitive understanding. Having a clear and intuitive message is consequential to my overall assessment, as it tells future work not to rely only on the eigenvalue distribution.

  • Explicit formulas for both outlier location and eigenvector overlap to determine actual thresholds and step sizes that are verified in the simulations, showing the theory is reliable, which is consequential to my overall assessment.

Critical weaknesses
  • Having one synthetic input, initialization at zero error, only local linear stability, and only the top eigenvalue at initialization (and not how it changes during training) makes it far from real training settings.

  • When the output dimension is small relative to the hidden width, which is a typical case in practical settings, averaging over hidden directions removes the spectral difference. This limits where the main message applies.

These two points are consequential to my overall assessment because they reduce the paper’s practical usage.

Questions for authors

Please address the weaknesses.

Reviewer_Hnx4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper illustrates the phenomenon that two networks' NTK can have the same limiting empirical spectral distribution as width goes to infinity, but they have the different largest eigenvalues (which can not be detected by the bulk distribution). Since gradient descent stability depends on the largest eigenvalue, the stable step sizes can differ.

Critical strengths

The paper seems to illustrate a simple but important observation from both theoretically and empirically. The main message is somewhat clearly illustrated in Fig 1.

Critical weaknesses

The paper has significant presentation problems that make it almost totally unreadable. The following are representative examples rather than an exhaustive list. In particular, I found section 3 difficult to follow from beginning to end, despite familiarity with NTKs and random-matrix theory.

  • The main set-up, such as the precise network architecture, scaling limit and trainable parameterization are not given at the start of section 3. Is the network defined in section 3.3, eq(4) the main set-up used throughout? Why is it only defined at section 3.3? Currently section 3.1 does not make sense itself.
  • Terminology like "derivative gate" "center/tail activations" is stated without any even informal explanation of what it means.
  • Notation are not defined when they are first mentioned, like the learning rate η\eta, the residual rtr_t, and the activation ϕA\phi_A.
  • Many mathematical statements are unexplained and not given proper context, and does not make any sense in reading.
  • The relationship between various parts of section 3 is not explained clearly, making the logic hard to be understood.

Because the accessibility problem is too significant, it is also very hard for me to access the content, particularly the theoretical correctness of the paper. I had to use AI tool to even get an idea of what the paper is claiming. Based on my understanding through that, i also have the following concerns regarding the scope and practical relevance:

  • The theory is conditional on very strong (and unrealistic) assumptions on the network: the activation is non-standard; there's a specific limiting regime where input, hidden and output dimensions all goes to infinity at the same rate; choosing the target function so that initialization is already an zero-loss solution is strange. Hence the scope of the theoretical analysis seems to be limited.
  • The "stability" of lr is also not carefully defined. It seems also that the notion of stability here is also narrow in scope and is separate from practical meaning of stability.
  • The main message is that the bulk distribution is insufficient --- this is already generally expected and not surprising at all. Largest eigenvalue can be easily empirically found, and is also a generally available spectral statistics. Not sure whether this paper's claim has a strong practical implication.
Questions for authors

No additional questions beyond weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.