acceptodds
Under review as a conference paper at ICLR 2027

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer

Abstract

We study the evolution of hidden-weight spectra in wide neural networks trained by (stochastic) gradient descent. We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for spiked ensembles whose spike directions remain statistically dependent on the random bulk. We apply this framework to two settings: (1) infinite-width nonlinear networks in mean-field/P scaling and (2) deep linear networks in the proportional high-dimensional limit, where width, input dimension, and sample size diverge with fixed ratios. Our theory predicts how outliers evolve with training time, width, output scale, and initialization variance. In deep linear networks, P yields width-consistent outlier dynamics and hyperparameter transfer, including width-stable growth of the leading NTK mode toward the edge of stability (EoS). In contrast, NTK parameterization exhibits strongly width-dependent outlier dynamics, despite converging to a stable large-width limit. We show that this bulk+outlier picture is descriptive of simple tasks with small output channels, but that tasks involving large numbers of outputs (ImageNet classification or GPT language modeling) are better described by a restructuring of the spectral bulk. We develop a toy model with extensive output channels that recapitulates this phenomenon and show that edge of the spectrum still converges for sufficiently wide networks.

Primary area:
learning theory
Keywords:
Spectral dynamics, hyperparameter transfer, dynamical mean-field theory, random matrix theory.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PrPSOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper develops a general DMFT in an abstract spiked matrix setting. It then applies this theory in two important cases: Infinite Width network (with constant input dimension, batch size and training time) and deep linear networks (with width, input dimension, batch size and training size large and comparable). These results are then used to study hyperparameter transfer. The authors support their theoretical findings with numerical experiments.

Critical strengths

The problem of identifying outliers in the dynamical setting of training neural networks is hard in general. In that sense, the authors not only develop a theory that works, but they also map two seemingly different cases in the same framework of spiked random matrices with coupled bulk and spikes. The theoretical contribution itself is already enough to justify acceptance. Beyond the theoretical contribution, the paper is clearly written and understandable (although introducing DMFT is never an easy task in a paper).

Critical weaknesses

None.

Questions for authors
  1. Through the paper the authors always assume that initialization of the parameters is done with a particular mean and variance. I imagine there is a large family of distributions that match the necessary assumptions, but I don't manage to find it in the main text nor the appendix.
  2. Appendix G states that the ImageNet and GPT experiments, and in particular Figures 6 and 7, use NNs trained by Adam, while the developed theory is written for (S)GD . However, the authors claim that the difference comes from the large output channel. Why is the output dimension the "main suspect" and not the training algorithm, which could also be modifying the bulk of the matrix spectrum?
1 comment

Critical Strengths

Thank you for such an encouraging review. Bringing both infinite-width nonlinear networks and deep linear networks in the proportional regime into the same framework under SGD dynamics was one of our main goals, so we especially appreciate that you highlighted this connection.

Questions

  1. Thank you for this question. You are right, we never stated the assumptions on the initialization. We fixed it and now state them explicitly after Eq. (1). The assumptions are: the entries of each Wℓ(0)W^\ell(0) are i.i.d., mean zero, variance σ2\sigma^2, with finite fourth moment. Gaussianity is not needed, since the initialization enters only through the disorder average in Eq. (72), which at leading order in NN only depends on the second cumulant. The finite fourth moment is needed for the bulk-plus-outlier picture. This is shown in the new Fig. 11 of App. C.2: with Rademacher, uniform and Student-t5t_5 initializations, λmax⁡(T)\lambda_{\max}(T) coincide with the Gaussian DMFT prediction, whereas with a Student-t3t_3 initialization (finite variance, infinite fourth moment) the bulk is still Marchenko–Pastur but O(10)O(10) eigenvalues, which are produced by large entries of W(0)W(0), already lie above the edge at T=0T=0.
  2. This is a very good question. We thank you for this point, with which we actually agree and will clarify in the main text. Our reason for pointing to the output dimension is the toy model of Sec. 5: it is trained with plain (S)GD and already reshapes the bulk as soon as C/D=O(1)C/D=O(1) (Figs. 8, 18–20), while with C=10C=10, for the ResNet in Fig. 2, the bulk stays Marchenko–Pastur even after many steps. An extensive number of outputs is therefore sufficient for bulk reshaping, and it is the mechanism our theory captures. This is also consistent with evidence that bulk deformation does not require Adam: the heavy-tailed spectra of Martin & Mahoney (2021) are measured on ImageNet CNNs trained with SGD for instance. However, the output dimension is only one possibility, and we totally agree that Adam's preconditioned update makes each update full rank and could therefore deform the bulk even when the gradient itself is low-rank, i.e. even for tasks with few outputs. We will state this in the captions of Figs. 6 and 7, add this discussion to Sec. 5, and leave the study of optimizer effects on the bulk structure for future work.
Reviewer_WAZeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper develops a two-level dynamical mean-field framework for predicting spectral outliers of trained weight matrices when the update directions depend statistically on initialization. It applies the framework to nonlinear networks in the fixed-data, infinite-width regime and to deep linear networks in a proportional limit. It also studies a one-step linear model with extensively many outputs, motivated by empirical bulk deformation in ImageNet models and language models.

Critical strengths

Below, I list what I believe are the strengths of the paper.

  1. Accounting for dependence between learned update directions and initialization is a worthwhile contribution.
  2. Studying the spectral distribution of non-linear neural networks in the width limit, so far, I have only seen it done for deep linear networks under quite strong assumptions.
  3. The distinction between a limiting bulk distribution and isolated eigenvalues is conceptually valuable.
Critical weaknesses

I identified the following weaknesses:

  1. As a reader less familiar with this proof strategy, I found the arguments difficult to follow in their current form. A worked example involving a shallow nonlinear network in the main text would help make the approach more accessible. I would also appreciate a clearer introduction to the general technique, starting with the simple case of a GOE matrix with an independent rank-one perturbation that is provided in the appendix, but it is still quite simplified and hard to follow. More explanation of the key steps and the intuition behind them would substantially improve readability.

  2. The lemmas and main results in Sections 3–5 are stated rather informally, making their precise assumptions and conclusions difficult to identify. More explicit statements would help clarify the scope of the results. In particular, the authors should explicitly distinguish predictions obtained through formal DMFT calculations from rigorously established results, and specify the assumptions under which each result applies.

Questions for authors

I would ask the authors to address my weaknesses. Would it be possible to address weaknesses 1 and 2? I really like the idea and the intention of the paper, and I think this is a nice contribution; however, I would like it to be easier to follow and the proof technique more understandable for someone who does not have a strong background in this kind of argument.

5 comments

We thank the Reviewer for the careful reading and for the constructive criticism. We are glad that the three main points of the paper came across as the contributions we intended. The two weaknesses raised both concern presentation, and we address each in detail below, with the expanded toy-model derivation, a new worked example on a nonlinear network, and explicit assumption statements for every result. We hope the Reviewer will find the paper substantially easier to follow with these changes and will consider revising their score.

Critical Weaknesses

1. Making the two-level DMFT easier to follow Thank you for suggesting these two examples. They helped us see where the presentation needed more intuition. We have expanded Appendix A to explain how the spectral probe detects an independent rank-one spike and revised Section 2 to introduce the two levels of DMFT before stating Result 1 (see also our response to Reviewer RUTb). Plus, we have also added a new Appendix D.1 on the explicit derivation for the toy model. Below, we first walk through the independent-spike example and then show how one gradient step in a shallow nonlinear network can produce a spectral outlier (which can also be found in the updated pdf).

1(a). An independent rank-one spike: how the probe detects an outlier.

Consider a symmetric D×DD\times D matrix M=M0+γ0Dvv⊤M=M_0+\frac{\gamma_0}{D}vv^\top, where M0M_0 is a GOE matrix and vv a fixed vector with ∥v∥2=D\|v\|^2=D, which we assume independent of M0M_0 for this toy example. We would like to know its spectrum: the density of the bulk and whether an eigenvalue detaches from it.

Define the linear probe dynamics

∂τh(τ)=−M h(τ)+j(τ),\partial_\tau h(\tau)=-M\,h(\tau)+j(\tau),

where τ\tau is a fictitious time and jj a source. Its solution is h(τ)=e−Mτh(0)+∫0τe−M(τ−τ′)j(τ′)dτ′h(\tau)=e^{-M\tau}h(0)+\int_0^\tau e^{-M(\tau-\tau')}j(\tau')d\tau' hence the linear response of h(τ)h(\tau) to a kick j(τ′)j(\tau') at an earlier time τ′\tau' is the propagator

{\partial j(\tau')} = e^{-M(\tau-\tau')}\Theta(\tau-\tau').$$ Its Fourier transform in $\tau-\tau'$ is $(i\omega+M)^{-1}$: the resolvent of $M$. **The probe is just a way of writing the resolvent as a response function.** Two scalars extracted from it contain everything we want. The normalized trace of the response $$R_h(\omega)=\tfrac1D\mathrm{Tr}\,(i\omega+M)^{-1}=\tfrac1D\sum_k\frac{1}{i\omega+\lambda_k}$$ is the Stieltjes transform: at $i\omega=-\lambda-i0^+$ its imaginary part is $\pi\rho(\lambda)$, i.e. the spectral density. Since each eigenvalue in this sum has weight $1/D$, an isolated eigenvalue is invisible in $R_h$ in the large $D$ limit. To see it we must look along the direction that created it. If we start the probe at $h(0)=v$ with $j=0$, then $h(\omega)=(i\omega+M)^{-1}v$ and the correlation between $v$ and the probe is $$C_{vh}(\omega)=\tfrac1Dv\cdot h(\omega)=\sum_k\frac{|\langle v,u_k\rangle|^2/D}{i\omega+\lambda_k},$$ i.e. the resolvent projected on $v$. At an isolated eigenvalue this correlation function has a pole, and the residue of the pole is the squared overlap of that eigenvector with $v$, which is now finite in the large $D$ limit. **So: bulk is given by the trace of the response, while the outliers by the poles of the projected resolvent.** The whole problem is now to compute $R_h$ and $C_{vh}$ after averaging over $M_0$, which is what DMFT does. To compute these quantities through DMFT, we introduce a conjugate field $\hat h$ that enforces the probe equation of motion through a Dirac delta function: $$Z=\int \mathcal{D}h\,\mathcal{D}\hat h \exp\left(i\int d\tau\,\hat h\cdot \left[\partial_\tau h+M_0h+\gamma_0 C_{vh}(\tau)v-j\right]\right)=1.$$ The random matrix appears linearly in the exponent, so averaging over the GOE gives the factor $$\exp\left[-\frac{1}{2}\int d\tau\,d\tau' \left(C_h(\tau,\tau')\,\hat h(\tau)\cdot\hat h(\tau') -D R_h(\tau,\tau')R_h(\tau',\tau)\right)\right]$$ with $R_h(\tau,\tau')=-\frac iD h(\tau)\cdot\hat h(\tau')$ and $C_h=\frac1Dh\cdot h$. Because the action is now proportional to $D$ and a function of $C_h,R_h$, the coordinates decouple at large $D$ and each one obeys a **single-site scalar process** $$\partial_\tau h(\tau)=u_h(\tau)+\int d\tau'\,R_h(\tau,\tau')\,h(\tau')-\gamma_0C_{vh}(\tau)\,v_i,\qquad u_h\sim\mathrm{GP}(0,C_h),$$ closed by $C_h=\langle hh\rangle$ and $R_h=\langle\delta h/\delta u_h\rangle$ which converge to deterministic quantities at $D\to \infty$. The bulk follows by differentiating the single-site process with respect to the noise $u_h$. In Fourier space this obeys $R_h=1/(i\omega-R_h)$, i.e. $R_h(\omega)=\tfrac12\big(i\omega-\sqrt{(i\omega)^2-4}\big)$, whose imaginary part at $i\omega=-\lambda$ is the semicircle law which defines the bulk (Eq. (24) Appendix A).

The spike follows by projecting the single-site process on vv. Since vv is independent of M0M_0, ⟨vu⟩=0\langle v u\rangle=0, and with ⟨v2⟩=1\langle v^2\rangle=1 and Cvh(0)=C0C_{vh}(0)=C_0 one finds in Fourier space (Eq. (26) Appendix A) Cvh(ω)=C0iω+γ0−Rh(ω).C_{vh}(\omega)=\frac{C_0}{i\omega+\gamma_0-R_h(\omega)} .

Notice that the poles do not depend on C0C_0, which is why the initial condition plays no role in locating outliers; it matters only for the residue. We want to study \begin{equation} i\omega + \gamma_0 - R_h(\omega) = 0. \end{equation} Analytically continuing to the real spectral axis with iω=−λ−i0+i\omega=-\lambda-i0^+, the outlier condition becomes \begin{equation} -\lambda + \gamma_0 = \frac{ -\lambda + \sqrt{\lambda^2-4} }{2}. \end{equation} Solving gives the classical spiked-Wigner outlier location \begin{equation} \lambda_{\rm out} = \gamma_0 + \frac{1}{\gamma_0}. \end{equation} This solution lies outside the Wigner bulk only when γ0>1\gamma_0>1.

1(b). Worked example: a three-layer nonlinear network. We now apply Result 1 in the regime of this section: N→∞N\to\infty at fixed input dimension DD. We consider the smallest network that has a nontrivial spectrum. With a single hidden layer the spectrum is trivial: the only matrix is W0∈RN×DW^0\in\mathbb R^{N\times D}, and 1NW0⊤W0\frac1NW^{0\top}W^0 is D×DD\times D with all eigenvalues →1\to1. We therefore take two hidden layers,

f=1γN w⋅ϕ(h2),h2=1NW1ϕ(h1),h1=1DW0x,f=\tfrac{1}{\gamma N}\,w\cdot\phi(h^2),\qquad h^2=\tfrac{1}{\sqrt N}W^1\phi(h^1),\qquad h^1=\tfrac{1}{\sqrt D}W^0x,

with random intial weights N(0,1)\mathcal N(0,1), and study the hidden matrix M1=1NW1⊤W1M^1=\frac1NW^{1\top}W^1 (notice that α=N/N=1\alpha=N/N=1, so Marchenko–Pastur edge is at λ=4\lambda =4). To keep every quantity explicit and simplify the derivation as much as possible, we train on a single example xx with ∥x∥2=D\|x\|^2=D and target yy, so that h1(0)=χ1∼N(0,1)h^1(0)=\chi^1\sim\mathcal N(0,1) entrywise, and all three layers are updated with GD. One step. At initialization, in the large NN limit f=O(N−1/2)f=O(N^{-1/2}), so Δ0=y\Delta_0=y and the update equation is W1(1)=W1(0)+1Ng(0)ϕ1(0)⊤W^1(1)=W^1(0)+\frac1{\sqrt N}g(0)\phi^1(0)^\top with ϕ1(0)=ϕ(χ1),g(0)=ηγy  w⊙ϕ′(χ2),χ2=1NW1(0)ϕ1(0).\phi^1(0)=\phi(\chi^1),\qquad g(0)=\eta\gamma y\;w\odot\phi'(\chi^2),\qquad \chi^2=\tfrac1{\sqrt N}W^1(0)\phi^1(0).

This is Eq. (1) with the structure (2): the right vector ϕ1\phi^1 is independent of W1(0)W^1(0), and the left vector g(0)g(0) is an elementwise function of χ2\chi^2. The spike is a function of the random matrix whose spectrum we want.

First-level DMFT: statistics of the spike vectors. Result 1 needs four numbers for S=1S=1: the correlations Cϕ=1N∥ϕ1∥2C^\phi=\frac1N\|\phi^1\|^2, Cg=1N∥g∥2C^g=\frac1N\|g\|^2 and the responses Rϕ,RgR^\phi,R^g of Lemma 1. They come from averaging over the three random objects W0(0)W^0(0), W1(0)W^1(0), ww. As before, the averages over these quantities can be easily perfomed by enforncing the definitions of the fields with conjugate variables and Dirac delta functions.

Tthe algebra of these simple Gaussian averages is the same as the above toy example. We then state the result: the action again factorizes across neurons, and the entries of the two spike vectors become i.i.d. draws from the single-site stochastic processes

ϕ1=ϕ(χ1),χ1∼N(0,1),\phi^1=\phi(\chi^1),\qquad \chi^1\sim\mathcal N(0,1), g=ηγy w ϕ′(χ2),χ2=uχ+Rϕg,uχ∼N(0,Cϕ),  w∼N(0,1),g=\eta\gamma y\,w\,\phi'(\chi^2),\qquad \chi^2=u^\chi+R^\phi g,\qquad u^\chi\sim\mathcal N(0,C^\phi),\;w\sim\mathcal N(0,1),

(and ξ=uξ+Rgϕ1\xi=u^\xi+R^g\phi^1, uξ∼N(0,Cg)u^\xi\sim\mathcal N(0,C^g)), with correlations and responses fixed self-consistently by Cϕ=⟨(ϕ1)2⟩C^\phi=\langle(\phi^1)^2\rangle, Cg=⟨g2⟩C^g=\langle g^2\rangle, Rϕ=⟨∂ϕ1/∂uξ⟩R^\phi=\langle\partial\phi^1/\partial u^\xi\rangle, Rg=⟨∂g/∂uχ⟩R^g=\langle\partial g/\partial u^\chi\rangle. This is Lemma 1 for S=1S=1, σ2=α=1\sigma^2=\alpha=1. After one step there is no feedback yet: ϕ1\phi^1 does not involve uξu^\xi, so Rϕ=0R^\phi=0 and Cϕ=E[ϕ(χ1)2]C^\phi=\mathbb E[\phi(\chi^1)^2]; then χ2=uχ∼N(0,Cϕ)\chi^2=u^\chi\sim\mathcal N(0,C^\phi) and

Cg=η2γ2y2 E[ϕ′(uχ)2],Rg=ηγy E[w] E[ϕ′′(uχ)]=0.C^g=\eta^2\gamma^2y^2\,\mathbb E\big[\phi'(u^\chi)^2\big],\qquad R^g=\eta\gamma y\,\mathbb E[w]\,\mathbb E\big[\phi''(u^\chi)\big]=0 .

This concludes the first level DMFT.

Second level DMFT: the spectral probe. As in 1(a), we probe the spectrum with a linear dynamics, ∂τψ0=M1(1)ψ0\partial_\tau\psi_0=M^1(1)\psi_0. Since M1(1)M^1(1) is a product, we write it in two steps:

ψ1=1NW1(0)ψ0+c0 g,∂τψ0=1NW1(0)⊤ψ1+c1 ϕ1\psi_1=\tfrac{1}{\sqrt N}W^1(0)\psi_0+c_0\,g,\qquad \partial_\tau\psi_0=\tfrac{1}{\sqrt N}W^1(0)^\top\psi_1+c_1\,\phi^1 with c0=1N ψ0⋅ϕ1,c1=1N ψ1⋅g.c_0=\tfrac1N\,\psi_0\cdot\phi^1,\qquad c_1=\tfrac1N\,\psi_1\cdot g .

By enforcing these dynamics as for the toy model, we can now average over W(0)W(0). The average does two things. The memory terms give the bulk: the response of the probe is the Stieltjes transform of the square Wishart matrix

G(z)=12z(z−z2−4z),edge at λ=4,G(z)=\frac{1}{2z}\Big(z-\sqrt{z^2-4z}\Big),\qquad\text{edge at }\lambda=4,

the analogue of RhR_h in 1(a). The projections give the outliers: we project the ψ0\psi_0 equation on ϕ1\phi^1 and the ψ1\psi_1 equation on gg, exactly as we projected on vv in 1(a). The only new ingredient is that the probe noise is now correlated with the noise that generated gg, so where 1(a) used ⟨vuh⟩=0\langle v u_h\rangle=0 we now have, by Stein’s lemma,

⟨uψ1 g⟩=c0 Rg.\langle u^{\psi_1}\,g\rangle=c_0\,R^g .

In this one-step example, both responses vanish. Equation (8) of Result 1 then reduces to two equations for the projections we just defined:

(1-G(z))c_1=C^g c_0.$$ At an outlier these equations admit a nonzero solution. Substituting the first into the second gives $$\frac{1-G(z)}{G(z)}=C^\phi C^g\equiv\kappa, \qquad\text{so}\qquad G(z)=\frac{1}{1+\kappa}.$$ At the upper bulk edge, $G(4)=1/2$, and $G(z)$ decreases for $z>4$. Thus a solution lies above the bulk precisely when $\kappa>1$. Using $z=1/[G(1-G)]$, its location is $$\lambda_{\rm out}=\frac{(1+\kappa)^2}{\kappa}.$$ For ReLU, the two Gaussian averages above give $C^\phi=1/2$ and $C^g=(\eta\gamma y)^2/2$. A single gradient step therefore produces an outlier precisely when $|\eta\gamma y|>2$.
Reviewer_RUTbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper uses dynamic mean field theory (DMFT) to study the spectral dynamics during neural network training, especially the emergence of outliers in the spectra of weight matrices. The authors first analyzed the spikes in the infinite-width nonlinear networks in mean-filed setting. Then the authors characterized the linear networks with proportional width regime with linear teacher models. This spectral dynamics is connected to learning-rate transfer and this learning rate transfer is stable in μ\muP. Additionally, the authors provided a potential model to study for practical neural networks like ImageNet-scale ResNet18 and transformers, where the output dimension is much larger than hidden width and the spectral bulk is not fixed over training anymore. By considering the number of classes is also growing proportionally with input dimension, this paper shows a potential analysis for bulk dynamics of two-layer neural networks.

Critical strengths

This paper analyzes the training time BBP transition for weight matrices during different training regimes. This is a hard question in deep learning and many literatures have worked on this. This paper uses DMFT to compute this spectral dynamics with clear formulas, which help researchers to further understand the training dynamics and feature learning in neural networks. The motivation of this research paper is strong.

Critical weaknesses
  1. The paper is not well-written and the organization is not clear to me. The section 2 introduces the DMFT results for the dynamics of the outliers but there is no clear setup of the model, assumptions, and explanation of how this section is related to later section. There are some notations which are not defined in the main text. This reduces the readability of the main text of this paper. Besides, I think two-level DMFT is one of the main contributions of this paper, but it is not clearly introduced in the main text.

  2. The paper claims that this paper is a feature learning result but the full paper only studies the dynamics of the spikes during training. There is no generalization error analysis or explicit explanations of the feature learning from the training-time BBP transitions. The machine learning conclusion of this paper is not clearly stated.

Questions for authors
  1. What is the condition of W(0)W(0) in Section 2.1? You define W(S)W(S) as a random matrix initialized with mean-zero i.i.d. entries of variance σ2\sigma^2 with SS spikes. Do you mean W(0)W(0) has i.i.d. entries with variance σ2\sigma^2? What are the deterministic functions ϕt\phi_t and gtg_t in (2)? Do they have explicit formulas or are iteratively defined?

  2. In Section 2.1, you assumed that spikes g(t)g(t) and ϕ(t)\phi(t) have asymptotically independent entries. Have you check empirically that this is a reasonable assumption in practical neural networks? During training, I think these entries of spikes cannot be independent anymore because of feature learning.

  3. In Result 1, what is R^{\phi},R^g,C^\{\phi},C^g\in\mathbb{R}^{T\times T} What is TT? This is not introduced in this section, although later you used it as the training time.

  4. In section 3, what is the loss function ℓ\ell and what is the data distribution for training? What are the the structural assumptions of batchsize?

  5. What are the conditions of βc\beta_c, data distribution xx and noise ϵc\epsilon_c in Section 4? In your analysis, how does the spikes βc\beta_c in the teacher model affect the dynamics of the spikes in (13)?

  6. In your analysis, you only considered finite steps TT? What if T∞T\infty scaled with the dimensions? Figure 6 indicates the bulk is also changing when training longer time. And this is not covered in you DMFT regime.

  7. There are several additional references of theoretical studies characterize spectra in shallow networks after one gradient step and BBP transition of spike in the weight matrix related to feature learning.

Damian et al. 2022. Neural networks can learn representations with gradient descent.

Ba et al. 20222. High-dimensional asymptotics of feature learning: How one gradient step improves the representation.

Bietti et al. 2022. Learning single-index models with shallow neural networks.

Nichani et al. 2023. Provable guarantees for nonlinear feature learning in three-layer neural networks.

Wang et al. 2024 Nonlinear spiked covariance matrices and signal propagation in deep neural networks.

4 comments

Critical Strengths

We thank you for acknowledging the contribution of our work, especially in recognizing that tracking the evolution of the weight spectra is a hard problem. Your questions also helped us see where the paper could explain the connection to feature learning more clearly. We address your concerns below.

With these changes, we hope the Reviewer will find the paper clearer, and will consider revising their score.

Weaknesses

1. The paper is difficult to follow. Section 2 setup, notation, and connection to later sections are unclear, and the two-level DMFT needs a clearer introduction in the main text.

Thank you for this feedback. We agree that Section 2 can be improved, as well as the general organization of the main text. We are currently working on implementing the following changes (see updated pdf version):

  1. Clarify the general setting of Eq. (1) and its connection to Sections 3–5. We now open Section 2 by stating the general contribution from the theory: how do the singular values of a random matrix evolve as it accumulates outer-product updates whose directions are statistically coupled on its initialization. Equation (1) formulates this problem for the evolution of a matrix W(S)∈RN1×N0W(S)\in\mathbb R^{N_1\times N_0}, whose initialization W(0)W(0) is random with centered i.i.d. entries. To see how Eq. (1) naturally arises under (S)GD, consider a fully connected layer with incoming feature vector ϕ\phi and preactivation h=Wϕh=W\phi, suppressing scalar normalization factors. For a single example loss L\mathcal L, let g=∇hLg=\nabla_h\mathcal L be the gradient backpropagated to hh. Since hi=∑jWijϕjh_i=\sum_j W_{ij}\phi_j, the chain rule gives
=\frac{\partial\mathcal L}{\partial h_i}\,\phi_j =g_i\phi_j.$$ Thus each example contributes to an outer product $g\phi^\top$ for the update of $W$, and this identity only requires that the layer $W$ apply linearly to its incoming activation. The network and the target function can both be nonlinear. We now have a short list in Section 2 explaining what the sum in Eq. (1) represents in each of the settings we study, why it contains finitely many terms in Sections 3 and 4, and how this changes in Section 5. Precisely: - **Section 3**. In infinite-width nonlinear networks under SGD, each minibatch example contributes an outer product per training step. The sum runs over **(training step, minibatch example)** pairs, giving $S=BT$ at fixed batch size $B$ and training horizon $T$. - **Section 4.** We study deep linear networks as width $N$ and input dimension $D$ grow at fixed $\nu =N/D$, with $\alpha = P/D$ fixed for full-batch GD or $\alpha =B/D$ fixed for online SGD. At each step $t$, the downstream weights determine a backward direction $g_c^{\ell+1}(t)$ for each output channel $c$. Because the network is linear, this direction is shared across examples and can be factored out of the data sum. The resulting forward vector in Eq. (13) is $\phi_c^\ell(t)\propto\sum_{\mu\in\mathcal B_t}\Delta_{\mu c}(t)\,\phi_\mu^\ell(t)$, where $\mathcal B_t$ denotes the full dataset or the current minibatch. Given $\phi_c^\ell(t)$, the sum in Eq. (1) runs over **(training step, output channel)** pairs $(t,c)$, giving $S=CT$ at fixed output dimension $C$ and training horizon $T$. - **Section 5.** When $C$ itself grows proportionally with dimension, even one gradient step can produce an update of extensive rank. This motivates our analysis of how training can deform the spectral bulk. 2. **Introduce the two levels of DMFT before stating Result 1.** We now explain the two steps that connect the weight updates in Eq. (1) to the spectral prediction in Result 1. - **The first level DMFT describes the directions generated by training.** Each update in Eq. (1) is an outer product of two vectors, $\phi(t)$ and $g(t)$. At large width, the coordinates of each vector behave as independent copies of a scalar stochastic process defined by our DMFT. Lemma 1 determines the two-time correlations and responses of these processes, capturing how the update directions evolve and how they depend on the initial weights. - **The second level asks when these updates produce a spectral outlier.** At a chosen training time, after $S$ outer-product terms have accumulated, we hold $W(S)$ fixed and introduce a linear probe evolving under $M(S)=W(S)^\top W(S)/N_1$. The probe evolves in an auxiliary time $\tau$. Its response in the Laplace domain is the resolvent $(zI-M(S))^{-1}$. The normalized trace of this resolvent captures the bulk, but each isolated eigenvalue contributes only $1/N_0$ of the spectral mass and is therefore invisible in the limiting density at large $N_0$. To detect those eigenvalues, we project the probe equations onto the update directions. The correlations and responses from the first level then determine the matrix $A_S(z)$ in Result 1. Its zeros, given by $\det A_S(z)=0$ outside the bulk, predict where outliers appear.

2. The connection between spectral outliers dynamics and feature learning is unclear, no explicit generalization error analyses.

We agree that the text should explain more explicitly what the training-time BBP transitions tell us about feature learning, thank you for pointing this out.

First, we would like to clarify that the first level DMFT (Lemma 1) gives access to the test loss. For instance, for the proportional deep linear networks of Section 4, the population test loss takes the explicit form Ltest(T)=Cv(T,T)+σ2\mathcal L_{\rm test}(T)=C_v(T,T)+\sigma^2 (Eq. 207), where Cv(T,T)C_v(T,T) is the equal-time correlation of the prediction-error field vv which our DMFT computes.

Second, to clarify what the BBP transition tells us about feature learning, we added an entire new Section in Appendix E.12.4. We use the second level DMFT to track the evolution of the squared alignment cos⁡2(u1(T),w⋆)\cos^2(u_1(T),w_\star) between the leading right singular vector u1(T)u_1(T) of the first-layer weight matrix W0(T)W^0(T) and the teacher w⋆w_\star in a deep linear network (see new Appendix E.12.4). This alignment tells us whether the direction most strongly amplified by the first layer weight is predictive of the target vector.

Notice that this quantity is already accessible from our second level DMFT through the spectral probe. At each training step TT, we fix the first layer covariance of the weights M(T)=W0(T)⊤W0(T)/NM(T)=W^0(T)^\top W^0(T)/N and initialize the probe at the teacher value

\qquad \psi_0(0)=w_\star.$$ We track the normalized dot product $D^{-1}w_\star^\top\psi_0(\tau)$. Its Laplace transform is $$f_T(z)=\frac{1}{D}w_\star^\top(zI-M(T))^{-1}w_\star =\sum_j\frac{(w_\star^\top u_j(T))^2/D}{z-\lambda_j(T)}.$$ Once $\lambda_1(T)$ separates from the bulk, its residue isolates the contribution of the leading right singular vector and $$\lim_{z\to\lambda_1(T)}(z-\lambda_1(T))f_T(z) =\frac{(w_\star^\top u_1(T))^2}{D}.$$ Since $|w_\star|^2/D\to1$, this residue gives the squared teacher alignment in the proportional limit. The new Figure 16 compares both DMFT predictions with simulations. In the example shown, the outlier separates earlier under $\mu$P than under NTK-P, and its right singular vector aligns with the teacher more rapidly. The BBP transition thus identifies when the leading feature captures a nonzero fraction of the target signal in this model.

Questions

  1. Yes, the i.i.d. assumption applies to Wij(0)∼N(0,σ2)W_{ij}(0)\sim\mathcal N(0,\sigma^2). We will clarify this wording.

    The maps ϕt\phi_t and gtg_t in Eq. (2) are defined implicitly by each model’s training equations. For instance, for the nonlinear networks of Section 3, their values are computed recursively in training time. Writing the neuron index ii explicitly, Eq. (116) gives

h_{\mu,i}^{\ell}(t) &=\chi_{\mu,i}^{\ell}(t) +\eta\gamma\sum_{s<t}\mathbb E_{\nu\in\mathcal B_s} \big[\Delta_\nu(s)C_{\mu\nu}^{\phi^{\ell-1}}(t,s)g_{\nu,i}^{\ell}(s)\big],\\ z_{\mu,i}^{\ell}(t) &=\xi_{\mu,i}^{\ell}(t) +\eta\gamma\sum_{s<t}\mathbb E_{\nu\in\mathcal B_s} \big[\Delta_\nu(s)C_{\mu\nu}^{g^{\ell+1}}(t,s)\phi_{\nu,i}^{\ell}(s)\big]. \end{aligned}$$ At each step $t$, these equations give the forward feature $\phi_{\mu,i}^{\ell}(t)=\phi(h_{\mu,i}^{\ell}(t))$ and the backward gradient $g_{\mu,i}^{\ell}(t)=\dot\phi(h_{\mu,i}^{\ell}(t))z_{\mu,i}^{\ell}(t)$. Using this recursion up to step $t$ gives $\phi_t$ and $g_t$. At infinite width, correlations and response functions appearing in these equations concentrate to deterministic values. Our DMFT defines the resulting single-site stochastic processes that describe the evolution of the feature trajectory: Eqs. (3)–(4) give the law of the fields $\chi$ and $\xi$, and Eqs. (5)–(6) determine their correlation and response kernels self-consistently. Given a realization of the fields, the recursion determines the feature trajectory. This is the sense in which the maps are deterministic while the features themselves remain random. We will make this connection explicit after Eq. (2). 2. Thanks for this question. For the networks studied, this property follows asymptotically from DMFT in the large-width limit. We will clarify this distinction. **The independence is across distinct neurons in the same layer.** For example, in Section 3, the trajectories of any fixed number $N$ of distinct neurons converge as $N\to\infty$ to independent draws from the same single-neuron process. Thus, for a given input, layer $\ell$, and training step $t$, the limiting joint density of their preactivations factorizes as $$p(h_1^\ell(t),\ldots,h_N^\ell(t)) =\prod_{i=1}^{N}p(h_i(t)).$$ **Feature learning changes the law of that single-site process. In particular, the responses of one neuron to different inputs can become strongly correlated during training.** For an empirical check, Bordelon & Pehlevan (2022) compare the distribution of preactivations across neurons in a trained nonlinear network layer with the single-neuron distribution predicted by DMFT. Under strong feature learning, the empirical distribution becomes non-Gaussian, as the theory predicts. This comparison tests the single-neuron law. We also added this check in Appendix ??. 3. Thank you for catching this typo. Correlation and response matrices in Eqs. (5)–(6) are $S\times S$. We changed this in Result 1 and keep $T$ for the number of training steps. 4. $\ell(f_\mu,y_\mu)$ is a generic scalar per-example loss, differentiable in the prediction $f_\mu$. It enters the dynamics through $\Delta_\mu(t)=-\partial_{f_\mu}\ell(f_\mu(t),y_\mu)$. The theory conditions on the observed inputs, labels, and minibatch sequence, so **it requires no particular training-data distribution**. In the synthetic regression experiment, we use squared loss, $\ell(f,y)=\tfrac12(f-y)^2$, with inputs drawn from $\mathcal N(0,I_D)$ and labels generated by a single-index polynomial target, but in principle **the theory can accomodate inputs with an arbitrary covariance and labels generated by a general target function**. In Section 3, the scaling assumption is that batch size $B$, training horizon $T$, and input dimension $D$ remain fixed as the width $N\to\infty$. Examples may be reused across steps or drawn fresh. We will state these assumptions explicitly in Section 3.
  1. In Section 4, the inputs are i.i.d. xμ∼N(0,ID)x_\mu\sim\mathcal N(0,I_D), and the labels are yμc=βc⊤xμ/D+ϵμcy_{\mu c}=\beta_c^\top x_\mu/\sqrt D+\epsilon_{\mu c}, with independent Gaussian label noise of variance σ2\sigma^2. The teacher vectors βc\beta_c are fixed independently of the inputs and network initialization, with finite limiting overlaps Qcc′=lim⁡D→∞βc⊤βc′/DQ_{cc'}=\lim_{D\to\infty}\beta_c^\top\beta_{c'}/D. These overlaps give the covariance of the noiseless targets. Appendix E.13 retains cross-channel correlations, so the teachers need not be orthogonal.

    For output cc, the teacher βc\beta_c enters Eq. (13) through the residual Δμc(t)=βc⊤xμ/D+ϵμc−fc(xμ,t)\Delta_{\mu c}(t)=\beta_c^\top x_\mu/\sqrt D+\epsilon_{\mu c}-f_c(x_\mu,t). At each step, the network weights each example’s layer-ℓ\ell activation by its residual and sums over the batch, giving ϕcℓ(t)∝∑μ∈BtΔμc(t) ϕμℓ(t)\phi_c^\ell(t)\propto\sum_{\mu\in\mathcal B_t}\Delta_{\mu c}(t)\,\phi_\mu^\ell(t). This is the direction paired with gcℓ+1(t)g_c^{\ell+1}(t) in the rank-one update to WℓW^\ell. As the network learns the target, the residuals change, so different combinations of activations contribute at each step. Result 1 predicts when these updates produce an outlier.

  2. Yes. Result 1 takes N→∞N\to\infty at fixed training horizon TT, with S=BT=O(1)S=BT=\mathcal O(1) in Section 3 and S=CT=O(1)S=CT=\mathcal O(1) in Section 4. Here SS bounds the rank of the accumulated update. If S=o(N)S=o(N), the limiting bulk of a width-NN hidden layer remains unchanged. Once SS is of order NN or larger, whether the bulk changes depends on the target and the resulting feature dynamics. Strong correlations across examples and training times can leave only a few spectrally active directions even when SS is large.

    Figure 2 illustrates this possibility on CIFAR-10 for instance: after 1,000 steps with B=128B=128, only a few outliers are visible above a MP-like bulk despite S=BT≫N\mathcal S = BT\gg N. Instead in Figure 6, the ImageNet bulk itself shifts during training. We agree that this shift lies outside Result 1. Section 5 shows how bulk deformation can arise after one gradient step when the number of output channels grows proportionally with width and the update can have extensive rank. Predicting the spectral dynamics when TT grows with width remains open. Since DMFT retains the training history through two-time correlations and responses, the dimension of A(z)A(z) grows linearly with TT, so a dense evaluation of the outlier condition costs O(T3)\mathcal O(T^3) per value of zz, which is prohibitive. We specify this in the limitations.

  3. Thank you for pointing us to these papers, which are very relevant to us! We added all five citations the related-work discussion (see updated pdf). Ba et al. (2022) is especially relevant to our question. They show that even the first gradient step can produce a spike in the first-layer weights aligned with the linear component of the teacher. Wang et al. (2024) show how this learned direction appears in the conjugate kernel on test data. What we wanted to understand is how this picture evolves in deep settings as training continues. Our two-level DMFT lets us follow the weights layer by layer and predict when an outlier separates from the bulk and how it moves over subsequent steps.

Comments on the paper

1 comment

We reject the accusation of “plagiarism of the core research agenda and framing.” This is a serious allegation, yet the review identifies no copied text, derivations, results, or specific appropriated contribution. High-level overlap in research interests, the use of standard terminology (such as “spectral dynamics”), and an unintentional missing citation do not substantiate plagiarism. We were not aware of the work at the time of writing, and the omission was an oversight. We therefore ask that the allegation be withdrawn.

We would like to clarify two points:

  1. Spectral dynamics and feature learning are longstanding research topics. Martin and Mahoney (2021) studied training-induced spectral structure years earlier; Olsen et al. also cite this work when introducing the bulk-plus-outlier picture. There is an extensive empirical and theoretical literature on neural-network spectra, the emergence of informative directions during training, and feature learning. A non-exhaustive list includes Thamm et al. (2022), Damian et al. (2022), Ba et al. (2022), Bietti et al. (2022), Wang et al. (2023), Nichani et al. (2023), Dandi et al. (2024), Cui et al. (2024), Wang et al. (2024), and Dandi et al. (2025). Dyson-Brownian descriptions of weight dynamics also appear in Aarts et al. (2024), before Olsen et al.

This research agenda is shared by a broad community, and overlap in research questions does not, by itself, support an allegation of plagiarism. Here “spectral dynamics” describes the subject being studied. Neither this terminology nor the use of common network architectures establishes that a contribution has been appropriated. A scientific comparison should address the actual assumptions, methods, and results of the papers. We provide that comparison below.

  1. The dynamical assumptions, methods, and results are fundamentally different. Olsen et al.’s principal SDE analysis models minibatch fluctuation of SGD as isotropic white noise with a constant diffusion coefficient. Their stationary Gamma-type law requires an additional closure that replaces the interacting spectral dynamics with a scalar Markov diffusion process for each squared singular value. Our theory imposes neither assumption. We derive finite-time spectral evolution from the actual gradient updates of the networks analyzed, through a self-consistent description of their forward and backward features. These updates retain their feature-dependent structure with statistical dependence on initialization. In our derivation, the history of training is retained through correlation and response functions. We thereby predict BBP-type outlier escape and its dependence on parameterization and width, with implications for learning-rate transfer and the edge of stability. These are distinct learning mechanisms and results, not another derivation of their stationary spectral law.

Regarding the reviewer’s specific questions:

  1. We added a citation to their paper. This addresses the omission in our related-work discussion; it does not substantiate the allegation of plagiarism.
  2. Please see point 2 above for the technical comparison. An allegation of plagiarism requires specific evidence of appropriation; thematic overlap alone is insufficient. The comparison above demonstrates clear and substantive differences between the two works in assumptions, methods, and results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.