acceptodds
Under review as a conference paper at ICLR 2027

Weight Anisotropy in Mean-Field Theory: Learning on Isotropic Data

Abstract

with low-dimensional target structure (canonical examples include -sparse parity and sparse multi-index models), yet their neural tangent kernel limits require samples in the ambient dimension . We trace this gap to a single mechanism: during training, networks develop strong anisotropy between weight coordinates aligned with the task and those that are not. We call this input feature selection (IFS) and show, through analysis of stochastic gradient Langevin dynamics, that it arises from coordinate-dependent effective regularisation that kernels structurally cannot exhibit. Mean-field (MF) theory is the natural interpretable framework for feature learning beyond the kernel regime, but standard MF tracks only first moments of the weight distribution and so cannot represent IFS. We introduce MF-ARD, which augments MF with coordinate-wise precisions through automatic relevance determination. With this single additional set of order parameters, MF-ARD (i) captures the sharp generalisation transitions of SGLD-trained networks on -sparse parity and single-index models, and (ii) provably breaks the curse of dimensionality: its phase-transition threshold depends on the intrinsic task dimension rather than the ambient dimension .

Primary area:
interpretability and explainable AI
Keywords:
mean field theory, generalization, feature learning, parity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fSnUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the generalization properties of two-layer NNs for tasks with isotropic data distribution and low-dimensional target functions. While MF is used as a natural framework for feature learning beyond kernel regime, standard MF fails to explain input feature selection (IFS). To address this issue, the paper introduces MF-ARD, an extension of the standard MF using the second-order moments of the weights. The proposed framework are shown to be able to predict sharp phase-transition of generalization of 2-layer NNs trained with stochastic gradient Langevin dyanamics.

Critical strengths
  1. The proposed framework, MF-ARD, is simple and interpretable, and it successfully predicts generalization properties of two-layer NNs while standard MF cannot.
Critical weaknesses
  1. The results in Section 3 seems obvious. Eq. (6) is immediately follows from Ito's formula. To my understanding, Section 3 is not the main contribution of this paper and just introduces the notion of gradient-weight alignment.
  2. The precision parameter ρj\rho_j and the conditional distribution of wjw_j are assumed to be Gamma and Gaussian distribution, respectively. These are reasonable choices but not justified sufficiently. Thus, it's unclear how well it approximates the true posterior.
  3. Comparison with existing results on mean-field theory of NNs is not fully explained. For example, [Suzuki et al. 2023] have also analyzed sparse parity problems using mean-field dynamics.
Questions for authors
  1. What does "implicit regularization" in Section 3 mean? There seems to be explicit l2l^2-regularization.
  2. Could you elaborate on the relationship with existing works on mean-field dynamics (e.g. [Suzuki et al. 2023])?
  3. How does the setting of α0\alpha_0 influences the results?
2 comments

We thank the reviewer for the assessment and for the clear questions.

W1

We agree that Eq. (6) follows directly from Itô's formula, which is why we call it exact, and we do not count it as a main result. Section 3 names the quantity that matters, the gradient-weight alignment, and shows on trained networks that it separates relevant from irrelevant rows before the test error drops (Fig. 2d) and that the stationary balance holds (Fig. 2c). This balance is the law MF-ARD is built to reproduce.

W2

Under linear response, ⟨ρ^j∥wj∥2⟩≈ρjr⟨∥wj∥2⟩\langle\hat\rho_j\|\mathbf w_j\|^2\rangle\approx\rho^r_j\langle\|\mathbf w_j\|^2\rangle, the loss acts on coordinate jj like a quadratic potential 12ρjr∥wj∥2\frac12\rho^r_j\|\mathbf w_j\|^2 (Sec. 4.3), which gives the Gaussian conditional p(wj∣ρj)=N(0,ρj−1)p(w_j\mid\rho_j)=\mathcal N(0,\rho_j^{-1}), and the Gamma prior is the conjugate choice that keeps the update (Eq. 13) in closed form. We do not claim that this reproduces the full SGLD posterior, and the shape of the marginals does differ between SGLD and MF-ARD. What the closure is meant to capture, the separation of the coordinate variances and the test error, does agree (Fig. 1b, Fig. 4), and it depends little on the Gamma shape over a wide range (see Q3).

W3 and Q2

Suzuki et al. (2023) prove that mean-field Langevin dynamics, the infinite-width limit of noisy gradient descent with 1/N1/N output scaling, learns sparse parities with a sample complexity in which kk is decoupled from the exponent of dd. Their "mean field" is that infinite-width limit and their result is a guarantee, while ours is a self-consistent approximation of the finite-width SGLD posterior that predicts the test error and the weight anisotropy across the whole (P,κ)(P,\kappa) plane.

Q1

The explicit weight decay λ\lambda is the same for every coordinate, and by implicit we mean the extra term that the loss adds through the gradient-weight alignment ρ^j\hat\rho_j. Eq. (6) can be written as

ddt⟨∥wj∥2⟩=−2η ⟨(λ+ρ^j) ∥wj∥2⟩+2TηN,ρ^j=⟨wj,∇wjℓ⟩∥wj∥2,\frac{d}{dt}\langle\|\mathbf w_j\|^2\rangle=-2\eta\,\big\langle(\lambda+\hat\rho_j)\,\|\mathbf w_j\|^2\big\rangle+2T\eta N,\qquad \hat\rho_j=\frac{\langle\mathbf w_j,\nabla_{\mathbf w_j}\ell\rangle}{\|\mathbf w_j\|^2},

so the net decay of row jj is λ+ρ^j\lambda+\hat\rho_j, which is reduced on the relevant rows (ρ^j<0\hat\rho_j<0) and stays at λ\lambda on the irrelevant ones (ρ^j≈0\hat\rho_j\approx0) although the explicit decay is the same for all. We will call it coordinate-dependent effective regularisation, as in the abstract, to avoid the confusion.

Q3

α0\alpha_0 sets how strongly the precisions are held at their isotropic value, since

ρj=α0+N/2α0σw2/d+(N/2) ⟨wj2⟩ → α0→∞  dσw2(plain MF),ρj → α0→0  1⟨wj2⟩(full ARD feedback).\rho_j=\frac{\alpha_0+N/2}{\alpha_0\sigma_w^2/d+(N/2)\,\langle w_j^2\rangle}\,\xrightarrow{\,\alpha_0\to\infty\,}\,\frac{d}{\sigma_w^2}\quad (\text{plain MF}),\qquad \rho_j\,\xrightarrow{\,\alpha_0\to0\,}\,\frac{1}{\langle w_j^2\rangle}\quad (\text{full ARD feedback}).

In a sweep on sparse parity the location of the transition moved little over a wide range of small α0\alpha_0, including the value used in the paper, and only large α0\alpha_0 shifted it gradually toward plain MF. We will add the sweep to the appendix and recommend α0≪N\alpha_0\ll N, the condition of Theorem 4.3.

If our answers address the reviewer's concerns, we kindly ask the reviewer to reconsider the score.

Thank you for the clarification. Regarding Q1, ρ^j\hat{\rho}_j appears to be just a loss gradient term, so I still do not understand why it is referred as implicit regularization. Overall, the paper offers an interesting perspective on generalization in neural networks. However, the analysis relies on approximations while the setting is limited to two-layer networks. There is also room for improvement in the presentation. I therefore maintain my score.

Reviewer_pTk4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies a two-layer NN model trained by Langevin Dynamics, focusing on the particular setting of k-sparse parity. The paper is based on two claims: 1) The gap between kernel sample complexity and trained NN comes from the anisotropy in the NN weights, and 2) The classical MF approximation of the weight "only tracks first moments" and is hence insufficient to correctly predict the generalization error of trained NNs. The aim of the paper is therefore to solve this by adding corrections to MF Theory.

Critical strengths

The main strength of the paper is its empirical study. The paper provides extensive numerical backup for the phenomena the study.

Critical weaknesses
  1. The paper repeatedly says that "MF theory only tracks first moments", and this is not true. This wouldn't be a problem in general, however, the authors use this as one of the main motivations to develop their theory. It would be nice to understand what do the authors mean by this. (See Questions).
  2. The writing of the paper is not clear. In particular, a lot of notation (such as P, κ\kappa) is used before introducing it, which is confusing.
  3. The claims of the paper seems a bit exaggerated given their results: at the beginning the paper argues that IMF is the reason there is a gap between kernel limits and trained NNs, however, all the theoretical set up and experiments are for the particular case of k-sparse parities. In that sense, the paper is not transparent on what it shows from the beginning and this claim about IMF in particular is never proved.
  4. There are other technical issues I would like to clarify. Please refer to the questions section for this.
Questions for authors
  1. From what I understand in the paragraph before Equation (9), m_{A}=0 is a fixed point in the equations. However, it seems that taking m_{S} = 0 in Equation (9) might not yield a fixed point. Could you clarify this? It could be that I am missing something.
  2. Theorem 4.3 presents itself as an (asymptotic) variance matching Theorem for the Langevin Dynamics and for the MF-ARD model. However, I don't clearly see why would the two precisions ρ\rho match (which assume would be necessary necessary to have the asymptotically matching moments). Could you clarify this point?
1 comment

We thank the reviewer for the careful reading and for the precise questions.

W1

The phrase is wrong as written, since MF describes the full single-neuron distribution and so all of its moments. What we mean is that its only self-consistent order parameters are the mean coefficients mAm_A, so nothing feeds the coordinate-wise second moments back into the prior, and the coordinate term d2σw2∑jwj2\frac{d}{2\sigma_w^2}\sum_j w_j^2 in Eq. (5) gives every coordinate the same precision at every fixed point, which is why plain MF cannot develop strong anisotropy (Theorem 4.1). MF-ARD replaces this term by 12∑jρjwj2\frac12\sum_j\rho_j w_j^2 with self-consistent ρj\rho_j, and we will replace the phrase with this statement throughout.

W2

We will define PP, κ\kappa, TT and every other symbol at first use, and give PP and κ\kappa in the introduction and in the caption of Table 1. We will also remove a clash, since TT is the number of time steps in Table 1 and the temperature in Eq. (1).

W3

We agree that we do not prove that IFS explains the kernel gap in general, and our claim holds for coordinate-sparse targets on isotropic data. There, SGLD networks generalise only when they develop IFS (Fig. 6), suppressing IFS with a penalty brings them back to the kernel curve (Fig. 7), and plain MF, which cannot represent it (Theorem 4.1), misses the gain. We will rewrite the abstract and introduction to claim exactly this.

Q1

The sentence before Eq. (9) is imprecise. Eq. (9) has the form mS=c (1−mS) ω0m_S=c\,(1-m_S)\,\omega_0 with c=N1−2γ/κ2c=N^{1-2\gamma}/\kappa^2, so every solution satisfies

mS=c ω01+c ω0,ω0=⟨JS(w)2σa−2+Σ(w)/(κ2N2γ)⟩p(w∣mS)>0,m_S=\frac{c\,\omega_0}{1+c\,\omega_0},\qquad \omega_0=\Big\langle\frac{J_S(\mathbf w)^2}{\sigma_a^{-2}+\Sigma(\mathbf w)/(\kappa^2N^{2\gamma})}\Big\rangle_{p(\mathbf w\mid m_S)}>0,

so the target acts as an external field and the lower branch sits at a small mSm_S, suppressed by a power of dd because JS(w)2J_S(\mathbf w)^2 is small under the isotropic prior, rather than at zero. For A≠SA\neq S, mA=0m_A=0 is exact in the population limit, because p(w∣mS)p(\mathbf w\mid m_S) is invariant under wj→−wjw_j\to-w_j while JSJAJ_SJ_A changes sign for jj in exactly one of SS and AA. Only in the saddle-point evaluation of App. D.5 is mS=0m_S=0 an exact fixed point above κc\kappa_c, because there the optimal weight scale is zero, and we will rewrite the main text in these terms.

Q2

You are right that the theorem does not show that the two precisions take the same value, and its wording should say so. If we define ρjr:=⟨ρ^j∥wj∥2⟩∞/⟨∥wj∥2⟩∞\rho^r_j:=\langle\hat\rho_j\|\mathbf w_j\|^2\rangle_\infty/\langle\|\mathbf w_j\|^2\rangle_\infty, the stationary balance in Eq. (7) gives, with no linear-response assumption,

SGLD:ρeff,j:=λ+ρjrT=N⟨∥wj∥2⟩∞,MF-ARD:ρj=1+O(α0/N)⟨wj2⟩pARD,\text{SGLD:}\quad \rho_{\mathrm{eff},j}:=\frac{\lambda+\rho^r_j}{T}=\frac{N}{\langle\|\mathbf w_j\|^2\rangle_\infty},\qquad\text{MF-ARD:}\quad \rho_j=\frac{1+O(\alpha_0/N)}{\langle w_j^2\rangle_{p_{\mathrm{ARD}}}},

so both obey the same law, namely that the precision is the inverse of the second moment per neuron. The two precisions therefore coincide exactly when ⟨∥wj∥2⟩∞=N⟨wj2⟩pARD\langle\|\mathbf w_j\|^2\rangle_\infty=N\langle w_j^2\rangle_{p_{\mathrm{ARD}}}, that is when MF-ARD reproduces the SGLD second moments, and this is the closure assumption, which we test numerically (Fig. 1b, Fig. 2c). We will restate the theorem in this form, with ρj≡ρeff,j\rho_j\equiv\rho_{\mathrm{eff},j} named as the assumption.

If our answers address the reviewer's concerns, we kindly ask the reviewer to reconsider the score.

Reviewer_ffpvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how two-layer networks learn to select relevant input coordinates when the input distribution is isotropic and the target depends on only a few coordinates. The authors connect the resulting weight anisotropy to coordinate-dependent effective regularization in SGLD and introduce MF-ARD, a mean-field model with self-consistent coordinate-wise precisions. They analyze the critical noise threshold for learning and compare the predictions with simulations on sparse-parity and sparse single-index tasks.

Critical strengths

I find the paper generally well written, and the main equations are easy to follow. I particularly appreciate that the proposed extension leads to relatively simple self-consistency equations, with a clear interpretation of the additional parameters. I also appreciate the related-work section and Table 1. The table provides a useful overview of existing theoretical approaches, the regimes they consider and the order parameters they track. This helps place the proposed framework in the context of previous works.

Critical weaknesses
  1. I find the set of tasks considered rather restrictive. Depending on a low-dimensional subspace does not necessarily mean depending on only a few input coordinates. For example, a target could depend only on x1+⋯+xdx_1+\cdots+x_d: the task is still one-dimensional, but every coordinate contributes equally. A network could learn this direction while the weight variances remain identical across coordinates, since what matters is how the weights on different coordinates relate to each other. I appreciate that the authors acknowledge this limitation, but their additional single-index experiment also uses a sparse target vector. Could they consider a dense or rotated target direction to clarify how their mechanism extends from selecting input coordinates to learning combinations of them?
  2. I would like to better understand the justification for modifying the prior. The authors introduce coordinate-dependent precisions that allow the model to favor some input coordinates over others. This does not tell the model which coordinates are relevant, but it introduces a structure particularly suited to the tasks considered. My concern is that input selection is partly built into the modified model, rather than fully explained as a consequence of the original training dynamics (for e.g. by studying how scaling up data naturally arises in input selection and sparsity of the weights).
  3. The role of width and noise scaling is unclear to me. The authors use κ\kappa without additional width-dependent rescaling. How does this choice affect the comparison between plain MF, MF-ARD, and SGLD?

More generally, I find the scope of the conclusions somewhat broader than the setting studied. The analysis concerns shallow networks, sparse-coordinate tasks, and the stationary distribution of SGLD. While this is a useful setting, I would like to better understand which conclusions are expected to hold during finite-time training, with other optimizers, or in deeper networks. A clearer discussion of these points would help me assess the broader relevance of the proposed mechanism.

Questions for authors

See above.

2 comments

We thank the reviewer for the careful reading. We address all points below.

W1

Your example is the right test. With Gaussian inputs SGLD is rotation-equivariant, so a dense direction has the same learning curve as a sparse one, and the anisotropy then appears as one large eigenvalue of the second-moment matrix ⟨ww⊤⟩\langle \mathbf w\mathbf w^\top\rangle instead of on its diagonal, which diagonal MF-ARD cannot see. The fix replaces the dd precisions by a precision matrix Λ\Lambda with the matrix version of our Gamma prior (Wishart, with mean (d/σw2) I(d/\sigma_w^2)\,I), which changes the prior term in Eq. (12) and the update in Eq. (13) to

12∑jρjwj2 ⟶ 12 w⊤Λ w,Λ=(2α0+N)(2α0σw2d I+N ⟨ww⊤⟩pARD)−1.\frac{1}{2}\sum_{j}\rho_j w_j^2\,\longrightarrow\,\frac{1}{2}\,\mathbf w^\top\Lambda\,\mathbf w,\qquad \Lambda=\big(2\alpha_0+N\big)\Big(\frac{2\alpha_0\sigma_w^2}{d}\,I+N\,\langle\mathbf w\mathbf w^\top\rangle_{p_{\mathrm{ARD}}}\Big)^{-1}.

This update is rotation-equivariant and reduces exactly to Eq. (13) when ⟨ww⊤⟩\langle\mathbf w\mathbf w^\top\rangle is diagonal. On a single-index task with a dense random teacher it recovers the teacher direction as the leading eigenvector of ⟨ww⊤⟩\langle\mathbf w\mathbf w^\top\rangle from an isotropic start and reaches low test error while the diagonal version does not, and we will add the extension and this experiment to the appendix.

W2

The ARD prior is built to reproduce the stationary balance of SGLD coordinate by coordinate (Sec. 3, Eq. 7), and it carries no information about which coordinates matter. All precisions start equal and the update in Eq. (13) depends on jj only through ⟨wj2⟩\langle w_j^2\rangle, so the anisotropy can only come from the data, and it appears only once PP is large enough, close to where it appears in SGLD (Fig. 4), while below the transition both stay isotropic (Fig. 1b). What the diagonal form does build in is the coordinate basis (see W1), and Theorem 4.4 assumes a small initial gap, so that theorem describes how the selection is amplified and not how it starts, which we will state plainly.

W3

All three are compared at the same width, the same γ=1/2\gamma=1/2 and the same raw κ\kappa, and κ\kappa enters the SGLD posterior, plain MF and MF-ARD only through the couplings 1/(κ2N2γ)1/(\kappa^2N^{2\gamma}) and 1/(κ2Nγ)1/(\kappa^2N^{\gamma}) (Eqs. 3, 5, 12), so at a given width a width-dependent rescaling of κ\kappa would only relabel the κ\kappa axis for all three alike. Width does move the transition, since the onset formula (App. D.6) gives

κc2 ≃ ∣Dk∣ σaNγC,CMF=dkσw2,CARD=∑j∈Sρj=Θ(k),\kappa_c^2\,\simeq\,\frac{|D_k|\,\sigma_a}{N^{\gamma}\sqrt{C}},\qquad C_{\mathrm{MF}}=\frac{dk}{\sigma_w^2},\qquad C_{\mathrm{ARD}}=\sum_{j\in S}\rho_j=\Theta(k),

with DkD_k depending only on kk. The factor N−γN^{-\gamma} is the same for plain MF and MF-ARD, so a temperature scaled as κ2∝N−γ\kappa^2\propto N^{-\gamma} would hold the transition in place as the network grows, while the dd versus kk difference of Theorem 4.4 sits in CC and does not depend on this choice. We will state the convention and its effect in Sec. 2.2.

W4

We agree and will narrow the abstract and introduction to two-layer networks, coordinate-sparse targets and the stationary state of SGLD, and say that deeper networks are open. The mechanism is less tied to this setting than the theory, because the row-energy equation (Eq. 6) holds at every time and the relevant rows separate during training, before the test error drops (Fig. 2). For minibatch SGD with weight decay and no injected noise, with gjt\mathbf g_j^t the minibatch gradient of row jj, the row energy obeys exactly

∥wjt+1∥2=(1−ηλ)2 ∥wjt∥2−2η(1−ηλ) ⟨wjt,gjt⟩+η2 ∥gjt∥2.\|\mathbf w_j^{t+1}\|^2=(1-\eta\lambda)^2\,\|\mathbf w_j^{t}\|^2-2\eta(1-\eta\lambda)\,\langle\mathbf w_j^{t},\mathbf g_j^{t}\rangle+\eta^2\,\|\mathbf g_j^{t}\|^2 .

Averaged over the minibatch, the first-order change is −2η(λ+ρ^j)∥wj∥2-2\eta(\lambda+\hat\rho_j)\|\mathbf w_j\|^2, the same anti-decay term as in Eq. (6), and the minibatch noise in η2∥gj∥2\eta^2\|\mathbf g_j\|^2 takes the place of the injected noise 2TηN2T\eta N. In a direct comparison SGD matched SGLD in test error and in the anisotropy through the transition, so we expect the mechanism, though not the exact phase boundary, to carry over.

If our answers address the reviewer's concerns, we kindly ask the reviewer to reconsider the score.

I thank the authors for their response and appreciate the proposed extension to dense target directions. However, my main concern remains. My concern is that the original SGLD dynamics already determine a stationary posterior, including its prior. Replacing this prior with an adaptive ARD prior generally defines a different posterior. I therefore do not see what justifies interpreting this modification as a correction to the mean-field approximation of the original model.

I would have found the mechanism more convincing if it emerged from a derivation starting with the original model, for instance through finite-width corrections to mean-field theory. As presented, MF-ARD seems better justified as a phenomenological model supported by numerical agreement, and I think the theoretical claims should reflect this distinction. Since the response does not sufficiently resolve this concern, I am maintaining my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.