- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the generalization properties of two-layer NNs for tasks with isotropic data distribution and low-dimensional target functions. While MF is used as a natural framework for feature learning beyond kernel regime, standard MF fails to explain input feature selection (IFS). To address this issue, the paper introduces MF-ARD, an extension of the standard MF using the second-order moments of the weights. The proposed framework are shown to be able to predict sharp phase-transition of generalization of 2-layer NNs trained with stochastic gradient Langevin dyanamics.
Critical strengths
- The proposed framework, MF-ARD, is simple and interpretable, and it successfully predicts generalization properties of two-layer NNs while standard MF cannot.
Critical weaknesses
- The results in Section 3 seems obvious. Eq. (6) is immediately follows from Ito's formula. To my understanding, Section 3 is not the main contribution of this paper and just introduces the notion of gradient-weight alignment.
- The precision parameter and the conditional distribution of are assumed to be Gamma and Gaussian distribution, respectively. These are reasonable choices but not justified sufficiently. Thus, it's unclear how well it approximates the true posterior.
- Comparison with existing results on mean-field theory of NNs is not fully explained. For example, [Suzuki et al. 2023] have also analyzed sparse parity problems using mean-field dynamics.
Questions for authors
- What does "implicit regularization" in Section 3 mean? There seems to be explicit -regularization.
- Could you elaborate on the relationship with existing works on mean-field dynamics (e.g. [Suzuki et al. 2023])?
- How does the setting of influences the results?
2 comments
Thank you for the clarification. Regarding Q1, appears to be just a loss gradient term, so I still do not understand why it is referred as implicit regularization. Overall, the paper offers an interesting perspective on generalization in neural networks. However, the analysis relies on approximations while the setting is limited to two-layer networks. There is also room for improvement in the presentation. I therefore maintain my score.
We thank the reviewer for the assessment and for the clear questions.
W1
We agree that Eq. (6) follows directly from Itô's formula, which is why we call it exact, and we do not count it as a main result. Section 3 names the quantity that matters, the gradient-weight alignment, and shows on trained networks that it separates relevant from irrelevant rows before the test error drops (Fig. 2d) and that the stationary balance holds (Fig. 2c). This balance is the law MF-ARD is built to reproduce.
W2
Under linear response, ⟨ρ^j∥wj∥2⟩≈ρjr⟨∥wj∥2⟩, the loss acts on coordinate j like a quadratic potential 21ρjr∥wj∥2 (Sec. 4.3), which gives the Gaussian conditional p(wj∣ρj)=N(0,ρj−1), and the Gamma prior is the conjugate choice that keeps the update (Eq. 13) in closed form. We do not claim that this reproduces the full SGLD posterior, and the shape of the marginals does differ between SGLD and MF-ARD. What the closure is meant to capture, the separation of the coordinate variances and the test error, does agree (Fig. 1b, Fig. 4), and it depends little on the Gamma shape over a wide range (see Q3).
W3 and Q2
Suzuki et al. (2023) prove that mean-field Langevin dynamics, the infinite-width limit of noisy gradient descent with 1/N output scaling, learns sparse parities with a sample complexity in which k is decoupled from the exponent of d. Their "mean field" is that infinite-width limit and their result is a guarantee, while ours is a self-consistent approximation of the finite-width SGLD posterior that predicts the test error and the weight anisotropy across the whole (P,κ) plane.
Q1
The explicit weight decay λ is the same for every coordinate, and by implicit we mean the extra term that the loss adds through the gradient-weight alignment ρ^j. Eq. (6) can be written as
dtd⟨∥wj∥2⟩=−2η⟨(λ+ρ^j)∥wj∥2⟩+2TηN,ρ^j=∥wj∥2⟨wj,∇wjℓ⟩,
so the net decay of row j is λ+ρ^j, which is reduced on the relevant rows (ρ^j<0) and stays at λ on the irrelevant ones (ρ^j≈0) although the explicit decay is the same for all. We will call it coordinate-dependent effective regularisation, as in the abstract, to avoid the confusion.
Q3
α0 sets how strongly the precisions are held at their isotropic value, since
ρj=α0σw2/d+(N/2)⟨wj2⟩α0+N/2α0→∞σw2d(plain MF),ρjα0→0⟨wj2⟩1(full ARD feedback).
In a sweep on sparse parity the location of the transition moved little over a wide range of small α0, including the value used in the paper, and only large α0 shifted it gradually toward plain MF. We will add the sweep to the appendix and recommend α0≪N, the condition of Theorem 4.3.
If our answers address the reviewer's concerns, we kindly ask the reviewer to reconsider the score.