acceptodds
Under review as a conference paper at ICLR 2027

Mean-Field Langevin Dynamics for Two-Layer ReLU Networks

Abstract

Mean-field Langevin dynamics provide a framework for analyzing convergence of wide two-layer neural networks while capturing feature learning. But standard uniform log-Sobolev arguments do not apply to ReLU activations with weak weight decay, where the associated Gibbs potentials may be non-coercive. For square loss, under boundedness, regularity of the activation indicator, and a mild initialization condition, we prove existence of global weak solutions and establish exponential convergence of the free energy to its global minimum for every positive weight-decay and temperature parameter. The proof uses a positive coercivity threshold to separate two regimes. In the low-coercivity regime, activation regularity and two-homogeneity yield a uniform lower bound on the dissipation; in the coercive regime, radial rescaling yields a log-Sobolev inequality. Together, these estimates imply a Polyak–Lojasiewicz inequality on every free-energy sublevel. For fixed input dimension and other problem parameters, the resulting convergence rate deteriorates at most polynomially in the inverse temperature.

Primary area:
optimization
Keywords:
mean-field Langevin dynamics, ReLU neural networks, global convergence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_37JqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper analyzes the learning dynamics of two-layer ReLU networks in the mean-field limit. While prior works have established exponential convergence guarantees for bounded and smooth activation functions, the quantitative convergence guarantee for ReLU networks remains still open. The paper have developed novel techniques to handle non-smoothness and unboundedness of ReLU activation and derived exponential convergence rate to the global optimum.

Critical strengths
  1. Quantitative convergence guarantees for ReLU networks in the mean-field regime is an important open question and the paper have addressed this by demonstrating the exponential convergence guarantee under mild assumptions. The dependence of the convergence rate on the inverse temperature τ\tau is polynomial, which is milder than that established in the other setup.
  2. The paper have introduced several techniques to resolve technical difficulties posed by the non-smoothness and unboundedness of ReLU.
Critical weaknesses
  1. The obtained PL constant has exponential dependence on the dimension dd while the dependence on the inverse templerature τ\tau is polynomial. Thus, the obtained convergence guarantee is not effective for high-dimensional problems.
  2. The convergence analysis heavily depends on the 2-homogeneity of ReLU activation. Thus, it is not straightforward to extend it to other unbounded activation functions.
Questions for authors
  1. Could you elaborate how the PL constant depends on the dimension? Is it milder than that established in the previous works?
  2. Is it possible to extend the analysis to other activation functions like GeLU?
Reviewer_hwKPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyses mean-field Langevin dynamics, in the limit that corresponds to an infinitely wide two-layer neural network with ReLU activations, and the appropriate initialisation scaling. They prove that there exists a ‘global energy-dissipating weak trajectory’ weakly satisfying the probability current equation, and that for any weak solution we have exponential convergence in free energy.

Critical strengths

S1: The paper identifies a gap in the theory of mean field Langevin dynamics, proving that for the ReLU activation we have existence of weak solutions, and exponential convergence in free energy.

S2: The paper is mathematically sound and rigorous.

Critical weaknesses

W1: More motivation is needed for the setting of the problem. How does this relate to the limit of finite width networks? Do these correspond to the stated weak solutions?

Questions for authors

Q1: Solutions are defined to be those that satisfy Definition 4, please expand on this relates to the dynamics 2.8?

Q2: If we are performing (noisy) SGD in a finite width setting, does this converge to a weak solution as the width increases.

Reviewer_NqZHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether mean-field Langevin dynamics for exact ReLU networks converge globally, and whether a quantitative rate can be given. The analysis is carried out on the free energy. This is a different setting from most of the mean-field Langevin dynamics literature, where the principal quantity of interest is the distribution over parameters. Most of the theoretical effort of the past eight to ten years has gone toward establishing convergence properties of the distribution itself. That's not really an issue here because the Langevin yields an easy argument for dynamical convergence.

The setting applies only to ReLU networks. ReLU networks have been studied extensively in the mean-field context, and the subgradient properties near the cusp are known to affect convergence. The paper also includes weight decay. No finite-N or propagation-of-chaos results are given, and the results require a specific formulation of the loss, bounded inputs, and a finite fourth moment at initialization. The theory contribution is marginal, and the dimension dependence of the constants makes the overall convergence rate less impactful.

Critical strengths

The usual proof strategy does not apply here: because the ReLU feature is 2-homogeneous in its parameters, weak weight decay leaves the proximal Gibbs measure non-normalizable, so there is no distribution on which to assert a log-Sobolev inequality in the first place. The authors handle this by splitting on coercivity, recovering an LSI in the coercive case through a rescaling matched to the homogeneity, and abandoning the LSI entirely in the non-coercive case in favor of a direct lower bound on the dissipation. Gluing the two cases together is what yields a global PŁ inequality, and this adaptation is the substantive technical contribution of the paper.

Critical weaknesses

The result is a statement about the free energy rather than about the parameter distribution, proving the existence of a fixed point of the parameter distribution is typically hard without noise and trivial with it. The guaranteed rate carries an exponent of (d+4)(d+8)/6(d+4)(d+8)/6, which is e(d2/6)⋅log(1/τ)e^{(d^2/6)·log(1/\tau)}, so the advertised contrast with rates that are exponential in inverse temperature holds only along τ\tau at fixed dd, and is comparable to the eO(D)e^{O(D)} convergence time already in Mei, Montanari and Nguyen. Finally the paper gives no finite-N or propagation-of-chaos result, and Assumption 2 rules out any data distribution with atoms and hence all empirical risk, so the setting sits away from the standard machine-learning problem in two independent directions.

Questions for authors
  1. Can these results be made considerably broader than the ReLU case?
  2. Is the d2d^2 in the exponent empirically observable?
Reviewer_divjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies quantitative convergence of mean-field Langevin dynamics for jointly trained two-layer ReLU networks. To address the failure of standard convergence arguments under weak weight decay, it establishes a positive dissipation bound in the regime where the associated Gibbs distribution may not exist. Combined with a log-Sobolev argument in the complementary regime, this gives exponential convergence of the regularized objective and polynomial dependence on inverse temperature at fixed dimension.

Critical strengths
  • Quantitative convergence for exact ReLU networks with arbitrarily small positive weight decay appears to be a valuable technical contribution.
  • The proof strategy addresses a real limitation of existing analyses. The main estimates appear coherent, and the improvement in temperature dependence is interesting.
Critical weaknesses
  • Insufficient explanation of the main proof idea: The dissipation bound is central to the contribution, but its interpretation is not explained well. For instance, the paper would benefit from explaining why small dissipation would force excessive mass along a low-potential direction and how this contradicts the moment bound. The current presentation introduces the estimates before making this argument sufficiently transparent.
  • Dense and repetitive writing: Several parts of the proof contain unnecessarily verbose paragraphs and many auxiliary quantities. The tube construction and entrance-mass estimate are particularly difficult to follow. Although Section 4 provides a roadmap, the least obvious step is largely deferred to the appendix. Separating the conceptual argument from the choice of constants would considerably improve readability.
  • Limited discussion of applicability: The activation-indicator assumption excludes empirical input distributions on finite datasets. The paper should make this restriction more prominent and explain its implications for the connection to neural-network training. More generally, the result concerns continuous-time, infinite-width dynamics and convergence of a regularized population objective.
  • Interpretation of the rate: The polynomial temperature dependence is meaningful at fixed dimension, but the exponent grows quadratically with dimension and other constants also deteriorate. Appendix G discusses this dependence. The main text would benefit from a clearer explanation of what the rate establishes about learning and which aspects are likely limitations of the proof.
Questions for authors
  • Intuitively, why does weak confinement imply a positive dissipation bound? What ensures sufficient mass near the entrance of the tube?
  • Is the activation-indicator assumption essential, or could the argument be extended to empirical input distributions?
  • What conclusions about prediction error follow when temperature and weight decay decrease, beyond convergence to the corresponding regularized optimum?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.