Neural networks have parameter symmetries: parameter transformations that do not affect the represented function. How do these symmetries affect the functions that neural networks ultimately learn? We study the implicit bias arising from these symmetries by calculating the equilibrium distribution over represented functions under Langevin dynamics with Gaussian regularization. To isolate the effect of symmetries on this distribution, we study the limit in which the strength of the regularization is taken to zero. In the low temperature limit, we recover known effective regularizations. At positive temperature, a renormalization procedure allows us to derive a novel entropy which predicts symmetry-induced preference for certain functions. For linear networks, we obtain explicit entropy terms that favor singular and low-rank functions. For networks with homogeneous activations, we find entropy terms that favor inactive neurons, which can lead to neural "heat death" in which equilibrium mass concentrates on networks with vanishing output. We interpret these results through the neural tangent kernel, as a simplicity bias selecting for functions that vary slowly under changes in the parameters.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.Reviewer_mgM8OpenReview ↗
Rating
4
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary
It deal with a fundamental question in deep learning theory: how do non-compact parameter symmetries (like layer-wise rescaling in homogeneous/ReLU networks or invertible basis changes in linear networks) shape the implicit bias of noisy training dynamics in function space? Interesting results regards multiple topics:
Two Limits of the Fiber Partition Function (Section 2)
Explicit Renormalized Entropies Across Architecture (diagonal, 2layers nets, homogeneous nets)
NTK volune
Time-dependent unregularized bridge
It is original theory as far as I can say, formally deep and rigorous
Critical strengths
It is original theory as far as I can say, formally deep and rigorous
The mathematics seems very nicely done connecting in a unifying framework a number of previous results
Critical weaknesses
The main weakness is that the theory does not directly apply to SGD and similar optimization techniques since it is focused on Langevin equations, but I think the insights are likely to extend to SGD.
Questions for authors
A small typo here
Surprisingly, we that these limits do agree empirically.
Thank you for the positive review, and pointing out the typo. We have gone through the paper to correct the spelling errors we could find. We have also added clarifications throughout. We hope this improves the paper’s readability.
The reviewer is correct that our theory does not apply to SGD, but rather to the equilibrium of Langevin dynamics. Such equilibrium analyses have indeed proved useful across the literature (lines 74-97). In our case, we are able to connect the weak-regularization equilibrium to unregularized Langevin dynamics quantitatively for diagonal networks, with Theorem 1 in our updated submission.
This paper studies how parameter symmetries affect the distribution over functions induced by Langevin dynamics. It introduces quadratic parameter regularization to obtain a normalizable equilibrium, then analyzes the weak-regularization limit. The low-temperature regime recovers effective regularization costs, while the fixed-positive-temperature regime yields renormalized entropy corrections. The paper derives these corrections for diagonal, two-layer square linear, and homogeneous networks. It also proves concentration on the globally zero function for qualifying deep ReLU networks under radially coercive losses, and relates local function-space entropy to the neural tangent kernel. Numerical Langevin simulations illustrate the predictions and explore a heuristic connection to unregularized dynamics.
Critical strengths
The paper provides explicit calculations of the contribution of non-compact parameter fibers to function-space distributions.
The distinction between the low-temperature and fixed-positive-temperature limits is useful. The framework recovers familiar representation costs in the former regime and produces additional entropy factors in the latter. The contrast between diagonal networks, whose leading renormalized correction is constant, and two-layer square linear networks, whose correction depends on the singular values, gives concrete information about how different parameterizations affect the induced distribution. These calculations constitute a substantive theoretical contribution.
The heat-death result identifies a precise mechanism through which equilibrium selection can favor a higher-loss function.
For the specified homogeneous ReLU networks with at least two hidden layers and the stated loss assumptions, the paper shows that the equilibrium probability of the globally zero function tends to one as regularization vanishes at fixed positive temperature. This establishes that the volume associated with zero-function representations can overcome a finite loss disadvantage. Figures 3-5 provide numerical illustrations of the predicted distributions and zero-output behavior in the models studied.
Critical weaknesses
The broader significance of the asymptotic results and their connection to finite-time learning need a clearer explanation.
The introduction asks how symmetries affect the functions ultimately learned by neural networks. The strongest rigorous conclusions concern weak-regularization limits of equilibria in particular model families. The connection to long-time unregularized Langevin dynamics is explicitly heuristic, and the simulations do not establish the time scale or regularization range over which the limiting predictions become relevant. In particular, the heat-death theorem should not be interpreted as showing that prolonged ordinary SGD or Adam training generally produces zero-output networks.
The structural and loss assumptions also matter: the demonstrated ReLU mechanism uses a bias-free layered homogeneous architecture and a loss that grows along every nonzero represented function ray. Its applicability to networks with residual connections or normalization, and to standard classification objectives, is not established here. Please identify which assumptions are essential, state the scope alongside the main claims, and explain a concrete new prediction or conceptual consequence beyond the general observation that parameterization affects implicit bias. A finite-regularization study of the same ReLU model, reporting zero-function probability over time for several regularization strengths, would help establish where the asymptotic result is informative. Large-scale application benchmarks are not necessary to address this concern.
The sparsity interpretations require more precise assumptions and a clearer distinction between local density and global probability.
In Appendix C.4, the feature-sparsity argument integrates a node scale using ds_v/s_v. However, the measure derived earlier in the same appendix has s_v^(n-2) ds_v for nodes in the last hidden layer. Moreover, the readout weights disappear from the leading renormalized factor displayed there. The argument therefore needs to specify which hidden layers it applies to; its current wording appears broader than the derivation supports. Please clarify the scope and handle the last hidden layer separately.
In Section 6 and Appendix C.6, the determinant factorization assumes a full-rank feature Gram matrix. This is a substantive condition on the input distribution Q and the feature configuration, rather than merely a notational convenience. Please state sufficient assumptions or explain the rank-deficient case. The small-amplitude term in the resulting function-space density must also be interpreted relative to the chosen L2(Q) volume measure. A divergent local density does not by itself establish that a normalized limiting distribution concentrates on sparse functions; the volume element, global integrability, and singular regions also matter. The discussion should distinguish this local density statement from the separate global zero-function concentration result in Section 5.
Questions for authors
What concrete new prediction best demonstrates the significance of renormalized entropy?
Please give one explicit example of a quantitative prediction or distinction obtained here that is not supplied by representation-cost analyses, entropy calculations on balanced manifolds, or existing finite-width prior analyses. The related-work discussion explains several differences in setup; making the consequence of those differences explicit would strengthen the motivation.
When does the heat-death limit become informative at finite regularization and finite time?
For the ReLU model in Figure 5, can you report zero-function probability over time for several positive regularization strengths, using the existing noise setting? Please identify the observed regime of agreement with the weak-regularization prediction and clearly separate this evidence from the unproved interchange of limits discussed in Sections 3 and 7.
Which nodes are covered by the feature-sparsity argument in Appendix C.4?
Please reconcile the ds_v/s_v measure used in that argument with the different measure for last-hidden-layer nodes. Does the stated divergence apply only to non-final hidden layers? Please revise the statement and derivation accordingly.
What assumptions justify the function-space sparsity interpretation in Section 6?
Please specify conditions on Q and the feature configuration that ensure the Gram matrix in Appendix C.6 is full rank. Is the small-amplitude result intended only as a local density statement, or is there also a globally normalizable limiting distribution with a corresponding probability claim? Please state the supported conclusion explicitly.
Thank you for the positive assessment of our concrete calculations of the renormalized entropy and heat death, and for the attention to the details of our assumptions. We welcome the opportunity to strengthen our paper and address your questions.
Weakness 1:
We agree the difference from SGD/Adam and the conditions for heat death could be made more precise. To this end, we have updated the introduction (lines 74-79), and added a precise theorem statement for heat death, Theorem 2 (with a discussion of when its assumptions hold). We have also made the connection of the λ→0 limit to long-time Langevin more precise:
(a) We update to the heuristic and give a formal proof for diagonal networks (Theorem 1); we conjecture it extends to the linear case, leaving the question open to future work.
(b) We update the ReLU experiment to include the requested comparison of zero-function-probability across different regularizations, showing a smooth increase as regularization vanishes, with long-time limits agreeing with theory (see Figure 5). (We thank you kindly for the suggestion.)
Weakness 2:
We appreciate the push to clarify the role of different strata, as this is a subtle point in the theory for which we try to balance clarity and rigor. We have updated Sections 2 and 6 (as well as the discussion after Theorem 2) to make this clearer (lines 188-191; 451-454; 480-482; 496-500). In particular, different strata are compared by their growth rates Cα, while the renormalized entropy on a particular regular stratum only reveals the preference within that stratum.
The reviewer is correct that the ReLU examples in Appendix C.4 and Section 6/Appendix C.6 require more care. We have updated Appendix C.4 to correctly state the sparsity bias on the regular stratum is only for nodes in a hidden layer which is not the last. (The final readout weights do not appear in the entropy, and so do not have this bias). For the ReLU example ins Section 6, we add conditions guaranteeing positive definiteness of the NTK. (The general discussion in Section 6 still allows for the positive semi-definite case.)
Question 1:
To our mind the two most important examples of concrete predictions (beyond existing representation-cost and balanced manifold analyses) would be precisely the strengths that you mentioned above. First, renormalization allows us to go beyond balance and derive a concrete correction for unbalanced linear networks in terms of singular values (and contrast this to the diagonal case). Second, heat death gives a “precise mechanism” describing how equilibrium selects a higher-loss function.
Question 2:
We have updated Figure 5 to include non-zero regularizations, allowing one to visualize the crossover into the heat-death regime. The interchange of limits in Section 3 is now proved for diagonal networks (Theorem 1).
Question 3:
The feature-sparsity argument applies to nodes v which are in a hidden layer, but not the last hidden layer. We have updated this in Appendix C.4.
Question 4:
We have given precise conditions for a full-rank Gram matrix in the example in Section 6/Appendix C.6. The small-amplitude result is a local density statement, and we have updated Sections 2 and 6 to make this clearer.
The paper introduces a posterior over functions in neural networks. The posterior is motivated as essentially a Gibbs stationary distribution, but for the case where the normalisation integral is not finite. More precisely, their posterior is the limit limλ→0 of the Gibbs posterior with L2 regularisation term λ∣θ∣22/2 at some temperature T>0. They show that taking the T→0 limit first and the λ→0 limit second recovers a distribution that has been previously studied (it just selects loss minimisers with the smallest effective regularisation). Taking the λ→0 limit at fixed T>0 gives a new distribution. At finite T, the distribution can be written as ∝e−ℓ(f)/T+logΩ(f) where logΩ(f) is a correction term they choose to call the renormalised entropy.
They motivate this distribution as possibly being the large time limit of unregularised Langevin dynamics under isotropic noise. They calculate the renormalised entropy for various toy models and show that their distribution often seems to exhibit what might be termed a simplicity bias. They give particular prominence to the result that it strongly favors the zero function in bias-free ReLU MLPs with at least two hidden layers under certain conditions (loss bounded below and radially coercive), a phenomenon they term ‘heat death’.
Critical strengths
As far as I can tell, the formal derivations seem correct and novel.
Critical weaknesses
Weakness 1)
I think this could use a lot more words in the introduction laying out what setting their results actually apply to. I feel it’s kind of insufficiently clear early on how far removed any of this analysis is from applicability to real neural network training.
Any equilibrium analysis like this can’t say much about results far below the mixing time. The dynamics before equilibrium is reached are where a lot of the key and interesting questions about modern learning algorithms actually live (See sparse parity learning as a central example. All the results about what is difficult to learn in this setting concern the training time it takes to learn them, which ends up being exponential in the complexity of the solution. Assuming equilibrium throws these dynamics away entirely.). Giving any useful guarantee on convergence time to reach equilibrium is often intractable for real settings.
One could say we must start somewhere. We might ultimately want and need a dynamic theory out of equilibrium to describe neural network training in practice, but to get there maybe we ought to understand the simpler equilibrium cases first.
I do agree with that sentiment, but I still feel like it should be laid out much more and much earlier in the paper that this isn’t about neural network training as currently practiced. Somewhat relatedly, it says in the discussion section
“In contrast, we study the entire non-compact fiber (though we do recover the Menon & Yu (2025) entropy as a subleading term in the T →0 expansion of Zα; Appendix C.7). Our results complement empirical studies of simplicity bias which suggest parameter volume as a mechanism (Valle-Perez et al., 2019), and our explicit linear network entropy may help explain observed low-rank bias (Huh et al., 2023).“
Which I feel overstates the relevance of the results to this. I do not see how the demonstrations of simplicity bias being linked to parameter volume in this paper’s setting are any more relevant to explaining the observation that simplicity bias seems linked to parameter volume and low-rank bias in real training than other such theoretical demonstrations that have come before, such as for example Singular Learning Theory, which also predicts both of these things in its setting.
Weakness 2)
The paper justifies the relevance of the posterior they study in part through a conjectured connection between that posterior and Langevin dynamics. Specifically, they argue that it might correspond to the large time limit of the unregularised isotropic Langevin posterior. But I’m not very convinced by those arguments. They don’t have a derivation. Just a heuristic case, and some toy experiments.
The toy experiments only look at a few tiny models, and even there, looking at the plots I often find myself only dubiously convinced that the distribution actually fits the data well. They often consider only relatively small times that might be far away from convergence for all I know. The fit of the theory to the data seems basically eyeballed from histograms without much quantification, and often doesn’t look fantastic to me.
Figure 3, t=0.5, looks like an underfit?
Figure 4: For s2, the naive e−ℓ/T law makes almost the same predictions as λ→0. So this doesn’t tell us that much. And for s1, are we sure that the remainder of the spike that distinguishes the prediction of their distribution from the simple baseline wouldn’t go away at t=20 or t=100?
Figure 5 shows neurons dying more as time goes on and the fraction of networks outputting zero approaching one, but doesn’t make an explicit quantitative comparison for finite times.
Lastly, Astra appears to have found at least one counterexample where the posterior and the unregularised isotropic noise Langevin dynamics at large times clearly don’t agree, which I will show below. You can argue that the setting of that counterexample isn’t very realistic, but the mere fact that this exists makes me nervous that there might be counterexamples closer to real settings of interest as well.
Astra:
The weak-regularization Gibbs limit can differ from the large-time function law of unregularized isotropic Langevin dynamics, even when both limits exist and initialization is standard Gaussian. In the construction below, at the fixed input (1,0), the probability that the random predictor outputs a positive value is exactly 1/2 under the limiting Gibbs law and at least 39/64 under the large-time unregularized Langevin law. The construction uses a smooth nonlinear parameterization of a linear predictor and a bounded-below loss that is radially coercive in coefficient space, with ℓ(0)=+∞. It addresses the proposed connection at the level of the general parameter-to-function framework.
Construction. Let θ∈R3 be the trainable parameter, with θ0∼N(0,I3) independently of the Brownian motion. Set
e1=(1,0,0),y=θ+8e1=(a,b,c),q=∣y∣2.
Define χ(q)=exp[−1/(1−q)] for q<1 and χ(q)=0 for q≥1, and put D(q)=q+χ(q)2. For x∈R2, define
w(θ)=D(q)(a,χ(q)),fθ(x)=w(θ)⊤x.
The denominator never vanishes, so the parameter-to-function map is smooth everywhere. Take temperature T=1, regularizer R(θ)=∣θ∣2/2, and loss
ℓ(w)=∣w∣2−2log∣w∣2(w=0),ℓ(0)=+∞.
Write L(y)=ℓ((a,χ(q))/D(q)). The dynamics are
dθtλ=−∇L(θtλ+8e1),dt−λθtλ,dt+2,dBt.
The Gaussian initialization, isotropic noise, and squared-norm regularizer all refer to the same coordinates θ. In y coordinates, the finite-loss domain is
D=R3∖a=0,∣y∣≥1.
It is connected: the two signs of a communicate through the unit ball. The infinite-loss wall is inaccessible to the diffusion, as verified below.
Unregularized escape from Gaussian initialization. For r=∣y∣≥1 and s=a/r, we have
w=(s,0),L(y)=s2−4log∣s∣.
Thus the loss is independent of radius outside the unit ball. For λ=0, until the process returns to that ball,
drt=rt2,dt+2,dβt.
This is a three-dimensional Bessel process with a constant time rescaling. Its probability of ever hitting radius 1 from radius r>1 is 1/r. On the event ∣θ0∣≤4, the initial translated point has a0≥4 and r0≥4. Conditional on any such initial point, the probability of never returning to the unit ball is at least 3/4, and on that event the sign stays positive. Since E∣θ0∣2=3, Markov's inequality gives
Pr(∣θ0∣≤4)≥1613.
Consequently, the probability p of eventual escape with positive sign satisfies
p≥431613=6439>21.
Existence of the full unregularized function-law limit. The process exits every bounded ball almost surely, as shown below. Start the first escape attempt at time zero if r0>1; otherwise wait for radius 2. After each return to radius 1, wait until radius 2 to start another attempt. Each attempt from radius 2 has conditional probability 1/2 of never returning to radius 1. The number of failures therefore has a geometric tail. Almost surely, the process eventually escapes to infinity with a fixed sign.
Outside the unit ball, the generator separates into
L=∂rr+r2∂r+r21A,
where the angular generator, acting on the first spherical coordinate, is
Ah(s)=(1−s2)h′′(s)+(s4−8s+2s3)h′(s).
The intrinsic angular diffusion can be constructed independently of the radial process. On a successful attempt starting at time σ, its clock is At=∫σtru−2,du, which diverges almost surely. Indeed,
dlogrt=rt−2,dt+2,rt−1,dβt;
a finite clock would force logrt to converge to a finite value, contradicting radial escape.
On 0<s<1, the angular diffusion has integrable speed density s4e−s2 and scale density [(1−s2)s4e−s2]−1, whose integrals diverge at both endpoints. It is positive recurrent and its law converges to the normalized speed measure. Let μ+ be the law of (s,0) when s has density
C−1s4e−s2,0<s<1,C=∫01s4e−s2,ds,
and let μ− be its reflection in w1=0.
For each attempt, conditional on its starting history, success depends only on the future radial process. Conditioning on success therefore leaves the independent intrinsic angular diffusion unchanged. Angular convergence applies on each successful attempt. Summing over the attempts is justified by the geometric tail bound. This proves
Law(w(θt0))⟹pμ++(1−p)μ−,p≥6439.
The argument can first be conditioned on the initial point and then integrated over the Gaussian initialization by bounded convergence. Thus this is convergence of the full function law, not merely a lower bound on an escape event.
The weak-regularization Gibbs limit. For every λ>0, the Gibbs density in y coordinates is
gλ(y)∝exp[−L(y)−2λ∣y−8e1∣2].
It is integrable and reversible. The diffusion is nonexplosive, locally uniformly elliptic, and irreducible on the connected domain D. Strong Feller regularity and Doob's theorem give uniqueness of this equilibrium and convergence to it from the Gaussian initialization.
To compute its limit, rescale y=z/λ. For each z=0, outside the measure-zero equatorial plane, the unnormalized rescaled density tends to
\exp\left[-\left(\frac{z_1}{\\|z\\|}\right)^2-\frac{\\|z\\|^2}{2}\right].$$
Dominated convergence applies: $e^{-L}$ is bounded, and $e^{-\\|z-8\sqrt\lambda e_1\\|^2/2}$ is bounded by a constant times $e^{-\\|z\\|^2/4}$ for $0<\lambda\le1$. The limiting spherical measure has first-coordinate density proportional to $|s|^4e^{-s^2}$ on $(-1,1)$. Hence
$$\operatorname{Law}_{g_\lambda}(w)\Longrightarrow\frac12\mu_++\frac12\mu_-.$$
The two component laws assign zero mass to $w_1=0$, so the limiting positive-sign probabilities differ by at least $7/64$.
Well-posedness and absence of bounded-region trapping. These points matter because the loss is singular. Set
$$F(y)=a^2+\chi(q)^2,\qquad z=F/D,\qquad H=D^{-2}e^{-F/D}.$$
Then $e^{-L}=F^2H$, with $H$ smooth and strictly positive, and the unregularized drift is $2\nabla F/F+\nabla\log H$. At every point of the excluded wall, $F=0$, $\nabla F=0$, and $\Delta F=2$. Therefore
$$\mathcal L(F^{-1})=-\frac{\Delta F+\nabla\log H\cdot\nabla F}{F^2}<0$$
near each compact part of the wall. The same conclusion holds after adding the smooth regularization drift. On any fixed ball, the generator of $F^{-1}$ is bounded above. Stopped Ito calculus bounds the probability of reaching $F=\varepsilon$ before a fixed time and before exiting that ball by a constant times $\varepsilon$. Letting $\varepsilon$ tend to zero proves wall avoidance.
At fixed direction, $z=F/D$ decreases with radius, while $z-2\log z$ decreases with $z$ for $0<z\le1$. Thus $y\cdot\nabla L(y)\ge0$, yielding
$$\mathcal L r^2\le6,\qquad
\mathcal L_\lambda r^2\le6-\lambda r^2+64\lambda.$$
These bounds exclude explosion. Together with local smoothness and wall avoidance, they give a unique global strong solution.
Finally, $D'(q)>0$, and
$$\partial_a z=\frac{2a(b^2+c^2)D'(q)}{D(q)^2}.$$
Since $d(z-2\log z)/dz<0$ on $0<z\le1$, the unregularized generator satisfies $\mathcal L(a^2)\ge2$. If $\tau_M$ is the first exit from a ball of radius $M$ containing the initial point in $y$ coordinates, stopped Ito calculus gives $\mathbb E\tau_M\le(M^2-a_0^2)/2$. This justifies the repeated escape attempts above.
Failure of limit interchange. Finite-time continuous dependence of the SDE on $\lambda$, using wall avoidance and nonexplosion with localization bounds uniform over $0\le\lambda\le1$, identifies the $\lambda\downarrow0$ process with the unregularized process. Localize conditional on the initial point, then integrate over its Gaussian law. With the same Gaussian initialization for every $\lambda$, the two iterated limits are therefore
$$\lim_{t\to\infty}\lim_{\lambda\downarrow0}\operatorname{Law}(w(\theta_t^\lambda))
=p\mu_++(1-p)\mu_-,\qquad p\ge\frac{39}{64},$$
$$\lim_{\lambda\downarrow0}\lim_{t\to\infty}\operatorname{Law}(w(\theta_t^\lambda))
=\frac12\mu_++\frac12\mu_-.$$
Both limits exist and are unequal.
Questions for authors
Could you state a list of any informal heuristics or formal properties, beyond those brought up in the paper, that you believe must hold for the conjectured correspondence between the posterior you study and the large time behaviour of unregularised, isotropic noise Langevin dynamics to apply, and does this list exclude the setting of the counterexample above? If it does not, do you have some other reason to consider the counterexample irrelevant?
Thank you for the careful reading and questions in your review. They have pushed us to re-examine and update some of the claims in our paper, including a new theorem justifying the limit interchange for diagonal networks.
There seem to be two concerns here:
What does equilibrium tell us?
What can we say about equilibrium?
We try to address these below. Briefly:
We agree with the reviewer the paper should be clear that the main analysis is of Langevin equilibrium and not of SGD or transient effects, and have updated some of our phrasing to reflect that (lines 74-79); we stand by equilibrium being an established and useful perspective (which in some cases, as in our diagonal theorem can provide insight even before equilibrium).
We strengthen the heuristic derivation of effective regularization and provide a theorem showing it, and limit interchange, are formally justified for diagonal networks; for all models, we update the figures and experiments, including simulations over much longer times (Figures 3, 4 and 5) and quantitative comparisons of the distances between the different distributions using the Kolmogorov-Smirnoff distance (diagonal case: Figures 9 and 10; matrix case: Figures 12 and 13; ReLU case: Figure 14).
Further details on 1
The paper should indeed not give the impression that it characterizes general “real neural network training”, and we thank the reviewer for pointing that out. We update the introduction, and take the opportunity to emphasize:
(i) Langevin is a solvable model, meant to provide insights, but not necessarily a precise description of algorithms such as gradient flow or SGD (lines 74-79)
(ii) As the reviewer points out, Langevin equilibrium corresponds to Bayesian posterior for the network. There is a substantial literature that studies Bayesian posteriors over functions induced by Gaussian weights (e.g. the works mentioned in our introduction: Li & Sompolinsky, 2021; Izmailov et al., 2021; Zavatone-Veth & Pehlevan, 2021; Noci et al., 2021; Kobialka et al., 2026, among many others). That literature is subject to the same criticism, but is broadly considered of interest. We do agree that our introduction is framed somewhat differently, but we believe it’s a useful perspective because it allows one to ask whether unregularized Langevin dynamics behave the same way (as proved in Theorem 1, and examined in our experiments).
We also agree the connection to simplicity bias in real training was stronger than we intended, and have toned it down in our discussion of related work.
Further details on 2
We have added a proof that the zero-regularization and infinite-time limits commute in the diagonal case, based on our effective regularization idea, giving a quantitative rate under mild assumptions. We agree that our heuristic argument was previously a weak link in our reasoning, and see it now on firmer ground. We believe the proof technique may be extendable to the matrix case, but are more skeptical in the ReLU case (see the discussion after Theorem 1, and the discussion of the empirical linear network results on lines 373-377).
In all cases, we have run the simulations for much longer (up to t=10000 for the linear networks, and t=320 for ReLU), which hopefully go some way to address concerns about convergence; we have also modestly increased the size of the linear models (from 2x2 to 4x4). For a quantitative comparison, we have added plots of the Kolmogorov-Smirnoff distance between the empirical and predicted equilibrium distributions, and find it decreases and stabilizes around zero, while the same does not hold for the naive distribution without the entropic correction. Rigorously proving or checking convergence of Markov chains to equilibrium is a difficult and active area of research, and is outside the scope of this paper.
The heuristic also sheds light on the reviewer’s counterexample. The rescaling detailed in Appendix C.2 gives the effective potential Vt(θ)=L(θ)+λeff(t)R(θ), but tracking the corresponding instantaneous Gibbs distribution requires θ to equilibrate on a timescale comparable that over which Vt changes. In the counterexample there are two regions, on which the effective function has opposite sign, and which communicate only through a fixed central ball. Under the rescaling, this ball shrinks in radius at rate O(t−1/2), creating a bottleneck between two regions of non-vanishing Gibbs probability. Mixing between these regions thus becomes slow enough that the dynamics “remember” the initialization. In the case of diagonal networks, a uniform Poincaré inequality (Lemma 1) rules out such slow equilibration. Whether this generalizes to other networks is an interesting and important question we leave open for future work.
We hope we have addressed the reviewer's concerns, and welcome any further comments.
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.
Thank you for the positive review, and pointing out the typo. We have gone through the paper to correct the spelling errors we could find. We have also added clarifications throughout. We hope this improves the paper’s readability.
The reviewer is correct that our theory does not apply to SGD, but rather to the equilibrium of Langevin dynamics. Such equilibrium analyses have indeed proved useful across the literature (lines 74-97). In our case, we are able to connect the weak-regularization equilibrium to unregularized Langevin dynamics quantitatively for diagonal networks, with Theorem 1 in our updated submission.
We welcome any other further questions or edits.