acceptodds
Under review as a conference paper at ICLR 2027

A Stochastic–Geometric Theory of Scaling Laws in Grokking

Abstract

Delayed generalization (\ie grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell–core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.

Primary area:
learning theory
Keywords:
Grokking, Scaling Laws of Grokking, Stochastic-Geometric Theory, Theoretical Deep Learning, Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iaDyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

A fully-connected two-layer neural network trained with Adam on modular arithmetic data can exhibit grokking. Assuming that the parameter space consists of a generalizing core and an overfitting shell outside of the core, the authors derived expressions for the core and shell radii, and presented numerical results supporting their theory.

Critical strengths

None.

Critical weaknesses
  1. The presentation of the mental model is not clear. The introduction appear to tell me, the parameter space consists of a generalizing core and an overfitting shell, but what is the justification? Also, if that is the case, the core and shell radii should be the property of the loss function and risk function themselves, so why would they depend on the learning rate and batch size? The authors indeed say the shell-core structure is "optimizer-induced", but I think the meaning of that deserves elaboration.

  2. The authors claim Eq. (7) is one of their main result, but I don't see what can be learned from the equation. Eq. (7) tells me that the shell radius depend on the ratio of the learning rate and the batch size, but this is not surprising because the minibatch noise depends on the same ratio. How does the shell radius depend on this ratio? Eq. (7) says their relation is a Taylor expansion, which is true for any smooth function. If, for example, the authors can show that the 3rd-order term must be orders or magnitudes smaller than the constant, linear, and quadratic terms, then that is indeed something new for me to learn. Unfortunatelly I don't see the authors proving that.

  3. The numerical results are presented with "theoretical fits." However, they did not clearly write down the fitting functions. Therefore, how falsifiable are their predictions is not clear to me.

Questions for authors

What is the justification of the shell-core model of the parameter space?

Eqs. (7) and (15) appers to be just Taylor expansions. What can I learn from them, since all smooth functions can be written as Taylor series?

7 comments
AuthorsOpenReview ↗

Responses to Reviewer iaDy (part 1)

Dear Reviewer iaDy,

We very much appreciate that you have read and assessed our work. We will organise our responses point by point to address your concerns.

Regarding

A fully-connected two-layer neural network trained with Adam on modular arithmetic data can exhibit grokking.

We appreciate your careful reading and the summary.

We would like to clarify that this paper does not focus specifically on two-layer neural networks; the two-layer architecture is used only in the experimental setting. Our work studies training dynamics.

To validate our equations, lemmas, theorems, and theoretical predictions, we need to compute the gradient, with O(N)O(N) entries, and the Hessian, with O(N2)O(N^2) entries, at every training step, where is the number of parameters. For larger networks, this computation becomes impractical.

Regarding

Assuming that the parameter space consists of a generalizing core and an overfitting shell outside of the core, the authors derived expressions for the core and shell radii, and presented numerical results supporting their theory.

We appreciate your careful reading and summary.

The topological configuration comes from empirical observations under weight-shrinkage regularisers. Please refer to the experiments conducted in Figure 1 and Appendix (A.2 ADDITIONAL RESULTS: ADAM–INDUCED SHELL–CORE RADIUS & STOPPING-TIME CONCENTRATION).

Regarding

the authors derived expressions for the core and shell radii, and presented numerical results supporting their theory.

We appreciate that you highlight our contribution.

Regarding

Critical Weaknesses:

  • The presentation of the mental model is not clear. The introduction appear to tell me, the parameter space consists of a generalizing core and an overfitting shell, but what is the justification?

We appreciate your concerns, welcome criticism and open to academic disagreements.

  • First, from literature. Please refer to the 1st paragraph of introduction (page 1)
`Empirical findings consistently suggest that training dynamics in the grokking regime exhibit three-stage behavior (Power et al., 2022; Nanda et al., 2023; Kumar et al., 2024). In the initial stage, the model rapidly interpolates the training set and reaches the memorization regime. In the second stage, the trajectory remains near the memorization solutions for an extended period before escaping and eventually reaching the generalization regime. The length of this extended period scales with the task, learning rate, batch size, and regularization coefficient. In the third stage, drift and diffusion equilibrate in the optimization-induced dynamics, and the trajectory settles within the generalization solution region centered at its minimizer.` 
  • Second, from experiments/observations and optimisation dynamics. Please refer to the 2nd paragraph of introduction (page 1)
`Under the regime with ℓ2 weight-shrinkage regularization, these observations imply the existence of two distinct classes of solutions: memorization solutions, which only interpolate the training data and does not generalize, and generalization solutions, which additionally achieve low loss with respect to the underlying data distribution. Their topological configuration is thus induced by optimization dynamics, and characterized into three stages: the early stage, the late stage, and the equilibrium, as shown in Figure 1a. The early-stage dynamics are driven by deterministic drift from large gradients, which dominates the optimization trajectory and rapidly drives it toward the memorization manifold. At this stage, the gradient fluctuations are anisotropic. The late-stage dynamics are driven more slowly by decaying but nonzero gradients, together with isotropic gradient fluctuations, which guide the optimization trajectory across the trivial memorization solutions. Once the trajectory reaches the generalization set, the drift and diffusion in the optimization dynamics equilibrate and confine the trajectory within the generalization manifold, referred to as the equilibrium. Figure 1b–1c provide empirical evidence consistent with this optimization-induced topological configuration by visualizing optimization trajectories via isometric manifold learning. This optimizerinduced topological configuration, as illustrated in Figure 1a, gives rise to grokking and forms a shell–core configuration: initialization solutions concentrate on a thin hyperspherical shell, which encloses a shell of memorization solutions containing a compact core of generalization solutions. Detailed empirical results of Adam–induced shell–core radii are provided in Appendix A.2.`
AuthorsOpenReview ↗

Responses to Reviewer iaDy (part 2)

Regarding

Also, if that is the case, the core and shell radii should be the property of the loss function and risk function themselves, so why would they depend on the learning rate and batch size?

We appreciate your concerns. Let us put it simply.

  • The radii are a property of neither the loss function nor the risk function.
  • Their dependence on hyperparameters such as the learning rate and the batch size comes directly from Adam's joint SDE (Lemma 1):

dSt=μ(St),dt+ηb,σ(St),dWt,St:=(θt⊤,mt⊤,vt⊤)⊤∈R3p\mathrm{d} S_t = \mu(S_t)\\,\mathrm{d} t + \sqrt{\frac{\eta}{b}}\\,\sigma(S_t)\\,\mathrm{d} W_t, \qquad S_t := (\theta_t^\top, m_t^\top, v_t^\top)^\top \in \mathbb{R}^{3p}

μ(St)=(−B(t),mt⊘vt−α1(mt−gˉt)−α2(vt−g⊙g‾t)),σ(St)=(0α1Σt1/2α2Dt)\mu(S_t) = \begin{pmatrix} -B(t)\\,m_t \oslash \sqrt{v_t} \\\\ -\alpha_1\bigl(m_t - \bar g_t\bigr) \\\\ -\alpha_2\bigl(v_t - \overline{g\odot g}_t\bigr) \end{pmatrix}, \qquad \sigma(S_t) = \begin{pmatrix} 0 \\\\ \alpha_1 \Sigma_t^{1/2} \\\\ \alpha_2 D_t \end{pmatrix}

αi:=1−βiη,B(t):=1−e−α2t1−e−α1t,g⊙g‾t:=gˉt⊙gˉt+1b,diag(Σt),Dt:=2,diag(gˉt),Σt1/2\alpha_i := \frac{1-\beta_i}{\eta}, \qquad B(t) := \frac{\sqrt{1 - e^{-\alpha_2 t}}}{1 - e^{-\alpha_1 t}}, \qquad \overline{g\odot g}_t := \bar g_t \odot \bar g_t + \frac{1}{b}\\,\mathrm{diag}(\Sigma_t), \qquad D_t := 2\\,\mathrm{diag}(\bar g_t)\\,\Sigma_t^{1/2}

Intuition. Why learning rate and batch size enter into Adam's SDE.

In mini-batch sampling, the optimiser uses the gradient computed from a mini-batch to estimate the true full-batch gradient. The error (variance) of this estimate appears as the diffusion term. The batch size controls the noise magnitude: a larger batch gives a smaller estimation error.

SGD as example. Let me use SGD's SDE as an example, step by step. (You can also find in appendix A5)

For example, suppose a batch contains bb samples

x1,x2,⋯ ,xbx_1, x_2, \cdots, x_b

with per-sample gradients

g1,g2,⋯ ,gbg_1, g_2, \cdots, g_b

whose mean and covariance are

gˉ:=E[gi],Σ:=Cov[gi]\bar g := \mathbb{E}[g_i], \qquad \Sigma := \mathrm{Cov}[g_i]

The mini-batch gradient is their average

g:=g1+g2+⋯+gbbg := \frac{g_1 + g_2 + \cdots + g_b}{b}

By the central limit theorem (CLT), if the batch size bb is sufficiently large, gg converges in distribution to

N(gˉ,1bΣ)\mathcal{N}\left(\bar g, \frac{1}{b}\Sigma\right)

where Σ\Sigma is the per-sample covariance. The covariance of the batch average is scaled by 1b\frac{1}{b} because the samples are independent:

Cov[g1+⋯+gbb]=1b2,Cov[g1+⋯+gb]=1bΣ\mathrm{Cov}\left[\frac{g_1 + \cdots + g_b}{b}\right] = \frac{1}{b^2}\\,\mathrm{Cov}[g_1 + \cdots + g_b] = \frac{1}{b}\Sigma

Wiener process. Therefore, for any mini-batch, the estimated gradient gg can be written as two terms:

g=gˉ−ϵ,ϵ=1bΣ1/2ζ,ζ∼N(0,Ip)g = \bar g - \epsilon, \qquad \epsilon = \frac{1}{\sqrt{b}}\Sigma^{1/2}\zeta, \qquad \zeta \sim \mathcal{N}(0, I_p)

where ϵ∼N(0,1bΣ)\epsilon \sim \mathcal{N}(0, \frac{1}{b}\Sigma) is the sampling noise. We use the negative-sign convention because gradient descent uses the negative gradient; this simplifies the later derivation.

SDE. Adam is too complicated to serve as an example, so let us use SGD. Consider the update at iteration kk (a stochastic process)

θk+1=θk−η,gk\theta_{k+1} = \theta_k - \eta\\,g_k

where η\eta is the learning rate. Substituting gkg_k gives

θk+1=θk−η,gˉk+ηbΣk1/2ζk\theta_{k+1} = \theta_k - \eta\\,\bar g_k + \frac{\eta}{\sqrt{b}}\Sigma_k^{1/2}\zeta_k

Each iteration corresponds to a time step Δt=η\Delta t = \eta, i.e. t=kηt = k\eta. A Wiener process (Brownian motion) WtW_t has increments ΔWk=Δt,ζk∼N(0,Δt,Ip)\Delta W_k = \sqrt{\Delta t}\\,\zeta_k \sim \mathcal{N}(0, \Delta t\\,I_p), so η,ζk=η,ΔWk\eta\\,\zeta_k = \sqrt{\eta}\\,\Delta W_k and

θk+1−θk=−gˉk,Δt+ηb,Σk1/2,ΔWk\theta_{k+1} - \theta_k = -\bar g_k\\,\Delta t + \sqrt{\frac{\eta}{b}}\\,\Sigma_k^{1/2}\\,\Delta W_k

Taking Δt→0\Delta t \to 0 gives the SDE of SGD:

dθt=−gˉt,dt+ηb,Σt1/2,dWt\mathrm{d}\theta_t = -\bar g_t\\,\mathrm{d} t + \sqrt{\frac{\eta}{b}}\\,\Sigma_t^{1/2}\\,\mathrm{d} W_t

Where η\eta and bb enter. The batch size enters through the covariance

AuthorsOpenReview ↗

Responses to Reviewer iaDy (part 3)

Regarding

The authors indeed say the shell-core structure is "optimizer-induced", but I think the meaning of that deserves elaboration.

We appreciate your concerns. The shell-core structure is indeed induced by the optimiser: it follows from Adam's joint-state SDE, which we developed and validated.

Adam's joint-state SDE is

dSt=μ(St),dt+ηb,σ(St),dWt,St:=(θt⊤,mt⊤,vt⊤)⊤∈R3p\mathrm{d} S_t = \mu(S_t)\\,\mathrm{d} t + \sqrt{\frac{\eta}{b}}\\,\sigma(S_t)\\,\mathrm{d} W_t, \qquad S_t := (\theta_t^\top, m_t^\top, v_t^\top)^\top \in \mathbb{R}^{3p}

where θt\theta_t is the parameter at time tt, and mtm_t and vtv_t are the first- and second-moment estimates at time tt.

The SDE has two terms. In SDE theory, μ(St),dt\mu(S_t)\\,\mathrm{d} t is called the drift, which is deterministic, and ηb,σ(St),dWt\sqrt{\frac{\eta}{b}}\\,\sigma(S_t)\\,\mathrm{d} W_t is called the diffusion, which is stochastic.

Training starts from an initial state S0=(θ0,0,0)S_0 = (\theta_0, 0, 0), where θ0\theta_0 is the initial parameter given by the initialiser (e.g. PyTorch's Kaiming initialiser), and the first- and second-moment estimates are initialised to zero.

  • Initialisation shell. As we derived in Section 3, θ0\theta_0 concentrates on a thin hyperspherical shell: for a given initialiser, the initial radius is nearly the same whatever the random seed.
  • Memorisation. Starting from S0S_0, as optimisation progresses, the trajectory first reaches the memorisation boundary ∂M\partial M at the stopping time τM(S0)\tau_M(S_0). Because the initial states concentrate, the trajectories towards memorisation behave alike. This is why, in Fig. 1, we observe very symmetric trajectories using manifold learning (MDS). It also suggests that, for any starting parameter θ0\theta_0 (with the initialiser kept the same), the radius at which the trajectory hits the memorisation boundary, ∥θτM(S0)−θ0∥22\Vert \theta_{\tau_M(S_0)} - \theta_0\Vert_2^2, is expected to be nearly identical.
  • Generalisation. Once the trajectory has reached the memorisation manifold, as optimisation continues, it finally settles in the generalisation manifold, where drift and diffusion equilibrate.

This is the optimisation-induced topology. The memorisation and generalisation manifolds MM and GG are defined as the solutions reachable under Adam's dynamics.

Empirically, the three radii concentrate across seeds (Appendix A.2; S5S_5, 10 seeds, mean ±\pm std): initialisation 248.25±0.60248.25 \pm 0.60, memorisation 185.53±0.59185.53 \pm 0.59, generalisation 69.50±1.0269.50 \pm 1.02.

Regarding

  1. The authors claim Eq. (7) is one of their main result, but I don't see what can be learned from the equation. Eq. (7) tells me that the shell radius depend on the ratio of the learning rate and the batch size,

We appreciate your concerns.

We would like to clarify the scope of this work: we study the scaling laws of grokking, i.e. how grokking scales with the learning rate, the batch size and the regularisation coefficient. Eq. (7) is the result that theoretically derives the relationship between these hyperparameters and the memorisation radius. Several things can be learned from it.

  • The learning rate and the batch size act through the single ratio η/b\eta/b. They modulate the stochastic diffusion of Adam's SDE through the factor η/b\sqrt{\eta/b}. To the best of our knowledge, the scaling of the memorisation radius with η/b\eta/b has not been derived in prior work.
  • The regularisation coefficient acts differently. λ\lambda modulates the deterministic drift, and enters only through the leading constant, (ρM(0))2∝exp⁡(−O(λ))(\rho_M^{(0)})^2 \propto \exp(-O(\lambda)).
  • Eq. (7) predicts a U-shape. It is consistent with Fig. 2.

Regarding

but this is not surprising because the minibatch noise depends on the same ratio. How does the shell radius depend on this ratio?

We appreciate your concerns. We welcome criticism and are open to academic disagreement.

However, knowing that the mini-batch noise depends on the ratio η/b\eta/b does not tell us how the scaling laws are quantitatively governed by η/b\eta/b and λ\lambda.

Reviewer_ZmhoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies grokking in a standard setting of two-layer networks trained with Adam and weight decay. The authors propose a geometric picture of parameter space: initializations lie on a thin spherical shell, memorizing networks (low training loss) lie on a shell of smaller radius, and generalizing solutions (low test loss) on an even smaller one. The authors frame the problem as afirst-passage problem between these spheres and analyze the weight norm when the optimizer first reaches the memorization and the generalization solutions. The first passage problem is then analyzed with a continuous-time SDE limit of Adam which in late stages of training is reduced to a one-dimensional SDE for the parmeter norm.

Critical strengths
  1. Insight: The first passage formulation is insightful and, in my knowledge, a new way to compute the grokking time from theory. the reduced SDE offers a clean and computable way to quantify the delay.

  2. Quantiative: the theory tracks the numerical experiments very well: the theoretical expressions for ρM2\rho_M^2, ρG2\rho_G^2 and τM→G\tau_{M\to G} closely match the measured values across the sweeps over η\eta, bb and λ\lambda.

Critical weaknesses
  1. Applicability: the theory relies heavily on the weight decay, though grokking has been observed also without it (eg Kumar et al 2024, cited in the manuscript), and with full batch optimization (eg the leading term in theorem 3 diverges when λ→0\lambda\to0). So the theory only describes the regime with strong weight decay, and the learning rate mostly sets the time scale. meanwhile, the number of samples used in training, which is the main control parameter in Power et al and Liu et al, is absent. The number used in experiments is not even reported. This is the main weakness in my opinion.

  2. The Scaling laws carry little content and the constants are fitted and not discussed:

  • theorem 1 is just a second order taylor expansion in $\eta/b $ around a noiseless trajectory, so the fit in Fig. 2 is basically the statement that ρM\rho_M is analytic to 2nd order
  • similarity, theorem 2 is just the standard stationary variance of noisy dynamics in a quadratic well.
  • theorem 3 assumes that the norm decays exponentially to leading order from ρM\rho_M to ρG\rho_G at rate λ/sˉ\lambda/\bar s, giving τM→G≈sˉηλlog⁡(ρM/ρG)\tau_{M\to G} \approx \frac{\bar s}{\eta\lambda}\log(\rho_M/\rho_G). However, since sˉ\bar s is not computed independently but it is defined as the average decay rate measured along the trajectory (Eqs. 437-438), so the formula holds for any trajectory by construction.

Minor:

  1. Section 3 and Appendix A.3–A.4 establish that the norm of a Gaussian initialization concentrates on a thin shell, which is textbook chi-square result. It uses heavier tools than needed and does not contribute to readability.
  2. Lots of missing details: what's the loss used? size of training set? how was "theoretical curves fitted"? what epsilon was used? these gaps do not meet basic standards of sceintifc reproducability
Questions for authors
  1. Training data size. What training fraction was used? How do ρM\rho_M, ρG\rho_G and τM→G\tau_{M\to G} depend on it, and can the framework predict how the solution transition time scales with the number of training samples?

  2. Grokking without weight decay. How does the theory account for grokking with λ=0\lambda = 0, or with full-batch training? In these cases the leading term of Theorem 3 diverges, or the noise vanishes, yet grokking may occur.

  3. Definitions of memorization and generalization. What value of ϵ\epsilon was used? Can the authors show training and test curves (loss and accuracy) with τM\tau_M and τG\tau_G marked, to show that they correspond to the start and end of the plateau in learning? Do the results change with ϵ\epsilon?

  4. Theoretical curves. How were the "theoretical" curves in Figs. 2, 4 and 5 obtained? Which constants are fitted, and how many free parameters does each curve have? As a simple check of Theorem 1, does ρM2\rho_M^2 stay the same when η\eta and bb are scaled by the same factor?

9 comments
AuthorsOpenReview ↗

Responses to reviewer Zmho (part 1)

Dear Reviewer Zmho,

We very much appreciate that you have read and assessed our work. We will organise our responses point by point to address your concerns.

Regarding

The paper studies grokking in a standard setting of two-layer networks trained with Adam and weight decay. The authors propose a geometric picture of parameter space: initializations lie on a thin spherical shell, memorizing networks (low training loss) lie on a shell of smaller radius, and generalizing solutions (low test loss) on an even smaller one. The authors frame the problem as a first-passage problem between these spheres and analyze the weight norm when the optimizer first reaches the memorization and the generalization solutions. The first passage problem is then analyzed with a continuous-time SDE limit of Adam which in late stages of training is reduced to a one-dimensional SDE for the parmeter norm.

We appreciate your careful reading of our work and thank you for highlighting our theoretical analysis.

We would like to clarify that this paper does not focus specifically on two-layer neural networks; the two-layer architecture is used only in the experimental setting. Our work studies training dynamics.

To validate our equations, lemmas, theorems, and theoretical predictions, we need to compute the gradient, with O(N)O(N) entries, and the Hessian, with O(N2)O(N^2) entries, at every training step, where NN is the number of parameters. For larger networks, this computation becomes impractical.

Regarding

Critical Strengths:

  • Insight: The first passage formulation is insightful and, in my knowledge, a new way to compute the grokking time from theory. the reduced SDE offers a clean and computable way to quantify the delay.
  • Quantiative: the theory tracks the numerical experiments very well

We appreciate that you carefully read our work and highlight the insights. In fact, our core contribution and positioning are:

  • Core contribution. Before the responses, we want to clarify the claimed contribution of our work comes from the analytical analysis regarding how hyper parameters such as learning rate and batch size modulate grokking, using stopping analysis based on the rigorously-derived SDE and the radial-form SDE.

  • Positioning. Unlike the largely empirical literature on scaling laws in grokking, our work provides a purely theoretical framework based on SDE theory that derives the observed scaling laws under weight-shrinkage regularisation.

Regarding

Critical Weaknesses:

  • Applicability: the theory relies heavily on the weight decay, though grokking has been observed also without it (eg Kumar et al 2024, cited in the manuscript), and with full batch optimization (eg the leading term in theorem 3 diverges when ). So the theory only describes the regime with strong weight decay, and the learning rate mostly sets the time scale. meanwhile, the number of samples used in training, which is the main control parameter in Power et al and Liu et al, is absent. The number used in experiments is not even reported. This is the main weakness in my opinion.

We appreciate and thank you for raising your concerns. We welcome criticism and open to academic disagreements.

We want to clarify two aspects:

  • First, this is the scope and limitation of the theoretical analysis. The reason is because deriving the scaling laws purely from SDE theory needs the weight-evolution SDE and its radial form (manifold radius) has to be analysable so that we are able to derive useful and verifiable results. Without the weight-shrinkage regulariser, we have the difficulty to explicitly derive the radial SDE. Thus, analytical form of the scaling laws is impossible.
  • Second, for the learning tasks we've tested in our setting. Without weight-shrinkage regularisers, training doesn't grok after significantly elongated steps.

To avoid to complicate the problem, and deliver a neat theory to the community, we treat this as the elimination of our work. We wish to leave this into future work if that is mathematically possible.

AuthorsOpenReview ↗

Responses to reviewer Zmho (part 2)

Regarding

The Scaling laws carry little content and the constants are fitted and not discussed:

  • theorem 1 is just a second order taylor expansion in η/b\eta/b around a noiseless trajectory, so the fit in Fig. 2 is basically the statement that is analytic to 2nd order

We appreciate your concerns, and would like to clarify the following.

  • In the training dynamics, when trajectories hit the memorisation boundary, the SDE is dominated by the drift, and the diffusion terms act only negligibly in this regime. Unlike an empirically based study, a purely analytical derivation is very difficult: without these simplifications and assumptions, we would not be able to derive the analytical form of the scaling law.

  • Theorem 1 is not a Taylor expansion. It is derived from the perturbation expansion of the infinitesimal generator LS\mathcal{L}_S. For a stochastic process driven by a Wiener process, a Taylor expansion is not applicable, since the process is continuous but not differentiable.

Let us briefly explain how Theorem 1 is derived.

  • We first establish a rigorous, non-approximated joint SDE for Adam on the state St=(θt⊤,mt⊤,vt⊤)⊤∈R3pS_t = (\theta_t^\top, m_t^\top, v_t^\top)^\top \in \mathbb{R}^{3p} (Lemma 1):

dSt=μ(St),dt+ηb,σ(St),dWt\mathrm{d} S_t = \mu(S_t)\\,\mathrm{d} t + \sqrt{\frac{\eta}{b}}\\,\sigma(S_t)\\,\mathrm{d} W_t

  • We then define the stopping time τM(S0)\tau_M(S_0), starting from the initial state S0S_0 and hitting the memorisation boundary ∂M\partial M:

τM(S0):=inf⁡t≥0:θt(S0)∈∂M\tau_M(S_0) := \inf\\{t \ge 0 : \theta_t(S_0) \in \partial M\\}

  • Based on τM(S0)\tau_M(S_0), we define the memorisation radius ρM2\rho_M^2:

ρM2:=ES0∈ΘS[∥θτM(S0)∥22]\rho_M^2 := \mathbb{E}_{S_0 \in \Theta_S} \bigl[ \Vert \theta_{\tau_M(S_0)} \Vert_2^2 \bigr]

Adam’s joint SDE→stateSt→stopping timeτM→memorisation radiusρM2\text{Adam's joint SDE} \to \text{state} \quad S_t \to \text{stopping time} \quad \tau_M \to \text{memorisation radius} \quad \rho_M^2

In SDE theory, one question of interest is: when StS_t changes by ΔSt\Delta S_t, how much does a scalar function of the state change in expectation? This is given by an operator called the infinitesimal generator, which is derived from Itô's lemma. For Adam's joint SDE it is

LS[∙]:=μ(S)⊤∇S[∙]+12,ηb,tr(ΣS(S),∇S2[∙]),ΣS(S):=σ(S)σ(S)⊤\mathcal{L}_S[\bullet] := \mu(S)^{\top}\nabla_S[\bullet] + \frac{1}{2}\\,\frac{\eta}{b}\\,\mathrm{tr}\bigl(\Sigma_S(S)\\,\nabla_S^2[\bullet]\bigr), \qquad \Sigma_S(S) := \sigma(S)\sigma(S)^{\top}

  • For a problem in which trajectories hit a memorisation boundary, the PDE must satisfy boundary conditions. In SDE theory this is referred to as the Dirichlet exit problem. Let u(S):=E[∥θτM(S0)∥22∣S0=S]u(S) := \mathbb{E}\bigl[\Vert \theta_{\tau_M(S_0)} \Vert_2^2 \mid S_0 = S\bigr] be the exit-value function, so that ρM2=ES0∈ΘS[u(S0)]\rho_M^2 = \mathbb{E}_{S_0 \in \Theta_S}[u(S_0)]. Then

LSu(S)=0(θ∈M∘),u(S)=S⊤EθS(θ∈∂M)\mathcal{L}_S u(S) = 0 \quad (\theta \in M^\circ), \qquad u(S) = S^\top E_\theta S \quad (\theta \in \partial M)

where EθE_\theta is the θ\theta-block projector, so that S⊤EθS=∥θ∥22S^\top E_\theta S = \Vert \theta \Vert_2^2.

This PDE admits no analytical solution. One way forward is to derive an approximate solution by operator perturbation, expanding the operator.

We agree with you that the "algebraic form" of the result may resembles Taylor expansion.

AuthorsOpenReview ↗

Responses to reviewer Zmho (part 3)

similarity, theorem 2 is just the standard stationary variance of noisy dynamics in a quadratic well.

We appreciate your concerns.

Thm2 differs from Thm1 in both its form and its derivation route. In the derivation of Thm1, the dynamics are dominated by drift. For Thm2, however, diffusion is no longer negligible: once the trajectory enters ∂G\partial G, the drift and diffusion equilibrate.

To derive thm2, the outline is

  • Radial SDE after slow-manifold reduction. From Adam's joint SDE, we derive the SDE of the manifold radius rt2:=∥θt∥22r_t^2 := \Vert \theta_t \Vert_2^2 (Lemma 2; please see Fig. 6, High-Level Sketch of Theoretical Analysis Framework) under the slow-manifold reduction and the small-isotropic gradient covariance (see Observations 1 and 2):

drt2≈[−2s(θt),θt⊤(gˉt∗+λθt)+η,p]dt+2η,rt,dWt(r)\mathrm{d} r_t^2 \approx \left[ -\frac{2}{s(\theta_t)}\\,\theta_t^\top \bigl( \bar g^{\ast}_t + \lambda \theta_t \bigr) + \eta\\,p \right] \mathrm{d} t + 2\sqrt{\eta}\\,r_t\\,\mathrm{d} W_t^{(r)}

where s(θ)s(\theta), with 1s(θ):=1ptr(π(θ))\frac{1}{s(\theta)} := \frac{1}{p}\mathrm{tr}\bigl(\pi(\theta)\bigr), is the effective scalar preconditioner obtained from Adam's preconditioner π(θ)\pi(\theta), and gˉ∗\bar g^{\ast} is the task-loss mean gradient.

  • First-hitting problem on ∂G\partial G. Starting from the state at which the trajectory first reaches ∂M\partial M, we define

τG(S0):=inf⁡t≥τM:θt(S0)∈∂G,ρG2:=ES0∈ΘS[∥θτG(S0)∥22]\tau_G(S_0) := \inf\\{t \ge \tau_M : \theta_t(S_0) \in \partial G\\}, \qquad \rho_G^2 := \mathbb{E}_{S_0 \in \Theta_S}\bigl[\Vert \theta_{\tau_G(S_0)} \Vert_2^2\bigr]

  • Equilibrium centre. Adam's SDE then approaches its equilibrium, centred at the regularised local minimiser (we do not assume a global minimiser):

θ⋆(S0):=arg⁡min⁡θ(S0)Lf∗(θ(S0))+λ2∥θ(S0)∥22\theta^\star(S_0) := \arg\min_{\theta(S_0)} \mathscr{L}^{\ast}_f(\theta(S_0)) + \frac{\lambda}{2}\Vert \theta(S_0) \Vert_2^2

  • Linearisation around θ⋆\theta^\star. Writing θt=θ⋆+δt\theta_t = \theta^\star + \delta_t and using gˉ∗(θ⋆)+λθ⋆=0\bar g^{\ast}(\theta^\star) + \lambda\theta^\star = 0, under the dominant-regularization approximation,

gˉt∗+λθt≈λδt+O(∥δt∥22)\bar g^{\ast}_t + \lambda\theta_t \approx \lambda\delta_t + O\bigl(\Vert \delta_t \Vert_2^2\bigr)

  • Local stationary approximation. Taking the conditional expectation of the radial SDE, with s⋆(S0):=s(θ⋆(S0))s^\star(S_0) := s(\theta^\star(S_0)),

0≈−2λs⋆(S0),E[∥δt∥22∣S0]+η,p+O(η2λ2)⇒E[∥δt∥22∣S0]≈ηλ,s⋆(S0)2,p0 \approx -\frac{2\lambda}{s^\star(S_0)}\\,\mathbb{E}\bigl[\Vert \delta_t \Vert_2^2 \mid S_0\bigr] + \eta\\,p + O\left(\frac{\eta^2}{\lambda^2}\right) \quad\Rightarrow\quad \mathbb{E}\bigl[\Vert \delta_t \Vert_2^2 \mid S_0\bigr] \approx \frac{\eta}{\lambda}\\,\frac{s^\star(S_0)}{2}\\,p

  • Averaging over S0∈ΘSS_0 \in \Theta_S gives Theorem 2:

ρG2≈(ρG(0))2+ηλ,cG+O(η2λ2)\rho_G^2 \approx \bigl(\rho_G^{(0)}\bigr)^2 + \frac{\eta}{\lambda}\\,c_G + O\left(\frac{\eta^2}{\lambda^2}\right)

(ρG(0))2:=ES0∈ΘS[∥θ⋆(S0)∥22],cG:=ES0∈ΘS[12s⋆(S0),tr(D⋆(S0))]>0\bigl(\rho_G^{(0)}\bigr)^2 := \mathbb{E}_{S_0 \in \Theta_S}\bigl[\Vert \theta^\star(S_0) \Vert_2^2\bigr], \qquad c_G := \mathbb{E}_{S_0 \in \Theta_S}\bigl[\tfrac12 s^\star(S_0)\\,\mathrm{tr}\bigl(D^\star(S_0)\bigr)\bigr] \gt 0

where D⋆(S0):=G(θ⋆(S0))G(θ⋆(S0))⊤D^\star(S_0) := G(\theta^\star(S_0))G(\theta^\star(S_0))^\top and G(θ)G(\theta) is the preconditioned diffusion factor, with G(θ)G(θ)⊤≈IpG(\theta)G(\theta)^\top \approx I_p.

The last step is a local stationary balance, but the result is specific to Adam. It rests on the reduction of Adam's joint SDE on St∈R3pS_t \in \mathbb{R}^{3p} to a one-dimensional radial SDE (Lemma 2), and the constant cGc_G is determined by Adam's preconditioner through s⋆s^\star and D⋆D^\star. As a result, the correction scales with η/λ\eta/\lambda, with only a weak batch-size dependence cG∝O(1b)c_G \propto O(\frac{1}{\sqrt b}), and (ρG(0))2∝exp⁡(−O(λ))(\rho_G^{(0)})^2 \propto \exp(-O(\lambda)).

Reviewer_evUyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors characterize the solution manifold in parameter space of the ADAM optimizer with the goal of understanding the phenomenon of grokking. In grokking, training error goes to zero much faster (in fewer iterations) than the generalization error. The results here show the time required to go from memorization to generalization on average in terms of the hyperparameters such as learning rate and regularization strength.

Critical strengths
  1. Clear explanation of grokking
  2. Non standard examples like learning group actions
Critical weaknesses

It is well known that ADAM dynamics can be written under appropriate assumptions (small learning rate) as an SDE and there is a vast literature (e.g Wilson et al 2017, Reddi Kale Kumar 2018) proving lack ergodicity and lack of convergence to solutions that generalize. The authors may want to specifically write what assumptions they are making on the optimization landscape to obtain their results; they assume the probability of the solutions reaching generalization error less than epsilon is always non-zero.

The results do not really tell us that the solution manifold is a sphere and there is an inner shell, outer shell etc. One can always define radii or diameter of any set but that doesn't mean the set is a sphere or the surface of a sphere. There is a lot of terminology from differential geometry, probability (e..g, what is p_ADAM) that is used without properly defining them.

Questions for authors

Is the asymptotic expansion of the solution of the SDE used? How exactly does this lead to Theorem 1? The authors cite a textbook on perturbation analysis and say the proof is in the appendix but this does not exactly convey any intuition. The same kind of questions apply to every result in the paper.

Why are we able to get explicit dependence on the hyperparameters of the transition time and how do we interpret them? What is the guarantee that solutions won't go back to ``shell'' of memorization?

3 comments
AuthorsOpenReview ↗

Responses to Reviewer evUy (part 1)

Dear Reviewer evUy,

We very much appreciate that you have read and assessed our work. We will organise our responses point by point to address your concerns.

Core contribution. Before the responses, we want to clarify the claimed contribution of our work comes from the analytical analysis regarding how hyper parameters such as learning rate and batch size modulate grokking, using stopping analysis based on the rigorously-derived SDE and the radial-form SDE.

Positioning. Unlike the largely empirical literature on scaling laws in grokking, our work provides a purely theoretical framework based on SDE theory that derives the observed scaling laws under weight-shrinkage regularisation.

Regarding

Critical Strengths:

  • Clear explanation of grokking
  • Non standard examples like learning group actions

We thank the reviewer for highlighting these strengths.

Regarding

It is well known that ADAM dynamics can be written under appropriate assumptions (small learning rate) as an SDE and there is a vast literature (e.g Wilson et al 2017, Reddi Kale Kumar 2018)

We appreciate the concerns and thank the reviewer.

We want to clarify that SDE approximations of Adam do exist. For example, we have credited such as [1-2] in our manuscript.

[1] Sadhika Malladi et al. On the SDEs and scaling rules for adaptive gradient algorithms. NIPS
[2] Enea Monzio Compagnoni et al. Adaptive methods through the lens of SDEs: Theoretical insights on the role of noise.

However, most of these approximations are first-order approximations. In our work, we need to study grokking from the perspective of the stochastic dynamics. In a stochastic dynamical system, studying the modulation effects on this stochastic process from such as learning rate and batch size needs the second-order information. To the best of our knowledge, the SDE form we rigorously proved does not exist in literature. We develop this for studying the stochastic dynamics of grokking under weight-shrinkage regulariser.

Regarding

proving lack ergodicity and lack of convergence to solutions that generalize.

We appreciate the concerns. First of all, the delayed generalisation on datasets without noise and under structures exists empirically in literature. This is the grokking.

For the literature [3-4], we've read their papers. Their papers do not overlap with grokking and do not discuss the generalisation from the perspective of SDE theory such as ergodicity. Probably you the literature [3-4] you mentioned are wrong?

One more note. Ergodicity is orthogonal to generalisation.

Despite the study regarding Adam by approximating exact SDE comes very late in literature [1-2], they also do not overlap our formulation (exact SDE).

[3] Ashia C. Wilson et al. The Marginal Value of Adaptive Gradient Methods in Machine Learning
[4] Sashank J. Reddi. On the convergence of Adam and Beyond

Regarding

The authors may want to specifically write what assumptions they are making on the optimization landscape to obtain their results; they assume the probability of the solutions reaching generalization error less than epsilon is always non-zero.

We appreciate the concerns. We agree with you.

In fact, our theoretical research regarding solution structure (in another paper, not in the scope of this paper) and experiments show that, from the lens of SDEs, in equilibrium state, weight trajectories will reach a ring likes the “event horizon of a black hole.” The solutions will stay on this ring-like region centring its local optimum, rather than zero loss. The confinement of this region is small.

This can be done by SDE and its FPE (Fokker-Planck Equation).

Regarding

The results do not really tell us that the solution manifold is a sphere and there is an inner shell, outer shell etc.

We appreciate your concerns. Please refer to Appendix A.2 in page 14. The empirical evidences show the manifold shape and the shell-core structure under several tasks.

We want to add a disclaimer. We only study the weight-shrinkage situation. In our experiments and literature, grokking needs weight-shrinkage regulariser. But this does not exclude that without weight-shrinkage regulariser, models can not grok. This is the limitation of this work. we only consider the classic setting and provide theoretical interpretation.

AuthorsOpenReview ↗

Responses to Reviewer evUy (part 2)

Regarding

Is the asymptotic expansion of the solution of the SDE used? How exactly does this lead to Theorem 1? [...]

We appreciate your concerns.

Theorem 1 is derived through an asymptotic expansion, but one obtained through perturbation theory on operators. Usual Taylor expansion doesn't apply to stochastic process which is continuous but not differentiable ANYWHERE.

Let me put the outline simply.

We use plain language below; if you are familiar with SDE theory and operator perturbation theory (e.g. Kato's), please ignore some of the terms we use.

  1. Establishing a rigorous Adam SDE. We first establish a rigorous parameter-evolution SDE for Adam, without approximations, so that the SDE captures all information during the evolution.

Please also see Appendix A.5 (PROOF: ADAM'S CLOSED-FORM CONTINUOUS-TIME SDE LIMIT) and Fig. 6.

The SDE is

dSt=μ(St),dt+ηb,σ(St),dWt,St:=(θt⊤,mt⊤,vt⊤)⊤∈R3p\mathrm{d} S_t = \mu(S_t)\\,\mathrm{d} t + \sqrt{\frac{\eta}{b}}\\,\sigma(S_t)\\,\mathrm{d} W_t, \qquad S_t := (\theta_t^\top, m_t^\top, v_t^\top)^\top \in \mathbb{R}^{3p}

where θ,m,v\theta, m, v are the parameters and the first- and second-moment estimates, respectively. Please see Lemma 1, which is also verified through experiments.

  1. Defining the stopping time. Suppose we train a model from the initial state S0=(θ0,0,0)S_0 = (\theta_0, 0, 0), since Adam and AdamW initialise mm and vv to zero.

We are interested in the time at which the trajectory StS_t, starting from S0S_0, reaches the memorisation boundary ∂M\partial M. We can now define the stopping time τM(S0)\tau_M(S_0) for a single realisation of the SDE. This is standard in SDE theory.

τM(S0):=inf⁡t≥0:θt(S0)∈∂M\tau_M(S_0) := \inf\\{t \ge 0 : \theta_t(S_0) \in \partial M\\}

  1. Defining the expectation over all realisations starting from anywhere in the initialisation manifold S0∈ΘSS_0 \in \Theta_S. We are interested in the expected radius on reaching the memorisation boundary.
  • We denote by θτM(S0)\theta_{\tau_M(S_0)} the parameter at the moment the trajectory hits the memorisation boundary.

ρM2:=ES0∈ΘS[∥θτM(S0)∥22]\rho_M^2 := \mathbb{E}_{S_0 \in \Theta_S} \left[ \Vert \theta_{\tau_M(S_0)} \Vert_2^2 \right]

  1. Infinitesimal generator. Let u(S):=E[∥θτM(S0)∥22∣S0=S]u(S) := \mathbb{E}\bigl[\Vert \theta_{\tau_M(S_0)} \Vert_2^2 \mid S_0 = S\bigr] be the exit-value function: the expected squared parameter norm at the first-hitting time of ∂M\partial M, starting from SS. Averaging it over the initialisation gives ρM2=ES0∈ΘS[u(S0)]\rho_M^2 = \mathbb{E}_{S_0 \in \Theta_S}[u(S_0)]. uu is a scalar function of the state, and StS_t evolves exactly according to its SDE, i.e.

SDE→St→u→ρM2\mathrm{SDE} \to S_t \to u \to \rho_M^2

In SDE theory (and in physics as well), this type of problem is called the Dirichlet exit problem, defined on the state StS_t.

We ask: if StS_t changes by ΔSt\Delta S_t, how much does uu change? In SDE theory, there is an operator, defined on the SDE and acting on the scalar function uu, called the "infinitesimal generator" LS[∙]\mathcal{L}_S[\bullet].

This is basic SDE theory. The generator is derived directly from Itô's lemma, which is why it has a second-order term: the stochastic effect usually has non-zero variance and does not admit the usual first-order Taylor expansion.

LS[∙]:=μ(S)⊤∇S[∙]+12,ηb,tr(ΣS(S),∇S2[∙])\mathcal{L}_S[\bullet] := \mu(S)^{\top}\nabla_S[\bullet] + \frac{1}{2}\\,\frac{\eta}{b}\\,\mathrm{tr}\bigl(\Sigma_S(S)\\,\nabla_S^2[\bullet]\bigr)

and uu solves the Dirichlet problem

LSu(S)=0(θ∈M∘),u(S)=∥θ∥22(θ∈∂M)\mathcal{L}_S u(S) = 0 \quad (\theta \in M^\circ), \qquad u(S) = \Vert \theta \Vert_2^2 \quad (\theta \in \partial M)

This is a PDE which doesn't admit analytical solution in usual sense.

  1. Perturbation solution for ρM2\rho_M^2.

However, directly solving for uu, and hence expressing ρM2\rho_M^2 in terms of the hyperparameters of the SDE, is impossible.

To solve it, we need some tricks. One way is to decompose the generator LS\mathcal{L}_S. In operator theory, an operator can be expanded (this is very common in quantum physics and functional analysis).

Note: there are several traditions for this theory; the one we are familiar with is Kato's framework for spectral theory/operator theory.

Please read A.12 PROOF: SCALING LAW OF MEMORIZATION RADIUS for the details.

AuthorsOpenReview ↗

Responses to Reviewer evUy (part 3)

Regarding

Why are we able to get explicit dependence on the hyperparameters of the transition time and how do we interpret them?

We appreciate your concerns.

Why hyperparaemters enter into scaling laws.

This is because the algebraic link: AdamSDE→stateSt→stoppingtimeandradiusρM2\mathrm{Adam SDE} \to \mathrm{state} S_t \to \mathrm{stopping time and radius} \rho_M^2. The explicit relationship is ruled by the infinitesimal generator L[∙]\mathcal{L}[\bullet] of the SDE (in SDE theory). The hyper-parameters of the SDE enter into L[∙]\mathcal{L}[\bullet].

This is why we get explicit dependence on the hyperparameters of the transition time.

Interpretation.

Intuitively, both learning rate η\eta and batch size bb modules the SDE.

  • The diffusion of the SDE is modulated through ηb\frac{\eta}{b}. Larger ηb\frac{\eta}{b} leads higher randomness; smaller ηb\frac{\eta}{b} leads lower randomness. Solution trajectories with smaller randomness will be faster to reach the memorisation boundary.

  • The drift of the SDE is modulated by η\eta. Larger η\eta gives faster step size, reaching to memorisation boundary is faster.

This is also verified by the experiments in Fig2: Scaling Laws of Manifold Radius

Please also read L458 Discussions and Limitations.

(L458) Discussions and Limitations. The scaling laws originate from how η\eta, bb, and λ\lambda modulate Adam's SDE: η\eta and bb module the stochastic diffusion by scaling the gradient covariance with a factor η/b\sqrt{\eta/b}, whereas λ\lambda modulates the deterministic gradient drift flow by regularization. Our results inherit the conditions required by Adam's SDE limit, including an adapted Wiener-process approximation. These conditions require a sufficiently small learning rate and a sufficiently large batch size; thus, our results do not directly apply to the large-η/b\eta/b regime.

Regarding

What is the guarantee that solutions won't go back to ``shell'' of memorization?

We appreciate your concerns.

The guarantee comes from the equilibrium state of the SDE. This can be expressed quantitatively using the Fokker–Planck equation (FPE) for the radial SDE. The FPE mathematically describes probabilities of solutions. Of course, the probability that a solution returns to any location in the space is nonzero. However, the probability measure is zero.

When solutions reach the vicinity of their respective optima, they remain ONLY on a ring very close to the optimum. This likes the event horizon of a black hole. The reason is that noise from mini-batch sampling is always present and nonzero. This persistent noise maintains a balance between the drift toward the optimum and the thermal fluctuations that prevent convergence to the optimum. This behavior is governed by the nature of the SDE.

We thank the reviewer again.

The authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

We've revised the manuscript as per the concerns from the reviewers.

Dear Reviewers and AC,

We heartfelt thank the reviewers for reading and assessing our theoretical work. We also thank the AC for organising the reviewing.

We revised the manuscript as per the concerns such as the training data setting and reproducibility (see Appendix L624 for the outlines).

We also made clarification that why we choose a 2-layer network for experiments which is NOT the theoretical limitation:

We use a two-layer MLP because our analysis tracks per-sample gradients, of size O(p)O(p), together with second-order quantities, the gradient covariance Σt\Sigma_t and the Hessian, of size O(p2)O(p^2), throughout training; this becomes impractical for larger networks. Grokking requires FP32 precision; with half precision (FP16), training exhibits the well-known slingshot effect (Hanqing et al., 2026) and does not grok.

We welcome criticism and are open to academic disagreements.

Thank you again.

The authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.