acceptodds
Under review as a conference paper at ICLR 2027

Grokking as Slow Transport: A Quantitative Framework of Delayed Generalization

Abstract

Delayed generalization, or grokking, has been explained by many different mechanisms in the literature, with conclusions that sometimes appear to conflict. We show that these mechanisms can be understood as different kinetic regimes of a common process: the motion of a slow coordinate out of a metastable state. Starting from the minibatch update rule, we reduce plateau training dynamics to a one-dimensional first-passage problem and derive a governing equation for the grokking delay. This yields a quantitative regime classification, where each regime has a distinct delay law and seed-to-seed variability law. We further derive explicit delay laws for SGD and Adam-family optimizers by mapping their hyperparameters to the effective drift and diffusion of the slow coordinate. Our framework reconciles existing explanations as regime-specific limits and accounts for diverse grokking phenomena observed in prior work. Code will be publicly available at https://github.com.

Primary area:
learning theory
Keywords:
Grokking, training dynamics, stochastic gradient descent, AdamW, phase diagram.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EAT2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper says grokking is slow motion along one hidden coordinate. It builds a potential Φ = −∫ µ/D and splits cases into three regimes. The idea is interesting. But the evidence is weak. The main problem is circularity: the paper uses the true grokking time N* to define the coordinate range, the regime, and the prediction point. Then it reports good agreement with N*. This is not convincing prediction.

Critical strengths

Some concrete observations are valuable. Especially, full-batch Adam often sits at the ε floor and behaves like GD at η/ε and coupled L2 through Adam is not the same as AdamW decoupled decay, so weight decay is necessary in some settings but not others.

It tries to be quantitative. It predicts delay scalings rather than only giving another qualitative mechanism.

Critical weaknesses

N* is used to define key quantities such as ℓ, the profile estimator Ω(t)=v(N*)/v(t)−1, and even the forecast time tobs≈0.25N*. So the model often predicts the answer after seeing the answer.

The 1D reduction is not proven on real networks. Appendix D.9 effectively says the decisive test cannot be run on the real hosts. The clean validation is mostly on constructed toy anchors designed to satisfy the assumptions.

Different tasks use different slow coordinates, and when estimators disagree the chosen regime tends to be the one that matches the delay law. With non-grokking phases N1-N3 absorbing failures, the framework is hard to falsify.

Many physics-inspired quantities are introduced without plain-language explanation, and Φ becomes central before it is clearly defined. The appendix has unresolved equations and references, e.g., Pages 21, 34 etc. This makes the paper hard to read and obscures which quantities are derived or are assumed.

Questions for authors
  1. Can you choose the slow coordinate, readout threshold, ℓ, Ω estimator and prediction time before seeing N*, then report blind predictions on new hyperparameter, even ideally one new task?
  2. Can you estimate the timescale separation g and the memory correction D_mem/D0 on at least one real grokking model, not only on constructed anchors?
  3. What observation would satisfy the three regime map? If regime C never occurs in real grokking, why is it evidence for a unified theory of grokking?
  4. If I choose a different reasonable slow coordinate on the same model, e.g., ln‖θ‖ vs. embedding IPR vs. teacher alignment, do I get the same regime and the same N* law? If not, what objective rule selects the coordinate before the outcome is known?
7 comments
AuthorsOpenReview ↗

Circularity Concern: Main Prediction Results Never Affected; Remaining Affected Parts Now Fully Addressed by Reruns with Circularity Removed

We thank the reviewer for a careful and fair review. We address each question and revise the paper accordingly.

Important Question in Overview & W1 (Circularity)

Part of the circularity concern is correct, but we want to clarify that circularity isn't involved in any of our main prediction experiments and results (which include §4.3, Table 2, Figure 5 Top Row; §4.2, Fig. 4, etc.) Notably, The headline transfer results (Table 2, 2.6% median error) never use the target's N∗N^{*}. Each prediction is either the source cell's delay rescaled by the law's ratio, or the source's constants plus the target's initialization offset at t = 0, while the regime label comes from the source cell. So the central object, the grokking delay's prediction is never affected by the circularity. Below, we clarify the places where N* circularity do exist, and we rerun experiments without N*:

  1. The Ω tracks in Figs. 2, 3 and 7 and Tables 6–7 do use N∗N^{*}. The stage width ℓ was measured to the exit value at N*, and the profile estimator is Ω(t)=v(N∗)/v(t)−1Ω(t) = v(N^{*})/v(t) − 1. These figures place settings on the regime map. They were not used to produce the transfers, but we agree they cannot be the evidence that a regime is determinable in advance.
  2. The single-run forecast in Appendix B.3 used tobst_{obs} = 0.25 N*, which also depends on the outcome.

To resolve these 2 circularities, we construct an Outcome-free pipeline. We replaced each N*-dependent quantity with one that is observable before grokking:

  • ℓ now comes from the exit value of other knob cells on the same host, never the target cell.
  • Ω is a local estimate (dv/dx)⋅ℓ/v(dv/dx)·ℓ/v fitted on [tmemt_{mem}, tobst_{obs}] only.
  • tobs=K⋅tmemt_{obs} = K·t_{mem}, where tmemt_{mem} is the first step with training accuracy ≥ 0.99.
  • Probe versus profile estimator is chosen by validity flags (probe isotropy, burst overlap, linearity in δ) that do not look at the delay. This addresses your point that the chosen regime tends to match the delay law.

Reruns for (1): (Regime labels). 28 of 31 cells get the same label as in the paper.

  • All 15 ridge cells stay in regime A, with Ω between 3 and 2,700, always above 1.
  • The diagonal net is B at every K (9/9 cells).
  • Of the 16 decay-driven cells, 13 are B at K = 4. The three disagreements are Power λ = 0.1, MNIST λ = 0.003, and MNIST λ = 0.1.
    • Power λ = 0.1: memorization ends at 2% of the delay, so the early window looks like a ridge on ln‖θ‖. Ω is 88, 12 and 2.7 at K = 2, 4 and 8.
    • MNIST λ = 0.003: it is B only from K = 8.
    • MNIST λ = 0.1: K·t_mem ≥ N*, so there is no admissible reading. It is unresolved, not contradicted.

Reruns for (2): Single-run forecasts without N in Appendix B.3* (median error factor, number of cells within 2×):

Law, observation pointCellsMedian ratio N̂/N*Median errorWithin 2×
B: ℓsib/vℓ_{sib}/v, tobs=4⋅tmemt_{obs} = 4·t_{mem}150.791.32×14/15
A: ln(ℓsib/u)/λln(ℓ_{sib}/u)/λ, earliest admissible panel151.021.54×8/15

The old Appendix B.3 numbers were 1.9× for regime B and a median ratio of 0.29 for regime A. So the N dependence did not inflate the old results.* The B forecasts are biased slightly early. The A rows use dense panels whose times were scheduled as fractions of each cell's corpus delay. The estimator does not use N*, but the panel schedule was laid out with it, and we say so in the paper. We still consider cross-setting transfer the primary evidence (which is never affected by circularity, as discussed) and the single-run forecast a secondary, weaker result.

Thus, we fully address the circularity concerns here.

AuthorsOpenReview ↗

Q1, W3 & Q4: We don't choose slow coordinates, it's given by regime and formula automatically; (New Experiments) Blind predictions on new parameter, new task,

W3: Different Slow Coordinates Are Given Automatically, Not Chosen.

First, an important clarification: In our theory, the slow coordinate is intrinsic. We do not select/choose it when running experiments/predictions. It is the direction of slowest relaxation once the fast modes have equilibrated (§2.4). A scalar ξ(θ) is a valid description of it if it (i) tracks that slow direction, (ii) is monotone over the stage, and (iii) is something the readout responds to.

It is the delay formula who determines the coordinate in the following cases, depending on whether it's regime A or B:

Decay-driven hosts. Table 1 writes the law as (1/ηλwd)⋅ln(ρ0/ρ∗)(1/ηλ_wd)·ln(ρ_0/ρ*). That form holds only on θ=ln‖θ‖θ = ln‖θ‖, where decoupled decay is an exactly constant drift (§3.1, App. E.5). So the formula identifies the coordinate.

Ridge hosts. The law is ln(1+Ω)/λln(1+Ω)/λ, with u0u_0 the offset from the ridge top along the unstable direction. The coordinate is the amplitude of that unstable mode, which is IPR for the modular-addition hosts, the learned-direction amplitude for the polynomial host, and alignment for cubic and parity.


Q1: Blind predictions on new parameter, new task.

New Experiment Results. Then we run 13 additional experiments with various new hyperparameter and new task settings, as demonstrated in the figure below. PASS means N̂/N in [0.75, 1.33], "within 2×" means [0.5, 2], and nulls PASS in [0.85, 1.18].

CellLawPredictedMeasured (seeds)RatioVerdict
Power λ=0.5B, λN* const4,8004,650 (32)1.03PASS
Power η=2e-3B, ηN* const4,0002,900 (32)1.38within 2×
MNIST AdamW λ=0.006B, 1/N* linear in λ72,43169,125 (16)1.05PASS
MNIST AdamW λ=0.02B40,91539,875 (16)1.03PASS
diag-net B=32B, batch-size null8,8958,480 (32)0.95PASS
diag-net λ=2e-3B13,34212,325 (32)1.08PASS
poly GD x_s=3e-6A, log law1,8961,900 (32)1.00PASS
mod-add (Nanda) init×2A, log law8,7765,200 (16)1.69within 2×
sparse parity B=256A, larger B slower38,12535,000 (16)1.09PASS (direction ✓)
Division λ=0.15B15,50014,700 (24)1.05PASS
Division λ=0.5B4,6504,500 (24)1.03PASS
Division λ=1B2,3252,550 (24)0.91PASS
Division η=2e-3B, ηN* const3,8752,700 (24)1.44within 2×

Totals: 10 of 13 PASS, 13 of 13 within 2×. The new task is modular division (x/y mod 97, Power et al.). It is a different operation and dataset from the subtraction task in the paper. On it, 3 of 4 predictions pass and all 4 are within 2×.


Q4 (coordinate).

As discussed above, the slow coordinate is not a free choice. It is the one on which the driving mechanism acts linearly (ln‖θ‖ for decoupled decay, the unstable-mode amplitude for a ridge), and our laws are derived on it. N* itself is defined by the readout, so it and its knob laws do not depend on the coordinate we use to describe the plateau. However, as you asked, we also ran the same outcome-free pipeline (no N) on a second coordinate per host, with results below:*

New Experiment Results. When both coordinates are valid, they agree. On MNIST AdamW (ln‖θ‖ vs test loss) both give regime B and forecast within 4% of each other (1.09 vs 1.04). On Power λ = 1 (ln‖θ‖ vs IPR) both give B (0.71 vs 0.66). When the alternative is not monotone over [tmemt_{mem}, tobst_{obs}], the label changes. Teacher alignment on the diagonal net gives C/C/A with forecasts of 0.26–0.51×, while ln‖θ‖ gives B on every cell with forecasts of 0.72–1.11×. A “C” here flags a non-monotone coordinate. No host is in regime C by its delay statistics (CV ≤ 0.42, power-law knobs).

So the rule is known before the outcome: use the mechanism’s coordinate, and require it to be monotone over the observation window, which is checkable before grokking. On this rule the paper’s coordinate gives stable labels on the decay hosts.

Q2 (gg and DmemD_{mem}/D0D_0 on real networks)

Additional Experiments. We measured both on 3 real networks. The reduction to one coordinate holds on the SGD hosts. On the AdamW hosts the memory correction is large and our earlier claim was wrong, but the delay laws we rely on do not depend on it.

What the reduction needs. Two conditions: the slow coordinate must be much slower than the fast modes (g ≫ 1), and the fast modes must add little extra diffusion (Dmem/D0D_{mem}/D_0 small, i.e. DeffD_{eff} ≈ D0D_0).

How we measured, on the real plateau. At 0.3 and 0.6 of the delay we freeze the exact training state (parameters and optimizer buffers) and launch 32 replicas that differ only in future minibatch order. The shared drift cancels, so the replica variance of ln‖θ‖ is its noise, which gives DeffD_{eff}. The lag correlation of the increments gives the fast timescale τfτ_f, and then g = (remaining delay)/τfτ_f. (An earlier estimate from single detrended trajectories depended on the window length, so we replaced it.) We used 2 seeds × 2 branch points per host.

HostgD_eff/D0Replica spread / drift displacement
diag-net, SGD600–10000.004–0.0112×10⁻⁵ – 4×10⁻⁴
MNIST-1k, SGD+clip50–20000.4–1.05~3×10⁻⁸
MNIST-1k, AdamW35–2901.4–100.7–1.4×10⁻³

Reading the table.

  • SGD hosts: both conditions hold. g is large, and D_eff is at most about D0 (the fast oscillations make the increments of ln‖θ‖ anti-correlated). So the memory term is not positive and the one-coordinate picture is valid as stated.
  • AdamW hosts: g is acceptable, and D_eff is well above D0 (momentum and the preconditioner make the increments positively correlated). Our claim that "D_eff reduces to the projected noise" doesn't hold strictly for AdamW, so we corrected §2.4 in the paper to restrict our claim.
  • However, the delay laws still hold. The decay-driven law N* = ℓ/v uses only the measured drift and contains no D. Noise would matter only if it were comparable to the drift, and the last column in the table shows it is not (at most 0.11 of the displacement, usually far less). So the transfers and the sealed predictions on these hosts do not rest on the value of D.
Reviewer_pgxEOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper formulates grokking as first-passage transport along an effective coordinate, hoping to reconcile existing explanations limits and accounts for diverse grokking phenomena observed in prior work. The authors derive a governing equation for the generalization delay and show a quantitative three regime classification, where each regime has a distinct delay law and seed-to-seed variability law. They further derive explicit delay laws for SGD and Adam-family optimizers by mapping their hyperparameters to the effective drift and diffusion and test experimentally across several previously studied grokking settings.

Critical strengths

The paper proposes a new framework (perspective) to understand grokking, unifying several existing explanations of the grokking phenomena. Then, it conducts thorough experiments via examining multiple grokking tasks, architectures and optimizer families.

Critical weaknesses
  1. I found the paper hard to follow. This is perhaps because I don't have enough background to understand the analysis presented in this paper. But I believe the paper also needs to be further polished.

  2. It seems that the main one-dimensional hypothesis is insufficiently justified. The main issue is that what makes it reasonable and indeed correct to assume that only one relevant slow coordinate remains during the grokking plateau. Furthermore, the analysis consists of several approximation arguments, requiring substantially more clarification and corrections.

Questions for authors
  1. The authors claim that "a unified quantitative framework for grokking is needed to compare these findings, predict grokking delays, and explain why different experiments exhibit different signatures." If I understand correctly, the paper has made a strong assumption that when training loss is small, the progress toward generalization is controlled almost only by a single slow dimension. For different grokking scenarios, this coordinate could be the parameter norm, amplitude of a learned feature, alignment with the teacher, etc. Hence, grokking can be explained as the movement of a slow coordinate. However, I don't see any strong argument/theory to show that this assumption is reasonable and widely-applicable. Note that empirically observing this in several previous studied grokking tasks does not suffice.

  2. After reading, I realized that I probably don't have enough background to understand the notations and equations stated in this paper. (I'd be happy to learn from them if other reviewers understand it.)

Here, I would like to simply state some examples of the math that confuse me in the paper.

(1) Regarding the Equation (1), what is an arrow notation inside a function map? Moreover, you later defined the function f(θ)f(\theta) as the expected number of steps to first reach θe\theta_{e} from θ\theta in Section 2.1.

(2) In Section 2.1, regarding c(θ)=Vari[∂θℓi(θ)]c(\theta)=\mathrm{Var}_{i}[\partial_{\theta}\ell_{i}(\theta)], per-example variance is the same if the data is i.i.d., right? Regarding f(θ)=1+E[f(θ+Δ)]f(\theta) = 1+\mathbb{E}[f(\theta+\Delta)], did you define Δ\Delta before? (I could only find Δt\Delta_{t}.) Moreover, it'd be better to specify that the expectation for the second summand is taken over the randomness of Δ∣θ\Delta|\theta. More seriously, I don't understand the footnote why the bias is negligible compared to the variance near a plateau. Can the authors explain it? What is the so-called "reflecting lower boundary" θa\theta_{a} and why f′(θa)=0f^{'}(\theta_{a})=0?

Moreover, the paper consists of a mixture of exact identities and approximations such as using ≈\approx, ≃\simeq, etc. Many terms in their analysis are claimed negligible and then neglected. Hence, I believe that the current version needs substantially more clarifications to make the results convincing.

7 comments
AuthorsOpenReview ↗

W2 & Q1 Clarifications About One-dimensional Analysis In the Paper

Thank you very much for your valuable feedback.

We want to clarify that we are not assuming there's physically only one single direction moving in parameter space. The actual movement in the parameter space can be complicated; other directions can keep jittering simultaneously. But those jitters are not the deciding factor of the clock. After the training loss is already close to zero, directions deviating from the manifold are quickly pulled back by curvature, while motion along the valley floor (which continues pushing the generalization readout forward) lags behind. The core idea of our paper is to find a way to quantify the grokking delay (in a rough approximation), and the only thing we care inside the actual complexity is the "bottleneck" direction that keeps holding up in a slow progression, until grokking finally reaches destination. When we grasp the slowest progressing direction (which is 1D) from this multi-dimensional movement, we grasp what ultimately contributes the delay. And as we discussed in Section 2.4, there can be multiple sections on the path of grokking (where each section has a different dominating direction). In this case, we can add up the delay calculated on each section to roughly quantify the entire delay.

Thus we'd like to justify and convince you that the assumption is reasonable and applies to different settings, as we're assuming clock is determined by the dominant direction, not assuming other directions are frozen. We discuss in the following 3 cases, which cover the vast majority of grokking cases in literature:

Weight decay. Decoupled decay changes ln⁡∥θ∥\ln\|\theta\| by exactly ln⁡(1−ηλwd)\ln(1-\eta\lambda_{\mathrm{wd}}) at every step, in any dimension. Every other direction enters through one measured number, the inner product of θ\theta with the preconditioned gradient. No separation of timescales is required. MNIST-1k, Power, and the diagonal net are in this class.

Ridges. Near an unstable point each mode grows as eλite^{\lambda_i t}. The first mode to reach the exit is the one with the largest rate, and the delay is the log law of that mode. If several modes are close, the delay is the minimum of several log laws, which has the same form. Modular addition, the polynomial, the cubic, and sparse parity are in this class.

After memorization, in general. Fast relaxation off the interpolation manifold and slow motion along it is a standard fact for SGD near a manifold of minimizers. The extra assumption is that the test readout depends on one function of the position along that manifold. The condition is measurable: the slow stage must be much longer than the correlation time of the fast jitter. Additional Experiment: We measured it by branching 32 replicas from a frozen checkpoint, identical except for the future minibatch order. On the two SGD hosts the slow stage is 10210^{2}–10310^{3} times that correlation time, and noise along ln⁡∥θ∥\ln\|\theta\| is negligible beside the drift. Thus the one-coordinate description holds as stated.

Our vision in this pilot study to provide some initial perspectives and insights on bringing quantitative analysis to grokking. And this is a step that every science needs to step onto before a complete picture: throughout the history of science, we rarely begin with exact formulas or a complete perfect theory that captures every detail; instead, people establish quantitative methods with practical approximations, which still offer real value to practitioners and become the key to building up today's engineering. This is the value we see in this study compared to existing qualitative grokking literature, and we hope our initial quantitative perspective offers a helpful foundation for both researchers and practitioners in the community.

AuthorsOpenReview ↗

W2 Clarifications for All Approximation Arguments Made in this paper

Here, we list all 6 places where approximations are involved in the analysis and clarify them. We also show how small the dropped piece is numerically on the runs in our paper.

1. One step is described by its mean and its variance. (Taylor Expansion) The parameter update in one step is replaced by a drift μ\mu (the mean step) and a diffusion DD (half the variance of the step). Every third-and-higher cumulant of the step is left out. Those cumulants are smaller than the variance by powers of the step size divided by the stage width, (η/(Bℓ))k−2(\eta/(B\ell))^{k-2}. On the runs, the excess kurtosis of a step has median at most 0.160.16, with one cell that needs care. Cutting at the second cumulant is also the only consistent finite cut: a cut at order three or higher can assign negative probability.

2. The μ2\mu^2 piece of the step variance is left out. The exact second moment of a step is 2D+μ22D+\mu^2. The equation for the mean delay keeps 2D2D. The dropped piece contributes 12μ2f′′\tfrac12\mu^2 f'', and this is smaller than the drift term μf′\mu f' by the size of one step divided by the width of the stage. On every noisy plateau that ratio equals Ψ/(2N∗)\Psi/(2N^*) and is at most 10−210^{-2}. On a full-batch run the diffusion is zero, so the same comparison is made against the drift term. (We correct footnote 1 to clarify more clearly.)

3. Diffusion is taken as constant on one stage. The exact delay integral still contains ln⁡(D/D0)\ln(D/D_0), the change in diffusion from the bottleneck value D0D_0. That change is small next to the change in Φ\Phi across the same stage. The ratio Δln⁡D/∣ΔΦ∣\Delta\ln D/|\Delta\Phi| is at most 1.1×10−31.1\times 10^{-3} on every host except sparse parity, where it is 0.0640.064. In regimes A and B the leading delay is the travel time ∫dθ/μ\int d\theta/\mu, which does not contain DD, so this error sits only in the noise correction. In regime C the barrier exponent uses the true D(θ)D(\theta); only the prefactor uses DD at the bottom of the well.

4. Near a bottleneck the drift is a straight line plus a slope. (Taylor Expansion) On one stage, μ(θ)=v+λ(θ−θ∗)\mu(\theta)=v+\lambda(\theta-\theta^*). Here vv is the drift at the bottleneck and λ\lambda is how fast the drift changes. The shape of the stage is the single number Ω=λℓ/v\Omega=\lambda\ell/v, the fractional change in the drift from entrance to exit. Three shapes follow. The slope stays about constant when ∣Ω∣≲1|\Omega|\lesssim 1. The slope steepens when Ω≫1\Omega\gg 1. The slope reverses before the exit when Ω<−1\Omega<-1. This description is local to one stage. When the shape changes inside a run, the run is a sequence of stages and the delays add.

5. After the training loss is small, the test readout moves with one slow coordinate. Directions that leave the set of interpolating solutions relax quickly. The slow coordinate is what remains, and the fast directions are averaged at fixed slow coordinate. The separation is g=λf,min⁡/∣λslow∣g=\lambda_{f,\min}/|\lambda_{\mathrm{slow}}|, using the slowest of the fast rates, because that rate leaves the largest memory. The diffusion of the slow coordinate is the projected minibatch noise plus that memory, Deff=D0+DmemD_{\mathrm{eff}}=D_0+D_{\mathrm{mem}}, and DmemD_{\mathrm{mem}} is of order R/gR/g when RR is order one. On the SGD hosts, gg is 10210^{2} to 10310^{3} and Deff/D0≤1D_{\mathrm{eff}}/D_0\le 1. On the AdamW hosts the memory term is larger than the projected noise. In regimes A and B the leading delay does not contain DD. Along ln⁡∥θ∥\ln\|\theta\| the spread between replicas is at most 1.4×10−31.4\times 10^{-3} of the drift on the diagonal net and on both MNIST hosts, and 0.060.06 to 0.110.11 on Power.

6. Adam’s buffers change the noise, and they do not change the drift that sets the delay. The buffers forget in tens to about a thousand steps, a small fraction of the escape time N∗N^*. Over that long time the first moment has unit gain, so β1\beta_1 does not rescale the average step μ\mu. The same comparison does not decide the noise. Increments of the slow coordinate stay correlated for as long as the buffer remembers, about 0.90.9 from one step to the next on the AdamW hosts, and the accumulated spread is D0+DmemD_0+D_{\mathrm{mem}} with DmemD_{\mathrm{mem}} larger than the one-step noise D0D_0. The Adam formulas give D0D_0. In regimes A and B the leading delay is the travel time at drift μ\mu, and on these hosts that drift remains larger than the spread.

Above we show that these approximations are practically reasonable with only minor gaps abstracted away. Importantly, these are what makes quantitative analysis of grokking delay realistic, since otherwise we'll have to deal with such a complicated parameter space and high-dimensional movements, which likely is the reason that keeps almost all existing grokking literature on the qualitative level. We hope that through this first quantitative pilot study with approximations, we can take a step closer to the underlying reality of grokking.

AuthorsOpenReview ↗

W1 & Q2 Improving Readability (We Hope Intuitions Provided Here Can Help With Understanding!)

Thanks a lot for this candid feedback. We indeed condensed the paper too much during writing (as we seriously ran out of space), which made it hard to explain everything in an intuitive and clear enough way. Also, although we presented every derivation directly from the mini-batch optimizer update rule and elementary probability theory without assuming any physical laws, most of these ideas have directly comparable intuitions from Thermodynamics and Statistical Mechanics, the actual place where the author initially gets the modeling inspiration & intuition. We would like to bridge the intuitions and correct the inconsistent notations in this reply, and we've updated the paper.

Intuition

Imagine the network’s parameters as a ball with millions of directions to move in, rolling in a very long canyon with steep walls. Across the canyon, the walls are steep, so the ball bounces to the floor within a few steps and then jitters around it. These are the fast modes, and in grokking this is memorization: fitting the training set. Along the canyon, the floor is nearly level, so progress is slow. This is the slow coordinate θ, and the test readout depends on how far along the canyon you are. To predict the arrival time you don’t need the million wall-bouncing coordinates, only the profile of the canyon floor and the jitter along it. Everything else has already averaged out.

In Statistical Mechanics, this is known as adiabatic elimination or a reaction coordinate. In a chemical reaction, thousands of atomic vibrations equilibrate in femtoseconds while “how far the reaction has progressed” evolves slowly. A gas is another example: collisions equilibrate in nanoseconds, so a few slow macroscopic variables (an order parameter, a temperature) are enough to describe the system. Our “fast modes relax to a conditional equilibrium at fixed θ” is equivalent to local equilibrium.

Back to our "ball moving in canyon analogy". The delay is the first-passage time down the canyon floor. The only question is how the push along the floor changes between entrance and exit, and that is exactly what Ω measures. Exhaustively, this creates 3 distinctive landscape shapes, corresponding to our Regime A, B, and C. We have an visual image of what each landscape intuitively looks like in Figure 2 (Left). On a ridge (Regime A), it's like a stone at the tip of a cliff; the push keeps growing. While it's slow to reach the exact falling point, it falls very suddenly once it reaches the point of falling. On a flat stratch (Regime B), it's moving with almost constant push / speed, like a leaf floating on a steady current. On a barrier (Regime C), the ball is in a bowl on a shaking table; the force that makes the ball possibly escape the bowl is the shake (noise), but the landscape prevents it from escaping, so it spends a very long time keep trying to escape via those random noise. This process is so long, and that's why Regime C is mathematically valid for exhaustive discussion, but only exists in some manually constructed grokking cases, not observed in natural grokking settings so far.

And the formulas in Table 1 (E[N*] Column) are just describing the distance-velocity-time relationship of the 3 possible movement processes described above. However, the driving forces of the movement (drift and diffusion) involved in the formulas are determined by optimizers. Thus, we need to derive the drift μ\mu and diffusion DD from optimizer update rules and hyperparameters, and then eventually get the formulas directly mapping optimizer type & hyperparameters to the delay time, i.e., Table 1 (SGD Column & AdamW Column).

Reviewer_SMsjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies grokking, proposing a single mechanism underlying the diverse observations of this phenomenon. Specifically, the authors use a Markov process framework to analyze the time it takes for the iterates to move from one point to another. Using this analysis, they identify three regimes with different behavior. They also consider different optimizers, linking their hyperparameters to the effective drift and diffusion of the Markov process.

Critical strengths
  • Useful separation of drift-driven and noise-activated delays
  • The identification of regimes with different behavior
  • Attempt to unify the previously observed phenomena under one framework
  • Connection of the optimizer's choice to measurable quantities
Critical weaknesses

The analysis holds for univariate problems, and there is no clear way to extend it. The single-coordinate extension suggested by the authors is not rigorous. The issue is that this analysis is formulated in terms of the expected time to move from one value to another in a Markov process. In the one-dimensional case, this time is finite (under reasonable assumptions). However, if the dimension is two or more, the expected value is infinite. Furthermore, in three dimensions or above, all states are transient, meaning that the process has zero probability of visiting the point. Therefore, the method is inherently restricted to univariate problems.

Questions for authors

Why is a plateau important in your story? It is known that optimizers, especially SGD, waste a lot of time just oscillating between the banks of a valley of minima, while hardly making any progress along the valley. Why is a plateau a better explanation for grokking than oscillations?

2 comments
AuthorsOpenReview ↗

Clarifications About One-dimensional Analysis (Addressing W1)

Thanks a lot for your valuable feedback.

"The expected hitting time is infinite for d ≥ 2, and states are transient for d ≥ 3"

We agree that for driftless Brownian motion on an unbounded ℝ^d, hitting a point has probability zero for d ≥ 2. For d ≥ 3 the process is transient, so even a ball is missed with positive probability. Without drift or confinement, the mean hitting time of any target is infinite. Actually this is already true in one dimension: driftless Brownian motion on the whole line also has infinite expected hitting time. So 1D finiteness already comes from drift (or from walls), not from the dimension. Eq. (3) has a finite value because the drift toward the exit is positive on the stage. That ingredient does not depend on dimension.

However, this does not apply to our target. In the paper, the first-passage target is never a point of parameter space. It is the set {θ : ξ(θ) ≥ ξ_e}, where ξ is a scalar observable (log weight norm, Fourier concentration, teacher alignment). Equivalently, it is the open set {test accuracy > 0.99}. This set has codimension 1 or positive measure, and it is hit in finite expected time under the drift we measure. We should have said this explicitly, and the revised §2 now does.

Clarifications On Why Reduction to 1D is Important for the Theory

We are not assuming there's physically only one single direction moving in parameter space. The actual movement in the parameter space can be complicated; other directions can keep jittering simultaneously. But those jitters are not the deciding factor of the clock. After the training loss is already close to zero, directions deviating from the manifold are quickly pulled back by curvature, while motion along the valley floor (which continues pushing the generalization readout forward) lags behind. The core idea of our paper is to find a way to quantify the grokking delay (in a rough approximation), and the only thing we care inside the actual complexity is the "bottleneck" direction that keeps holding up in a slow progression, until grokking finally reaches destination. When we grasp the slowest progressing direction (which is 1D) from this multi-dimensional movement, we grasp what ultimately contributes the delay. There can also be multiple sections on the path of grokking (each section has a different dominating direction, Sec 2.4). In this case, we can add up the delay calculated on each section to roughly quantify the entire delay.

In short, we're assuming clock is determined by the dominant direction, not assuming other directions are frozen. We'd like to convince you that the assumption is reasonable and applies to different settings. We discuss in the following 3 cases, which cover the vast majority of grokking cases in literature:

Weight decay. Decoupled decay changes ln⁡∥θ∥\ln\|\theta\| by exactly ln⁡(1−ηλwd)\ln(1-\eta\lambda_{\mathrm{wd}}) at every step, in any dimension. Every other direction enters through one measured number, the inner product of θ\theta with the preconditioned gradient. No separation of timescales is required. MNIST-1k, Power, and the diagonal net are in this class.

Ridges. Near an unstable point each mode grows as eλite^{\lambda_i t}. The first mode to reach the exit is the one with the largest rate, and the delay is the log law of that mode. If several modes are close, the delay is the minimum of several log laws, which has the same form. Modular addition, the polynomial, the cubic, and sparse parity are in this class.

After memorization, in general. Fast relaxation off the interpolation manifold and slow motion along it is a standard fact for SGD near a manifold of minimizers. The extra assumption is that the test readout depends on one function of the position along that manifold. The condition is measurable: the slow stage must be much longer than the correlation time of the fast jitter. Additional Experiment: We measured it by branching 32 replicas from a frozen checkpoint, identical except for the future minibatch order. On the two SGD hosts the slow stage is 10210^{2}–10310^{3} times that correlation time, and noise along ln⁡∥θ∥\ln\|\theta\| is negligible beside the drift. Thus the one-coordinate description holds as stated.

Our vision in this pilot study to provide some initial perspectives and insights on bringing quantitative analysis to grokking. And this is a step that every science needs to step onto before a complete picture: throughout the history of science, we rarely begin with exact formulas or a complete perfect theory that captures every detail; instead, people establish quantitative methods with practical approximations, which still offer real value to practitioners and become the key to building up today's engineering. This is the value we see in this study compared to existing qualitative grokking literature, and we hope our initial quantitative perspective offers a helpful foundation for both researchers and practitioners in the community.

AuthorsOpenReview ↗

Clarification About Physical Picture (Addressing Q1); We also Provide More Intuitions Here to Help wiht the Understanding of the Paper

"Why is a plateau a better explanation than oscillations along the valley?"

We do not see these as competing observations, and we have both plateu (corresponding to slow modes) and oscillations along the valley in our story. Bank-to-bank oscillation is what we call the fast modes. We measured it: increments of ln‖θ‖ are anti-correlated (period-2 on MNIST-1k with clipping), and the path length exceeds the net displacement by up to ~10³. The delay is nonetheless set by the drift along the valley. For example, λ_wd·N* is constant to 1% over a tenfold change in weight decay, and blind predictions of this scaling on new values were accurate to within 3–8%. Instability bursts do not time the transition. By “plateau” we mean only that we are at a bottleneck, and the test readout is flat, not that the parameters are stationary, and we have clarified this in the revised text.


To help the reviewer better understand the physical picture, we would like to bridge the intuitions in the reply. The paper feels hard to grasp, with a ton of derivations we have to condense into a few pages. However, though we presented every derivation directly from the mini-batch optimizer update rule and elementary probability theory without assuming any physical laws, most of these ideas have directly comparable intuitions from Thermodynamics and Statistical Mechanics, the actual place where the author initially gets the modeling inspiration & intuition:

Intuitions.

Imagine the network’s parameters as a ball with millions of directions to move in, rolling in a very long canyon with steep walls. Across the canyon, the walls are steep, so the ball bounces to the floor within a few steps and then jitters around it. These are the fast modes, and in grokking this is memorization: fitting the training set. Along the canyon, the floor is nearly level, so progress is slow. This is the slow coordinate θ, and the test readout depends on how far along the canyon you are. To predict the arrival time you don’t need the million wall-bouncing coordinates, only the profile of the canyon floor and the jitter along it. Everything else has already averaged out.

In Statistical Mechanics, this is known as adiabatic elimination or a reaction coordinate. In a chemical reaction, thousands of atomic vibrations equilibrate in femtoseconds while “how far the reaction has progressed” evolves slowly. A gas is another example: collisions equilibrate in nanoseconds, so a few slow macroscopic variables (an order parameter, a temperature) are enough to describe the system. Our “fast modes relax to a conditional equilibrium at fixed θ” is equivalent to local equilibrium.

Back to our "ball moving in canyon analogy". The delay is the first-passage time down the canyon floor. The only question is how the push along the floor changes between entrance and exit, and that is exactly what Ω measures. Exhaustively, this creates 3 distinctive landscape shapes, corresponding to our Regime A, B, and C. We have an visual image of what each landscape intuitively looks like in Figure 2 (Left). On a ridge (Regime A), it's like a stone at the tip of a cliff; the push keeps growing. While it's slow to reach the exact falling point, it falls very suddenly once it reaches the point of falling. On a flat stratch (Regime B), it's moving with almost constant push / speed, like a leaf floating on a steady current. On a barrier (Regime C), the ball is in a bowl on a shaking table; the force that makes the ball possibly escape the bowl is the shake (noise), but the landscape prevents it from escaping, so it spends a very long time keep trying to escape via those random noise. This process is so long, and that's why Regime C is mathematically valid for exhaustive discussion, but only exists in some manually constructed grokking cases, not observed in natural grokking settings so far.

And the formulas in Table 1 (E[N*] Column) are just describing the distance-velocity-time relationship of the 3 possible movement processes described above. However, the driving forces of the movement (drift and diffusion) involved in the formulas are determined by optimizers. Thus, we need to derive the drift μ\mu and diffusion DD from optimizer update rules and hyperparameters, and then eventually get the formulas directly mapping optimizer type & hyperparameters to the delay time, i.e., Table 1 (SGD Column & AdamW Column).

We hope this provides more intuitions to help the reviewers better understand this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.