- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper says grokking is slow motion along one hidden coordinate. It builds a potential Φ = −∫ µ/D and splits cases into three regimes. The idea is interesting. But the evidence is weak. The main problem is circularity: the paper uses the true grokking time N* to define the coordinate range, the regime, and the prediction point. Then it reports good agreement with N*. This is not convincing prediction.
Critical strengths
Some concrete observations are valuable. Especially, full-batch Adam often sits at the ε floor and behaves like GD at η/ε and coupled L2 through Adam is not the same as AdamW decoupled decay, so weight decay is necessary in some settings but not others.
It tries to be quantitative. It predicts delay scalings rather than only giving another qualitative mechanism.
Critical weaknesses
N* is used to define key quantities such as ℓ, the profile estimator Ω(t)=v(N*)/v(t)−1, and even the forecast time tobs≈0.25N*. So the model often predicts the answer after seeing the answer.
The 1D reduction is not proven on real networks. Appendix D.9 effectively says the decisive test cannot be run on the real hosts. The clean validation is mostly on constructed toy anchors designed to satisfy the assumptions.
Different tasks use different slow coordinates, and when estimators disagree the chosen regime tends to be the one that matches the delay law. With non-grokking phases N1-N3 absorbing failures, the framework is hard to falsify.
Many physics-inspired quantities are introduced without plain-language explanation, and Φ becomes central before it is clearly defined. The appendix has unresolved equations and references, e.g., Pages 21, 34 etc. This makes the paper hard to read and obscures which quantities are derived or are assumed.
Questions for authors
- Can you choose the slow coordinate, readout threshold, ℓ, Ω estimator and prediction time before seeing N*, then report blind predictions on new hyperparameter, even ideally one new task?
- Can you estimate the timescale separation g and the memory correction D_mem/D0 on at least one real grokking model, not only on constructed anchors?
- What observation would satisfy the three regime map? If regime C never occurs in real grokking, why is it evidence for a unified theory of grokking?
- If I choose a different reasonable slow coordinate on the same model, e.g., ln‖θ‖ vs. embedding IPR vs. teacher alignment, do I get the same regime and the same N* law? If not, what objective rule selects the coordinate before the outcome is known?
7 comments
Q1, W3 & Q4: We don't choose slow coordinates, it's given by regime and formula automatically; (New Experiments) Blind predictions on new parameter, new task,
W3: Different Slow Coordinates Are Given Automatically, Not Chosen.
First, an important clarification: In our theory, the slow coordinate is intrinsic. We do not select/choose it when running experiments/predictions. It is the direction of slowest relaxation once the fast modes have equilibrated (§2.4). A scalar ξ(θ) is a valid description of it if it (i) tracks that slow direction, (ii) is monotone over the stage, and (iii) is something the readout responds to.
It is the delay formula who determines the coordinate in the following cases, depending on whether it's regime A or B:
Decay-driven hosts. Table 1 writes the law as . That form holds only on , where decoupled decay is an exactly constant drift (§3.1, App. E.5). So the formula identifies the coordinate.
Ridge hosts. The law is , with the offset from the ridge top along the unstable direction. The coordinate is the amplitude of that unstable mode, which is IPR for the modular-addition hosts, the learned-direction amplitude for the polynomial host, and alignment for cubic and parity.
Q1: Blind predictions on new parameter, new task.
New Experiment Results. Then we run 13 additional experiments with various new hyperparameter and new task settings, as demonstrated in the figure below. PASS means N̂/N in [0.75, 1.33], "within 2×" means [0.5, 2], and nulls PASS in [0.85, 1.18].
| Cell | Law | Predicted | Measured (seeds) | Ratio | Verdict |
|---|---|---|---|---|---|
| Power λ=0.5 | B, λN* const | 4,800 | 4,650 (32) | 1.03 | PASS |
| Power η=2e-3 | B, ηN* const | 4,000 | 2,900 (32) | 1.38 | within 2× |
| MNIST AdamW λ=0.006 | B, 1/N* linear in λ | 72,431 | 69,125 (16) | 1.05 | PASS |
| MNIST AdamW λ=0.02 | B | 40,915 | 39,875 (16) | 1.03 | PASS |
| diag-net B=32 | B, batch-size null | 8,895 | 8,480 (32) | 0.95 | PASS |
| diag-net λ=2e-3 | B | 13,342 | 12,325 (32) | 1.08 | PASS |
| poly GD x_s=3e-6 | A, log law | 1,896 | 1,900 (32) | 1.00 | PASS |
| mod-add (Nanda) init×2 | A, log law | 8,776 | 5,200 (16) | 1.69 | within 2× |
| sparse parity B=256 | A, larger B slower | 38,125 | 35,000 (16) | 1.09 | PASS (direction ✓) |
| Division λ=0.15 | B | 15,500 | 14,700 (24) | 1.05 | PASS |
| Division λ=0.5 | B | 4,650 | 4,500 (24) | 1.03 | PASS |
| Division λ=1 | B | 2,325 | 2,550 (24) | 0.91 | PASS |
| Division η=2e-3 | B, ηN* const | 3,875 | 2,700 (24) | 1.44 | within 2× |
Totals: 10 of 13 PASS, 13 of 13 within 2×. The new task is modular division (x/y mod 97, Power et al.). It is a different operation and dataset from the subtraction task in the paper. On it, 3 of 4 predictions pass and all 4 are within 2×.
Q4 (coordinate).
As discussed above, the slow coordinate is not a free choice. It is the one on which the driving mechanism acts linearly (ln‖θ‖ for decoupled decay, the unstable-mode amplitude for a ridge), and our laws are derived on it. N* itself is defined by the readout, so it and its knob laws do not depend on the coordinate we use to describe the plateau. However, as you asked, we also ran the same outcome-free pipeline (no N) on a second coordinate per host, with results below:*
New Experiment Results. When both coordinates are valid, they agree. On MNIST AdamW (ln‖θ‖ vs test loss) both give regime B and forecast within 4% of each other (1.09 vs 1.04). On Power λ = 1 (ln‖θ‖ vs IPR) both give B (0.71 vs 0.66).
When the alternative is not monotone over [, ], the label changes. Teacher alignment on the diagonal net gives C/C/A with forecasts of 0.26–0.51×, while ln‖θ‖ gives B on every cell with forecasts of 0.72–1.11×. A “C” here flags a non-monotone coordinate. No host is in regime C by its delay statistics (CV ≤ 0.42, power-law knobs).
So the rule is known before the outcome: use the mechanism’s coordinate, and require it to be monotone over the observation window, which is checkable before grokking. On this rule the paper’s coordinate gives stable labels on the decay hosts.
Q2
Q2 ( and / on real networks)
Additional Experiments. We measured both on 3 real networks. The reduction to one coordinate holds on the SGD hosts. On the AdamW hosts the memory correction is large and our earlier claim was wrong, but the delay laws we rely on do not depend on it.
What the reduction needs. Two conditions: the slow coordinate must be much slower than the fast modes (g ≫ 1), and the fast modes must add little extra diffusion ( small, i.e. ≈ ).
How we measured, on the real plateau. At 0.3 and 0.6 of the delay we freeze the exact training state (parameters and optimizer buffers) and launch 32 replicas that differ only in future minibatch order. The shared drift cancels, so the replica variance of ln‖θ‖ is its noise, which gives . The lag correlation of the increments gives the fast timescale , and then g = (remaining delay)/. (An earlier estimate from single detrended trajectories depended on the window length, so we replaced it.) We used 2 seeds × 2 branch points per host.
| Host | g | D_eff/D0 | Replica spread / drift displacement |
|---|---|---|---|
| diag-net, SGD | 600–1000 | 0.004–0.011 | 2×10⁻⁵ – 4×10⁻⁴ |
| MNIST-1k, SGD+clip | 50–2000 | 0.4–1.05 | ~3×10⁻⁸ |
| MNIST-1k, AdamW | 35–290 | 1.4–10 | 0.7–1.4×10⁻³ |
Reading the table.
- SGD hosts: both conditions hold. g is large, and D_eff is at most about D0 (the fast oscillations make the increments of ln‖θ‖ anti-correlated). So the memory term is not positive and the one-coordinate picture is valid as stated.
- AdamW hosts: g is acceptable, and D_eff is well above D0 (momentum and the preconditioner make the increments positively correlated). Our claim that "D_eff reduces to the projected noise" doesn't hold strictly for AdamW, so we corrected §2.4 in the paper to restrict our claim.
- However, the delay laws still hold. The decay-driven law N* = ℓ/v uses only the measured drift and contains no D. Noise would matter only if it were comparable to the drift, and the last column in the table shows it is not (at most 0.11 of the displacement, usually far less). So the transfers and the sealed predictions on these hosts do not rest on the value of D.
Circularity Concern: Main Prediction Results Never Affected; Remaining Affected Parts Now Fully Addressed by Reruns with Circularity Removed
We thank the reviewer for a careful and fair review. We address each question and revise the paper accordingly.
Important Question in Overview & W1 (Circularity)
Part of the circularity concern is correct, but we want to clarify that circularity isn't involved in any of our main prediction experiments and results (which include §4.3, Table 2, Figure 5 Top Row; §4.2, Fig. 4, etc.) Notably, The headline transfer results (Table 2, 2.6% median error) never use the target's N∗. Each prediction is either the source cell's delay rescaled by the law's ratio, or the source's constants plus the target's initialization offset at t = 0, while the regime label comes from the source cell. So the central object, the grokking delay's prediction is never affected by the circularity. Below, we clarify the places where N* circularity do exist, and we rerun experiments without N*:
To resolve these 2 circularities, we construct an
Outcome-free pipeline. We replaced each N*-dependent quantity with one that is observable before grokking:Reruns for (1): (Regime labels). 28 of 31 cells get the same label as in the paper.
Reruns for (2): Single-run forecasts without N in Appendix B.3* (median error factor, number of cells within 2×):
The old Appendix B.3 numbers were 1.9× for regime B and a median ratio of 0.29 for regime A. So the N dependence did not inflate the old results.* The B forecasts are biased slightly early. The A rows use dense panels whose times were scheduled as fractions of each cell's corpus delay. The estimator does not use N*, but the panel schedule was laid out with it, and we say so in the paper. We still consider cross-setting transfer the primary evidence (which is never affected by circularity, as discussed) and the single-run forecast a secondary, weaker result.
Thus, we fully address the circularity concerns here.