- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies grokking by looking at how weight decay affects the residual error and the NTK in homogeneous networks trained with squared loss. The main idea is that residual error persists in weakly represented task directions and can drive further feature learning after training accuracy becomes high. By focusing on one task direction and assuming that the residual relaxes faster than the kernel evolves, the authors obtain a reduced model of these dynamics. Introducing a kernel-strength threshold for generalization then gives a grokking timescale approximately proportional to . They compare the predicted timing and learning-rate/weight-decay phase boundaries with experiments on modular addition using MLPs and a Transformer.
Critical strengths
I appreciate the idea of simplifying the coupled residual and kernel dynamics to obtain tractable analytical expressions. The learning-rate and weight-decay sweeps provide a systematic comparison with the predicted phase structure. I also appreciate the transformer experiments, which explore whether similar behavior persists beyond the homogeneous networks considered in the theory setting.
Critical weaknesses
- I find the paper difficult to follow, particularly when distinguishing exact results from the assumptions introduced to obtain the reduced dynamics. Making these steps more explicit would help clarify what the theory establishes.
- I would like to better understand how well the assumptions behind the reduced dynamics hold during training in practice. The analysis assumes that the selected task direction stays approximately aligned with a kernel eigenvector and evolves largely independently of the other directions. It also assumes that the coefficient , which controls how strongly the residual drives kernel growth, is positive and approximately constant. However, showing that changes slowly compared with the residual does not mean that it stays constant during the much longer period over which the kernel evolves and generalization improves. Could the authors check these assumptions directly, for example by measuring the alignment, interactions between modes, and variation of during the transition?
- I find the connection between kernel growth and generalization insufficiently explained. The authors describe how the kernel strength evolves, but then assume that generalization occurs once it reaches a certain threshold. It remains unclear to me why reaching this threshold should imply good performance on unseen examples. Could the authors better justify this connection and determine the threshold independently, rather than fitting it to the observed transition?
- Novelty: I would appreciate a clearer explanation of what is new compared with previous work connecting grokking to delayed feature learning, particularly Kumar et al. and Lyu et al. Which predictions distinguish the proposed mechanism from these earlier accounts?
- How does this weight-decay mechanism relate to grokking without weight decay? More generally, which conclusions do the authors expected to persist beyond full-batch GD with squared loss? I think discussing these questions would help place the results within a broader picture of grokking.
I'm willing to reconsider my score if these questions/concerns would be addressed by the authors.
Questions for authors
See above.
5 comments
Points 3,4
- The connection between kernel growth and generalization is central to our research paper. We will implement expository steps in our manuscript to make sure that this is more clear. To sketch the connection, we hope the next paragraphs will make the connection more clear:
- Shared Fourier decomposition: Because both training and held-out pairs originate from the same generator over , every seen and unseen sample decomposes into the same group Fourier basis, with ground-truth targets supported exclusively on matched-frequency modes:
- NTK basis of homogeneous networks: By Euler's identity (), a -homogeneous network in the adiabatic regime acts as an evolving kernel ridge machine . We can transform into the task aligned directions, which correspond to the Fourier Eigenbasis used to build seen and unseen examples. The eigenvalues of in the kernel ridge machine corresponding to the Fourier Features define how much each feature is used to predict a new sample. We can therefore decompose the error.
- Spectral error governance: Decomposing this kernel predictor into the aligned Fourier eigenbasis gives the corresponding fractional residual error along each mode:
- Transfer to unseen examples: Because unseen inputs follow the same generator in our setting and share the global Fourier characters across , amplifying reduces approximation error on held-out examples as well.
- Regarding the threshold, a held-out sample changes predicted class when its margin crosses zero, which depends jointly on all Fourier pairs and residual interference through the rest of the Features. Crucially, our predictions do not rely on fitting the exact threshold. In the revision, we will add a dedicated section to compare kernel machines with homogeneous networks and make the connection between generalization and kernel features more clear.
- We thank the reviewer for this question. While Kumar et al. (2024) and Lyu et al. (2024) establish the connection between grokking and delayed feature learning, our work differs in the mechanism it derives and the quantitative predictions it makes:
- Compared with Kumar et al. (2024): They characterize delayed feature learning primarily through initial NTK–target alignment and a network-output scaling parameter controlling laziness, including in the absence of regularization. We identify a complementary, explicit source of continued feature learning under weight decay: the persistent ridge residual is largest relative to the target in weak NTK directions (Theorem 1), and directly drives their subsequent evolution through the neural tangent hierarchy (Eq. 8). Unlike Kumar et al., this yields a dynamical competition between residual-driven spectral growth and weight-decay-induced NTK contraction.
- Compared with Lyu et al. (2024): They prove a sharp transition from early kernel predictors to late min-norm/max-margin solutions, occurring around in the large-initialization regime. Our analysis does not require this large-initialization limit and instead characterizes the intermediate evolution of individual task-aligned NTK strengths. This yields a mode-specific equilibrium (Eq. 11) and a critical decay beyond which the required spectral strength becomes unreachable (Eq. 16), neither of which is derived in their analysis.
- Distinguishing predictions: Beyond decay-dependent timing, which is not itself novel, our reduced dynamics predict the finite-training boundary (Eq. 15), its inverse-decay regime, and a logarithmic divergence near the mode-reachability cutoff (Appendix D.3). These jointly predict the geometry of the learning-rate–weight-decay phase diagram, tested in our MLP and Transformer sweeps (Figures 3–4). Unlike the earlier accounts, these predictions arise explicitly from task-aligned NTK evolution, conditional on the reduced-model assumptions and effective generalization threshold. We will improve the related works section in our revised version with these points.
Point 5
- Grokking without weight decay. Our mechanism is not intended as a necessary explanation for all grokking. Without weight decay, the persistent ridge residual in Theorem 1 disappears, but nonzero residuals can still drive feature learning through the NTK evolution equation (Eq. 8). Such learning can continue after training accuracy saturates, provided the training loss has not vanished. Our theory thus identifies a specific mechanism by which weight decay sustains post-fit feature learning, complementing accounts of grokking without regularization (e.g., Kumar et al., 2024).
- Beyond full-batch GD and squared loss. Appendix E establishes that residual-dependent NTK evolution remains exact for differentiable, non-homogeneous architectures under squared loss, although the weight-decay terms become architecture-dependent. Our Transformer experiments (Section 3.3) empirically recover similar phase structure and timing despite violating homogeneity. For other differentiable losses, the output-loss gradient replaces the squared-loss residual as the driver of NTK evolution. However, the persistent ridge-residual formula and resulting quantitative timescale need not carry over. Likewise, minibatch SGD introduces stochastic effects absent from our gradient-flow analysis. We therefore expect the residual/gradient-driven feature-learning mechanism to be more general than the specific quantitative predictions, whose extension requires additional assumptions.
We thank the reviewer for the constructive feedback, and address all five points below. We hope our responses resolve all remaining concerns and would be grateful if the reviewer could reconsider their score in light of these clarifications.
Combining H5 with Assumption H6 (an effective scalar threshold Λ summarizing held-out margin crossings) and first-order discretization (t∼NηT) yields the finite-training boundary in Corollary 1 (Eq. 15) alongside the reachability, fitting, and stability bounds (Eqs. 16–18).
To make this progression transparent in the manuscript, we will introduce numbered Assumption blocks (Assumptions 1–6) directly in Section 2 prior to each theoretical statement. We will also explicitly cite the active assumptions in the headers of Theorem 1, Theorem 2, Corollary 1, and Eqs. (16)–(18) to distinguish exact identities from reduced-model approximations at a glance. Finally, we will expand the Appendices B–D.
These experiments directly address all three requests and support the reduced dynamics as an increasingly appropriate approximation during the later stages of grokking.