acceptodds
Under review as a conference paper at ICLR 2027

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

Abstract

In grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NTK) regime to one in which task-relevant kernel eigendirections continue to evolve. We provide a quantitative theory for how this transition from lazy to rich learning can produce delayed generalization. For homogeneous networks trained with squared loss and weight decay, we show that a finite residual remains after memorization, with larger residual fractions in target components associated with smaller NTK eigenvalues. These residuals feed back into the dynamics of the NTK itself, and projecting the resulting dynamics onto task-relevant spectral directions yields a reduced system in which residual-driven kernel growth competes with weight decay. This system predicts that the grokking timescale is controlled by the product of learning rate and weight decay, that feature learning slows logarithmically near a critical decay above which task-aligned NTK structure can no longer support generalization, and that stronger decay can prevent fitting altogether. We test these predictions in modular addition. In a homogeneous MLP, task-aligned Fourier structure continues to emerge in the NTK after training accuracy has saturated, and an 8490-grid of trained networks across varying learning rate and weight decay recovers the predicted phase geometry and inverse-product scaling of the generalization time with learning rate and weight decay. A one-block Transformer shows similar macroscopic phase structure in a 4245-grid, as well as the same transition-time scaling despite violating exact homogeneity. Together, these results provide a mechanistic derivation connecting post-fit feature learning to both the onset of generalization and its phase structure in the learning rate and weight decay plane.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
neural tangent kernel, neural tangent hierarchy, feature learning, grokking, optimization dynamics, spectral dynamics, lazy-to-rich transition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dHnxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies grokking by looking at how weight decay affects the residual error and the NTK in homogeneous networks trained with squared loss. The main idea is that residual error persists in weakly represented task directions and can drive further feature learning after training accuracy becomes high. By focusing on one task direction and assuming that the residual relaxes faster than the kernel evolves, the authors obtain a reduced model of these dynamics. Introducing a kernel-strength threshold for generalization then gives a grokking timescale approximately proportional to 1/(ηλW)1/(\eta \lambda_W). They compare the predicted timing and learning-rate/weight-decay phase boundaries with experiments on modular addition using MLPs and a Transformer.

Critical strengths

I appreciate the idea of simplifying the coupled residual and kernel dynamics to obtain tractable analytical expressions. The learning-rate and weight-decay sweeps provide a systematic comparison with the predicted phase structure. I also appreciate the transformer experiments, which explore whether similar behavior persists beyond the homogeneous networks considered in the theory setting.

Critical weaknesses
  1. I find the paper difficult to follow, particularly when distinguishing exact results from the assumptions introduced to obtain the reduced dynamics. Making these steps more explicit would help clarify what the theory establishes.
  2. I would like to better understand how well the assumptions behind the reduced dynamics hold during training in practice. The analysis assumes that the selected task direction stays approximately aligned with a kernel eigenvector and evolves largely independently of the other directions. It also assumes that the coefficient aa, which controls how strongly the residual drives kernel growth, is positive and approximately constant. However, showing that aa changes slowly compared with the residual does not mean that it stays constant during the much longer period over which the kernel evolves and generalization improves. Could the authors check these assumptions directly, for example by measuring the alignment, interactions between modes, and variation of aa during the transition?
  3. I find the connection between kernel growth and generalization insufficiently explained. The authors describe how the kernel strength evolves, but then assume that generalization occurs once it reaches a certain threshold. It remains unclear to me why reaching this threshold should imply good performance on unseen examples. Could the authors better justify this connection and determine the threshold independently, rather than fitting it to the observed transition?
  4. Novelty: I would appreciate a clearer explanation of what is new compared with previous work connecting grokking to delayed feature learning, particularly Kumar et al. and Lyu et al. Which predictions distinguish the proposed mechanism from these earlier accounts?
  5. How does this weight-decay mechanism relate to grokking without weight decay? More generally, which conclusions do the authors expected to persist beyond full-batch GD with squared loss? I think discussing these questions would help place the results within a broader picture of grokking.

I'm willing to reconsider my score if these questions/concerns would be addressed by the authors.

Questions for authors

See above.

5 comments
AuthorsOpenReview ↗

We thank the reviewer for the constructive feedback, and address all five points below. We hope our responses resolve all remaining concerns and would be grateful if the reviewer could reconsider their score in light of these clarifications.

  1. We will improve clarity by adding assumptions directly to theoretical result statements in the main text, while also enumerating them in the appendix of the manuscript and being more descriptive about individual steps taken. The assumptions we use are:
  • Exact dynamics (Proposition 1; Theorem 1, Eq. 5; Eq. 8): Under continuous-time gradient flow on squared loss with coupled L2 weight decay, Proposition 1 requires only differentiability, whereas adding Assumption H1 (D-homogeneity) closes the equations to yield the exact residual dynamics (Theorem 1, Eq. 5) and, for D=2, the exact empirical NTK hierarchy (Eq. 8).
  • Fixed-kernel limit (Theorem 1, Eqs. 6–7): Adding Assumption H2 (frozen NTK during initial fitting) yields the exact ridge equilibrium σ∗=−DλWnorm(K+DλWnormI)−1y\boldsymbol{\sigma}_* = -D\lambda_W^{\mathrm{norm}}(K + D\lambda_W^{\mathrm{norm}} I)^{-1}\mathbf{y}, which we use for motivating our analysis.
  • Modal closure and stability (Theorem 2, Eqs. 10–11): Projecting Eq. (8) onto a task-aligned Fourier mode u introduces Assumption H3 (single-mode decoupling: Ku≃ΛuK\mathbf{u} \simeq \Lambda\mathbf{u}, σ≃σu\boldsymbol{\sigma} \simeq \sigma\mathbf{u}) and Assumption H4 (locally constant, positive third-order coupling: a(θ)=⟨∇θΛ,∇θqθ⟩≃a>0a(\boldsymbol{\theta}) = \langle \nabla_{\boldsymbol{\theta}}\Lambda, \nabla_{\boldsymbol{\theta}} q_{\boldsymbol{\theta}} \rangle \simeq a > 0). Given the resulting reduced system (Eq. 10) under H1, H3, and H4, Theorem 2 provides an exact characterization of its unique non-negative equilibrium (Λ∗,σ∗)(\Lambda_*, \sigma_*) and local asymptotic stability.
  • Slow timescale and phase boundaries (Corollary 1, Eqs. 12–18): Assumption H5 (adiabatic timescale separation) reduces Eq. (10) to the slow mode equation on the normalized timescale.

Combining H5 with Assumption H6 (an effective scalar threshold Λ\Lambda summarizing held-out margin crossings) and first-order discretization (t∼NηTt\sim N\eta T) yields the finite-training boundary in Corollary 1 (Eq. 15) alongside the reachability, fitting, and stability bounds (Eqs. 16–18).

To make this progression transparent in the manuscript, we will introduce numbered Assumption blocks (Assumptions 1–6) directly in Section 2 prior to each theoretical statement. We will also explicitly cite the active assumptions in the headers of Theorem 1, Theorem 2, Corollary 1, and Eqs. (16)–(18) to distinguish exact identities from reduced-model approximations at a glance. Finally, we will expand the Appendices B–D.

  1. We trained eight additional grokking networks using the same experimental setup as our MLP phase diagram, with different combinations of η\eta and λW\lambda_W. The referenced figures can be found here (Figures):
  • Eigenvector alignment, Figure 2(d). For each task-aligned Fourier direction uiu_i, we compute its NTK strength Λi=ui∗Kui\Lambda_i=u_i^*Ku_i and the full-space eigenvector residual ri=∥Kui−Λiui∥2∥Kui∥2.r_i=\frac{\lVert Ku_i-\Lambda_i u_i\rVert_2}{\lVert Ku_i\rVert_2}. This quantity vanishes for an exact NTK eigenvector. The median residual decreases from 0.945 at initialization to 0.936 at training fit, 0.674 shortly before generalization, and 0.521 afterward. This indicates that the one-mode approximation is more accurate later in the transition than at its onset.
  • Interactions between modes, Figure 2(a–c). We measure the normalized pairwise NTK coupling Cij=∣ui∗Kuj∣ΛiΛjC_{ij}=\frac{|u_i^*Ku_j|}{\sqrt{\Lambda_i\Lambda_j}} between the 22 task-aligned Fourier directions un,nu_{n,n}, for n=1,…,22n=1,\ldots,22. Here, Cij=0C_{ij}=0 indicates no coupling under KK, while Cii=1C_{ii}=1. The heatmaps show the eight-run averages at initialization, training fit, and post-generalization. Their pronounced diagonal structure indicates weak coupling between most distinct task-aligned modes. Together with the improving alignment in panel (d), these results support progressively stronger spectral organization and mode decoupling.
  • In Figure 3, coupling coefficient aa. Across all eight configurations, aka_k remains positive at the sampled checkpoints and evolves smoothly, without abrupt fluctuations or sign reversals. Its gradual evolution supports treating aa as locally approximately constant over the faster residual-relaxation timescale, as motivated in Appendix C.1, while allowing its value to change over the longer grokking interval.
  • Figure 1 provides complementary evidence for continued feature learning. We measure the task-aligned NTK strengths Λn,n(t)=un,n∗K(t)un,n\Lambda_{n,n}(t)=u_{n,n}^*K(t)u_{n,n} throughout training. These strengths continue evolving after memorization, demonstrating ongoing task-aligned tangent-feature development during the grokking delay.

These experiments directly address all three requests and support the reduced dynamics as an increasingly appropriate approximation during the later stages of grokking.

AuthorsOpenReview ↗

Points 3,4

  1. The connection between kernel growth and generalization is central to our research paper. We will implement expository steps in our manuscript to make sure that this is more clear. To sketch the connection, we hope the next paragraphs will make the connection more clear:
  • Shared Fourier decomposition: Because both training and held-out pairs originate from the same generator over Zp×ZpZ_p\times Z_p, every seen and unseen sample decomposes into the same group Fourier basis, with ground-truth targets supported exclusively on matched-frequency (k,k)(k,k) modes: yc(a,b)=1p∑k=0p−1e−2πikc/pχk(a)χk(b).y_c(a,b) = \frac{1}{p}\sum_{k=0}^{p-1} e^{-2\pi i k c / p}\chi_k(a)\chi_k(b).
  • NTK basis of homogeneous networks: By Euler's identity (f=1DJθ\mathbf{f} = \frac{1}{D}J\boldsymbol{\theta}), a DD-homogeneous network in the adiabatic regime acts as an evolving kernel ridge machine f≃(K+DλWnormI)−1Ky\mathbf{f} \simeq (K + D\lambda_W^{\mathrm{norm}} I)^{-1}K\mathbf{y}. We can transform into the task aligned directions, which correspond to the Fourier Eigenbasis used to build seen and unseen examples. The eigenvalues of KK in the kernel ridge machine corresponding to the Fourier Features define how much each feature is used to predict a new sample. We can therefore decompose the error.
  • Spectral error governance: Decomposing this kernel predictor into the aligned Fourier eigenbasis gives the corresponding fractional residual error along each mode: ∣σad(Λk)∣yk=2λWnormΛk+2λWnorm.\frac{|\sigma_{\mathrm{ad}}(\Lambda_k)|}{y_k} = \frac{2\lambda_W^{\mathrm{norm}}}{\Lambda_k + 2\lambda_W^{\mathrm{norm}}}.
  • Transfer to unseen examples: Because unseen inputs follow the same generator in our setting and share the global (k,k)(k,k) Fourier characters across Zp×Zp\mathbb{Z}_p \times \mathbb{Z}_p, amplifying Λk\Lambda_k reduces approximation error on held-out examples as well.
  • Regarding the threshold, a held-out sample changes predicted class when its margin crosses zero, which depends jointly on all p−12\frac{p-1}{2} (k,k)(k,k) Fourier pairs and residual interference through the rest of the Features. Crucially, our predictions do not rely on fitting the exact threshold. In the revision, we will add a dedicated section to compare kernel machines with homogeneous networks and make the connection between generalization and kernel features more clear.
  1. We thank the reviewer for this question. While Kumar et al. (2024) and Lyu et al. (2024) establish the connection between grokking and delayed feature learning, our work differs in the mechanism it derives and the quantitative predictions it makes:
  • Compared with Kumar et al. (2024): They characterize delayed feature learning primarily through initial NTK–target alignment and a network-output scaling parameter controlling laziness, including in the absence of regularization. We identify a complementary, explicit source of continued feature learning under weight decay: the persistent ridge residual is largest relative to the target in weak NTK directions (Theorem 1), and directly drives their subsequent evolution through the neural tangent hierarchy (Eq. 8). Unlike Kumar et al., this yields a dynamical competition between residual-driven spectral growth and weight-decay-induced NTK contraction.
  • Compared with Lyu et al. (2024): They prove a sharp transition from early kernel predictors to late min-norm/max-margin solutions, occurring around t∼λW−1log⁡αt\sim\lambda_W^{-1}\log\alpha in the large-initialization regime. Our analysis does not require this large-initialization limit and instead characterizes the intermediate evolution of individual task-aligned NTK strengths. This yields a mode-specific equilibrium Λ∗\Lambda^{*} (Eq. 11) and a critical decay λWmode\lambda_W^{\mathrm{mode}} beyond which the required spectral strength becomes unreachable (Eq. 16), neither of which is derived in their analysis.
  • Distinguishing predictions: Beyond decay-dependent timing, which is not itself novel, our reduced dynamics predict the finite-training boundary η∗(λW)\eta^*(\lambda_W) (Eq. 15), its inverse-decay regime, and a logarithmic divergence near the mode-reachability cutoff (Appendix D.3). These jointly predict the geometry of the learning-rate–weight-decay phase diagram, tested in our MLP and Transformer sweeps (Figures 3–4). Unlike the earlier accounts, these predictions arise explicitly from task-aligned NTK evolution, conditional on the reduced-model assumptions and effective generalization threshold. We will improve the related works section in our revised version with these points.
  • Grokking without weight decay. Our mechanism is not intended as a necessary explanation for all grokking. Without weight decay, the persistent ridge residual in Theorem 1 disappears, but nonzero residuals can still drive feature learning through the NTK evolution equation (Eq. 8). Such learning can continue after training accuracy saturates, provided the training loss has not vanished. Our theory thus identifies a specific mechanism by which weight decay sustains post-fit feature learning, complementing accounts of grokking without regularization (e.g., Kumar et al., 2024).
  • Beyond full-batch GD and squared loss. Appendix E establishes that residual-dependent NTK evolution remains exact for differentiable, non-homogeneous architectures under squared loss, although the weight-decay terms become architecture-dependent. Our Transformer experiments (Section 3.3) empirically recover similar phase structure and (ηλW)−1(\eta\lambda_W)^{-1} timing despite violating homogeneity. For other differentiable losses, the output-loss gradient replaces the squared-loss residual as the driver of NTK evolution. However, the persistent ridge-residual formula and resulting quantitative timescale need not carry over. Likewise, minibatch SGD introduces stochastic effects absent from our gradient-flow analysis. We therefore expect the residual/gradient-driven feature-learning mechanism to be more general than the specific quantitative predictions, whose extension requires additional assumptions.
Reviewer_iA2YOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper aims to provide a quantitative theory explaining how the transition from lazy to rich learning can lead to delayed generalization (grokking). The authors derive residual and NTK dynamics under weight decay, introduce a reduced dynamical system, and use it to predict grokking timescales and phase boundaries. The theoretical predictions are compared with experiments on MLPs and Transformers.

Critical strengths

The paper addresses an interesting and important problem that how grokking depends on the the transition from lazy to rich learning. The predictions are tested on homogeneous MLPs and Transformers.

Critical weaknesses

My main concern is that the theoretical analysis does not establish a clear connection to grokking or delayed generalization, which is the central claim of the paper.

Specifically:

  1. Theorem 1 derives the residual dynamics for homogeneous networks and characterizes the residual under a fixed NTK. However, this result does not directly concern generalization.

  2. Theorem 2 characterizes a reduced NTK dynamical system under additional assumptions. However, it is unclear how this result establishes grokking. In particular, the theorem does not address generalization or the separation between training and generalization timescales.

  3. Corollary 1 introduces a so-called "finite-training grokking boundary". However, I do not find the connection between this boundary and actual grokking sufficiently justified. Corollary 1 appears to only characterize the time required for an approximate system to reach a prescribed threshold.

Additional Comments

  1. The assumptions should be stated more precisely and formulated explicitly in mathematical terms.

  2. The problem setting should be presented more clearly. The relationship between the original neural network dynamics, the reduced NTK dynamics, and held-out performance should also be clarified.

Questions for authors

See weakness.

4 comments

We thank the reviewer for the comments regarding positioning and clarity of the manuscript. As the other reviewers also mentioned the issues raised in the additional comments, we address

  1. stating assumptions directly, when they are needed in the main text as well as enumerating them in the appendix will help with clarity.
  2. We are also adding expository and motivational sections in each of the theoretical statements when needed.

We believe that these additions will also reconcile some of the main concerns raised by Reviewer iA2Y. Nevertheless, we directly answer all of the three concerns here. As a summary, we consider homogeneous neural networks, which obey the Euler equation. This allows us to rewrite the output of such a network as a kernel machine. A kernel machine linearly extrapolates a distinct feature set evaluated on the training set to form a prediction. In our case these features correspond to Fourier components of the target. As we can decompose the held out set into individual fourier components, we can directly compute the generalization error, by following the expression of fourier features in the network’s kernel. This allows us to transfer metrics extracted during training and make predictions about evaluation data and therefore generalization. In regards to the three main concerns:

  1. The residual under a fixed NTK does not directly characterize generalization. However projection of the kernel onto the fourier components (defined in equation (2)) characterizes the generalization residual
  2. Once we understand that the generalization error is a function of the fourier components of the NTK, we can study the dynamics of the NTK components related to generalization ((n,n) fourier modes) and training (all other modes). These then define the training and grokking time scale.
  3. Grokking is connected to the Generalization time scale, which occurs when the fourier modes of the NTK related to (n,n) frequencies reach a threshold. To compute the time scale needed to reach this boundary we integrate our equations of motion and compute the dependence of the learning rate on the hyperparameters of the system. Corollary 1 therefore analytically establishes the phase space boundary, we observe in Figure 3.

We hope these points are able to clear up any remaining confusion about our paper, and that you might consider revisiting your assessment and updating your score accordingly.

Thank you for the clarification. However, my fundamental concern remains that grokking is inherently a phenomenon of delayed generalization, whereas the theoretical results in this paper characterize only the training residuals and the evolution of the NTK, without establishing a connection to generalization.

The authors argue that the Fourier decomposition of the target function allows the generalization error to be computed from the NTK. However, I could not find a mathematical derivation establishing this connection. If such a connection can indeed be established, I encourage the authors to provide an explicit derivation in the revised manuscript. Otherwise, the theoretical results do not establish delayed generalization or a separation between training and generalization timescales, which is the defining characteristic of grokking.

AuthorsOpenReview ↗

Explicit connection

We thank the reviewer for this clarification. We agree that the connection to held-out generalization should be stated more explicitly. This connection follows through a short extension of our existing derivation, which we outline below.

First, extending Eq. (4) to held-out outputs fHf_H, the same homogeneity identity JHθ=DfHJ_H\theta=Df_H used in Theorem 1 gives

f˙H=−KHTσ−DλWnormfH,\dot f_H=-K_{HT}\sigma-D\lambda_W^{\mathrm{norm}}f_H,

where KHT=JHJT⊤K_{HT}=J_HJ_T^\top is the cross-NTK between held-out and training examples. This identity is exact under our assumptions and explicitly connects the training residual to held-out predictions.

Next, consider corresponding training and held-out Fourier directions uk,vku_k,v_k motivated by Eq. (2). Extending the approximate modal decoupling of Appendix C.1 to the cross-NTK, assume vk⊤KHT≃ckΛkuk⊤v_k^\top K_{HT}\simeq c_k\Lambda_k u_k^\top, with time-independent normalization ckc_k. Writing ak=uk⊤fTa_k=u_k^\top f_T and bk=vk⊤fHb_k=v_k^\top f_H, Eq. (5) then gives, for our two-homogeneous MLP,

-2\lambda_W^{\mathrm{norm}}(b_k-c_ka_k).$$ Thus, the mismatch between corresponding training and held-out Fourier coefficients decays exponentially. Combining this with the adiabatic gain already derived in Eq. (14) gives $$b_k(t)\simeq c_kg_k(t)y_k+ e^{-2\lambda_W^{\mathrm{norm}}(t-t_0)} [b_k(t_0)-c_ka_k(t_0)],$$ where $g_k(t)=\Lambda_k(t)/(\Lambda_k(t)+2\lambda_W^{\mathrm{norm}})$. This explicitly relates the evolving NTK Fourier modes to held-out predictions. In particular, using the Fourier decomposition of Eq. (2), the held-out correct-class margin against class $c_*+d$ becomes $$\Delta_d(t)\simeq\frac{1}{p}\sum_{k=1}^{p-1} \tilde g_k(t)\left[1-\cos\left(\frac{2\pi kd}{p}\right)\right] +\delta_d(t),$$ where $\tilde g_k=c_kg_k$, assuming conjugate modes have matching real gains, and $\delta_d$ collects the remaining contributions, including the initial mismatch. This follows by subtracting the Fourier expansions of the correct and competing logits: each frequency contributes $1-e^{-2\pi ikd/p}$, whose real part is $1-\cos(2\pi kd/p)$. Therefore, increasing task-aligned NTK strengths can drive held-out margins across zero once their Fourier contribution exceeds the remaining terms. This provides an explicit mathematical connection between the spectral dynamics and the held-out accuracy thresholds discussed in Appendix D.1, and motivates the effective generalization threshold $\Lambda_g$ in Section 2.3. Finally, Eq. (12) describes the post-fit evolution of these strengths, while Eq. (13) establishes the slow time $\tau\simeq\eta\lambda_Ws$. For a generalization threshold reached after an approximately fixed slow-time increment, this yields the predicted delay $s_{\mathrm{grok}}-s_{\mathrm{fit}}\propto(\eta\lambda_W)^{-1}$, consistent with Figure 3. We will incorporate this short extension into the manuscript. The residual dynamics, spectral evolution, and timescale separation are already established in the manuscript. This addition makes their conditional connection to held-out generalization mathematically explicit. We hope this addresses the concern and that you might consider revisiting your assessment and updating your score accordingly. We are also open to further discussion to clarify remaining issues.

I appreciate the authors' additional clarification. However, their response introduces a new, rather strong assumption that is not stated in the manuscript. Moreover, even under this assumption, the resulting expression does not establish delayed generalization. I encourage the authors to revise the manuscript and provide a more explicit derivation demonstrating how delayed generalization follows from their theoretical framework.

Nevertheless, in light of the clarifications provided, I have decided to raise my score to 2.

Reviewer_YeBPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies grokking through the evolution of the NTK. The main idea is that generalization requires sufficiently strong task-aligned directions in the NTK. If these directions are not sufficiently represented by the initial NTK, further feature learning may be needed for generalization. The authors show that weight-decay leaves non-zero residual errors, particularly along directions associated with small NTK eigenvalues. These residuals can drive further changes in the NTK and potentially increase the strength of task-aligned directions until they become sufficient for generalization. Although this evolution happens smoothly, classification accuracy can increase suddenly as many examples cross their decision boundaries around the same time. Based on a simplified model of these dynamics, the authors derive predictions for the grokking timescale and the phase boundaries in the learning rate-weight decay plane. They empirically test their hypothesis on MLPs and Transformers on modular addition benchmarks.

Critical strengths

NTK has turned out to be quite useful in understanding many phenomena surrounding neural networks. The main strength of the paper is that it uses this nice mathematical framework to provide an explanation of how weight decay can contribute to grokking through residual errors and continued feature learning. The optimization dynamics analysis is interesting, and the resulting quantitative predictions about the grokking timescale and phase boundaries in the learning-rate/weight-decay plane are particularly nice. I also find the connection between the evolution of task-aligned NTK directions and the emergence of generalization interesting, even though this connection relies on additional assumptions. Finally, despite being conducted on relatively simple datasets, the experiments are quite extensive, covering large hyperparameter sweeps on both MLPs and Transformers, and provide good support for the predicted scaling laws and phase structure.

Critical weaknesses

I found parts of the arguments difficult to follow, particularly the transition from the optimization dynamics to generalization. The analysis of the optimization dynamics is reasonably clear, modulo its assumptions, but the connection to grokking is less convincing to me. In particular, I think a missing theoretical link is an explanation of how the initial NTK can support near-perfect fitting of the training data without generalizing, and how the subsequent increase in task-aligned NTK strength resolves this discrepancy. In my understanding, the analysis establishes that weight decay can leave persistent residuals, especially in directions associated with small NTK eigenvalues, but it does not establish that the network has already successfully fitted the training data at this stage. Moreover, in the grokking setting (e.g., Nanda et al., 2023), training loss becomes very small well before generalization occurs. In the present formulation, however, the residual can remain substantial before the generalization transition. While it sounds like the authors use training accuracy rather than near-zero training loss as their fitting criterion, the analysis does not explicitly derive even this early high accuracy regime. My understanding is therefore that the current theory primarily explains possible subsequent feature evolution, conditional on an initial fitting stage having occurred, rather than explaining the complete grokking phenomenon from initialization.

The readability of the paper could also be improved. Many important details, definitions, and assumptions are postponed to the appendices, requiring the reader to constantly refer back to them to follow the main arguments. I believe the main text should be sufficiently self-contained to understand the central analysis, with only very technical or secondary details left to the appendices.

Questions for authors
  • Could the authors clarify the concern above regarding the connection between the optimization dynamics and generalization? In particular, how does the theory establish the initial high-training-accuracy, low-test-accuracy regime, and how does the subsequent NTK evolution resolve this difference?

  • I am not sure I agree with the statement in L147–148. Why does a model necessarily have to learn such a Fourier decomposition of the task? This seems to be one possible way to learn modular addition, but not necessarily the only one.

  • The assumption that a>0 seems to be quite important for the analysis. Is this checked empirically? Also, I am not sure I fully understand the argument for the slow evolution of a.

3 comments

We thank the reviewer for taking the time to carefully read our paper and for the questions and positive description of the strengths of our paper. Below we address all 3 points: 1. We clarify the connection:

  • i) Initial training fit without generalization. During the initial approximately fixed-NTK regime, the network can reach high training accuracy while retaining a nonzero squared-loss residual. Theorem 1 shows that, under weight decay, each NTK eigendirection retains a residual fraction ∣σk,∗∣∣yk∣=DλWnormΛk+DλWnorm.\frac{|\sigma_{k,*}|}{|y_k|}=\frac{D\lambda_W^{\mathrm{norm}}}{\Lambda_k+D\lambda_W^{\mathrm{norm}}}. Thus, directions with smaller NTK eigenvalues retain larger relative errors even after training classification accuracy saturates. In modular addition, the target-relevant directions are the matched-frequency Fourier components in Eq. (2). High training accuracy therefore need not imply that these globally useful features are already sufficiently developed to generalize to unseen input pairs.
  • ii) Why the residual drives subsequent feature learning. Crucially, the residual does not disappear after fitting. Equation (8) shows that it continues to affect the NTK through the neural tangent hierarchy: K˙=−K(2)σ−2λWnormK.\dot K=-K^{(2)}\sigma-2\lambda_W^{\mathrm{norm}}K. Under the task-aligned one-mode approximation, this yields Λ˙≃−aσ−2λWnormΛ\dot\Lambda\simeq-a\sigma-2\lambda_W^{\mathrm{norm}}\Lambda. For a>0a>0 and the negative post-fit residual, the residual-driven contribution increases task-aligned NTK strength. This provides a mechanism for continued feature learning after the training-accuracy plateau.
  • iii) How this improves generalization. Training and held-out inputs share the same modular-addition rule and its global Fourier decomposition. As task-aligned spectral structure develops across this domain, the model can increasingly express the Fourier components required for correct predictions on unseen inputs. In the adiabatic reduction, the gain of a task-aligned component increases with its NTK strength: g(Λ)=ΛΛ+2λWnorm.g(\Lambda)=\frac{\Lambda}{\Lambda+2\lambda_W^{\mathrm{norm}}}. Held-out accuracy changes when the resulting correct-class margins cross zero. Appendix D.1 (Eqs. 56–59) explains how smoothly evolving spectral strengths can produce an abrupt accuracy transition through these margin crossings.

Thus, early fitting can occur before globally transferable task-aligned features have developed, while persistent post-fit residuals drive the subsequent NTK evolution that enables held-out margin crossings. We will make this connection more explicit in the manuscript.

  1. We thank the reviewer for this observation. We agree that Fourier-based computation is not the only possible implementation of modular addition. However, the assumption that networks learn Fourier-aligned features is well motivated, both mathematically and empirically. First, Eq. (2) gives the exact Fourier decomposition of the one-hot modular-addition target, involving only matched-frequency components k(a)k(b). More importantly, Nanda et al. (2023) reverse-engineered Fourier-based circuits learned by grokking Transformers and demonstrated their causal importance through frequency ablations. Li et al. (2025) further established Fourier-aligned features theoretically for stylized neural networks solving modular addition. Consistent with these findings, our Figure 2 directly observes the emergence of Fourier-like structure in the NTK after training fit. Thus, our spectral reduction adopts a well-established and empirically supported learning mechanism, rather than asserting that every possible generalizing network must implement Fourier circuits.

  2. We now directly test the positivity and temporal evolution of a in an additional experiment (Figure 3).

  • Empirical verification. We evaluate the geometric coefficient ak(t)=⟨∇θΛk,∇θqk⟩a_k(t)=\langle\nabla_\theta\Lambda_k,\nabla_\theta q_k\rangle for 11 task-aligned Fourier modes across eight additional grokking configurations. Figure 3 shows that aka_k remains positive at all sampled post-fit checkpoints and evolves smoothly, without abrupt fluctuations or sign reversals. This directly supports the sign assumption underlying Eq. (10).
  • Why does a evolve slowly. Using two-homogeneity and the one-mode parameter approximation, Eq. (37) gives
\qquad b=\langle\nabla_\theta a,\nabla_\theta q_\theta\rangle.$$ Once the residual has relaxed to its instantaneous ridge value, Eq. (38) yields $$\dot a\simeq 2\lambda_W^{\mathrm{norm}} \left(\frac{by}{\Lambda+2\lambda_W^{\mathrm{norm}}}-a\right).$$ Thus, after residual relaxation, weight decay introduces no additional fast timescale for a. Its evolution is governed by the slower decay-controlled dynamics, provided the remaining geometric factors vary regularly. Figure 3 supports this timescale separation empirically.

Thanks to the authors for their response and additional experiments.

I think I understand the mechanism better now, but I am still not fully convinced that it addresses my main concern. I still do not see how the analysis establishes that the initial NTK can support high training accuracy without generalizing. In particular, Thm. 1 characterizes the remaining residuals but does not establish that the network has already achieved high training accuracy at this stage. Similarly, Eq. 14 describes how the gain of a task-aligned component increases with its NTK strength, but the connection to held-out predictions seems to rely on additional assumptions. My concern is therefore that the analysis explains possible subsequent feature evolution, but does not fully establish the initial optimization–generalization gap or how the subsequent evolution resolves it.

Regarding the assumption (a>0), I thank for the additional experiment supporting this assumption. However, I am not sure how Figure 3 of the submission (I tried to download the latest version) shows the evolution of aa. Doesn't Λ\Lambda also evolve on the same timescale according to Eq. 12? I am not entirely sure how this justifies assuming aa as constant during the evolution of Λ\Lambda.

About Fourier features, I believe either the authors agree with my comment that the Fourier decomposition is one possible way to learn modular addition, rather than a necessary one, or they maintain the original wording in L147–148, not sure I see a middle ground here.

Despite the remaining weaknesses, I maintain my original score and still weakly support accepting this paper at ICLR.

Please note that the new figure 3 can be found under the link that opens when clicking "(Figure 3)" in the rebuttal comment above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Score raise reviewer dHnx

We would like to point out that reviewer dHnx in his comment on the 19th of November, at 06:50 states: "I am raising my score to weak acceptance". However, the numerical score was not actually raised, and is currently left at 2, weak reject, instead of 3, weak accept. We kindly ask the conference chairs to take this into consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.