acceptodds
Under review as a conference paper at ICLR 2027

Understanding Learning Dynamics via Symmetry-Aware Spectra

Abstract

Overparameterized neural networks in certain settings have been observed to generalize long after perfect memorization, a phenomenon known as grokking. What drives this transition, and how to detect it during training, remains only partially understood. Common diagnostics are defined on the model weights, but parameter symmetries allow the same function to be represented by weight matrices of different norms and ranks. We study instead the singular-value spectrum of the parameter Jacobian , called the neural spectrum, as a symmetry aware measure of the locally accessible functional directions. We find that across different architectures, grokking is accompanied by spectral polarization: sensitivity becomes concentrated in a small effective subspace. Comparing standard, NTK, and maximal-update (P) parameterizations shows that the timing and form of this restructuring depend strongly on the feature-learning regime. Moreover, regularizing to encourage spectral concentration accelerates grokking, while encouraging a diffuse spectrum deters grokking. Beyond grokking, harder learnable tasks across four supervised-learning benchmarks generally produce broader, higher-rank spectra. These results support the neural spectrum as a common diagnostic of feature-learning dynamics and learned functional complexity.

Primary area:
optimization
Keywords:
parameter symmetry, feature learning, Jacobian, parameters, functions, grokking, task complexity, feature learning dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_1rVHOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper explores the spectrum of the neural network parameter Jacobian (the empirical-NTK spectrum) during training. Grokking coincides with "spectral polarisation", whose timing depends on the parameterisation. Regularising the spectrum speeds up or prevents grokking, and harder tasks produce higher-rank spectra.

Critical strengths
  • The authors have taken an interesting and very explanative approach to understanding neural network behaviour.
  • The study includes several relevant architectures, thus providing reasonable validation.
Critical weaknesses
  • The evolution of eNTK spectra and effective rank during training is well studied (e.g. Fort et al. 2020; Baratin et al. 2021; Tovey et al. 2025; Nikolaou et al. 2026), and grokking has been characterised as a lazy-to-rich transition (Kumar et al. 2024). Polarisation at grokking is close to what this work predicts.
  • Spectral polarisation means a few dominant eigenvalues, and hence eigenmodes, emerge during training. Under gradient flow the residual decays along these modes at rates set by their eigenvalues, so dominant modes correspond to the directions in which learning proceeds with the leading mode typically corresponds to the dataset mean. The paper does not connect its statistical measure back to these dynamics, so it remains unclear why polarisation should cause or signal generalisation and what this means for datasets that are not well represented by the mean. Is the top mode the trivial mean/uniform-logit direction, and do the results hold once it is removed?
  • It seems most experiment were run with only a few seeds.
Questions for authors
  • How would the analysis apply to strongly inhomogeneous datasets (e.g. multimodal classes or heavy class imbalance), where the data mean fits no class well and the leading eigenmode is therefore uninformative? Does polarisation still track generalisation in that case?
  • What would the neural spectrum look like in other learning paradigms such as reinforcement learning, where the targets are non-stationary? Could the polarising regulariser be used to accelerate training more generally, beyond the grokking setting?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1rVH (1/2)

Thank you for a careful review and for calling the approach “interesting and very explanative”; your question about the mean mode and class imbalance directly added results that we feel greatly strengthen the paper. We split JJ exactly into the softmax shift (your uniform-logit direction), the per-output mean directions (your mean modes) and an input-dependent part. With the first two removed the results hold (W2_c), and JJ’s energy still concentrates as the network groks under 32×32\times class imbalance (Q1).

W1 · Prior eNTK work

The evolution of eNTK spectra and effective rank during training is well studied (e.g. Fort et al. 2020; Baratin et al. 2021; Tovey et al. 2025; Nikolaou et al. 2026), and grokking has been characterised as a lazy-to-rich transition (Kumar et al. 2024). Polarization at grokking is close to what this work predicts.

As you note, lazy-to-rich theory predicts concentration once features are learned. These works do not show which part concentrates, a moving kernel that fails to grok, or a lever:

WorkWhat it showsWhat we add
Fort et al. 2020The kernel’s distance and velocity from initializationThe spectrum’s shape, part by part, through grokking
Baratin et al. 2021; Tovey et al. 2025The eNTK compresses while networks fitJJ’s input-dependent part changes as the network generalizes, long after the fit (−12-12%, against +0.6+0.6% on shuffled labels)
Nikolaou et al. 2026The eNTK’s effective rank as an effective dimension of learningWhich part of that dimension tracks generalization
Kumar et al. 2024Grokking as a late lazy-to-rich transitionUnder NTK parameterization the kernel moves far, yet the network never groks; penalizing JJ’s spectrum moves grokking both ways (4.8–7.4×7.4\times faster, or never)

The related work now includes the above comparison.

W2_a · Connecting the measure to the dynamics

Spectral polarisation means a few dominant eigenvalues, and hence eigenmodes, emerge during training. Under gradient flow the residual decays along these modes at rates set by their eigenvalues, so dominant modes correspond to the directions in which learning proceeds with the leading mode typically corresponds to the dataset mean. The paper does not connect its statistical measure back to these dynamics,

We tracked them directly. As you describe, the residual decays mode by mode: by memorization, 0.00 of it lies in the top 8 input-dependent directions and 0.01 in the top 99, so the plateau’s progress is not decay along the dominant modes. What changes over the plateau is the modes themselves, and on MNIST they come to carry the labels (W2_b). This is consistent with concurrent theory with weight decay, in which the residual left after fitting grows the kernel’s task-supported directions while the rest decay, concentrating the spectrum (Pracher et al. 2026); our grokking times follow its 1/(ηλ)1/(\eta\lambda) clock for decay ≥0.1\ge 0.1 (ηλtgrok\eta\lambda t_{\text{grok}} = 0.93–1.10 for decay 0.1–1, against 0.92 in their Transformer cohort).

W2_b · Why should polarization signal generalization?

…so it remains unclear why polarisation should cause or signal generalisation and what this means for datasets that are not well represented by the mean.

On MNIST the leading directions come to carry the labels: the label vector’s share in the top 10 eigenvectors of the mean-removed kernel rises from 0.05 at memorization to 0.26 at generalization and 0.36 at the end. And acting on JJ’s spectrum moves grokking: the penalty speeds it 7.4×7.4\times even with every weight tensor’s norm tied.

W2_c · Is the top mode the mean or the uniform logit?

Is the top mode the trivial mean/uniform-logit direction, and do the results hold once it is removed?

Often, yes, so we removed both exactly, and the results hold. Early in training the mean directions are the top 99 directions in all four architectures, holding 0.73–0.98 of JJ’s energy; late in training the shift takes 0.78 of the Transformer’s top 10 directions. With both removed, effective rank still falls 8–32% as the models grok, and penalizing only the input-dependent part still speeds up 17 of 18 seeds.

References

  • Baratin et al. 2021. Implicit regularization via neural feature alignment. AISTATS.
  • Fort et al. 2020. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. NeurIPS.
  • Kumar et al. 2024. Grokking as the transition from lazy to rich training dynamics. ICLR.
  • Nikolaou et al. 2026. Beyond scaling curves: internal dynamics of neural networks through the NTK lens. Mach. Learn.: Sci. Technol.
  • Pracher et al. 2026. A spectral theory of grokking: weight decay induces feature learning. arXiv.
  • Tovey et al. 2025. Collective variables of neural networks: empirical time evolution and scaling laws. Mach. Learn.: Sci. Technol.
AuthorsOpenReview ↗

Response to Reviewer 1rVH (2/2)

W3 · Seeds

It seems most experiment were run with only a few seeds.

Every experiment we report uses at least 3 seeds (5 for MNIST and parity), as did most of the paper’s (Fig. 2 and Tables 10–12 use 3, the complexity benchmarks 5); only Table 13 (2 seeds) and the widths 64 and 256 (one) did not. Our reruns reproduce Table 11 (4.8×4.8\times vs the submitted 5.2×5.2\times) and Table 13 (7.4×7.4\times vs 6.6×6.6\times from 2 seeds), and the penalty is faster in 18 of 18 seeds across architectures. In the three new tasks (table in G3), JJ’s change around generalization exceeds 1.5×1.5\times its shuffled-label baseline’s in every seed except one MNIST seed on the full JJ.

Q1 · Multimodal classes and heavy imbalance

How would the analysis apply to strongly inhomogeneous datasets (e.g. multimodal classes or heavy class imbalance), where the data mean fits no class well and the leading eigenmode is therefore uninformative? Does polarisation still track generalisation in that case?

JJ’s spectrum still tracks generalization in both cases, in its input-dependent part. With multimodal classes (sparse parity, where both classes share one input mean), JJ’s spectrum changes 10.4×10.4\times as much as on shuffled labels around generalization and concentrates after it. For heavy imbalance we trained the paper’s SP Transformer (decay 0.1) on the same 4,656 training pairs, repeating each class so that the commonest appears 32×32\times as often as the rarest (41,638 rows), and read generalization on class-balanced accuracy (3 seeds):

32×32\times imbalanceBalanced
Steps to memorization / to 0.95 class-balanced validation accuracy0.87k / 4.5±0.84.5 \pm 0.8k0.43k / 9.7±0.79.7 \pm 0.7k
Final accuracy: overall / rarest fifth of classes / worst class0.997 / 0.995 / 0.9721.000 / 1.000 / 0.980
Share of JJ’s top direction lying in the mean directions, init →\to grokking1.00 →\to 0.021.00 →\to 0.31
Share of JJ’s energy in the mean directions, init →\to memorization →\to grokking0.93 →\to 0.09 →\to 0.060.93 →\to 0.14 →\to 0.21
Full JJ’s effective rank, init →\to memorization →\to grokking3,042 →\to 5,807 →\to 5,385 (−7-7% in 3.6k steps)2,938 →\to 6,028 →\to 4,745 (−21-21% in 9.3k steps)
How the spectrum moves over the plateau’s first half →\to second halfspreads (2 of 3 seeds) →\to concentrates (3 of 3)concentrates →\to concentrates (3 of 3)
DD over the second half: full JJ / input-dependent part0.073 / 0.0800.146 / 0.126
DD over the same window, same network on shuffled labels—0.052 / 0.045

Under imbalance the top mode leaves the mean directions almost entirely (0.02 at grokking, against 0.31 balanced), and those directions hold only 0.06 of JJ’s energy, so the spectrum’s head is not the mean mode. The spectrum spreads over the plateau’s first half and concentrates over the second, as the network groks: L(k)L(k) of the input-dependent part rises by up to 0.080, against 0.045 for the same network on shuffled labels over a window 2.6×2.6\times longer. Effective rank falls less (−7-7% vs −21-21%) over a plateau 2.6×2.6\times shorter, and part of the balanced fall is the mean directions regaining energy (0.14 →\to 0.21).

Q2 · Beyond grokking

What would the neural spectrum look like in other learning paradigms such as reinforcement learning, where the targets are non-stationary? Could the polarising regulariser be used to accelerate training more generally, beyond the grokking setting?

Both are promising follow-ups and we thank the reviewer for their expert insight here. In particular, JJ needs only a differentiable model and a probe set and the penalty needs no labels, making it extremely general by construction, so both carry over to the reinforcement learning setting, though we have tested the penalty only where generalization lags fitting in the supervised case. However, we think JJ and the neural spectrum being related to functional and parameter space geometry means that it should carry over meaningfully in terms of its connection to learning dynamics, even in non-supervised settings beyond what the paper explicitly considers.

Thank you again for these questions; we would be glad to hear whether the results address them, and if so, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Common terms used in replies

Terms used in all our replies (copied from general response)

  • JJ: the Jacobian of the logits on a fixed probe batch with respect to all parameters (its squared singular values are the eNTK’s eigenvalues); effective rank: Roy & Vetterli’s (2007) soft count of its active directions.
  • Split of JJ into three orthogonal parts: the softmax shift (the same change to every logit of an input, invisible to the loss), the per-output mean directions (the same change to an output on every input), and the input-dependent part, the only part that can tell inputs apart.
  • L(k)L(k): the share of JJ’s energy ∑iσi2\sum_i \sigma_i^2 in its top kk directions. A window concentrates if L(k)L(k) rises at every kk and spreads if it falls at every kk; DD is the largest change of L(k)L(k).
  • Phases: memorization = first checkpoint at 99% training accuracy; grokking = first step at 95% validation accuracy; mid-plateau = their midpoint, which splits the plateau in half.
  • Shuffled labels: the same network, seed and data order on permuted labels; it memorizes but cannot generalize.
  • Speed-up: the unpenalized run’s steps to grokking over the penalized run’s (same seed); tied: every weight tensor’s norm reset to the unpenalized run’s after each step.
  • Orbit excess: (squared weight norm −- that of the least-norm point on its symmetry orbit) / the latter; 0% means balanced.
Reviewer_zUFXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes the neural spectrum, i.e., the singular values of the parameter Jacobian of the network outputs on a fixed probe set, as a diagnostic of training dynamics that is less sensitive to parameter symmetries than weight-based statistics. The main testbed is grokking on modular arithmetic, where the authors observe spectral polarization near the generalization transition across four architectures, with a timing that depends on whether the model is trained under SP, NTK or µP. They also add a regularizer on the spectrum and show that encouraging or suppressing polarization respectively accelerates or prevents grokking. Finally, the authors argue that the spectrum reflects functional complexity more generally, reporting higher spectral rank at convergence on harder supervised tasks.

Critical strengths

One of the main contributions is the characterization of grokking through a direct intervention on the spectrum, which provides insights into the role it might play in the transition. Additionally, the comparisons across different parameterizations, and their interpolation, give spectral polarization a clear interpretation in terms of the transition between lazy and rich regimes.

Critical weaknesses
  1. In section 4.5, the paper interprets higher rank on harder tasks as reflecting a more complex learned solution, while in several benchmarks the harder settings seem to be those where the model fits the target poorly, or where difficulty comes from added noise (as reflected by R2R^2 / accuracy in Table 1). In these cases, an alternative explanation might be that higher rank instead reflects memorization of training points rather than a richer learned function. This would be consistent with Section 4.1, where, on the same task, rank is high during memorization and drops once the model generalizes. If so, the rank interpretation as a measure of learned functional complexity, as stated in the abstract, would be less clear. See Question 1.

  2. Minor remark: the appendix appears somewhat disorganized. Several figures and tables seem to be placed outside their respective subsections, leaving these empty (A.1, A.4-A.5, A.7-A.13 and B). This affects readability and clarity.

Questions for authors
  1. Regarding Weakness 1: could the authors provide evidence that rank tracks task complexity rather than memorization? For instance, they could compare tasks of different difficulty that reach similar final performance. It would also help to clarify how the learnable frontier is defined in Section 4.5, since some settings presented as learnable seem to be poorly fit (see Table 1).
  2. The abstract states that grokking is accompanied by spectral polarization. As the authors note in Section 4.2, however, under µP generalization seems to precede polarization, and in Figure 2b the polarization of some architectures appears to fade after grokking. Could the authors clarify which quantity they consider the signature of grokking, and possibly align the framing of the abstract with this picture?
  3. Have the authors verified whether the polarizing regularizer also accelerates grokking for other architectures or under µP, beyond the SP Transformer? In addition, have they considered whether the effect of the anti-polarizing regularizer in preventing grokking is a generic effect of a penalty that keeps the model closer to the lazy regime, rather than an effect specific to the spectrum? A comparison with a non-spectral penalty with a similar effect might help clarify this point.
5 comments
AuthorsOpenReview ↗

Response to Reviewer zUFX (1/2)

Thank you for a constructive review and for seeing the intervention as “one of the main contributions”. We ran each experiment you suggested and feel that it greatly strengthens the paper’s results: tasks of different difficulty at the same test performance (W1), the penalty in every architecture and under μ\muP (Q3_a), and a non-spectral pull toward initialization (Q3_b). Below, JJ’s input-dependent part means JJ without the softmax shift and the per-output mean directions.

W1, Q1_a · Does rank track complexity or memorization?

W1: In section 4.5, the paper interprets higher rank on harder tasks as reflecting a more complex learned solution, while in several benchmarks the harder settings seem to be those where the model fits the target poorly, or where difficulty comes from added noise (as reflected by R2R^2 / accuracy in Table 1). In these cases, an alternative explanation might be that higher rank instead reflects memorization of training points rather than a richer learned function. This would be consistent with Section 4.1, where, on the same task, rank is high during memorization and drops once the model generalizes. If so, the rank interpretation as a measure of learned functional complexity, as stated in the abstract, would be less clear.

Q1: Regarding Weakness 1: could the authors provide evidence that rank tracks task complexity rather than memorization? For instance, they could compare tasks of different difficulty that reach similar final performance.

Rank tracks the complexity of whatever function the network fits, target or noise, and with generalization held fixed it rises with the target’s complexity. Tasks of different difficulty at the same final performance, as you suggest: a student MLP (4×5124 \times 512, GELU) learns random ReLU teachers of growing width on 16-dimensional Gaussian inputs (20,000 samples, 3 seeds):

Teacher width (ReLU units)12481632
Test R2R^21.0001.0001.0001.0000.9990.999
Student’s effective rank64102150179254281

Effective rank rises with teacher width (Spearman 0.98), and so does every Rényi rank (0.79–0.98; at α=0.05\alpha = 0.05, from 356 to 450). Explicit memorization: the width-8 teacher (5,000 samples, 3 seeds) with a fraction of the training targets replaced by noise of the same variance, all fit:

Fraction replaced by noise00.250.51
Training R2R^2 on the noisy labels1.0000.9980.9970.996
Test R2R^21.0000.7060.294−0.595-0.595
Effective rank234401437444

And on modular arithmetic the generalizing Transformer's input-dependent rank sits 5–21% below the same network's on shuffled labels from memorization to grokking, and 38% below 4.6k steps later. Memorizing noise is fitting a more complex function, as the paper's Hermite-with-noise benchmark already shows (App. A.9); this is what the abstract means by learned functional complexity, as §4.5 now says. This also matches §4.1: rank is high at memorization and falls as the generalizing solution replaces the memorizing one.

Q1_b · How is the learnable frontier defined?

It would also help to clarify how the learnable frontier is defined in Section 4.5, since some settings presented as learnable seem to be poorly fit (see Table 1).

§4.5 now defines the frontier as the degree at which relative test error reaches 1, no better than predicting the mean, which here is degree 20 (Fig. 26 maps this error over degree and coefficient spread). By that definition, Table 1's degree 4 is learned (R2=0.98R^2 = 0.98), 10 is partially learned (0.16), 20 is the transition (−0.04-0.04) and 60 lies beyond it (−0.18-0.18); its rows now carry these labels.

Minor · Appendix layout

…the appendix appears somewhat disorganized. Several figures and tables seem to be placed outside their respective subsections, leaving these empty (A.1, A.4-A.5, A.7-A.13 and B).

Fixed: every appendix figure and table now sits in its own subsection.

AuthorsOpenReview ↗

Response to Reviewer zUFX (2/2)

Q2 · Which quantity is the signature?

The abstract states that grokking is accompanied by spectral polarization. As the authors note in Section 4.2, however, under μ\muP generalization seems to precede polarization, and in Figure 2b the polarization of some architectures appears to fade after grokking. Could the authors clarify which quantity they consider the signature of grokking, and possibly align the framing of the abstract with this picture?

The signature is a change of JJ's whole singular-value distribution: as the model groks, the share of the input-dependent energy held by the top kk directions rises at every kk (all four architectures of Fig. 2, all seeds). Effective rank, the Rényi ranks and σ1/σk\sigma_1/\sigma_k all move with it. σ1/σ50\sigma_1/\sigma_{50} fades in the MLP and CNN because σ50\sigma_{50} sits inside the 99 per-output directions, which the mean directions reclaim after grokking, while σ1/σ1000\sigma_1/\sigma_{1000} rises as they grok (kk-ablation, Fig. 8). Under μ\muP the input-dependent structure forms while the network fits and already reflects the task (its effective rank is 26–30% below the same network's on shuffled labels), and the polarization that follows grokking lies outside that part. The abstract now reads: "Across four architectures under standard parameterization (SP), grokking is accompanied by spectral polarization: the spectrum of the Jacobian's input-dependent part shifts toward its top directions as the model generalizes. The training regime sets its timing: under maximal-update parameterization (μ\muP) this structure forms while the model fits, and under NTK parameterization the kernel moves but the model does not grok."

Q3_a · Other architectures and μ\muP

Have the authors verified whether the polarizing regularizer also accelerates grokking for other architectures or under μ\muP, beyond the SP Transformer?

Yes: it speeds grokking in 18 of 18 seeds across all four architectures and μ\muP. Each penalized run against the same seed unpenalized (same initialization and data order), 3 seeds each:

SettingSpeed-up, penalty on the full JJSpeed-up, penalty on the input-dependent part only
Transformer, no decay7.4±1.1×7.4 \pm 1.1\times7.7±2.5×7.7 \pm 2.5\times
Transformer, decay 0.12.0±0.1×2.0 \pm 0.1\times2.1±0.3×2.1 \pm 0.3\times
LSTM, decay 0.11.8±0.5×1.8 \pm 0.5\times3.6±2.1×3.6 \pm 2.1\times
CNN, decay 0.11.7±0.1×1.7 \pm 0.1\times1.8±0.2×1.8 \pm 0.2\times
MLP, decay 0.11.4±0.0×1.4 \pm 0.0\times1.3±0.0×1.3 \pm 0.0\times
μ\muP Transformer, decay 0.014.3±2.5×4.3 \pm 2.5\times3.0±0.3×3.0 \pm 0.3\times (mean of the 2 of 3 seeds that are faster)
Seeds faster than unpenalized18 of 1817 of 18

The penalty on the full JJ can be met by moving energy into the softmax shift and the mean directions: in the CNN and MLP they hold 0.98–1.00 of JJ’s energy at grokking. Confined to the input-dependent part, the penalty leaves at most 0.18 there and still speeds grokking in 17 of 18 seeds.

Q3_b · A non-spectral penalty toward the lazy regime

In addition, have they considered whether the effect of the anti-polarizing regularizer in preventing grokking is a generic effect of a penalty that keeps the model closer to the lazy regime, rather than an effect specific to the spectrum? A comparison with a non-spectral penalty with a similar effect might help clarify this point.

We ran that comparison: a penalty that keeps the model near initialization does not reproduce the effect (table in G2; SP Transformer without decay, read at 10k steps). Pulling the weights toward initialization either still generalizes (a=10a = 10) or prevents fitting (a=30,100a = 30, 100). The anti-polarized network instead moves further from initialization than the unpenalized one (relative distance 0.39 vs 0.32) and fits, while at decay 0.01 the same penalty prevents grokking (Table 11). The paper’s other non-spectral baseline, a gradient-norm penalty, is also the weaker lever (3.2×3.2\times vs 5.2×5.2\times in Table 11).

§4.4 now includes the architectures, μ\muP and the pull (App. D.6). Thank you again; we would be glad to hear whether these results answer your questions, and if they do, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Common terms used in replies

Terms used in all our replies (copied from general response)

  • JJ: the Jacobian of the logits on a fixed probe batch with respect to all parameters (its squared singular values are the eNTK’s eigenvalues); effective rank: Roy & Vetterli’s (2007) soft count of its active directions.
  • Split of JJ into three orthogonal parts: the softmax shift (the same change to every logit of an input, invisible to the loss), the per-output mean directions (the same change to an output on every input), and the input-dependent part, the only part that can tell inputs apart.
  • L(k)L(k): the share of JJ’s energy ∑iσi2\sum_i \sigma_i^2 in its top kk directions. A window concentrates if L(k)L(k) rises at every kk and spreads if it falls at every kk; DD is the largest change of L(k)L(k).
  • Phases: memorization = first checkpoint at 99% training accuracy; grokking = first step at 95% validation accuracy; mid-plateau = their midpoint, which splits the plateau in half.
  • Shuffled labels: the same network, seed and data order on permuted labels; it memorizes but cannot generalize.
  • Speed-up: the unpenalized run’s steps to grokking over the penalized run’s (same seed); tied: every weight tensor’s norm reset to the unpenalized run’s after each step.
  • Orbit excess: (squared weight norm −- that of the least-norm point on its symmetry orbit) / the latter; 0% means balanced.
Reviewer_ETyuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates grokking under the lens of the singular value spectrum of the model's Jacobian, referred to as the neural spectrum. The paper provides evidence that grokking is associated with a concentration of the neural spectrum, and that incentivising the model to either concentrate or spread its Jacobian (by regularising on the non-leading singular values) result in faster or slower grokking, respectively.

Critical strengths

The paper deals with an interesting and timely problem. It is refreshing to see four kinds of architectures (CNN, LSTM, MLP, and transformer) being reported in a grokking paper. The effort to test the paper's hypothesis by adding a new regularisation term (Sec 4.4) is good practice.

Critical weaknesses
  1. Novelty is very limited. This "neural spectrum" is nothing more than the eigenvalues of the empirical NTK (as stated by the authors), and therefore the contribution of the paper amounts to calculating the effective rank of a matrix already widely used in the study of model representations. I do not believe the concentration metrics to be a sufficient theoretical contribution or the results to be a sufficient empirical contribution to warrant publication in ICLR.
  2. The manipulation in Sec 4.4 is poorly controlled. It is unclear how the proposed intervention relates to other known factors of grokking such as weight norm, weight rank, logit scaling, etc.
  3. The headline association between grokking and spectral concentration is weak. For example, Figure 2 shows that concentration measures often begin changing only after grokking, and they continue changing even thousands of steps after grokking has finished. This calls into question the association and casts substantial doubt on the practical relevance of the results.
  4. The issue of initialisation is not dealt with in sufficient depth. The NTK and learning dynamics literature has established that weight initialisation is crucial to determine the dynamics of the NTK and whether and how quickly the system develops a rich representation. There is virtually no treatment of initialisation parameters in the paper.
Questions for authors

How does the neural spectrum relate to lazy and rich representations? And more specifically, how is your key result different from known reports of grokking as a transition to the rich learning regime?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ETyu (1/3)

We thank the reviewer for calling the problem “interesting and timely” and the regularization test “good practice”. Your request to test the intervention against known factors, and your excellent suggestion on initialization, led to some of our most important new experiments that we feel strengthen the paper substantially: with every weight tensor’s norm tied, the penalty still speeds grokking 7.4×7.4\times (W2), and moving one initial function along its rescaling orbit changes grokking time 4.9×4.9\times (W4). Below, JJ’s input-dependent part means JJ without the softmax shift and the per-output mean directions, the two parts that cannot tell inputs apart.

W1_a · Isn’t this just the effective rank of a known matrix?

Novelty is very limited. This “neural spectrum” is nothing more than the eigenvalues of the empirical NTK (as stated by the authors), and therefore the contribution of the paper amounts to calculating the effective rank of a matrix already widely used in the study of model representations.

The neural spectrum is widely studied and we use it deliberately; our contribution is what it reveals, not its effective rank (G1). JJ’s rank is the local functional dimension (Grigsby et al. 2025), and its spectrum shows how training reorganizes the directions in which the function can change. Through that lens:

  • The change is specific to generalizing: JJ’s input-dependent effective rank falls 12% as the network groks, against +0.6+0.6% for the same network on shuffled labels (W3).
  • Generalizing has no single spectral path: under NTK parameterization the kernel moves far, yet the network never groks; under maximal-update parameterization (μ\muP) it generalizes before its spectrum fully concentrates.
  • The spectrum is a lever: acting on it moves grokking in both directions (W2).

W1_b · Is the contribution sufficient?

I do not believe the concentration metrics to be a sufficient theoretical contribution or the results to be a sufficient empirical contribution to warrant publication in ICLR.

The contribution is these findings, and each now rests on matched baselines with at least 3 seeds. The regularization test, which you call good practice, holds with every weight tensor’s norm tied (W2); the spectral change is specific to generalizing at matched steps (W3); and it recurs in three more grokking tasks at 1.7–10×10\times the change on shuffled labels (G3).

W2 · Weight norm, weight rank and logit scale

The manipulation in Sec 4.4 is poorly controlled. It is unclear how the proposed intervention relates to other known factors of grokking such as weight norm, weight rank, logit scaling, etc

The speed-up follows none of these factors (table in G2; SP Transformer without weight decay, 3 seeds):

  • Weight norm: with every weight tensor’s norm tied to the unpenalized run’s, the speed-up stays at 7.4×7.4\times. The norm effect is real (holding the norm at its initial value speeds grokking 2.3×2.3\times, as Liu et al. 2023 predict), and the penalty adds 3.6×3.6\times on top.
  • Weight rank: the weights’ effective rank stays within 7% of the unpenalized run’s, while JJ’s input-dependent rank falls to 0.38–0.44 of it (Yunis et al. 2024 track the former).
  • Logit scale: the same speed-up comes with logits 1.24×1.24\times or 2.05×2.05\times larger (Prieto et al. 2025).

The paper's gradient-norm penalty, by contrast, grows the weights and logits (2.17×2.17\times, 5.47×5.47\times) and speeds grokking less (2.4×2.4\times vs 4.8×4.8\times). §4.4 and App. D.6 now include these baselines.

References

  • Grigsby et al. 2025. Functional dimension of feedforward ReLU neural networks. Advances in Mathematics.
  • Liu et al. 2023. Omnigrok: grokking beyond algorithmic data. ICLR.
  • Prieto et al. 2025. Grokking at the edge of numerical stability. ICLR.
  • Yunis et al. 2024. Approaching deep learning through the spectral dynamics of weights. arXiv.
AuthorsOpenReview ↗

Response to Reviewer ETyu (2/3)

W3_a · Does concentration only start after grokking?

The headline association between grokking and spectral concentration is weak. For example, Figure 2 shows that concentration measures often begin changing only after grokking,

JJ’s spectrum changes as test accuracy rises; what lags is the paper’s onset marker, a threshold on that change. The four architectures of Fig. 2 (decay 0.1, 3 seeds each), with the plateau split at mid-plateau; DD is read on JJ’s input-dependent part:

TransformerLSTMMLPCNN
Validation accuracy at memorization / mid-plateau / grokking0.29 / 0.41 / 0.870.02 / 0.07 / 0.940.01 / 0.01 / 0.930.01 / 0.01 / 0.94
Change in full JJ’s effective rank, plateau’s first half / second half−4-4% / −9-9%−2-2% / −22-22%−8-8% / −35-35%−3-3% / −15-15%
DD over the first half / second half0.041 / 0.0890.044 (spreads) / 0.1060.091 / 0.2570.040 / 0.132
Seeds whose spectrum concentrates over the second half3 of 33 of 33 of 33 of 3
log⁡10σ1/σ50\log_{10} \sigma_1/\sigma_{50} at memorization →\to grokking0.32 →\to 0.510.44 →\to 0.550.38 →\to 0.600.16 →\to 0.52
Paper’s onset marker (log⁡10σ1/σ50\log_{10} \sigma_1/\sigma_{50} up 0.5), as a multiple of the grokking step2.07×2.07\times0.97×0.97\timesnot reachednot reached

Every architecture's spectrum changes more over the half that ends at grokking than over the first half, while validation accuracy climbs. The marker waits for log⁡10σ1/σ50\log_{10} \sigma_1/\sigma_{50} to rise a fixed 0.5 above its first value: the Transformer crosses it at twice the grokking step and the MLP and CNN never do, because σ50\sigma_{50} sits inside the block of 99 per-output directions (one per output; Fig. 27). A memorizer changes far less (table in G3): over the plateau's second half, the generalizing SP Transformer's input-dependent effective rank falls 12% (5,640 →\to 4,955), while the same network on shuffled labels moves +0.6+0.6%. §4.1 now reports this change itself, half by half, rather than a threshold (App. D.2).

W3_b · Change after grokking, and practical relevance

and they continue changing even thousands of steps after grokking has finished. This calls into question the association and casts substantial doubt on the practical relevance of the results.

Change after grokking is expected. The network then removes its memorizing solution (Nanda et al. 2023; Varma et al. 2023), and part of the later change sits in the softmax shift and the mean directions rather than the input-dependent part: by the end, the shift holds 0.70 of the Transformer’s energy, and the mean directions 0.74–0.85 of the MLP’s and CNN’s. On relevance: the measure needs no labels, and acting on it moves grokking (W2).

References

  • Nanda et al. 2023. Progress measures for grokking via mechanistic interpretability. ICLR.
  • Varma et al. 2023. Explaining grokking through circuit efficiency. arXiv.
AuthorsOpenReview ↗

Response to Reviewer ETyu (3/3)

W4 · Initialization

The issue of initialisation is not dealt with in sufficient depth. The NTK and learning dynamics literature has established that weight initialisation is crucial to determine the dynamics of the NTK and whether and how quickly the system develops a rich representation. There is virtually no treatment of initialisation parameters in the paper.

Initialization is an important knob for the transition from lazy to rich learning (Woodworth et al. 2020), and we tested it three ways; each changes both grokking time and JJ’s spectrum. Initial scale (SP Transformer, decay 0.1, 3 seeds per scale):

Every initial weight ×α\times \alphaα=0.5\alpha = 0.5α=1\alpha = 1 (standard)α=2\alpha = 2α=4\alpha = 4
Steps to grokking13.3±3.113.3 \pm 3.1k9.1±0.89.1 \pm 0.8k20.2±0.620.2 \pm 0.6k27.7±1.227.7 \pm 1.2k
Change in full JJ’s effective rank, plateau’s first half / second half−8-8% / −16-16%−6-6% / −14-14%+1+1% / −24-24%+4+4% / −32-32%
Change in log⁡10σ1/σ1000\log_{10} \sigma_1/\sigma_{1000}, first half / second half+0.15+0.15 / +0.14+0.14+0.04+0.04 / +0.19+0.19+0.04+0.04 / +0.40+0.40−0.15-0.15 / +0.58+0.58
  • Initial scale (table): from α=1\alpha = 1 up, the lazier the start, the more of JJ’s change waits for the plateau’s second half and the later grokking comes (Spearman 0.78 over the 12 runs).
  • SP’s vs μ\muP’s initial weights: swapping only these moves JJ’s path with them; from SP’s weights, μ\muP groks at 13.0–25.9k steps instead of 1.8–10.6k.
  • Orbit position: one initial function moved along its rescaling orbit groks 4.9×4.9\times sooner (25.6k →\to 5.2k steps), while no weight matrix’s normalized spectrum changes (G4).

Our focus had been how functional dimension, a coarse measure of local geometry, tracks task complexity and generalization; these tests show how initialization enters that picture. We added them to §4.2–4.3 (App. D.4), with a short discussion of initialization and the eNTK's evolution in the related work, and feel the suggestion makes the paper much stronger as a result.

Q1 · How is this different from grokking as a lazy-to-rich transition?

How does the neural spectrum relate to lazy and rich representations? And more specifically, how is your key result different from known reports of grokking as a transition to the rich learning regime?

Kumar et al. (2024) show grokking as a late lazy-to-rich transition, but that does not determine how JJ’s spectrum evolves. Following JJ shows three things they do not: the change is specific to generalizing (W3_a), generalizing does not require one spectral path (under NTK the kernel moves yet the network never groks; W1_a), and acting on the spectrum moves grokking both ways (W2).

We hope the baselines in W2 and the matched comparison in W3 address your main concerns. If they do, we would be grateful if you would reconsider your score, and we would be glad to hear your view either way.

References

  • Kumar et al. 2024. Grokking as the transition from lazy to rich training dynamics. ICLR.
  • Woodworth et al. 2020. Kernel and rich regimes in overparametrized models. COLT.
Reviewer_Z3cLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes learning dynamics of overparameterized neural networks via the singular value spectrum of the Jacobian of the parameter-to-function mapping, which they refer to as the “neural spectrum”. Specifically, the paper investigates how the neural spectrum evolves during grokking across different architectures and feature learning parameterizations (muP, NTP, SP) and observe that sensitivity becomes concentrated in fewer functional directions during grokking, which they term “spectral polarization”. The authors further intervene directly on the spectrum by regularizing it and analyze how this affects grokking.

Critical strengths
  • The general direction of studying what diagnostics can track grokking across architectures and training regimes is interesting and relevant.
  • The paper does not only observe the effects but attempts a causal validation by an intervention on the spectrum.
Critical weaknesses
  • I wonder about the novelty of the main results, since the “neural spectrum” is, up to monotone transformation, just the spectrum of the empirical NTK (which is also stated in l. 161-162). That the NTK spectrum concentrates and aligns with the task as a neural network learns features seems well established (e.g., [1]), and that grokking appears at the transition from lazy to rich training (where lazy nets do not grok and faster feature learning shortens the delay) was already observed by [2]. Given this, it appears to me that the central results are close to restating phenomena that are already known, and it is unclear to me what is new beyond measuring this with the particular chosen metrics/architectures chosen here. I am similarly unsure about the intervention to the spectrum, where I suspect that forcing high effective rank would give an NTK that is roughly isotropic (which should have a similar effect as staying in the laze regime), so this result seems fairly unsurprising to me. Similarly, (given [1]), the connection of spectral rank to functional complexity in sec. 4.5 is expected by construction.
  • I am not convinced by the central motivation via parameter symmetries. The authors argue that common diagnostics based on weights are confounded by parameter symmetries, but the standard spectral quantities that have been studied before (e.g., [3] for singular values of layer-wise weights or also the NTK spectrum) are already invariant w.r.t a large group of parameter symmetries (e.g. permutations, usually orthogonal transformations wherever there are GL invariances), and the Jacobian spectrum itself is not invariant under rescaling symmetries either, just choosing a smooth approximation to the rank that is closer to hard thresholding decreases that sensitivity. What would matter in practice is whether the orbits of the rescaling/GL symmetries are explored for different training runs in practice, and this effect is often relatively small as they come with conserved quantities and, e.g., imbalances are driven to zero under weight decay (unlike, e.g. the permutation orbit which is fully explored, but also doesn’t change the spectrum). However, I didn’t find any argument for where, e.g., in related prior literature on grokking, parameter symmetries have actually meaningfully confounded any results, and it would improve the paper if it convincingly demonstrated how this matters.
  • The comparisons of the different parameterizations and their connection to the feature learning regimes lacks breadth and rigor. The experiments in the main body on muP/NTP/SP are only for a fixed LR and 3 small widths (and the main body results are just a single width), so they differ just in init scales and LR multipliers, but “parameterization” and “feature learning” only really make sense as a width-scaling statement. For example, at a fixed width, NTP with a correspondingly high LR can also look like muP. Hence, I feel that comparisons between both are more meaningful when drawn from multiple LRs and widths, to make sure that the conclusions are not specific to the one setting looked at for each parameterization. It is also unclear to me what the continuous interpolation between parameterizations would give, as if one interpolates the training algorithm, the trivial baseline expectation would be that the dynamics interpolate continuously as well.
  • The experiments for grokking are obtained from just a single task (modular multiplication), so it is unclear how the findings would generalize.

Miscellaneous (i.e., did not affect the score):

  • The font seems off, please check formatting instructions.
  • The appendix is rather messy. For example, many subsection titles appear to be empty headers (I suspect they just contain figures which don’t fit on the pages but then it might be helpful to use float barriers).

[1] Atanasov, Bordelon, Pehlevan. Neural Networks as Kernel Learners: The Silent Alignment Effect. ICLR 2022.

[2] Kumar, Bordelon, Gershman, Pehlevan. Grokking as the Transition from Lazy to Rich Training Dynamics. ICLR 2024.

[3] Yunis, Patel, Wheeler, Savarese, Vardi, Livescu, Maire, Walter. Approaching Deep Learning through the Spectral Dynamics of Weights. arXiv:2408.11804, 2024.

Questions for authors
  • Why are you choosing to frame the results around singular values of the parameter-to-function Jacobian and not the spectrum of the empirical NTK which, as it seems to me, is the more standard object in the literature? What does this framing add beyond the existing NTK alignment results?
  • Do you have any explanation for why SP and muP behave differently, especially why in muP “generalization precedes full polarization”?
  • In fig. 2, for the CNN it seems that the effective rank is going down only after grokking happens and for MLP/CNN the polarization index rise and then decrease again. Have you looked into why that might be?
  • Do you have an example where in a real training run, the orbit under rescaling symmetries is explored sufficiently so that this would meaningfully obscure the NTK spectrum?
  • How sensitive are the results to the Renyi rank α\alpha, and do you see any connection to parameter symmetries in practice, given that sensitivity to them should grow with α\alpha?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Z3cL (1/4)

Thank you for an expert and detailed review, and for noting that the paper “attempts a causal validation by an intervention on the spectrum”. We think your points on the eNTK literature, and specific questions on clarifying the practical importance of symmetry, and requests for more tasks beyond modular arithmetic and ablations for the causal intervention and feature learning parameterizations have made the paper stronger, and we hope you find the new results compelling.

W1_a · Relation to the eNTK

I wonder about the novelty of the main results, since the “neural spectrum” is, up to monotone transformation, just the spectrum of the empirical NTK (which is also stated in l. 161-162).

Yes: JJ’s singular values are the square roots of the eNTK’s eigenvalues (l. 161–162), and we say this explictly but should have been more careful about our framing in terms of thinking about the Jacobian as connecting to parameter and function space geometries (G1, W1_b). Fig. 2’s result also holds for JJ’s input-dependent part: its effective rank falls 8–32% as the models grok.

W1_b · Is this already known from Atanasov et al. [1] and Kumar et al. [2]?

That the NTK spectrum concentrates and aligns with the task as a neural network learns features seems well established (e.g., [1]), and that grokking appears at the transition from lazy to rich training (where lazy nets do not grok and faster feature learning shortens the delay) was already observed by [2]. […] it is unclear to me what is new beyond measuring this with the particular chosen metrics/architectures chosen here.

They are our starting point, but neither shows these three results:

  • When JJ changes. In [1] the kernel’s eigenstructure forms early, before the loss falls. In grokking the network fits first and JJ’s input-dependent part reorganizes long after: −12-12% as it groks, against +0.6+0.6% for the same network on shuffled labels.
  • A moving kernel that fails to grok. In [2] lazy networks do not grok. Our NTK network is not lazy: its kernel moves far (cosine distance 0.65–0.73) and it fits, yet it never groks.
  • A lever. Penalizing the spectrum speeds grokking 1.4–7.4×7.4\times across architectures, or prevents it.

W1_c · Is the anti-polarizing effect just laziness?

I am similarly unsure about the intervention to the spectrum, where I suspect that forcing high effective rank would give an NTK that is roughly isotropic (which should have a similar effect as staying in the laze regime), so this result seems fairly unsurprising to me.

Not quite: the kernel does become roughly isotropic, but the network is not lazy, and a lazy network is not isotropic (G2). It moves further from initialization than the unpenalized network (0.39 vs 0.32 at 10k steps) and changes its kernel almost as much (0.67 vs 0.75), and at decay 0.01 the same penalty prevents grokking (Table 11); a pull toward initialization either still generalizes or cannot fit, and leaves JJ far from isotropic (effective rank 3,250 vs 8,389).

W1_d · Is the link to complexity built in?

Similarly, (given [1]), the connection of spectral rank to functional complexity in sec. 4.5 is expected by construction.

It is not built in; we tested it. With generalization held fixed (test R2≥0.999R^2 \ge 0.999), random ReLU teachers of width 1 to 32 raise the student’s effective rank from 64 to 281.

W2_a · Aren’t standard spectra already invariant?

I am not convinced by the central motivation via parameter symmetries. […] the standard spectral quantities that have been studied before (e.g., [3] for singular values of layer-wise weights or also the NTK spectrum) are already invariant w.r.t a large group of parameter symmetries (e.g. permutations, usually orthogonal transformations wherever there are GL invariances),

Many objects are invariant to the symmetries you list, but both non-orthogonal and data-dependent symmetries also exist, and they matter for the links we study between data geometry, loss-landscape geometry, optimization and feature learning. Applying parameter symmetries during optimization speeds convergence and improves generalization (Zhao et al. 2024). Even when a symmetry preserves the function and the ranks of its Jacobian and Hessian, the local geometry around the parameters can differ considerably in the non-orthogonal case, and finite step sizes in gradient descent compound the difference.

References

  • Zhao et al. 2024. Improving convergence and generalization using parameter symmetries. ICLR.
  • [1]–[3] are your references: Atanasov et al. 2022, Kumar et al. 2024 and Yunis et al. 2024.
AuthorsOpenReview ↗

Response to Reviewer Z3cL (2/4)

W2_b · JJ’s spectrum is not invariant to rescalings either

and the Jacobian spectrum itself is not invariant under rescaling symmetries either, just choosing a smooth approximation to the rank that is closer to hard thresholding decreases that sensitivity.

Yes: along a rescaling orbit JJ’s rank and image are fixed while its singular values move, and usefully so: they track how far each optimizer step moves the function. After W2_d’s orbit move, each AdamW step on the unembedding moves the logits cc times as far (Adam’s step ignores a tensor’s scale), and JJ’s block for that tensor grows by cc. Low Rényi orders limit the remaining sensitivity (Q5).

W2_c · Are the orbits explored in practice?

What would matter in practice is whether the orbits of the rescaling/GL symmetries are explored […] in practice, and this effect is often relatively small as they come with conserved quantities and, e.g., imbalances are driven to zero under weight decay […].

Not under minibatch AdamW, the paper’s optimizer: training moves the weights along the orbits, away from balance. Orbit excess (see Terms), mean of 3–5 seeds, minibatch unless noted:

RunOptimizerAt memorizationAt grokkingAt the end
Paper’s ReLU Transformer, decay 0.01 / 0.1 / 1AdamW1.4% / 1.4% / 1.9%41% / 49% / 39%37% / 68% / 147%
Omnigrok’s MNIST (at their 60%-accuracy event)AdamW93%196%630%
Sparse parityAdamW119%24%30%
Sparse paritySGD + L243%3%1%
Nanda et al.’s modular additionfull-batch AdamW4.2–8.2%*0.1%0.1–0.2%
  • Of the measured norm.

Balance appears under SGD with L2 (as Kunin et al. 2021 predict) and under full-batch AdamW, not under minibatch AdamW: on the same parity task, 24–30% against 1–3%. Conserved quantities preserve orbit position rather than erase it (Zhao et al. 2023).

W2_d · Has symmetry confounded any grokking result?

However, I didn’t find any argument for where, e.g., in related prior literature on grokking, parameter symmetries have actually meaningfully confounded any results, and it would improve the paper if it convincingly demonstrated how this matters.

  • Two examples:
  • Omnigrok's weight norm is mostly not set by the function.* Liu et al. (2023) explain MNIST grokking by the weight norm entering a "Goldilocks zone", but in their run, at their grokking event, function-preserving ReLU rescalings could remove two-thirds of the squared norm. As minibatch AdamW keeps adding this excess (W2_c), the squared norm falls only 41% up to the event, while the orbit's least norm falls 62%.
  • Orbit position alone changes grokking time.* One initial function, trained from four points of its rescaling orbit, groks between 25.6k and 5.2k steps, 4.9×4.9\times apart (G4). JJ registers the move (effective rank at step 0: 730 vs 474); every weight matrix's normalized spectrum, which Yunis et al. [3] track, is identical at all four.

W3_a · Fixed width and “parameterization”

[…] The experiments in the main body on muP/NTP/SP are only for a fixed LR and 3 small widths, so they differ just in init scales and LR multipliers, but “parameterization” and “feature learning” only really make sense as a width-scaling statement.

Yes: at a fixed width these are training regimes, not width-scaling parameterizations, and the paper’s claim is about regimes (§4.2), as §4.2 now clearly says. Their order, μ\muP first and NTK never, holds at widths 64, 128 and 256 (Figs. 6–7; one seed at 64 and 256) and at both learning rates where we ran all three (W3_b).

W3_b · Multiple learning rates

For example, at a fixed width, NTP with a correspondingly high LR can also look like muP. Hence, I feel that comparisons between both are more meaningful when drawn from multiple LRs and widths […].

No learning rate we tried makes NTK look like μ\muP:

Learning rate (width 128, decay 0.01, 3 seeds each)1×1\times (η=10−3\eta = 10^{-3})3×3\times10×10\times
NTK: runs that grok in 100k steps0 of 30 of 30 of 3
NTK: kernel change 1−cos⁡(Kend,K0)1 - \cos(K_{\text{end}}, K_0)0.730.700.65
NTK: log⁡10σ1/σ1000\log_{10} \sigma_1/\sigma_{1000}, memorization →\to end1.42 →\to 1.351.33 →\to 1.131.16 →\to 1.10
μ\muP / SP: steps to grokking5.9k / 37.3k2.2k / 7.0knot run (NTK only)

At every rate NTK’s kernel moves far, yet its top singular values fall back and it never groks.

References

  • Kunin et al. 2021. Neural mechanics: symmetry and broken conservation laws in deep learning dynamics. ICLR.
  • Liu et al. 2023. Omnigrok: grokking beyond algorithmic data. ICLR.
  • Nanda et al. 2023. Progress measures for grokking via mechanistic interpretability. ICLR.
  • Zhao et al. 2023. Symmetries, flat minima, and the conserved quantities of gradient flow. ICLR.
  • [3] is your reference: Yunis et al. 2024.
AuthorsOpenReview ↗

Response to Reviewer Z3cL (3/4)

W3_c · What does the interpolation add?

It is also unclear to me what the continuous interpolation between parameterizations would give, as if one interpolates the training algorithm, the trivial baseline expectation would be that the dynamics interpolate continuously as well.

It tests whether one lazy-to-rich axis, not three hand-picked recipes, sets the outcome: it sets whether the network groks, but the endpoints do not predict how fast and this also shows the expectation that “dynamics interpolate continuously” is not necessarily true in terms of grokking speed (Table 10):

  • One boundary: no run groks for θ≤0.33\theta \le 0.33; every run does from SP (θ=0.5\theta = 0.5) on.
  • Not a blend: SP memorizes fastest (0.4k steps; NTK 10.7k, μ\muP 1.0k), and θ=0.67\theta = 0.67 groks slowest (46.1k; SP 27.6k, μ\muP 14.5k).

W4 · A single task

The experiments for grokking are obtained from just a single task (modular multiplication), so it is unclear how the findings would generalize.

We added three grokking tasks (G3). Against the same seed on shuffled labels, JJ’s spectrum changes around generalization 1.7×1.7\times as much on MNIST (3.2×3.2\times without the mean directions), 10.4×10.4\times on sparse parity and 3.7×3.7\times on the comparison task.

Misc · Font and appendix

The font seems off, please check formatting instructions. The appendix is rather messy. […]

We have fixed both in the revision.

Q1 · Why JJ rather than the eNTK?

Why are you choosing to frame the results around singular values of the parameter-to-function Jacobian and not the spectrum of the empirical NTK which, as it seems to me, is the more standard object in the literature? What does this framing add beyond the existing NTK alignment results?

JJ’s parameter side ties the kernel to local geometry and its symmetries (G1, point 1). A symmetry maps JJ to JT−1JT^{-1}, fixing its rank and image but not its singular values (W2_b). Our effective rank also uses JJ’s singular values, pi∝σip_i \propto \sigma_i (Roy & Vetterli 2007), which weighs the tail more than the eNTK’s σi2\sigma_i^2; §3 wrongly lists it as the α=1\alpha = 1 member of its σi2\sigma_i^2 family (now fixed). Beyond alignment, see W1_b.

Q2 · Why do SP and μ\muP differ?

Do you have any explanation for why SP and muP behave differently, especially why in muP “generalization precedes full polarization”?

The initial weights. Swapping μ\muP’s three differences from SP one at a time, only the initial weights change JJ’s path and μ\muP’s speed (from SP’s initial weights, μ\muP groks at 13.0–25.9k steps instead of 1.8–10.6k). μ\muP starts with 0.99 of JJ’s energy in the per-output mean directions, and its input-dependent structure forms while it fits: at 1k steps (validation accuracy 0.46) that part’s effective rank is already 26% below the shuffled-label network’s. The polarization after grokking happens outside that part, so generalization precedes full polarization.

Q3 · The CNN’s rank and the fading σ1/σ50\sigma_1/\sigma_{50} in Fig. 2

In fig. 2, for the CNN it seems that the effective rank is going down only after grokking happens and for MLP/CNN the polarization index rise and then decrease again. Have you looked into why that might be?

Both come from the 99 per-output mean directions. The CNN’s rank does fall as test accuracy rises (−15-15% from mid-plateau to grokking); its later drop is mostly those directions regaining energy (0.29 →\to 0.85 of JJ’s energy). σ50\sigma_{50} sits inside the top 99 directions, one per output (JJ has a cliff at the number of outputs, Fig. 27); after grokking the MLP’s and CNN’s top 99 become mean directions again, so σ1/σ50\sigma_1/\sigma_{50} falls, while σ1/σ1000\sigma_1/\sigma_{1000} rises in all four architectures as they grok (Fig. 8).

Q4 · A real run where symmetry obscures the spectrum

Do you have an example where in a real training run, the orbit under rescaling symmetries is explored sufficiently so that this would meaningfully obscure the NTK spectrum?

Function-preserving symmetries matter in real training: they change grokking speed and can distort the full eNTK. JJ’s parts capture both.

  • Rescalings change grokking speed, and JJ tracks it. Training explores these orbits (W2_c). One function started at four orbit points groks in 5.2k to 25.6k steps (G4). [3]’s weight spectra are identical at all four. JJ’s unembedding block scales ×0.1 to ×100 with each AdamW step’s effect on the logits; the larger it is, the faster the run groks.
  • The softmax’s translation symmetry distorts the full eNTK. AdamW grows the readout rows’ shared vector, which shifts an input’s logits equally and gets no gradient, until the shift holds 0.73 of JJ’s energy. At decay 0.01, the full spectrum then cannot tell a shuffled-label network from a generalizing one (effective rank 2,074 vs 2,049); JJ’s input-dependent part (Terms) can (6,539 vs 5,461).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3) - Novelty and relationship to prior work

We thank all four reviewers. Three singled out the intervention on the spectrum: it “attempts a causal validation” (Z3cL), is “good practice” (ETyu) and is “one of the main contributions” (zUFX). They also found the direction “interesting and relevant” (Z3cL), the problem “interesting and timely” (ETyu) and the approach “interesting and very explanative” (1rVH). We answer the shared concerns here: novelty, the intervention’s specificity and the breadth of the evidence (G1–G4).

G1 · Novelty and relationship to prior work (Z3cL, ETyu, 1rVH)

You are right that the neural spectrum is essentially the spectrum of the well-studied empirical NTK, and we didn’t mean to imply that this is where the overly of our work lies. Our contribution is to connect it to the geometry of neural-network function spaces, which gives three new findings. JJ, the Jacobian of the outputs with respect to the parameters, has singular values equal to the square roots of the eNTK’s eigenvalues; the name is Kopitkov & Indelman’s (2020). Prior work has tied the eNTK’s evolution to feature learning and grokking: Baratin et al. (2021) show feature alignment and spectral compression, Atanasov et al. (2022; Z3cL’s [1]) the early emergence of task-aligned kernel structure, and Kumar et al. (2024; Z3cL’s [2]) grokking as a lazy-to-rich transition.

1. A geometric view of the eNTK, through parameter-space symmetries. JJ’s null space holds the local, data-dependent symmetry directions, and its rank is the functional dimension (Grigsby et al. 2025; Bona-Pellissier et al. 2024) and this also is what makes the eNTK/neural spectra we study special in the first place, as it connects to the functional geometry because JJ is the pushforward map from parameter to function space . Its singular values measure how sensitive the function is along each direction, and unlike the rank they move under symmetries and reparameterizations. This matters in training: minibatch AdamW leaves the weights 39–49% above the smallest norm for the same function at grokking, and moving one initial function along a rescaling orbit changes grokking time 4.9×4.9\times, which JJ registers and weight spectra miss (G4). The softmax’s translation symmetry also lets training drift into a direction no prediction sees, which at weight decay 0.01 makes a memorizer’s full spectrum concentrate about as much as a generalizing network’s; so we split JJ exactly into that shift, the per-output mean directions and the input-dependent part, which alone can tell inputs apart.

2. Feature learning and spectral concentration are distinct. Previous works often discuss the eNTK aligning with task-relevant features as the model is trained and spectral concentration correlating strongly with generalization as the model transitions from lazy to rich regimes in rocking , but we found the feature learning regime differs based on how the model is parameterized (which is determined by initialization choice, scaling of activations, and learning rates), and that these dynamics directly affect what generalizing functions the optimization dynamics guide a model toward. Thus, a lazy-to-rich transition need not determine how the spectrum evolves as a network generalizes, and the training regime changes it. Under standard parameterization (SP) the spectrum concentrates as the network groks, and this is specific to generalizing: JJ’s input-dependent effective rank falls 12% over those steps, while the same network trained on shuffled labels barely changes (+0.6+0.6%). Under maximal-update parameterization (μ\muP) the network generalizes before the spectrum fully concentrates. Under NTK parameterization the kernel moves far from its initial value (cosine distance 0.65–0.73), yet the network never groks.

3. Acting on the spectrum changes generalization. A label-free penalty on JJ’s spectral mass outside its top eight directions speeds grokking 5.2×5.2\times (Table 11; 4.8×4.8\times in our rerun); reversed, it prevents grokking though the network still memorizes. The speed-up survives without weight decay and with every weight tensor’s norm tied to the unpenalized run’s (G2). Unlike interventions on weight spectra, representations or alignment with a target kernel, it acts directly on how sensitive the function is to its parameters.

References

  • Atanasov et al. 2022. Neural networks as kernel learners: the silent alignment effect. ICLR.
  • Baratin et al. 2021. Implicit regularization via neural feature alignment. AISTATS.
  • Bona-Pellissier et al. 2024. Geometry-induced implicit regularization in deep ReLU neural networks. arXiv.
  • Grigsby et al. 2025. Functional dimension of feedforward ReLU neural networks. Advances in Mathematics.
  • Kopitkov & Indelman 2020. Neural spectrum alignment: empirical study. arXiv.
  • Kumar et al. 2024. Grokking as the transition from lazy to rich training dynamics. ICLR.
AuthorsOpenReview ↗

General response (2/3) - The intervention: weight norm, logit scale or laziness?

Terms used in all our replies

  • JJ: the Jacobian of the logits on a fixed probe batch with respect to all parameters (its squared singular values are the eNTK’s eigenvalues); effective rank: Roy & Vetterli’s (2007) soft count of its active directions.
  • Split of JJ into three orthogonal parts: the softmax shift (the same change to every logit of an input, invisible to the loss), the per-output mean directions (the same change to an output on every input), and the input-dependent part, the only part that can tell inputs apart.
  • L(k)L(k): the share of JJ’s energy ∑iσi2\sum_i \sigma_i^2 in its top kk directions. A window concentrates if L(k)L(k) rises at every kk and spreads if it falls at every kk; DD is the largest change of L(k)L(k).
  • Phases: memorization = first checkpoint at 99% training accuracy; grokking = first step at 95% validation accuracy; mid-plateau = their midpoint, which splits the plateau in half.
  • Shuffled labels: the same network, seed and data order on permuted labels; it memorizes but cannot generalize.
  • Speed-up: the unpenalized run’s steps to grokking over the penalized run’s (same seed); tied: every weight tensor’s norm reset to the unpenalized run’s after each step.
  • Orbit excess: (squared weight norm −- that of the least-norm point on its symmetry orbit) / the latter; 0% means balanced.

G2 · Is the intervention just weight norm, logit scale or laziness? (Z3cL, ETyu, zUFX)

ETyu: The manipulation in Sec 4.4 is poorly controlled. It is unclear how the proposed intervention relates to other known factors of grokking such as weight norm, weight rank, logit scaling, etc

zUFX: …have they considered whether the effect of the anti-polarizing regularizer in preventing grokking is a generic effect of a penalty that keeps the model closer to the lazy regime, rather than an effect specific to the spectrum?

Z3cL: …I suspect that forcing high effective rank would give an NTK that is roughly isotropic (which should have a similar effect as staying in the laze regime)…

No. The speed-up survives with every weight tensor’s norm tied, it does not follow weight rank or logit scale, and the network it stops from grokking is not lazy. SP Transformer without weight decay, 3 seeds. Every column after the speed-up is a ratio to the same seed’s unpenalized run, read at the penalized run’s grokking step (—: not measured):

ArmSpeed-upSquared weight norm (ratio)Weights’ effective rank, mean of 11 matrices (ratio)Logit norm (ratio)JJ’s input-dependent effective rank (ratio)
Spectral penalty7.4±1.1×7.4 \pm 1.1\times1.080.972.05±0.232.05 \pm 0.230.44±0.100.44 \pm 0.10
Spectral penalty, every tensor’s norm tied7.4±1.5×7.4 \pm 1.5\times1.000.951.24±0.441.24 \pm 0.44—
Input-dependent penalty, every tensor’s norm tied9.5±1.2×9.5 \pm 1.2\times1.000.931.89±0.261.89 \pm 0.260.38±0.100.38 \pm 0.10
Spectral penalty with the total norm held at its initial value, against the unpenalized run held the same way (itself 2.3×2.3\times faster)3.6±0.4×3.6 \pm 0.4\times1.000.971.66±0.111.66 \pm 0.11—

Tying every tensor’s norm leaves the speed-up at 7.4–9.5×9.5\times; the weights’ effective rank stays within 7% of the unpenalized run’s while JJ’s input-dependent rank falls to 0.38–0.44 of it; and the speed-up is the same whether the logits grow 1.24×1.24\times or 2.05×2.05\times. The norm effect is real (holding the norm speeds grokking 2.3×2.3\times), and the penalty adds 3.6×3.6\times on top of it.

Laziness, read at 10k steps in the same setting; the pull toward initialization adds θ←θ−ηa(θ−θ0)\theta \leftarrow \theta - \eta a(\theta - \theta_0) after each step:

ArmDistance moved from initialization, ∥θ−θ0∥/∥θ0∥\lVert\theta - \theta_0\rVert / \lVert\theta_0\rVertKernel change 1−cos⁡(K,K0)1 - \cos(K, K_0) (0 = unchanged)Train / val. accuracyJJ’s effective rank (of 9,504 directions)
Unpenalized0.320.7511.000 / 0.3196,416
Anti-polarizing penalty0.390.6741.000 / 0.2778,389
Pull toward initialization, a=10a = 100.070.8030.980 / 0.9483,331
Pull, a=30a = 300.030.6910.079 / 0.0393,116
Pull, a=100a = 1000.010.0040.023 / 0.0183,250

The anti-polarized network moves further from initialization than the unpenalized one and fits, and at decay 0.01 the same penalty prevents grokking (Table 11). No pull strength reproduces this: at a=30a = 30 the kernel changes about as much (0.691 vs 0.674), but the network cannot fit, and every pulled network’s JJ is far from isotropic (effective rank 3,116–3,331, against 8,389). The speed-up also holds in 18 of 18 seeds across all four architectures and μ\muP (1.4–7.4×7.4\times; table in our reply to zUFX).

References

  • Roy & Vetterli 2007. The effective rank: a measure of effective dimensionality. EUSIPCO.
AuthorsOpenReview ↗

General response (3/3) - Other tasks, initialization, and the revisions

G3 · Is the change specific to generalization, and beyond one task? (Z3cL, ETyu)

Z3cL: The experiments for grokking are obtained from just a single task (modular multiplication), so it is unclear how the findings would generalize.

ETyu: The headline association between grokking and spectral concentration is weak.

JJ’s change tracks generalization in four tasks, each compared with the same seed trained on shuffled labels, which memorizes but cannot generalize. Modular arithmetic (SP Transformer, decay 0.1, 3 seeds):

Effective rank, read at the same step0.5k (memorized)5.2k (mid-plateau)9.7k (grokked)14.3k
Full JJ, generalizing run6,0285,5504,7453,452
Full JJ, same seed on shuffled labels6,3306,1886,1635,809
Input-dependent part, generalizing run6,1105,6404,9553,754
Input-dependent part, shuffled labels6,3986,2376,2726,012
Input-dependent part: generalizing run below shuffled labels5%10%21%38%

Over the plateau’s second half, DD is 0.126 for the generalizing run’s input-dependent part and 0.045 on shuffled labels. The three new tasks, each against the same seed on shuffled labels. DD is read over the plateau’s second half or the equal-length window after generalization, whichever is larger (in all three, the window after), and the shuffled-label run uses windows of the same lengths:

Omnigrok’s MNIST (Liu et al. 2023)Sparse parity (Merrill et al. 2023)Comparison task (Wang et al. 2024)
Setup; seedsAdamW, init scale 8; 53 of 40 bits, SGD; 58-layer GPT-2, decay 0.1; 3
DD of the run over DD of its shuffled-label run, full JJ1.72±0.14×1.72 \pm 0.14\times10.38±0.74×10.38 \pm 0.74\times3.68±0.64×3.68 \pm 0.64\times
Same ratio, mean directions removed3.23±0.41×3.23 \pm 0.41\times10.75±0.71×10.75 \pm 0.71\times—
DD itself, run / shuffled labels (mean directions removed; Wang et al.: full JJ)0.123 / 0.0380.108 / 0.0080.093 / 0.026
Shuffled-label run’s best test accuracy0.12 (chance 0.10)0.52 (chance 0.50)—

G4 · Initialization and parameter symmetries (Z3cL, ETyu)

ETyu: The issue of initialisation is not dealt with in sufficient depth. […] There is virtually no treatment of initialisation parameters in the paper.

Z3cL: …it would improve the paper if it convincingly demonstrated how this matters.

Where initialization sits on a symmetry orbit changes grokking time at a fixed function, and JJ registers this while weight spectra do not. One initial function placed at four points of its rescaling orbit: at step 0 the final LayerNorm’s gain and bias are multiplied by cc and the unembedding is divided by cc, which leaves every output unchanged (paper’s ReLU Transformer, decay 0.1, 3 seeds):

c=0.1c = 0.1c=1c = 1 (standard)c=10c = 10c=100c = 100
How far one AdamW step on the unembedding moves the logits, relative to c=1c = 1 (also the scale of JJ’s unembedding block)×0.1\times 0.1×1\times 1×10\times 10×100\times 100
Same for one step on the final LayerNorm×10\times 10×1\times 1×0.1\times 0.1×0.01\times 0.01
Steps to grokking25.6±2.125.6 \pm 2.1k20.7±3.420.7 \pm 3.4k16.7±0.616.7 \pm 0.6k5.2±0.75.2 \pm 0.7k
Rank of JJ3,1683,1683,1683,168
JJ’s effective rank at step 0733±42733 \pm 42730±37730 \pm 37481±19481 \pm 19474±19474 \pm 19
Singular values of every weight matrix, normalizedidenticalidenticalidenticalidentical
Total squared weight norm at step 0 (gains included), relative to c=1c = 11.21–1.2211.86–1.8887.6–89.7

Revisions, already in the uploaded PDF (every change is in blue):

  • §3: We split JJ exactly into its three parts, defined L(k)L(k) and DD, and fixed a typo in the effective-rank formula (pi∝σip_i \propto \sigma_i, as computed).
  • §4.1–4.3: We added the shuffled-label baseline (G3), initial scale and the orbit move (G4), and now call SP, NTK and μ\muP training regimes at a fixed width (Z3cL).
  • §4.4: We added G2's baselines and the penalty in every architecture and under μ\muP (zUFX).
  • §4.5: We defined learned functional complexity and the learnable frontier, and relabelled Table 1 (zUFX).
  • New §4.6: We added G3's three grokking tasks.
  • Abstract, introduction and related work: We restated the signature of grokking (zUFX), credited the prior eNTK literature, compared our results with it (1rVH) and cited the papers in our replies.
  • Appendix: Appendix D font/layout fixed and new tables (Z3cL, zUFX).

References

  • Liu et al. 2023. Omnigrok: grokking beyond algorithmic data. ICLR.
  • Merrill et al. 2023. A tale of two circuits: grokking as competition of sparse and dense subnetworks. arXiv.
  • Wang et al. 2024. Grokked transformers are implicit reasoners: a mechanistic journey to the edge of generalization. NeurIPS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.