- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper explores the spectrum of the neural network parameter Jacobian (the empirical-NTK spectrum) during training. Grokking coincides with "spectral polarisation", whose timing depends on the parameterisation. Regularising the spectrum speeds up or prevents grokking, and harder tasks produce higher-rank spectra.
Critical strengths
- The authors have taken an interesting and very explanative approach to understanding neural network behaviour.
- The study includes several relevant architectures, thus providing reasonable validation.
Critical weaknesses
- The evolution of eNTK spectra and effective rank during training is well studied (e.g. Fort et al. 2020; Baratin et al. 2021; Tovey et al. 2025; Nikolaou et al. 2026), and grokking has been characterised as a lazy-to-rich transition (Kumar et al. 2024). Polarisation at grokking is close to what this work predicts.
- Spectral polarisation means a few dominant eigenvalues, and hence eigenmodes, emerge during training. Under gradient flow the residual decays along these modes at rates set by their eigenvalues, so dominant modes correspond to the directions in which learning proceeds with the leading mode typically corresponds to the dataset mean. The paper does not connect its statistical measure back to these dynamics, so it remains unclear why polarisation should cause or signal generalisation and what this means for datasets that are not well represented by the mean. Is the top mode the trivial mean/uniform-logit direction, and do the results hold once it is removed?
- It seems most experiment were run with only a few seeds.
Questions for authors
- How would the analysis apply to strongly inhomogeneous datasets (e.g. multimodal classes or heavy class imbalance), where the data mean fits no class well and the leading eigenmode is therefore uninformative? Does polarisation still track generalisation in that case?
- What would the neural spectrum look like in other learning paradigms such as reinforcement learning, where the targets are non-stationary? Could the polarising regulariser be used to accelerate training more generally, beyond the grokking setting?
5 comments
Response to Reviewer 1rVH (2/2)
W3 · Seeds
It seems most experiment were run with only a few seeds.
Every experiment we report uses at least 3 seeds (5 for MNIST and parity), as did most of the paper’s (Fig. 2 and Tables 10–12 use 3, the complexity benchmarks 5); only Table 13 (2 seeds) and the widths 64 and 256 (one) did not. Our reruns reproduce Table 11 ( vs the submitted ) and Table 13 ( vs from 2 seeds), and the penalty is faster in 18 of 18 seeds across architectures. In the three new tasks (table in G3), ’s change around generalization exceeds its shuffled-label baseline’s in every seed except one MNIST seed on the full .
Q1 · Multimodal classes and heavy imbalance
How would the analysis apply to strongly inhomogeneous datasets (e.g. multimodal classes or heavy class imbalance), where the data mean fits no class well and the leading eigenmode is therefore uninformative? Does polarisation still track generalisation in that case?
’s spectrum still tracks generalization in both cases, in its input-dependent part. With multimodal classes (sparse parity, where both classes share one input mean), ’s spectrum changes as much as on shuffled labels around generalization and concentrates after it. For heavy imbalance we trained the paper’s SP Transformer (decay 0.1) on the same 4,656 training pairs, repeating each class so that the commonest appears as often as the rarest (41,638 rows), and read generalization on class-balanced accuracy (3 seeds):
| imbalance | Balanced | |
|---|---|---|
| Steps to memorization / to 0.95 class-balanced validation accuracy | 0.87k / k | 0.43k / k |
| Final accuracy: overall / rarest fifth of classes / worst class | 0.997 / 0.995 / 0.972 | 1.000 / 1.000 / 0.980 |
| Share of ’s top direction lying in the mean directions, init grokking | 1.00 0.02 | 1.00 0.31 |
| Share of ’s energy in the mean directions, init memorization grokking | 0.93 0.09 0.06 | 0.93 0.14 0.21 |
| Full ’s effective rank, init memorization grokking | 3,042 5,807 5,385 (% in 3.6k steps) | 2,938 6,028 4,745 (% in 9.3k steps) |
| How the spectrum moves over the plateau’s first half second half | spreads (2 of 3 seeds) concentrates (3 of 3) | concentrates concentrates (3 of 3) |
| over the second half: full / input-dependent part | 0.073 / 0.080 | 0.146 / 0.126 |
| over the same window, same network on shuffled labels | — | 0.052 / 0.045 |
Under imbalance the top mode leaves the mean directions almost entirely (0.02 at grokking, against 0.31 balanced), and those directions hold only 0.06 of ’s energy, so the spectrum’s head is not the mean mode. The spectrum spreads over the plateau’s first half and concentrates over the second, as the network groks: of the input-dependent part rises by up to 0.080, against 0.045 for the same network on shuffled labels over a window longer. Effective rank falls less (% vs %) over a plateau shorter, and part of the balanced fall is the mean directions regaining energy (0.14 0.21).
Q2 · Beyond grokking
What would the neural spectrum look like in other learning paradigms such as reinforcement learning, where the targets are non-stationary? Could the polarising regulariser be used to accelerate training more generally, beyond the grokking setting?
Both are promising follow-ups and we thank the reviewer for their expert insight here. In particular, needs only a differentiable model and a probe set and the penalty needs no labels, making it extremely general by construction, so both carry over to the reinforcement learning setting, though we have tested the penalty only where generalization lags fitting in the supervised case. However, we think and the neural spectrum being related to functional and parameter space geometry means that it should carry over meaningfully in terms of its connection to learning dynamics, even in non-supervised settings beyond what the paper explicitly considers.
Thank you again for these questions; we would be glad to hear whether the results address them, and if so, we would be grateful if you would consider raising your score.
Common terms used in replies
Terms used in all our replies (copied from general response)
- : the Jacobian of the logits on a fixed probe batch with respect to all parameters (its squared singular values are the eNTK’s eigenvalues); effective rank: Roy & Vetterli’s (2007) soft count of its active directions.
- Split of into three orthogonal parts: the softmax shift (the same change to every logit of an input, invisible to the loss), the per-output mean directions (the same change to an output on every input), and the input-dependent part, the only part that can tell inputs apart.
- : the share of ’s energy in its top directions. A window concentrates if rises at every and spreads if it falls at every ; is the largest change of .
- Phases: memorization = first checkpoint at 99% training accuracy; grokking = first step at 95% validation accuracy; mid-plateau = their midpoint, which splits the plateau in half.
- Shuffled labels: the same network, seed and data order on permuted labels; it memorizes but cannot generalize.
- Speed-up: the unpenalized run’s steps to grokking over the penalized run’s (same seed); tied: every weight tensor’s norm reset to the unpenalized run’s after each step.
- Orbit excess: (squared weight norm that of the least-norm point on its symmetry orbit) / the latter; 0% means balanced.
Response to Reviewer 1rVH (1/2)
Thank you for a careful review and for calling the approach “interesting and very explanative”; your question about the mean mode and class imbalance directly added results that we feel greatly strengthen the paper. We split J exactly into the softmax shift (your uniform-logit direction), the per-output mean directions (your mean modes) and an input-dependent part. With the first two removed the results hold (W2_c), and J’s energy still concentrates as the network groks under 32× class imbalance (Q1).
W1 · Prior eNTK work
As you note, lazy-to-rich theory predicts concentration once features are learned. These works do not show which part concentrates, a moving kernel that fails to grok, or a lever:
The related work now includes the above comparison.
W2_a · Connecting the measure to the dynamics
We tracked them directly. As you describe, the residual decays mode by mode: by memorization, 0.00 of it lies in the top 8 input-dependent directions and 0.01 in the top 99, so the plateau’s progress is not decay along the dominant modes. What changes over the plateau is the modes themselves, and on MNIST they come to carry the labels (W2_b). This is consistent with concurrent theory with weight decay, in which the residual left after fitting grows the kernel’s task-supported directions while the rest decay, concentrating the spectrum (Pracher et al. 2026); our grokking times follow its 1/(ηλ) clock for decay ≥0.1 (ηλtgrok = 0.93–1.10 for decay 0.1–1, against 0.92 in their Transformer cohort).
W2_b · Why should polarization signal generalization?
On MNIST the leading directions come to carry the labels: the label vector’s share in the top 10 eigenvectors of the mean-removed kernel rises from 0.05 at memorization to 0.26 at generalization and 0.36 at the end. And acting on J’s spectrum moves grokking: the penalty speeds it 7.4× even with every weight tensor’s norm tied.
W2_c · Is the top mode the mean or the uniform logit?
Often, yes, so we removed both exactly, and the results hold. Early in training the mean directions are the top 99 directions in all four architectures, holding 0.73–0.98 of J’s energy; late in training the shift takes 0.78 of the Transformer’s top 10 directions. With both removed, effective rank still falls 8–32% as the models grok, and penalizing only the input-dependent part still speeds up 17 of 18 seeds.
References