- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors consider three model/task combinations that are known to show grokking and train these models for long enough that the test accuracy decreases again (denoted as anti-grokking by the authors). The authors then define a metric, by counting the Marchenko-Pastur outliers of weight matrix spectra after shuffling their entries, and show that these outliers form essentially only in the anti-grokking phase. Feeding random data through the model and suppressing these spike directions, the authors identify spike directions relevant to the outputs of the models. Suppressing these directions continuously during training improves generalization in the anti-grokking phase compared to the unsuppressed baseline.
Critical strengths
In the examples presented by the authors, there is a strong agreement between formation of the "correlation traps" and the anti-grokking phase, which, if a convincing relation to overfitting in a more general setting was shown, would allow a detection of overfitting from weight matrices alone. The suppression analysis indicates that these spike directions in the shuffled weights may indeed be related to the loss in generalization performance in the anti-grokking phase.
Critical weaknesses
- The manuscript overall feels unfinished, poorly written, and it is very hard to follow. I list some examples that makes me come to this conclusion below:
a. Figures: All figures seem unfinished. The text is extremely small and the legends not legible (at any reasonable zoom level).
-
Fig. 1: The legend mentions a gray region, but in the figure it seems to be drawn in blue. The axis labels are tiny.
-
Fig. 2: There are no axis labels. The caption says but the outlier in panel b seems to be at . The panel titles contain which seems to be undefined. The vertical, dashed, red line is not explained.
-
Fig. 3: The curves seem to be way smoother despite it is supposed to show the same data as in Fig. 1. The text claims the "test loss reaches a minimum at some point. This is the regime (green) we call anti-grokking", but the test loss in panel a is minimal in the middle of the yellow (grokking) regime.
-
Fig. 4: is undefined. It is not clear what the points represent or how they are obtained. The text says: "Figure 4 validates random-probe JSD against the task-based consequence of the same intervention. [...] Within each model family, the ranking is strong. The results support JSD as a data-free prioritization score for behaviorally consequential traps", but I really do not see how this is supported by what is shown in the figure.
-
Fig. 5: The y-axis label is missing. It is described in the main text by the incomplete sentence: "The practical implication that one can detect layers for follow-up behavioral testing when only weights are available".
b. There are many dense sentences such as: "A Correlation Trap is a separated right-edge mode relative to an empirical MP/TW null fitted after entry-wise randomization of a layer weight matrix".
c. The appendices are not well motivated and integrated into the paper. For example the proofs of Appendix A are not clearly motivated in the paper. The paragraph starting on line 266 is unclear. It is hard to see how the proved theorem is related to the main text at all.
d. Table 2: The reported KS-test p-values are only either tiny or 1.0, which indicated wrong underlying statistics (if drawn according to the test distribution, the p-values should be uniform in [0,1] by definition).
e. The abbreviations JSD (I assume Jensen-Shannon divergence) is used all over the manuscript, but it is never defined, neither what it stands for nor how it is computed.
f. Lines 291-298 are completely ununderstandable if one is not an expert in the GrokkedTransformer of Wang et al. (2024).
g. On line 258 it is unclear what the authors mean by "fitted-MP distance".
h. Line 261 mentions some point or section "A" that is not mentioned before.
i. Line 303: The parameter is not defined or motivated and requires a reader to know Liu et al. (2023).
j. Line 362: It is unclear what "same Frobenius correction budget at the trap edit" means.
k. Line 385: It is not clear what "a matched Gaussian replacement for the mapped FC1 direction" means.
- The abstract makes overstated claims about overfitting detection from weights alone that are not convincingly demonstrated in the paper, which the authors also acknowledge in the limitations section, which is, however, rather hidden in appendix H. I was also irritated by the sentence "We do not claim that this archive alone is an end-to-end executable release" seeming to indicate that the code may not have been used to run the experiments and that the data may not be reproducible?
- References and the related work are weak. In the RMT paragraph of the related work all citations involve the same author, Charles H. Martin, despite there are many works on RMT of network weights in the literature (this is 20% of the only 25 references for this author only). Another example, in the footnote 1 on page 4, the citations do not establish that neural network bulk vectors follow Porter-Thomas statistics, but that the vectors of random matrices do.
Questions for authors
- How consistent are the results over different training runs? How are error bars computed in Figure 1 (both for accuracies and for spike outliers)? Not all curves seem to have error bars, or are they too small to be visible?
- In which sense does the modular addition task in Fig. 1b show grokking, both test and train seem to rise together?
- The authors discuss that the number of outliers is quite stable between reshuffling. How do the outlier compare to those of the unshuffled weights in the different training phases? In which phase do MP outliers form in the unshuffled weights?
- The text corresponding to Fig. 2 says it is for a "well behaved layer". How does a typical layer look like?
- The examples of global norm in line 375 shows strong changes in mean and fluctuations between anti-grokking and the other regimes (where the norms are very similar), while mean trap count has error bars almost as large as the mean itself. Why do the authors conclude that the trap count is a better metric than the norm?
- In what sense are spike directions (i.e. the singular vectors) stable between different shuffles?
- Does Fig. 5 show the mean outlier count over different shuffles? How large are the fluctuations?
3 comments
Response to 7s4R pt 2 addressing questions
1. Training runs, shuffles, and error bars
For the MLP in Figure 1(a), the accuracy curves summarize 10 independent training runs with different random seeds. The solid curves are the pointwise means, and the shaded regions are ±1 sample standard deviation across the 10 runs.
At each analyzed checkpoint for each run, every layer is independently shuffled 50 times. The blue curve is the mean trap count over these shuffles, and its faint band is ±1 sample standard deviation over the shuffles.
The main modular-addition and GPT-2 trajectories are individual long-horizon runs. They required nearly half a year of GPU training to reach anti-grokking (due to the fact that this occurs only on the long horizon) , so Figure 1 does not report training-run error bars for these panels.
We will revise the caption to state (n=10) for the MLP accuracy bands, (n=50) shuffles for the spectral bands, identify the bands as sample standard deviations, and explicitly mark the single-run panels. We will also clarify that Figure 3 shows the mean of two training runs ((n=2)). We have not included training-run error bars in Figure 3, since estimates of run-to-run variability based on only two runs are highly uncertain; we will state this limitation explicitly in the caption.
2. Grokking in Figure 1(b)
Figure 1(b) does show delayed generalization, which is characteristic of grokking. Training and test accuracy do not converge at the same time. Training accuracy reaches 100% around steps, whereas test accuracy reaches 100% around steps: a roughly 14-fold difference in convergence time.
One important point here is that the axis is logarithmic, unlike in Nanda et al. This can make the delay easy to miss visually, but is essential to show the long training period required for anti-grokking.
3. Randomized versus original weight spectra
We thank the reviewer for raising this distinction. Our diagnostic deliberately operates on the entry-wise randomized weight spectra, rather than the spectra of the original trained weight matrices.
Martin and Mahoney (2019, 2021) show that trained weight matrices can develop strong correlations and therefore non-Marchenko–Pastur ESDs, including heavy-tailed ESD regimes ( as seen in the alpha metric in their paper) in which a well-defined separation between an MP bulk and isolated outliers is absent.
Consequently, directly applying an MP/TW criterion to unshuffled weights would not generally have the same statistical interpretation. Our approach instead constructs a conditional randomized reference by permuting weight entries and evaluating the resulting spectrum against a fitted MP/TW baseline. The purpose of randomization is therefore to disrupt the learned correlations associated with the original arrangement of the weights while preserving their empirical distribution.
4. Meaning of a “typical layer”
By a “typical layer,” we mean one whose distribution of weight entries is compatible with our randomized MP/TW baseline: after entry-wise shuffling, its covariance spectrum is well described by the fitted MP bulk, without persistent large outliers beyond the finite-size right-edge threshold. This is the behavior described as a “well-behaved layer” in Section 3.2. Shuffling preserves the distribution of the weight entries while removing their learned arrangement, allowing us to assess this compatibility directly. The Gaussian-like and Laplace-like cases discussed in the paper are both relevant to the MP law.
5. Comparison with the L2 norm and interpretation of the reported spread
Consider a thought experiment: scale the same initial weights using init=10 versus init=100. The initial L2 norm differs tenfold by definition, whereas the MP/TW outlier count remains unchanged under this rescaling because the eigenvalues and fitted threshold scale together. Thus, there is no scale-independent absolute L2 cutoff: the norm can be modulated simply by changing the initialization scale. Our detector measures spectral atypicality independently of this overall scale, because the eigenvalues and the fitted MP/TW threshold scale together, leaving the trap count unchanged under uniform rescaling. Consistently, our experiments with init=4,8 show zero traps during pre-grokking and grokking.
The reported (4.56 + /- 3.03) represents the mean and sample standard deviation across layers, not a standard error. The large spread reflects differences in layer characteristics and mean trap counts 7.00, 1.17, and 5.50 while all three layers exhibit nonzero mean counts during anti-grokking versus zero earlier.
Furthermore, unlike a single global norm, our method identifies individual traps whose influence on predictions can be characterized using JSD. Labeled interventions can then determine whether these effects are beneficial or harmful.
Response to 7s4R pt 2 addressing questions contd.
6. Stability of trap directions across shuffles .
We thank the reviewer for suggesting this experiment, as it led to very interesting results.
We tested 100 independent entry-wise shuffles at the final MNIST MLP checkpoint without weight decay. For each shuffle, we reconstructed the leading 20 detected traps separately and undid the permutation to return them to the original weight coordinates. Specifically, if denotes the entry permutation and is a trap component of the shuffled matrix, its contribution in the original coordinates is . We compared the subspaces spanned by these reconstructed matrices across shuffles, so the comparison does not depend on their ordering or the choice of basis within each subspace. We also compared their sums, , using cosine similarity. This sum is exactly the change to the weights when all 20 traps are removed: the edited layer is . Thus, we compared both the subspaces identified by different shuffles and the direction of the combined weight change produced by removing the detected traps. We repeated both comparisons using 20 bulk singular-vector pairs as a control.
| Comparison after undoing the shuffle | 20 traps | 20 bulk singular-vector pairs |
|---|---|---|
| Backprojected subspace similarity | 0.0985 | 0.000506 |
| Same subspace comparison, excluding shared alignment with the original weights | 0.0901 | 0.000179 |
| Trap-removal matrix similarity | 0.616 | 0.087 |
| Same matrix comparison, excluding shared alignment with the original weights | 0.405 | 0.031 |
Values are means over all 4,950 shuffle pairs, in the original weight coordinates. Subspace similarity is 1 for identical subspaces and 0 for orthogonal subspaces; matrix similarity is the cosine between the reconstructed matrices.
The backprojected trap subspaces overlap substantially more than the bulk control.
Most interestingly, we examined 31 WeightWatcher-detected traps at this checkpoint and measured each trap’s stability as its mean squared projection onto trap subspaces from the same 100 shuffles, after undoing the permutations and retaining alignment with the original weights. We found a strong positive correlation between trap stability and output JSD on 8,192 Gaussian probes: Spearman for the Gaussian replacement of the trap vector.
So, in this experiment, more stable directions seemed to be related to traps that had greater functional influence on the predictions under these interventions: a very intriguing result!
7. GPT-OSS shuffle count
This is the mean count over five shuffles, with fluctuations of around 7%. Because of GPT-OSS’s size and our limited compute, we could shuffle and compute the spectra only five times, compared with 50 for the other main spectral analyses. We will make this explicit.
Response to 7s4R pt 1 addressing critical weaknesses
We thank you for your time reviewing this paper
1. Presentation, Appendix A, KS p-values, and JSD
We have now added more detail to the figure labels to prevent ambiguity and will shortly upload the updated manuscript. The dense sentences resulted from trying to include this information within the page limit. We have revised these passages and moved material to the appendix to ensure clarity, but would also like to clarify several important points in the review.
Interpretation of KS p-values. We respectfully disagree with the inference that these values indicate “wrong underlying statistics.” Uniformity applies to correctly calibrated continuous p-values under a true null, not to an arbitrary collection of tests on separate phases. Table 2 compares selected training phases, including anti-grokking, where departure from the MP baseline is precisely the phenomenon being investigated. There is no requirement that these three selected values appear uniformly distributed since these are separate tests on separate phases/distributions , and their failure to do so clearly does not establish that the statistics are wrong.
Definition and interpretation of JSD. JSD is already explicitly identified as Jensen–Shannon divergence in Section 5, lines 393–394. Equation (7) defines our score as the expected divergence between the original and intervened models’ softmax output distributions; lines 396–398 specify the probes and temperature. Its meaning and use are therefore already described in the manuscript.
Regarding what the JSD figures support, the idea is to quantify the effect of each trap. One way is to pass data through the original model, remove the trap, and measure the change in predictions. Another way and what our JSD test does, is to pass Gaussian probes through the original and intervened models and measure the change in their softmax output distributions. The plots show that the effects measured using data inputs and Gaussian probes are highly correlated, suggesting that the JSD test can identify traps with greater influence on predictions.
Modular addition does show delayed generalization, which is characteristic of grokking. Training and test accuracy do not converge at the same time. Training accuracy reaches 100% around (7\times10^2) steps, whereas test accuracy reaches 100% around (10^4) steps : a roughly 14-fold difference in convergence time.
One important point here is that the axis is logarithmic. This can make the delay easy to miss visually unlike the linear scale shown by Nanda et al. etc , but is essential to show the long training period required for anti-grokking.
Motivation for Appendix A. Appendix A connects spectral structure to generalization in a tractable Gaussian regression setting. Theorem A.1 quantifies how spike strength and target alignment affect excess test risk despite exact training fit. Proposition A.5 and Lemma A.6 then motivate assessing individual traps through their effects on predictions: JSD measures output sensitivity, while labeled interventions determine whether performance improves or worsens. This is the connection to the main text, and we will make it more explicit in the paragraph beginning at line 266.
2. Checkpoint availability and reproducibility
That sentence refers specifically to the GPT-2 checkpoints used for the anti-grokking analysis. They were excluded from the ZIP file because of their size and have been uploaded to Hugging Face; the link was withheld to preserve anonymity. All experiments are reproducible from scratch, and the tool is fully open source and available for others to test.
3. Porter–Thomas statistics and references
Martin and Mahoney (2021, Section 3.3) explicitly discuss Porter–Thomas statistics for bulk eigenvectors and examine eigenvector localization in trained networks in Figure 19.
Additionally, Thamm et al. (2022, Figures 4 and 8) directly test singular-vector components of trained neural-network weight matrices against Porter–Thomas predictions, finding broad agreement in the random-like bulk and deviations primarily among vectors associated with large singular values. We are happy to add Thamm et al. and other relevant references suggested by the reviewer.