- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors test whether one layer transformer models with no layer norm represent particular intermediate values when calculating a^b mod p, with p prime for 4 different prime numbers. They call these intermediate values the CRT decomposition (since they are derived from the Chinese Remainder Theorem) and they are related to the original problem though the discrete log. Relating to previous work by Nanda et. al., they also look for the presence of specific sparse Fourier components in the output of the models’ MLP– and find frequencies that are consistent with their predictions. They also do causal test to establish that the subspaces found by projecting the MLP output onto these Fourier components each carry the relevant intermediate value. Additionally, they show that the predicted fourier bands are the “grokked” circuits y showing that their frequencies are relatively boosted during the grokking phase of learning, when compared to the rest of the frequencies.
Critical strengths
The theoretical framework the authors lay out allows for the key frequencies to be predicted, rather than discovered post hoc. The authors can say in advance which residue of the answer should break when a given band is removed, which provides a satisfying prediction -> verification.
The causal tests are carefully done and match the predictions. They do both ablation and patching and the evidence is largely in favor of of their predicted outcomes.
Critical weaknesses
The manuscript leaves open some questions of the robustness of the findings. Every control is one that no hypothesis predicts will succeed: the control modulus for the probes, the random token orders for the spectra, and the randomly initialized network for restriction. Nothing tests a competing implementation, such as probing α mod m, b mod m, or (α + b) mod m. Such an investigation seems prudent, but it could be argued that it lies outside of the scope of the paper. If this is the case, the authors might want to indicate this in their “limitations section”. The evidence base also appears narrow: the authors only use 4 hand-picked primes, and the reason is left to the reader to imagine. Additionally, it is not clear wether the investigation was 20 models in which 20 showed the grokking solution-- or if 20 is just the number that did show the circuit out of an unspecified total.
The theoretical backbone of the paper is important for the study, but I found its presentation to be very confusing. Section 3 will be a roadblock for any reader not already familiar with all of the content. This is partly due to sloppy notation, for example equation 1 seems to have "mod p" as a condition for the congruence-- but equation 2 uses "mod p" as an operator that is specific to one side of the equation. However, I also think it could be remedied by offloading some more of the "mathy" stuff to the appendices. For the purpose of the investigation, the specifics the reader needs is actually quite straightforward. In 3.1: after relabeling tokens by discrete log, the prediction is that the model should represent an exponent mod each prime factor of p − 1 (the important thing is that you can make a quantitative prediction, not the math behind it). In 3.2: each of these residues has its own characteristic Fourier band. The details of the group isomorphisms and irreps are mostly a distraction here, in my opinion. If they are not, perhaps more time could be spent explaining why. I also found some of the experimental details to be obfuscated by the writing: the probe paragraph before Section 4.1 never says the target is γ mod m. More time can be spent explaining the setup in plain-language -- and I think it would make the paper much easier to apprecoate.
Questions for authors
The references section seems quite short for an academic paper, especially for a topic that has been heavily studied in recent years. For example a quick search for modular arithmetic and grokking revealed a 2024 paper on the arXiv: "Grokking Modular Polynomials" by Doshi et. al. This might very well not be relevant, but a quick glance makes it at least appear so. Did the authors do much of a literature review for this project? If so, why not include more references? The paper above, for example.
Why only 4 primes? I'd like to have seen a case with more factors (p = 2·3·5·7 + 1) or multiple prime powers (p = 4·9·11 + 1)
Did you only test 20 models, or was this circuit only observed in 20 models among many more tested?
3 comments
Thanks for the response; I appreciate the concision and clarity in addressing the points I raised.
My primary concern re: competing implementations has been somewhat answered. I suppose that my suggestion of (a+b) mod m as an alternative strategy might have been taken too literally-- I am not sure that is even a viable solution to the problem here, so perhaps looking for it doesn't make a lot of sense to begin with? Also, the fact that arbitrary functions of the residuals can be read off but not the sum feels like a contradiction. I remain somewhat unconvinced that alternatives have been ruled out by these tests.
The above, along with the fact that the additionally tested primes deviate somewhat from the framework is worth noting, but not enough for me to lower my overall positive assessment of the work. I will maintain my recommendation.
Other points:
The authors are now more transparent about the number of models tested in total. Filtering by grokking actually may provide a nice control group: I am curious if the authors have tried running the same tests on the models that fail to grok. This might provide some strong signal to support the authors main points, though perhaps it would simply replicate the data on the grokking models right before the onset of grokking?
The updated section 3 is significantly more readable for me. Thanks!
Analysis on non-grokked models
Thank you, we are glad §3 reads better.
Due to time constraints, we could not dive deeper into competing implementations analysis. However, we ran the un-grokked models through the analysis pipeline:
TL/DR: Almost all un-grokked models have their wrong answers wrong in exactly 1 residue
| models | accuracy | wrong answers confined to one residue | MLP prominence (compared to grokked counterpart) | streams at baseline after own-band ablation | |
|---|---|---|---|---|---|
| 2 | to | to | to (n/a) | 4 of 4 | |
| 2 | to | to | to ( to ) | 4 of 4 | |
| 4 | to | to ( to ) | 7 of 12 | ||
| 2 | to | to | to ( to ) | 4 of 8 | |
| 5 | to | to | to () | 7 of 15 |
The bands are there, with prominence close to their grokked counterparts. Through ablation, the models with small primes (p = 59, 83 and 131) seem to have the errors coming from 1 fail stream. Larger models (p = 211 and 397), the error seems to be compounded by many streams.
We thank reviewer 7DAN for recognizing the predictive framework and the careful causal tests. We have made significant adjustments to improve presentation, wording, and experimental data. We have added §6 on neurons and tori, which extends the experiments inside the MLP. We will address your concerns below.
Competing implementation:
"4 hand-picked primes", "20 of how many":
The 4 primes: Initially, we started with p=131 and walked backwards. (113, 127, 131 are consecutive primes). 83 was added in because of the large mod-41 band. 127 has mod-9 and 113 has mod-16, which are prime powers. We could not explain them at first and had to extend the research to cover these "special cases" (which is now §7 Inside a Prime Power). We don't handpick these based on behavior. We also trained at 47, 59 and 107 (Table 2).
We thank you for the suggestion and have extended the experiments to cover these. App. P covers this. For p=211, the framework applies (with a few deviations). For p=397, the filtration for prime-power streams applies (App. S), while its mod-11 stream is not carried by its CRT band alone. We report that as an anomaly.
We have documented this fully in the new App. C, Table 2. A total of 40 models with different activations (ReLU, GELU), seeds and primes were trained. 20 (now 25) have grokked (100% generalization) and were reported. We select models only by grokking (100% on all pairs), never by whether they show the circuit.
Presentation: §3 too mathematical
We agree that the math portion is taking valuable space and dilutes the paper from its results. Therefore, §3 The CRT Decomposition has been trimmed down. The new space is dedicated to §6, which brings more qualitative results. The setup is now in the main text (§4), in plain language, with full details in App. B.
The isomorphisms play an important role in the paper and are discussed in §9 Discussion. That being said, we have kept them to a minimum in the new revision.
Congruences are now written with (modp) throughout (Eqs. 1-2).
Every probe reads γmodm. Fig. 1's caption names it, and the new App. D states it with the probe details.
Questions:
Literature review: Thank you for pointing this out. This concern is also shared by other reviewers. We have reviewed the literature and cited it properly, as well as showing our delta in more detail in §2 Related Work. The number of cited works has increased from 10 to 34. Doshi et al. [1] is now included and credited for the discrete-log reindexing (§1, §3.2, §9).
References
[1] Darshil Doshi, Tianyu He, Aritra Das, Andrey Gromov. Grokking Modular Polynomials. arXiv:2406.03495, 2024. https://arxiv.org/abs/2406.03495