acceptodds
Under review as a conference paper at ICLR 2027

Initial Spectral Power Predicts Frequency Selection in Grokking and Steers It in Tied MLPs

Abstract

Networks that grok fit their training data early and generalize much later, using a handful of Fourier frequencies picked up along the way. We ask what decides which frequencies a network ends up with, and whether the same factors decide when it generalizes. We study small networks trained on modular arithmetic, fix our decision rules before running each experiment, and re-test the main results on fresh seeds. For the first question, the initialization matters a great deal. The frequencies with the most initial power in the embedding recover 4 to 13 times more of the final frequencies than chance, in bottlenecked MLPs on two tasks and in a one-layer transformer, and they do better when the initial leaders stand out. In tied MLPs, editing that power steers the outcome in a graded way: an outsider given 0.5, 0.75, 1 or 1.25 times the leader's power wins in 0, 13, 60 and 80% of networks, and a leader whose power is cut never wins. The same edit was not confirmed in the transformer, where it is a weaker lever. Phase alignment, which decides the equal-magnitude case studied in prior theory, carries information only once all powers are made equal. The outcome survives rounding, data order and weight noise, but not a change of data split. Timing behaves differently. Enlarging the winners' head start has no detectable effect, removing all initial contrast roughly doubles the time to grok in an exploratory comparison, and the one confirmed predictor is how late the network memorizes. In these full-batch runs, training never climbs the loss walls that straight lines between memorizing and generalizing solutions cross. We also report a circuit-free symmetry measure, the naturality error. In every run we measured, across three architectures, three modular tasks and the group , it rises during memorization and collapses as the network generalizes; its peak mostly reads prediction confidence, while its collapse exceeds what confidence explains by about an order of magnitude.

Primary area:
interpretability and explainable AI
Keywords:
grokking, modular arithmetic, Fourier features, training dynamics, initialization, progress measures, mode connectivity, pre-registration, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jhDeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks about input frequency fourier transform analysis on MLPs trained on modulo arithmetic (somewhat missing from the title). This setting was studied by He et al (https://openreview.net/forum?id=aVh9KRZdRk). Whereas the previous paper showed magnitude and phase influence at the neuron level, this paper shows that initial power of embeddings affects an entire network of paired MLPs output with domination at a few frequencies. The paper also checks whether this extends to transformers (positive for prediction, not for manipulating).

Critical strengths
  1. The paper extends previous work in showing: a . The Fourier power of init embeddings predicts the dominating final network frequencies, for modular arithmetic and both bottlenecked MLPs and transformers b. Changing the initial power affects the fequency selection for tied quadratic MLPs (but the main claim was not reported as holding for transformers)
  2. I would like to commend the authors on the execution of the experiments. Care was taken with seeds and splits to ablate the effects correctly. The authors were also careful to report what worked and what didn't.
Critical weaknesses

It is perhaps subjective, but, to begin with, the question of Fourier fequency over modular arithmetic seems to have limited scope, as the setting seems particularly structured for Fourier, and generalizations of insights are unclear. Due to this, I feel that the incremental value of the paper should be relatively high with respect to previous work.

  1. Network level prediction and determination results are useful extensions to He et al, but the previous paper already I feel covered magnitude and phase dependencies, so most of the conceptual value was there.
  2. The title overstates, as causality is shown only for tied quadratic MLPs on modular addition. I don't think that this is just nitpicking, as the accurately framed title would underscore the relative limitideness of the results.
Questions for authors
  1. The modular arithmetic task is favorable to Fourier. Could you please show more general positive results in terms of scope?
  2. He (see above) already analyzed magnitude and phase dependencies at the neuron level. Could you please highlight the conceptual delta regarding it?
  3. While the title implies MLP causality in the title (via "steers"), the paper seems to actually establish this for tied quadratic MLPs on modular addition. Could you please explain how the paper supports the title?
Reviewer_aLUROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates the effect of the initial power spectrum on the final frequencies that cause grokking to occur. They perform steering experiments to provide evidence for the hypothesis that the top-frequencies are determined by the initial power spectrum itself.

Critical strengths

The steering experiments provided important indirect evidence for their hypothesis.

Critical weaknesses

Clarity. Very poorly written paper. It is difficult to read with non-technical terms used without proper definition, eg. “leader”, “outsider” in abstract, “edits”, “twins” (line 130), and so on. These definitions are scattered. This makes the claims difficult to reconstruct. Experiments are not carefully described, with a lot of informal descriptions which are not scientifically accurate. Example, “"The outcome is not a fragile by-product of deterministic training (Figure 1d). Replicas that differ only in floating-point rounding end with identical winners in all 24 parents tested (72 comparisons), even though their raw parameters drift apart by 23–40% of their norm;" it’s not clear what exactly the definition of winners is? Where are the replicas trained on? How does one ensure difference only in floating-point rounding? The tone and style is also informal, and not suited for a scientific paper.

Non-standard definitions. The authors define the power-spectrum on “the function, not from its weight” (line 93). The justification for the same is not provided. This is in stark contrast to the usual Fourier spectra analysis in previous grokking literature which define these spectra for weights, see for instance, [1], [2].

Flawed methodology and analysis. The p-values reported are extremely small, this is usually an indication of some sensitivity at the beginning of Section 3. For example, the assumption of Independence across configurations needs clarification. The power inflation could be basically reproducing the methodology introduced in the Lottery Ticket Hypothesis by Frankle and Carbin 2018 [3], where masked neurons train almost as well as the full network. These neurons are recovered from the end of training, which is a possible explanation that the authors do not carefully control for.

Limited novelty. The paper draws heavily on the He et. al. [4]. In particular, it’s results in Section 3 (role of phase). In addition, the authors do not compare other experimental results in their work to [4]. More importantly, correlations between power spectra and grokking have been studied previously [5,6,7]. The current work provides no additional insight on the mechanism that drives the grokking delay.

[1] Doshi, D., Das, A., He, T., & Gromov, A. (2024, May). To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets. In International Conference on Learning Representations (Vol. 2024, pp. 3782-3810).
[2] Gromov, A. (2023). Grokking modular arithmetic. arXiv preprint arXiv:2301.02679.
[3] Frankle, J., & Carbin, M. (2018). The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635.
[4] Jianliang He, Leda Wang, Siyu Chen, and Zhuoran Yang. On the mechanism and dynamics of modular addition: Fourier features, lottery ticket, and grokking. arXiv preprint arXiv:2602.16849, 2026.
[5] Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217.
[6] Notsawo Jr, P., Zhou, H., Pezeshki, M., Rish, I., & Dumas, G. (2023). Predicting grokking long before it happens: A look into the loss landscape of models which grok. arXiv preprint arXiv:2306.13253.
[7] Kunin, D., Marchetti, G. L., Chen, F., Karkada, D., Simon, J., Deweese, M., ... & Miolane, N. (2026). Alternating gradient flows: A theory of feature learning in two-layer neural networks. Advances in Neural Information Processing Systems, 38, 4377-4424.

Questions for authors

I do not have any questions that could materially affect my assessment. I urge the authors to fix their presentation, clarity and novelty issues.

Reviewer_ZQjuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks which Fourier frequencies a grokking network ends up using, and what sets when it generalizes. According to the paper, the frequencies with the most initial embedding power Ek(0)E_k(0) recover 4 to 13 times more of the final winners than chance. In tied MLPs, editing that power steers the outcome with a dose response.

Critical strengths
  • The paper seems to have a robust experimental setup with experiments repeated to ensure reproducibility and robustness.
  • The paper seems to be honest about its reports, Tables 1 and 3 give status claims made by the paper.
Critical weaknesses
  • The paper's writing is not natural, contains many vague and undefined statements, and in general is hard to read. Almost nothing in table 1 is defined before reading it, and there's no hyperlink to where the definitions can be found. I had an extremely hard time reading the paper.

  • The paper is missing some important prior work that establish the learnt frequencies and their origins: https://arxiv.org/abs/2311.07568 and https://arxiv.org/abs/2407.12332.I'm curious how the "spectral power" established by the paper relates to these works' derivations on emergence of fourier features.

  • The phrase "a handful of Fourier frequencies picked up along the way" seems misleading. This has a dependence on the problem dimension, and is not "handful".

Questions for authors
  • What motivates the study of the proposed "functional power" metric? What literature is this metric coming from? Could the authors elaborate on this please?
  • Figure 1: What are "winner", "outsider", "leader"? These are undefined.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.