- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how Fourier features emerge in two-layer networks trained on modular addition. Using conditional statistics of the data, called probability signatures, the authors derive approximate gradient dynamics under small initialization. For the complete, clean addition table, these signatures are cyclic shift operators whose common eigenvectors form the Fourier basis. The resulting dynamics explain frequency selection and phase alignment, with weight decay acting as a growth threshold. The paper also bounds sampling and label-noise perturbations and explains why corrupted examples can show faster initial loss reduction. Experiments support these mechanisms, although the theory does not establish the full training trajectory on a sampled dataset or predict grokking itself.
Critical strengths
- Applies probability signatures to connect the data distribution directly to the emergence of Fourier features.
- Analyzes frequency selection, weight-decay filtering, and early noise fitting, providing insight into processes associated with grokking.
Critical weaknesses
- The small-initialization dynamics, frequency selection, and phase alignment closely resemble existing results. Beyond the probability-signature formulation, the paper offers limited new insight into the dynamics of modular addition.
- Although the paper analyzes reduced dynamics and bounds approximation errors, it does not establish the transition from memorization to generalization. The conditions and timing of this transition—the defining feature of grokking—remain unexplained.
Questions for authors
- The derived Fourier-mode dynamics appear closely related to those in [1]. Beyond an alternative derivation, what new dynamical insights or predictions does the probability-signature framework provide?
- Beyond frequency selection and early loss reduction, does the framework explain the transition from memorization to generalization? Can it establish conditions for grokking or predict when generalization begins?
[1] Jianliang He, Leda Wang, Fengzhuo Zhang, Siyu Chen, and Zhuoran Yang. Neural Networks Provably Learn Spectral Representations for Group Composition. arXiv:2606.02993, 2026.
2 comments
I appreciate the author's response. But in terms of the novelty, [1] has established the similar reduced dynamics under small intialization that decouples the neuron, and do further quatative analysis. I think the grokking anlysis is new, so overall I choose the keep the score
Thank you for the careful reading and constructive comments. We will address the questions one by one in the following text.
Q1. Contributions relative to He et al.
We became aware of this work at an early stage. The predecessor of [1] is [2] (arXiv:2602.16849), which was rejected at ICLR 2026 (https://openreview.net/forum?id=Qc4RLl7t3S). Some reviewers pointed out that they overclaim their contributions. The discussion of the mechanism of weight decay also remains unclear in the latest version. We already provided a comparison in the Related Work section of our original submission. In the current version of [1], all proofs, including convergence to a single frequency and phase alignment, proceed by prespecifying the Fourier decomposition coefficients of the parameters. Their work provides extensive results for modular addition and other finite-group operations.
Under the same conditions, our framework can obtain these conclusions: through Theorem 1, whose conditions are ensured by the group structure, linear algebra and group theory provide a route to finding this set of eigenvectors. In modular-addition, our work not only establishes the same conclusions as theirs, but also reveals features in the data. We studied how sampling and distribution affect model inference, which was not visible in prior work.
Our contributions extend beyond this, which are IMPORTANT:
First, we provide an explanation of the effect of weight decay on noise regularization. Proposition 2 states that an appropriate weight decay can select Fourier modes and eliminate the remaining neurons within finite time. For experiments, see Figure 1 with λ=0.0005 and Figure 2(A)(B)(C).
Second, when f(x,y) is not a group operation, signatures still have predictive value for training. In particular, first-order expansion terms generally take effect before second-order terms, in [3]. For example, the methodology in Section 6.1 does not depend on group operations. In both sets of experiments in Appendices F1 and F2, the target labels f(x,y) are not group operations, and we likewise provide valuable predictions that agree with the experiments.
In summary, the core of our work is to establish an analytical framework based on signatures, rather than solely to explain addition or grokking. Among our three main contributions—addition, the rapid early decrease of the loss on noisy examples, and extension to other distributions—modular addition accounts for only 1/3 of all. It serves as a sufficiently classical introductory example.
Q2. Memorization, generalization, and grokking
Regarding the boundary between memorization and generalization, Proposition 2 identifies weight decay as a core factor in generalization.
If λ is too small, many neurons that memorize noise survive, and the model overfits.
If λ is too large, all neurons die.
When λ is appropriate, random factors from initialization determine whether a neuron develops a Fourier frequency or dies (Proposition 2). Previous work has shown that neurons with Fourier frequencies are an important reason for grokking in modular addition [4], it is "Fourier mode --> grokking". We are building the bridge above "training distribution --> Fourier mode".
To characterize the time of grokking: In particular, when noise is present, the boundary for grokking must be strongly associated with the training-set fraction and the noise fraction. If there is no clear description or assumptions regarding the data and the specific training process, it becomes extremely difficult to obtain grokking time.
References
[1] He, Jianliang, et al. "Neural Networks Provably Learn Spectral Representations for Group Composition." arXiv:2606.02993, 2026.
[2] He, Jianliang, et al. "On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking." arXiv:2602.16849, 2026.
[3] Yao, Junjie, and Zhi-Qin John Xu. "Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models." arXiv:2509.20124, 2025.
[4] Nanda, Neel, et al. "Progress Measures for Grokking via Mechanistic Interpretability." ICLR, 2023.