acceptodds
Under review as a conference paper at ICLR 2027

Probability‑Signature Dynamics: Unpacking Modular Addition Learning Within Two‑Layer Networks

Abstract

Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization. While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear. We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution. For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics. This explains the emergence of Fourier sparsity, frequency matching, and phase alignment. The same framework resolves a puzzle under label noise: corrupted examples can show faster early loss decrease than clean examples, despite lacking a coherent generalization rule. We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement. Finally, this method can be applied to other operators. Taking XOR as an example, we observed the predicted frequency in experiments.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Two Layer Network, Gradient Dynamics, Modular Addition, Noise and regularization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_mqXbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how Fourier features emerge in two-layer networks trained on modular addition. Using conditional statistics of the data, called probability signatures, the authors derive approximate gradient dynamics under small initialization. For the complete, clean addition table, these signatures are cyclic shift operators whose common eigenvectors form the Fourier basis. The resulting dynamics explain frequency selection and phase alignment, with weight decay acting as a growth threshold. The paper also bounds sampling and label-noise perturbations and explains why corrupted examples can show faster initial loss reduction. Experiments support these mechanisms, although the theory does not establish the full training trajectory on a sampled dataset or predict grokking itself.

Critical strengths
  • Applies probability signatures to connect the data distribution directly to the emergence of Fourier features.
  • Analyzes frequency selection, weight-decay filtering, and early noise fitting, providing insight into processes associated with grokking.
Critical weaknesses
  • The small-initialization dynamics, frequency selection, and phase alignment closely resemble existing results. Beyond the probability-signature formulation, the paper offers limited new insight into the dynamics of modular addition.
  • Although the paper analyzes reduced dynamics and bounds approximation errors, it does not establish the transition from memorization to generalization. The conditions and timing of this transition—the defining feature of grokking—remain unexplained.
Questions for authors
  • The derived Fourier-mode dynamics appear closely related to those in [1]. Beyond an alternative derivation, what new dynamical insights or predictions does the probability-signature framework provide?
  • Beyond frequency selection and early loss reduction, does the framework explain the transition from memorization to generalization? Can it establish conditions for grokking or predict when generalization begins?

[1] Jianliang He, Leda Wang, Fengzhuo Zhang, Siyu Chen, and Zhuoran Yang. Neural Networks Provably Learn Spectral Representations for Group Composition. arXiv:2606.02993, 2026.

2 comments

Thank you for the careful reading and constructive comments. We will address the questions one by one in the following text.


Q1. Contributions relative to He et al.

We became aware of this work at an early stage. The predecessor of [1] is [2] (arXiv:2602.16849), which was rejected at ICLR 2026 (https://openreview.net/forum?id=Qc4RLl7t3S). Some reviewers pointed out that they overclaim their contributions. The discussion of the mechanism of weight decay also remains unclear in the latest version. We already provided a comparison in the Related Work section of our original submission. In the current version of [1], all proofs, including convergence to a single frequency and phase alignment, proceed by prespecifying the Fourier decomposition coefficients of the parameters. Their work provides extensive results for modular addition and other finite-group operations.

Under the same conditions, our framework can obtain these conclusions: through Theorem 1, whose conditions are ensured by the group structure, linear algebra and group theory provide a route to finding this set of eigenvectors. In modular-addition, our work not only establishes the same conclusions as theirs, but also reveals features in the data. We studied how sampling and distribution affect model inference, which was not visible in prior work.

Our contributions extend beyond this, which are IMPORTANT:

First, we provide an explanation of the effect of weight decay on noise regularization. Proposition 2 states that an appropriate weight decay can select Fourier modes and eliminate the remaining neurons within finite time. For experiments, see Figure 1 with λ=0.0005\lambda=0.0005 and Figure 2(A)(B)(C).

Second, when f(x,y)f(x,y) is not a group operation, signatures still have predictive value for training. In particular, first-order expansion terms generally take effect before second-order terms, in [3]. For example, the methodology in Section 6.1 does not depend on group operations. In both sets of experiments in Appendices F1 and F2, the target labels f(x,y)f(x,y) are not group operations, and we likewise provide valuable predictions that agree with the experiments.

In summary, the core of our work is to establish an analytical framework based on signatures, rather than solely to explain addition or grokking. Among our three main contributions—addition, the rapid early decrease of the loss on noisy examples, and extension to other distributions—modular addition accounts for only 1/3 of all. It serves as a sufficiently classical introductory example.


Q2. Memorization, generalization, and grokking

Regarding the boundary between memorization and generalization, Proposition 2 identifies weight decay as a core factor in generalization.

If λ\lambda is too small, many neurons that memorize noise survive, and the model overfits.

If λ\lambda is too large, all neurons die.

When λ\lambda is appropriate, random factors from initialization determine whether a neuron develops a Fourier frequency or dies (Proposition 2). Previous work has shown that neurons with Fourier frequencies are an important reason for grokking in modular addition [4], it is "Fourier mode --> grokking". We are building the bridge above "training distribution --> Fourier mode".

To characterize the time of grokking: In particular, when noise is present, the boundary for grokking must be strongly associated with the training-set fraction and the noise fraction. If there is no clear description or assumptions regarding the data and the specific training process, it becomes extremely difficult to obtain grokking time.


References

[1] He, Jianliang, et al. "Neural Networks Provably Learn Spectral Representations for Group Composition." arXiv:2606.02993, 2026.

[2] He, Jianliang, et al. "On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking." arXiv:2602.16849, 2026.

[3] Yao, Junjie, and Zhi-Qin John Xu. "Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models." arXiv:2509.20124, 2025.

[4] Nanda, Neel, et al. "Progress Measures for Grokking via Mechanistic Interpretability." ICLR, 2023.

I appreciate the author's response. But in terms of the novelty, [1] has established the similar reduced dynamics under small intialization that decouples the neuron, and do further quatative analysis. I think the grokking anlysis is new, so overall I choose the keep the score

Reviewer_Rc3DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors aim to study how gradient-based training gives rise to Fourier features for modular addition by using statistical quantities of the data distribution to prove theoretical results about leading gradient terms to conclude the appearance of a Fourier basis, the role of weight decay and learning under noisy labels. They track frequency selection, neuron growth and clean/noisy loss trajectories from a two-layer MLP with GeLU/ReLU/quadratic activations, L2 weight decay, AdamW, and label corruption.

Critical strengths
  • Although in an idealized setting (gradient flow under the full-distribution probability signatures) Proposition 1 and Theorem 1 connects data statistics and leading gradient terms which leads into a coherent recovery of the Fourier basis for modular addition and the Hadamard basis for XOR
  • Authors take care in empirical ablations that help with bridging their experiments with what their theoretical results may suggest, for instance the shared initialization parameterization in Appendix C.2. and the bijective-noise construction in Appendix F.2.
Critical weaknesses
  • I believe the writing organization could be improved where it is difficult to follow what certain theoretical results are trying to prove or empirical experiments are trying to validate, and how this fits in the broader motivation of the work. A more specific listing of contributions as well as a description of paper organization and a leading exposition to theorem statements and guidance on proofs would help with this. In particular, it is still unclear to me which conclusions about frequency selection follow under explicit assumptions, and which of these hold only for the truncated small-weight flow setting, and which are empirically supported.
Questions for authors
  • Are embedding-gradient contributions omitted in Theorem 3 and the corresponding proof in Appendix E.5.? These should have the same leading order for quadratic activations.
  • Can the authors clarify which conclusions about frequency selection follow under explicit assumptions, and which of these hold only for the truncated small-weight flow setting, and which are empirically supported?
  • Morwani et al. [1], Theorem 5 characterize Fourier representations through global margin maximization and is missing from related work.

[1] Morwani, Depen, et al. "Feature emergence via margin maximization: case studies in algebraic tasks." International Conference on Learning Representations. Vol. 2024. 2024.

1 comment

Thank you for your careful reading and constructive comments. We will address the questions one by one in the following text.


Q1. Embedding-gradient contributions in Theorem 3

Thank you for pointing this out. We have added the omitted contributions to Theorem 3 and revised the proof. In the updated statement, we define

κ~l=DE[σ(Hl)σ(Hl′)]+Dℓs2E[σ′(Hl)σ′(Hl′)].\widetilde\kappa_l=D\mathbb E[\sigma(H_l)\sigma(H'_l)]+D\ell s^2\mathbb E[\sigma'(H_l) \sigma'(H'_l)].

Compared with the previous version, the updated items are the inner-product of σ′(h)\sigma'(h).

  • Theorem 3.

Under the complete-table training and initialization specified above, assume that σ\sigma is locally Lipschitz and differentiable almost everywhere, with σ\sigma and its derivative of at most polynomial growth. There is a constant CinitC_{\rm init} common to all label assignments such that

=Cinit−ηN2[κ~0K0+(κ~1−κ~0)K1].=C_{\rm init}-\frac{\eta}{N^2} \left[\widetilde\kappa_0K_0+ (\widetilde\kappa_1-\widetilde\kappa_0)K_1\right].

BTW, the estimation of this value is still based on the statistics presented in the text. The corollaries and qualitative comparisons in Section 6 of the original manuscript continue to hold.

We have carried out a complete update in the main text.


Q2. Assumptions, truncated dynamics, and empirical support

Theorems 1 and 2 and Proposition 2 all require truncation under small initialization.

Theorems 1 and 3 require the training set to contain all pairs.

The experiments in Appendix C.1 support the gradient-flow approximations in Theorems 1 and 2; Appendix C.2 and Figure 3 in the main text support the frequency selection in Proposition 2.


Q3. The supplementary of related work

Thank you for your contribution to the relevant work. In the updated related work, we compared this reference.

They modeled and solved an optimization problem under the constraint of L2,3L_{2,3} norm, but there was no connection with the gradient flow with L2 decay. They have different settings.


Presentation Weakness

We express special thanks to you for carefully reading the mechanism analysis and for having identified the calculation error in Thm3.

Due to the limitation of the space, the initial version we submitted was rather abstract. We have optimized the organization of the article:

  • Described the boundaries of the contribution more precisely;
  • Revised the incorrect description of Theorem 3;
  • Comparative experiments between the approximate gradient and the true gradient have been moved from the appendix to the main text;
  • Conducted more numerical experiments in Appendix C to enhance the confidence of Theorem 1;
Reviewer_S3uCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper applies the Probability Signatures framework to address the question of how dataset statistics lead to certain features being learned during gradient descent for the modular addition task. This framework is also applied to explain how label noise reduces the loss more quickly earlier in training than correctly-labeled examples. Finally, they use this framework to predict the structure of the learned solution in the XOR task.

Critical strengths
  • The authors claim that application of the probability signatures framework can explain the features learned based on the statistics of the dataset and apply it to the modular addition task answering why fourier features are learned in modular addition and why label noise leads to earlier loss reduction. They also briefly address the XOR problem. One framework tackling multiple questions like this seems promising.
Critical weaknesses
  1. The text is somewhat difficult to read. I understand the very high-level goal of showing "why" certain features must be learned, this question tackled through the probability signatures framework, however, I struggle to understand how this is actually implemented in the text. Since I find it difficult to understand the framework, it's difficult to assess the experimental choices since they're meant to support the claims made by the framework.

  2. There are multiple citations missing, some very critical. [1] is a critical citation missing and it would be important to explain how your work is similar to and differs from theirs since your paper claims that most works are assuming the structure and not studying the forward question. Using the Alternating Gradient Flow framework they also: study 2-layer networks (in the small initialization regime) and theoretically characterize the learning dynamics trying to answer the forward question of how and why gradient descent arrives at the fourier features it does. Moreover, [2] is the same but applied to the broad class of group tasks in general, whereas your paper focuses on modular addition with the addition of XOR but doesn't expand on the XOR problem much. There are other citations that provide context on the state of the modular addition task that are missing. [3] directly studies modular addition and studies norm growth and Fourier sparsity throughout training, developing an ODE approach to study what representations survive. [4] studies modular addition as well and shows that the sparse Fourier features are robust across architectures and training regimes/hyperparamter settings. [5] also studies the modular addition task and shows the representational impact of phase alignment which your work also seeks to explain. [6] is a very early work characterizing the circuits of modular addition. [7] shows how phase alignment is critical to the structure of the circuit. [8] discusses dataset statistics and co-occurrences leading directly to the particular geometry of the representations, e.g. language statistics exhibit translation symmetry, so this is another work looking at dataset statistics themselves leading directly to the structure observed when reverse-engineering networks.

[1] "Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks", Kunin et al., NeurIPS 2025.

[2] "Sequential Group Composition: A Window into the Mechanics of Deep Learning", Marchetti et al., ICML 2026.

[3] "Survival of the Fittest Representation: A Case Study with Modular Addition", Ding et al., 2024.

[4] "Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks", McCracken et al., NeurIPS 2025.

[5] "On the geometry and topology of representations: the manifolds of modular addition", Moisescu-Pareja et al., ICLR 2026.

[6] "The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks", Zhong et al., NeurIPS 2023.

[7] "Modular addition without black-boxes: Compressing explanations of MLPs that compute numerical integration", Yip et al., 2024.

[8] "Symmetry in language statistics shapes the geometry of model representations", Karkada et al., ICML 2026.

Questions for authors
  1. I am not sure what determines which Probability Signatures are relevant. Why is it that for the modular addition problem, there are 3 types probability signatures (section 3.2): φ(a)(x,z)\varphi^{(a)(}x,z) and φ(b)(x,z)\varphi^{(b)(}x,z); Φ(a)(x,y,z)\Phi^{(a)}(x,y,z) and Φ(b)(x,y,z)\Phi^{(b)(}x,y,z); ψ(a)(x,y)\psi^{(a)(}x,y) and ψ(b)(x,y)\psi^{(b)(}x,y). That is, φ,Φ,ψ\varphi, \Phi, \psi are the probability signatures. Whereas section 6.2 on the XOR example only has Φx(a)\Phi_x^{(a)}. Why are the probability signatures so different in number and in form? What determines which ones are relevant? How did you choose the probability signatures for modular addition vs XOR? Moreover, the XOR example is quite less explained and elaborated so I don't understand where things come from.

  2. Can you explain how your framework differs from the works Alternating Gradient Flow framework which was applied to 2-layer networks for modular addition and the group-operation task more generally? This is a major question for me because this work wasn't cited and it also tackles the forward question rather than assuming the structure at convergence. See 2 in weaknesses.

  3. Figure 2C: How can top drive_noise become negative if it's measuring the norm of the neuron ∣∣θd∣∣||\theta_d|| which is non-negative?

  4. Why is the experiment for cosine similarity check of the gradients in the appendix? It seems it's supposed to support your model in Theorem 1 and 2 which seem like one of the major mathematical results in the paper.

  5. Equation is referred to in the text on line 175 and it links to an equation on line 141, however, this equation on line 141 is missing a label (1).

  6. The sentence on line 061 doesn't make sense to me (bolding mine). The sentence seems to both say that the probability signature framework studies data statistics but also doesn't? Could you please explain what this sentence is meant to convey?

"Our main analytical tool is the probability signature framework introduced by Yao & Xu (2025), which shows the correlation between data statistics and embedding without studying either modular addition learning or the key structure of data statistics."

  1. What is E∙\mathcal{E}_{\bullet} on line 256? Can you define it?
  2. What exactly is meant by "frequency" in the XOR setting? Is it the feature?

The idea seems interesting, but I'm not sure how much it differs from previous work (question 2) and exactly how it's implemented (question 1), which would make it difficult for it to be extended by others to other settings. I also don't quite understand the purpose of certain experiments but hopefully it will be made more clear once some questions about the framework are cleared up.

4 comments

Thank you for the careful reading and constructive comments. We will address the questions one by one in the following text.


Q1. Probability signatures for modular addition and XOR

In the XOR setting, P(z∣x)P(z\mid x) is always uniform, which satisfies the conditions of Theorem 1. Therefore, we can retain only the quadratic term in the expansion (the Φx(a)\Phi_x^{(a)} term in Thm 1).

(Intuitively, the uniformity of P(z∣x)P(z\mid x) means that the mapping y↦xXOR⁡yy\mapsto x\operatorname{XOR}y is one-to-one.)

In fact, the original gradient flow (Proposition 1) contains all these signatures. However, only the terms containing Φx(a)\Phi_x^{(a)} are of the lowest order and become the leading terms. This is what Theorem 1 states.

Theorems 1 and 3 hold for both addition and XOR tasks.

In the rebuttal of Q8, we provided a more detailed explanation on how to extend the prediction to the XOR problem.


Q2. Comparison with Alternating Gradient Flows

We considered more scenarios and also had different condition -- weight decay. Under the same assumptions, we can also reach similar conclusions.

  • Assumptions of theirs: the training set is the complete addition table P×PP\times P, the activation function is x2x^2, the initial amplitudes are pairwise distinct, and AGF trajectories approximate continuous GF trajectories.

  • Conclusion: the uu-phase of AGF is an optimization problem. Under the constraint ∣θ∣=1|\theta|=1, the optimal solution consists of exactly three sets of cosine functions.

Under the same assumptions, the derivations in our paper are also rigorous. Theorem 1 holds exactly (all first-order expansion terms vanish), and Theorem 2 is irrelevant here (there is no noise). The derivation of the Fourier coefficients involves no approximation. If the initial amplitudes are pairwise distinct, without weight decay, the condition of Proposition 2 (sr<sis_r<s_i for i≠ri\ne r) holds, yielding evolution along a unique frequency direction.

Beyond that,

(1) We consider GF with weight decay, which differs from their problem in its fundamental constraints and loss function.

(2) We can explain why different weight decay values lead to different solutions. (in prop.2)

(3) This framework readily extends to other problems because its conclusions follow solely from the structure of the data itself.


Q3. Negative top drive_noise in Figure 2C

The noise-drive term is defined as the average of −η⟨θd,∇dL⟩-\eta \langle \theta_d, \nabla_d L \rangle on noisy samples.

It means, this value depends on the angle between the parameter vector and the gradient direction. A negative value means that noisy examples suppress the growth of the norm of the Top neurons.


Q4. Gradient cosine-similarity validation

Your understanding is entirely correct. This is an important numerical validation of Theorem 1, and the main text also refers to this appendix near line 350. Due to the limitation of the space, we did not include any numerical verification results in the main text.


Q5. Equation numbering and cross-reference

This has been corrected in the updated version.


Q6. Relation to Yao & Xu (2025)

The contribution of [2] is that identical signatures correspond to identical evolution trajectories for embedding vectors, while different signatures correspond to different trajectories; lower-order signatures take effect earlier.

In comparison, our work focuses on the algebraic structure and density distribution of signatures. Particularly for modular addition, these special properties yield meaningful conclusions, such as the Fourier basis. We investigated the specific shapes that the parameters form, which was never discussed in [2].


Q7. Definition of E∙\mathcal{E}_{\bullet}

This notation refers to all of ExU\mathcal{E}_x^U, EyV\mathcal{E}_y^V, and EzW\mathcal{E}_z^W, which bound the sampling error for each parameter in the network. We have revised the text accordingly.


References

[1] Kunin, Daniel, et al. "Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks." NeurIPS, 2025.

[2] Yao, Junjie, and Zhi-Qin John Xu. "Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models." arXiv:2509.20124, 2025.

Q8. Meaning of frequency in the XOR setting

When we write out the signatures in the XOR setting, for example with N=4N=4, the joint probability signatures:

1 & & & \\\\ & 1 & & \\\\ & & 1 & \\\\ & & & 1 \end{array} \right), \Phi_1 = \left( \begin{array}{cccc} & 1 & & \\\\ 1 & & & \\\\ & & & 1 \\\\ & & 1 & \end{array} \right), \Phi_2 \hspace{-0.17em} = \hspace{-0.17em} \left( \begin{array}{ccccc} & & 1 & \\\\ & & & 1 \\\\ 1 & & & \\\\ & 1 & & \end{array} \right), \Phi_3 \hspace{-0.17em} = \hspace{-0.17em} \left( \begin{array}{ccccc} & & & 1 \\\\ & & 1 & \\\\ & 1 & & \\\\ 1 & & & \end{array} \right)$$ and their common eigenvectors are $$H_0 \hspace{-0.17em} = \hspace{-0.17em} \left( \begin{array}{c} 1\\\\ 1\\\\ 1\\\\ 1 \end{array} \right), H_1 \hspace{-0.17em} = \hspace{-0.17em} \left( \begin{array}{c} 1\\\\ - 1\\\\ 1\\\\ - 1 \end{array} \right), H_2 \hspace{-0.17em} = \hspace{-0.17em} \left( \begin{array}{c} 1\\\\ 1\\\\ - 1\\\\ - 1 \end{array} \right), H_3 \hspace{-0.17em} = \hspace{-0.17em} \left( \begin{array}{c} - 1\\\\ 1\\\\ 1\\\\ - 1 \end{array} \right)$$ Therefore, any $N$-dimensional vector admits an orthogonal projection decomposition, it is $$u=\sum_k E_{u,k} H_k.$$ here $k=0,1,2,\cdots,N-1$ are its `frequency` and the coefficient $E_{u,k}$ is the component. **Our main result in XOR:** Before training, we can predict that the neurons would become similar to some of $H_k$ during the process. ----- ### Presentation Weakness We express special thanks to you for carefully reading the mechanism analysis and for giving discussions on technical details. Due to the limitation of the space, the initial version we submitted was rather abstract. We have optimized the organization of the article: - Described the boundaries of the contribution more precisely; - Revised the incorrect description of Theorem 3; - Comparative experiments between the approximate gradient and the true gradient have been moved from the appendix to the main text; - Conducted more numerical experiments in Appendix C to enhance the confidence of Theorem 1;

Dear reviewer,

From your comment, I felt that you spent sufficient time understanding our work. I am extremely grateful for that. Therefore, we hope that our reply can address your doubts and clearly demonstrate our contributions. We are looking forward to further discussion with you.

I thank the authors for their response. The related work is now better, but while the rebuttal provides answers to some of my questions it still does not fully address my broad concern with presentation and understandability, a concern shared by reviews 2pzq and Rc3D. I believe the paper should be rewritten to be more clear and resubmitted. I maintain my score.

Reviewer_2pzqOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a mechanism by which 2-layer MLP learns algorithmic tasks, such as modular arithmetic and XOR. By assuming small weights and uniform logits at initialization, they show that the gradients can be approximated by leading quadratic terms involving shift-operators. They show that the dynamics decouples in the basis that diagonalize these operators (fourier/hadamard), and propose a mechanism that selects frequencies that eventually dominate each neuron. Using this framework they also provide an explanation for why models trained on corrupted labels initially train faster than clean labels.

Critical strengths
  • The mechanism for early-time speed-up in modular arithmetic with corrupted labels is new, to my knowledge.

  • The paper is technically sound. The authors have provided extensive empirical support for their claims, including explanations and/or empirical justifications for their assumptions. For example, how well the quadratic approximation holds as the training progresses is addressed in Appendix C.1 and figure 11.

Critical weaknesses

The two main concerns about the paper are novelty and presentation.

Novelty: It seems that the only novel contribution of the work is explaining the early-time speed-up with corrupted labels. The authors should make substantial efforts in highlighting their precise contributions. Here I list the main ideas of the paper (to my understanding) that already exist in literature:

  • The 2-layer MLP on algorithmic tasks, including label corruption, is obviously from existing literature.
  • The quadratic approximation of gradients in modular arithmetic setup using small initialization and uniform softmax assumption is from He et al, Feb 2026. (Albeit, the earlier work used quadratic activation while this work approximates smooth activations such as GELU to quadratic order -- the mechanism is the same.)
  • The isolation of dynamics in Fourier basis is also from He et al (Feb 2026) and Kunin et al (2025).
  • The frequency selection mechanism is also quite similar to He et al (Feb 2026)
  • Extension of these mechanisms to other group operations has also been done in He et al (June 2026)

Given this, the original contribution of the work seems very limited. If the authors think this assessment is inaccurate or unfair, I urge them to clarify their novelty. (Also some of these works are not in cited in the paper, and should be included.)

I have suggested strong rejection due these novelty concerns, but I am willing to increase the score if these concerns are even partially mitigated.

Presentation: The presentation of the paper requires substantial improvement. The main message of the paper (which is interesting) is obscured by the cumbersome writing style. My suggestion would be to parsimonious with notation and mathematical details in the main text and supplement that with clearer explanations.

He et al (Feb 2026): On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking He et al (June 2026): Neural Networks Provably Learn Spectral Representations for Group Composition Kunin et al (2025): Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks

Questions for authors

My main question remains for the authors to clarify their original contributions. See the first weakness.

4 comments

We thank the reviewer for the careful reading and constructive comments. We will address the weakness one by one in the following text.

W1. Original contributions

(i) compared with [1][2] by He et al.

We became aware of this work at an early stage. The predecessor of [1] is [2] (arXiv:2602.16849), which was rejected at ICLR 2026 (https://openreview.net/forum?id=Qc4RLl7t3S). Some reviewers pointed out that they overclaim their contributions. The discussion of the mechanism of weight decay also remains unclear in the latest version. We already provided a comparison in the Related Work section of our original submission. In the current version of [1], all proofs, including convergence to a single frequency and phase alignment, proceed by prespecifying the Fourier decomposition coefficients of the parameters. Their work provides extensive results for modular addition and other finite-group operations.

Under the same conditions, our framework can obtain these conclusions: through Theorem 1, whose conditions are ensured by the group structure, linear algebra and group theory provide a route to finding this set of eigenvectors. In modular-addition, our work not only establishes the same conclusions as theirs, but also reveals features in the data. We studied how sampling and distribution affect model inference, which was not visible in prior work.

Our contributions extend beyond this, which are IMPORTANT:

First, we provide an explanation of the effect of weight decay on noise regularization. Proposition 2 states that an appropriate weight decay can select Fourier modes and eliminate the remaining neurons within finite time. For experiments, see Figure 1 with λ=0.0005\lambda=0.0005 and Figure 2(A)(B)(C).

Second, when f(x,y)f(x,y) is not a group operation, signatures still have predictive value for training. In particular, first-order expansion terms generally take effect before second-order terms, in [4]. For example, the methodology in Section 6.1 does not depend on group operations. In both sets of experiments in Appendices F1 and F2, the target labels f(x,y)f(x,y) are not group operations, and we likewise provide valuable predictions that agree with the experiments.

In summary, the core of our work is to establish an analytical framework based on signatures, rather than solely to explain addition or grokking. Among our three main contributions—addition, the rapid early decrease of the loss on noisy examples, and extension to other distributions—modular addition accounts for only 1/3 of all. It serves as a sufficiently classical introductory example.

[1] He, Jianliang, et al. "Neural Networks Provably Learn Spectral Representations for Group Composition." arXiv:2606.02993, 2026.

[2] He, Jianliang, et al. "On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking." arXiv:2602.16849, 2026.

(ii) compared with [3] by Kunin et al.

We considered more scenarios and also had different condition -- weight decay. Under the same assumptions, we can also reach similar conclusions.

  • Assumptions of theirs: the training set is the complete addition table P×PP\times P, the activation function is x2x^2, the initial amplitudes are pairwise distinct, and AGF trajectories approximate continuous GF trajectories.

  • Conclusion: the uu-phase of AGF is an optimization problem. Under the constraint ∣θ∣=1|\theta|=1, the optimal solution consists of exactly three sets of cosine functions.

Under the same assumptions, the derivations in our paper are also rigorous. Theorem 1 holds exactly (all first-order expansion terms vanish), and Theorem 2 is irrelevant here (there is no noise). The derivation of the Fourier coefficients involves no approximation. If the initial amplitudes are pairwise distinct, without weight decay, the condition of Proposition 2 (sr<sis_r<s_i for i≠ri\ne r) holds, yielding evolution along a unique frequency direction.

Beyond that,

(1) We consider GF with weight decay, which differs from their problem in its fundamental constraints and loss function.

(2) We can explain why different weight decay values lead to different solutions. (in prop.2)

(3) This framework readily extends to other problems because its conclusions follow solely from the structure of the data itself.


W2. Presentation Weakness

Due to the limitation of the space, the initial version we submitted was rather abstract. We have optimized the organization of the article:

  • Described the boundaries of the contribution more precisely;
  • Revised the incorrect description of Theorem 3;
  • Comparative experiments between the approximate gradient and the true gradient have been moved from the appendix to the main text;
  • Conducted more numerical experiments in Appendix C to enhance the confidence of Theorem 1;

[3] Kunin, Daniel, et al. "Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks." NeurIPS, 2025.

[4] Yao, Junjie, and Zhi-Qin John Xu. "Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models." arXiv:2509.20124, 2025.

Reviewer_2pzqOpenReview ↗

Reply to rebuttal

Novelty:

  • For modular arithmetic, obtaining the known Fourier eigenbasis from the spectral properties of the conditional joint distribution of inputs and labels, using the existing probability-signature framework, does not seem like a particularly insightful new result.
  • The proposed explanation for the (empirically known) effect of weight decay on training with corrupted labels is interesting. Proposition 2 provides an intuitive explanation for the suppression of weak Fourier modes, but does not directly establish selective forgetting of corrupted labels. I also fail to see how Fig. 2 establishes the role of weight decay, since it does not vary weight decay values. Overall, I think this claim could be made sharper with targeted experiments (a suggestion for future versions).
  • The paper still does not support the authors' claim about their framework's utility beyond algorithmic tasks. For example, modular multiplication, studied in app.F1, is very similar to modular addition (Doshi et al 2024).

Overall, I remain unconvinced of the methodological novelty and generality of the paper's setup. My modified take on the paper's contributions is an explanation for initial speedup with corrupted labels, and a potential explanation for the effect of weight decay on corrupted labels.

Presentation: Based on the bullet points in the rebuttal, the authors addressed some but not all of my concerns with the presentation. I see that the revised manuscript has ~50% blue text in the main text. While I cannot go through the entire manuscript again, I take this to be efforts put by the authors to improve the presentation.

Based on the potentially distinct explanation of the effect of weight decay, I will increase my score from 1 to 2. However, I maintain my reservation that the paper falls short of a substantial contribution to the topic.

Doshi et al (2024): Darshil Doshi, Tianyu He, Aritra Das, Andrey Gromov. "Grokking Modular Polynomials". 2024.

Dear Reviewer,

We would like to express our sincere gratitude for the substantial and constructive feedback you have provided on our work. Your insightful suggestions have greatly contributed to the overall improvement of our manuscript. We will conduct more experiments with weight decay to support our viewpoint, rather than merely focusing on the mechanism explanation of the modular-addition task. We appreciate your decision to revise your initial evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.