acceptodds
Under review as a conference paper at ICLR 2027

The form of the weights in deep networks trained on modular addition

Abstract

Our work presents the form of the weights between neurons (and thus between layers) that deep neural networks (DNNs) learn on the popular task of modular addition. This task has been studied by many prior works, but only the activation geometry, which are the manifolds corresponding to the learned neural representations, are understood in the case of depth. The weights that are learned are only understood in 1-hidden layer networks and thus, this work serves to extend our understanding of the weights to multi-layer networks. To our knowledge, this is the first work to describe and validate the form that learned weights in deep networks take between layers across random seeds on a group multiplication task.

Primary area:
interpretability and explainable AI
Keywords:
Modular addition, mechanistic interpretability, group multiplication, group theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XNhJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the weights learned by deep neural networks on modular addition, extending earlier work on networks with a single hidden layer. The authors use the phases of neurons’ two-dimensional Fourier transforms to obtain meaningful coordinates for comparing connections across hidden layers. They find that weights between neurons dominated by the same frequency often follow a cosine of their relative phase. In some deeper models, particularly those using GELU, neurons mix multiple frequencies, requiring a small sum of Fourier components to describe the weights. Experiments across architectures, depths, and training settings show that these compact approximations closely reconstruct the learned weights and generally preserve task accuracy. The contribution is an empirical characterization of trained networks; it does not explain why optimization learns these structures or establish a computational advantage from depth.

Critical strengths
  • While previous work largely characterized the learned weights in networks with a single hidden layer, this paper examines connections between hidden layers in deeper models. It addresses an important gap by showing how sinusoidal weight structure extends to depth and how frequency mixing can complicate this structure.
  • Using Fourier phases to compare neurons provides a meaningful way to study weight patterns despite arbitrary neuron ordering. The authors validate the resulting formulas across architectures, activation functions, depths, and random seeds, and strengthen their findings by showing that replacing learned weights with fitted approximations preserves task accuracy in the main experiments.
Critical weaknesses
  • Fourier features in modular addition are well established, and Fourier-based analysis—including the 2D DFT—was already used by [1]. The paper should distinguish its specific contribution of fitting connections between hidden layers from these existing findings and techniques.
  • The claim of being the first to describe learned weights “with non-linearities and depth” needs a more precise scope. Prior work already analyzes nonlinear transformers and characterizes weights in ReLU and quadratic networks, including [1][2][3][4][5]. The authors should explicitly compare these results and restrict their priority claim to the particular connections between hidden layers that they characterize.
  • Prior work already establishes that networks collectively use multiple Fourier frequencies. The potentially new observation here concerns greater mixing of frequencies within individual neurons, rather than frequency diversity across the network. Although the paper discusses this mixing, it should more clearly explain and quantify what changes with depth and distinguish those effects from activation-function and training-setting differences.

[1] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress Measures for Grokking via Mechanistic Interpretability. ICLR, 2023.

[2] Jianliang He, Leda Wang, Siyu Chen, and Zhuoran Yang. On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking. arXiv:2602.16849, 2026.

[3] Andrey Gromov. Grokking Modular Arithmetic. arXiv:2301.02679, 2023.

[4] Depen Morwani, Benjamin L. Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham Kakade. Feature Emergence via Margin Maximization: Case Studies in Algebraic Tasks. ICLR, 2024.

[5] Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks. NeurIPS, 2023.

Questions for authors
  • What the advantange of trainign with deep model then the canonical single layer MLP and transformer? Does it speed use the training or the emergence of representation?
  • Despite teh representation, does the deep model help in generalization, i.e., grokking?
4 comments

Thanks for your review. We are confident that we can fully address all your concerns because they are all questions about prior work that have been clarified with minor revisions, as there is a substantial difference between our results and prior work that can be summarized as follows. First, prior work on the weights only inspected 1-hidden layer networks and this is why after more than 20 publications on modular addition, the forms of the weights between layers in deep networks have not yet been published. Second, our work achieves a massive improvement and gives these weights between layers, by using the same method as all prior work (the discrete fourier transform (DFT)), but we find a novel insight that was missed by prior work: the DFT contains information allowing us to make a human-interpretable coordinate system and thus interpret the weights of deep networks for the first time, providing equations fitting all weights between layers with very high R2R^2 and ablation accuracy.

Weakness 1.

Fourier features in modular addition are well established, and Fourier-based analysis—including the 2D DFT—was already used by [1]. The paper should distinguish its specific contribution of fitting connections between hidden layers from these existing findings and techniques.

We re-read our entire manuscript with your review in mind, and we can not find anywhere that we claim our novelty was the discovery of Fourier features, or that we were the first to use Fourier based analysis. To the latter part, we state that we discover how to use information in the 2d DFT in a new way that allows us to beat permutation symmetry within neurons in layers, and we even have an entire paragraph in the discussion and conclusion section speculating about why prior work missed this. No where do we claim, or want to claim that we are the first to use the 2D DFT, because that would distract from the impact of our main discovery: that we can and have modeled the weights between layers with very high R2R^2 and ablation accuracy.

In our discussion and conclusion section we immediately state:

The main discovery that made this work possible was that the phase information located in the 2d-DFT on the largest Fourier component (k,k) allows for neurons to be ``sorted'' into a permutation that allows for human-interpretable interpretation of the weights. We speculate that the reason this was missed before, is that the dominant Fourier components in the first layers are (0,k) or (k,0) and not (k,k) and thus it wasn't inspected by prior work. In fact, the strength of the (k,k) Fourier component in layer 1 preactivations is basically 0 (because neurons in the first layer haven't mixed a and b yet, unless they follow an attention layer) and, thus (k,k) begins to show up weakly only in the first ReLU or GELU non-linear postactivations, which is where we perform our 2d DFT.

We realize from your feedback that it would be good to put a similar statement in the related work section, and thus we modified the last paragraph in related work as follows:

It is worth noting that all of the preceding literature interpreting modular addition and group multiplication tasks relies on the 1d and 2d discrete Fourier transforms. This is where our work builds on preceding literature. We find information within the 2d DFT that prior work missed, which can be utilized to construct a method to normalize, or equivalently order neurons in layers, in a human-interpretable way such that the permutation invariance of neurons isn't an issue. Resultantly, we present the form of the weights in DNNs \textcolor{blue}{trained} on modular addition.

Furthermore, you state:

The paper should distinguish its specific contribution of fitting connections between hidden layers from these existing findings and techniques.

This was our goal with the long 2 page introduction to the paper. In it, we summarized prior work on activations, and prior work on weights, and then had a paragraph explaining exactly how our results are different to the 1-layer case: it lacks the permutation symmetry that deep networks have. Thus, we have made minor edits to absolutely clarify this and reduce all uncertainty and show them below in \textcolor{blue}{}:

\textbf{Prior work on activations.} The activations of individual neurons are sinusoidal \cite{nanda2023progress, chughtai2023neural, gromov2023grokking, he2026mechanism}, the activation manifolds of learned neural representations (a group of neurons collaborating together simultaneously within the network to compute the answer to the same subproblem) are 2-tori embedded in R4\mathbb{R}^4, or linear projections of these tori \citep{moisescu2026geometry} and the activation-geometry was found to globally implement a universal abstract template for an algorithm, being the Chinese Remainder Theorem \citep{mccracken2025uncoveringuniversalabstractalgorithm}.

\textbf{Prior work on weights.} As far weights are concerned, only 1-hidden layer models are understood and they are understood well. \citet{nanda2023progress} found that the learnable embedding matrix contained sinusoidal entries that could be read off the embedding vectors and avoided the permutation invariance of the weights in transformers by replacing the neurons with their fitted activations, replacing input and output weights together simultaneously with fitted second order sinusoidal functions cos⁡(a+b)\cos(a+b) and sin⁡(a−b)\sin(a-b). This work inspired a field seeking to explain why gradient descent methods learn sinusoids in 1-layer networks, starting with \citet{gromov2023grokking}, who gave analytical equations for a toy model of the weights of 1-hidden layer multilayer perceptrons (MLPs). Later, work proving sinusoids would be learned came, with \citet{morwani2024feature} and \citet{mohamadi2023grokking} using the assumption of margin maximization to show that neurons must be sinusoids to maximize the margin. Then heuristic descriptions and proofs of stochastic gradient descent (SGD) converging to sinusoidal weights in 1-hidden layer networks began with \citet{ding2024survival, tian2025provable, tian2026composing, he2026mechanism} and recently the alternating gradient flow from small weight initialization of \citet{kunin2026alternating}.

\textcolor{blue}{Why are the weights between layers an open problem?} Part of the difficulty stems from neurons in hidden layers having permutation symmetry: in a layer with ℓ\ell neurons there are ℓ!\ell! ways to permute neurons with each permutation resulting in the exact same network, with neurons simply located in different indices of the layer across different permutations (and these reorderings can be done in all layers). When considering a 1-hidden layer network trained on a+b≡c(modn)a+b \equiv c \pmod n, this permutation invariance within layers doesn't yet add difficulty because there is only 1 layer: the neurons can be permuted, but since they immediately connect to output logits, and the output logits have an ordering from 00 to n−1n-1, it is easy to plot the output weights from a neuron and see they are sinusoidal, as found by prior work \citep{gromov2023grokking, morwani2024feature, yip2024modularadditionblackboxescompressing, wu2025towards}. \citet{moisescu2026geometry} indirectly inspected the weights via principal component analyses and diffusion maps, finding that almost all of the computation about the answer c=(a+b) mod nc = (a+b)\bmod n was finished by the preactivations of the second layer, and that deeper layers just seemed to rotate the neural representation manifolds in the latent weight space of the network. With all of this said, the equations describing \textit{what} the weights are in deeper networks have remained unknown, even with the forms of the activations of neurons and the forms of the manifolds corresponding to neural representation activations being known \citep{moisescu2026geometry}.

\textbf{This is where this paper builds on prior work.} We uncover the equations that model weights between hidden layers in multi-layer networks. Achieving this creates, to our knowledge, the first description of what is learned between layers in DNNs trained on a task requiring non-linear activations to generalize to the test data (the modular addition dataset is not linearly separable). \textcolor{blue}{Resultantly, our work provides theoreticians attempting to create a scientific theory of deep learning with a dataset where they know in advance the forms of the weights after training. This should make it easier for future work to prove why these weights are what SGD learns, providing the first full exact model of deep learning on a dataset. Thus, we see our result as the first step toward this goal. Furthermore, our result is the natural follow-up to the papers that were able to describe deep non-linear activations on modular addition \citep{mccracken2025uncoveringuniversalabstractalgorithm}, and even their manifolds \citep{moisescu2026geometry}, but neither could describe the weights forming these activations.}

A summary of our contributions to literature interpreting modular addition follows:

  1. We discover the expressions that describe weights in trained networks with different activation functions (ReLU and GELU), depths, and architectures (both transformers and multi-layer perceptrons) over a diverse range of training settings (learning rates and weight decays). We believe we are the first work to provide such a description of learned weights \textcolor{blue}{in deep non-linear networks}. Please note, prior work has achieved this in the case of deep \textbf{linear} networks \citep{saxe2013exact} and did this for the training dynamics of these deep linear networks (no non-linear activations). While our work only gives the form of the weights found after training, it is a critical step for future work investigating the training dynamics of deep nonlinear networks learning modular addition.
  1. The key to this result is beating the permutation invariance between neurons within layers. We do this via uncovering a novel insight: \textit{the 2d discrete Fourier transform (DFT) of neuron activations contains phase information that can be used to canonically order weights between hidden layers. This information allows for neurons to be ordered in a human-interpretable way.} In essence, we use 2d DFTs of neuron activations to acquire a human-interpretable coordinate system, yielding the ability to fully describe the inter-layer weights in deep networks with sinusoids, which require an ordering (since their domain is totally ordered from 0 to 2pi). - With the 2d DFT, we define a \textit{relative phase} that gets around the problems caused by permutation symmetry within layers by giving a canonical coordinate system based on the activations of neurons. Thus, our work is the natural sequel to both \citet{mccracken2025uncoveringuniversalabstractalgorithm} and \citet{moisescu2026geometry}, who together fully described the activations of neurons, the activation-geometry of neural representations, and even the activation-geometry of a global algorithm networks implement across neural representations to solve modular addition in 1-4 hidden layer networks. - Resultantly, our work supplements this story and serves as the logical next step: we present the form of the equations for the weights between hidden layers, and validate them exhaustively over different training conditions and architectures. As we have a full understanding of how to order neurons, we can even study the weights of an entire neural representation simultaneously.

Could you please explain how this fails to “distinguish its specific contribution of fitting connections between hidden layers from these existing findings and techniques.”?

Weakness 2.

The claim of being the first to describe learned weights “with non-linearities and depth” needs a more precise scope. Prior work already analyzes nonlinear transformers and characterizes weights in ReLU and quadratic networks, including [1][2][3][4][5]. The authors should explicitly compare these results and restrict their priority claim to the particular connections between hidden layers that they characterize.

We believe the record should show the context of the quote “with non-linearities and depth”.

We discover the expressions that describe weights in trained networks with different activation functions (ReLU and GELU), depths, and architectures (both transformers and multi-layer perceptrons) over a diverse range of training settings (learning rates and weight decays). We believe we are the first work to provide such a description of learned weights with non-linearities and depth.

We agree that using the word “and” in its mathematical sense, to include both non-linearities and depth at the same times could be confusing. We have amended this to

We believe we are the first work to provide such a description of learned weights \textcolor{blue}{in deep non-linear networks}.

Furthermore, we cite and summarize four of the five papers you mention, and all five papers are about 1-hidden layer networks. Again as our introduction above shows, we never claim novel insights about the weights in 1-hidden layer networks (before the first layer, or after the last layer). We state, five times in the introduction the word “between” in the context of between hidden layers, after first laying out a paragraph explaining the weights in 1-layer nets are solved, and then explaining why those papers and their methods don’t generalize to multilayer networks (because of permutation symmetry). We added two citations to He et al, which is a recent work we didn't know about. Please clarify if the above introduction is not clear enough about this.

Weakness 3.

Prior work already establishes that networks collectively use multiple Fourier frequencies. The potentially new observation here concerns greater mixing of frequencies within individual neurons, rather than frequency diversity across the network. Although the paper discusses this mixing, it should more clearly explain and quantify what changes with depth and distinguish those effects from activation-function and training-setting differences.

No, the mixing of frequencies was already described by McCracken et al and Moisescu-Pareja et al. Our main result is the fact we’re first to offer a description of what the learned weights are after training in networks with >1hidden layer, which is a substantial improvement on the existing literature.

The main paper quantifies what changes and what holds over an exhaustively large space of training settings, including both where training settings are conducive to learning and where they aren’t conducive to learning. We investigate two architectures (transformer and MLP), with three different non-linearities, over four depths, and 25 different combinations of (learning rate, weight decay) and report results over the entire grid. We quantify how our solution performs over this entire space, and report that in region where training parameters are conducive to learning, our solution is learned. The appendix A.4 and A.5 further investigates different train/test splits and the affect of the Swish activation function respectively. As reviewer ohPo pointed out in their critical strengths, these quantifications are broad and suggest a level of generality since our solution is learned across such a diverse array of architectures and training settings.

Question 1.

What the advantange of trainign with deep model then the canonical single layer MLP and transformer? Does it speed use the training or the emergence of representation?

The study of the training dynamics is stated to be out of the scope of our paper multiple times as we focus on what is learned, and speak at length that this is a major breakthrough as we do it for deep non-linear networks. We mention multiple times the study of how and why it is learned would be a massive breakthrough, giving deep learning its first exactly solved model, where everything about what, how and why deep neural networks learn, behave and act on a dataset is understood. With that said, prior work has shown that deeper networks learning modular addition achieve lower final cross-entropy loss and larger logit margins, see e.g. McCracken et al.

Question 2.

Despite teh representation, does the deep model help in generalization, i.e., grokking?

See question 1, where we cite prior work that shows deeper networks achieve lower final cross-entropy loss and larger logit margins.

Refs:

Moisescu-Pareja et al. On the geometry and topology of representations: the manifolds of modular addition. In International Conference on Learning Representations, 2026.

McCracken et al. Uncovering a universal abstract algorithm for modular addition in neural networks. NeurIPS 2025.

Reviewer_ohPoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper looks at deep neural networks trained on modular addition. Previous work has found Fourier structure in representations, this work focus on the structure in the weights. Permutation symmetry make analysis difficult and the authors break the symmetry by considering phase information. The validate a "simple" and "general" form of the weights empirically.

Critical strengths

Breaking permutation symmetry through phase information is an interesting and enables the authors to analytically capture the structure of the weights. Empirical results are clear and are in agreement with the structures outlined by the authors. Results hold on MLPs and Transformers, ReLU and GELU, across depths, and hyperaparameters suggesting a level of generality.

Critical weaknesses

It is not obvious if a similar approach will work for abelian let alone problems with non-abelian groups (as the authors mention). The total extent of the impact of activation function is not clear, more experiments would be nice with other smooth activation functions (e.g. Switsh).

Questions for authors

How does the train/test split size (in your case 90%) effect the learned weights in connection to your analytic model?

1 comment

Thanks for your review. We’re glad you highlighted the main result, that we managed to break permutation symmetry and present the equations for the form of the weights between layers in deep non-linear networks learning modular addition. We’re also glad you liked the level of exhaustive experimentation we did to ensure our claims were robust and suggested generality over many different training settings. To this end, we added a lower train/test split and also the swish activation function, as you asked, and both these sets of experiments agreed with the results in the main paper.

Weakness 1.

It is not obvious if a similar approach will work for abelian let alone problems with non-abelian groups (as the authors mention).

We are confident a generalization of the approach will work for other abelian groups because of a paper on elementary p-groups and homocyclic groups that found the activations, and their activation manifolds were the same across abelian groups, being tori (McCracken et al). That paper reported having to use a more general “directional discrete Fourier transform”, so there would be additional steps required to generalize our method, similar to what that work on more general abelian groups had to do. After that generalization is done to the 2d DFT, the 2d directional DFT would contain the phase information required to give human-interpretable orderings of neurons in the layers, and thus allow for fitting the weights.

Extending to non-abelian groups would likely require even more work than this, but nonetheless, our paper serves as a proof of concept that modelling the weights between layers can be done and serves as a later chapter resolving open questions about what DNNs learn on modular addition.

Ref:

McCracken et al. Interpreting deep neural networks trained on elementary p groups reveals algorithmic structure. In Topology, Algebra, and Geometry in Data Science, 2026

Weakness 2.

The total extent of the impact of activation function is not clear, more experiments would be nice with other smooth activation functions (e.g. Switsh).

We added experiments with the Swish activation (see Appendix A.5) and found our claims matched as expected. We thank you for suggesting these additional experiments as they help widen the breadth of our results and demonstrate further generality.

Question 1.

How does the train/test split size (in your case 90%) effect the learned weights in connection to your analytic model?

We trained on a much lower train/test split (60%/40%) and found our analytic model still works as expected (a lot of prior work has shown different train/test splits don’t change what is learned on this task, but greatly change how quickly things are learned). Please see Appendix A.4.

We hope our rebuttal adequately addresses your concerns and that you’ll consider increasing your score if you think the paper now shows enough generality on modular addition and that the paper is impactful enough in making a significant step beyond prior work restricted to either 1-hidden-layer networks, or depth, but with linear activations. Given the literature we cite, our literature review, and the results of this reviewing process, we remain confident we are likely the first work ever to present analytical forms for the weights after training on a dataset using deep non-linear networks. This is impactful to the interpretability community, giving a dataset where the ground truth about the weights is known, and also to the theoretical community which hasn’t yet found a dataset where they can prove why and describe how stochastic gradient updates converge in deep non-linear networks.

Reviewer_5aACOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the structure of learned weights in deep networks trained on modular addition. The main contribution are:

  • Discovering expressions that describe learned weights in networks with different activation functions, depths, and architectures.
  • Using 2D DFTs of neuron activations to acquire a human-interpretable coordinate system despite arbitrary neuron ordering.
  • Proposing a simple form and a general form that combines a few Fourier components. The general form captures frequency mixing in deeper networks, especially those using GELU.

The experiments include MLPs and transformers with varying depths and nonlinearities, including ReLU and GELU. The fitted weights closely match the learned weights, and replacing the analyzed weights preserves perfect accuracy. Randomized control fits worse or requires more components, which supports the compactness of the learned structure. There's no experiment that shows applicability to other tasks.

Critical strengths
  • Compact description of networks trained on modular addition. Using activation Fourier phases as human-interpretable coordinates allows them to show patterns in connections between hidden layers despite arbitrary neuron ordering, extending prior understanding beyond 1-hidden layer networks.
  • Solid empirical validation within the modular addition task. The replacement experiments show that the fitted weights preserve perfect task accuracy across different model settings and seeds, and the control experiments show that randomly permuted weights require substantially more Fourier components to maintain perfect task accuracy. These results support the reliability of the main finding.
Critical weaknesses
  • The paper provides a task-specific finding rather than a general explanation of deep networks. All experiments are limited to modular addition, where its structure naturally favors Fourier analysis. The experiments varying architectures, depths, activations, and training settings only validate robustness within this task, and the stated advance towards understanding deep non-linear networks exceeds what is demonstrated in the paper.
  • The general form provides a weaker explanation than the simple form. Unlike the simple form, it separately fits the strength and phase of each Fourier component to the learned weights. Since the fitted phase offsets can absorb the target phase, it's unclear if the formula explains a relationship between connected neurons or mainly offers a compact approximation of their weights. This distinction is important as the general form is necessary precisely where the simple form breaks down.
Questions for authors
  • What insights about deep networks does the proposed weight description show beyond modular addition's Fourier structure? Are there any empirical results where this approach can explain the learned weights on tasks beyond modular addition?
  • In the general formula, the fitted phase offsets can absorb the target neuron's measured phase. Then, what does the phase difference between the source and target neurons help explain?
3 comments

Thanks for your review. We’re glad your strengths highlight what we want our work to provide the community: to our knowledge, the first work to extend understanding of the weights from 1-layer nets to depth. We felt we had to exhaustively show this over a massive diversity of training conditions, and appreciate your mention of this work. We are confident we can address your concerns and hope for more discussion.

Weakness 1.

The paper provides a task-specific finding rather than a general explanation of deep networks. All experiments are limited to mod addition, where its structure naturally favors Fourier analysis. The experiments varying architectures, depths, activations, and training settings only validate robustness within this task, and the stated advance towards understanding deep non-linear networks exceeds what is demonstrated in the paper.

This criticism would apply to almost all papers by the community interpreting what networks learn on math tasks, but this type of research is done with hopes of gaining intuition to help interpret nets trained on natural data like language. A main reason it’s researched comes from how physics was greatly influenced and progressed via “exactly solved models” (see book: Exactly Solved Models in Statistical Mechanics, Baxter). To summarize: exact models played a massive role in the evolution of physics by providing insights into the properties of complicated systems, by using simpler systems to approximate them. Mirroring this, many works we cite use modular addition (or other math tasks) to investigate the nature of various deep learning hypotheses.

Toward this goal our work is (to our knowledge) a major improvement on what exists. It’s first to present forms of the weights in deep non-linear nets, while prior work is far from how deep learning is practically implemented due to restricting to 1-hidden layer or, studying depth, but only with linear activations. Notably, prior work interpreting activations on mod addition showed a large discrepancy between 1-hidden layer and depth: McCracken et al. found the number of neural representations drops from O(n) to O(log(n)) by changing from 1-hidden ReLU layer to any depth >1. Thus, insights gained so far by papers studying depth 1 nets on mod addition don’t really generalize to depth on just the same task. This is what motivated our diverse experimental settings: to improve and give generality for the mod addition task.

To acquire a full exact model that includes depth, future work must prove why SGD learns the weights our paper describes. This has only been done for depth 1 on mod addition, and was only done after Gromov published what the weights were in 1-hidden layer networks. We hope that by providing the forms of the weights with depth, future work can prove why SGD learns this solution. As we state in the paper, it’s easier to prove something when you know what the result of training is.

The last part of your weakness:

“the stated advance towards understanding deep non-linear networks exceeds what is demonstrated in the paper”.

We don’t want to claim beyond modular addition and thus put in our limitations section: “This work only applies to DNNs trained on mod addition.” Thus, we would appreciate if you could point out where we may have overclaimed, or misworded our intent, we think possibly in the introduction quoted below?

“This is where this paper builds on prior work. We uncover the equations that model weights between hidden layers in multi-layer networks. Achieving this creates, to our knowledge, the first description of what is learned between layers in DNNs trained on a task requiring non-linear activations to generalize to the test data (the modular addition dataset is not linearly separable). This work will be of significant interest to theoreticians attempting to create a scientific theory of deep learning. After this, future work required to yield an exact model of modular addition would be a descriptive proof that SGD converges to the form of the weights we discover below, as it must explain the gradient dynamics. Our commentary on this can be found in the Discussion.”

We realized the bolded part wasn't clear about our intentions and changed it to:

“Resultantly, our work provides theoreticians attempting to create a scientific theory of deep learning with a dataset where they know in advance the forms of the weights after training. This should make it easier for future work to prove why these weights are what SGD learns, providing the first full exact model of deep learning on a dataset. Thus, we see our result as the first step toward this goal. Furthermore, our result is the natural follow-up to the papers that were able to describe deep non-linear activations on modular addition (McCracken et al), and even their manifolds (Moisescu-Pareja et al), but neither could describe the weights forming these activations.”

Does this address your concerns?

Weakness 2.

The general form provides a weaker explanation than the simple form. Unlike the simple form, it separately fits the strength and phase of each Fourier component to the learned weights. Since the fitted phase offsets can absorb the target phase, it's unclear if the formula explains a relationship between connected neurons or mainly offers a compact approximation of their weights. This distinction is important as the general form is necessary precisely where the simple form breaks down.

We see this weakness relates to the last question you asked so we will address that here too:

In the general formula, the fitted phase offsets can absorb the target neuron's measured phase. Then, what does the phase difference between the source and target neurons help explain?

We think there is a small confusion here that we only started to understand based on your question. The phase difference, which we define as the “relative phase”, offers a human interpretable way of ordering neurons. Every source neuron that inputs to the same target neuron has a different relative phase, the phase in the 2D DFT on the main frequency of the source neuron, subtracted from the phase in the main frequency of the target neuron, and circularly wrapped (normalized) to be between (-pi, pi). This phase is different for every source neuron, since each source neuron activates strongest on a different part of the activation manifold (each source neuron has a different phase) (Moisescu-Pareja et al showed that the activation manifold is spanned by all the neurons in the preceding layer with the same key frequency by having different phases). You can think of this as “different neurons need different phases so they take their max activations on different pieces of data to cover and span the full dataset”.

With this said, using the relative phase as a coordinate system to organize the source neurons and their output weight to the target neuron from -pi to pi gives a picture like Figure 1. Then fitting a simple form fit through those weights in the new coordinate system gives R^2 0.999. In the general form case, each Fourier component involved in the fit has its own target phase because it has its own coordinate system. Remember, the Fourier components here are orthogonal. So the weights inputting to a target neuron are a superposition (sum) of a few different fourier components that are all linearly independent from each other. It makes sense to us that each Fourier component has its own phase offset that needs to be fit because not every coordinate system is “centered” around the target neuron’s phase. With that said, for one target neuron and all of its many input weight source neurons, it is the case that fitting the offset phase can be done once per target neuron and Fourier component. This means that all the source neurons agree on the same coordinate system for that Fourier component, capturing this relationship between source neurons. We hope this answers your question about how the general form relates the source neurons and their outgoing weight to a target neuron in the next layer.

Question 1.

What insights about deep networks does the proposed weight description show beyond modular addition's Fourier structure? Are there any empirical results where this approach can explain the learned weights on tasks beyond modular addition?

As we state in our limitations (and is the case with the papers studying what’s learned by networks on modular addition), this work only applies to DNNs learning modular addition. For the time being, the paper serves first as a proof of concept, that it can be possible to get functional forms of the weights in deep networks, despite permutation symmetry making only the one “magic” permutation out of the factorial number of permutations have the ordering that would result in being able to fit a function through the weights. Our method finds that “magic” ordering, allowing us to show the form of the weights by fitting our forms with high scoring R^2. Secondly, we hope our result makes it possible for theorists to be able to prove why and how networks converge to these weights on modular addition, and hopefully after that, prove things about other math tasks too.

For the second part of the question, it is likely that further additions and generalizations can be made to our methods so that it works on all group tasks, which if all of which are fully understood give an infinitely large class of “exact models” where we would know what is learned, and how and why it is learned. This would be a gold standard for providing datasets with perfectly understood interpretations of what’s learned where researchers can study and question why, how and what is learned. Obviously, we won’t know until we get there whether the community learns generalizable insights into the mechanics of deep learning on natural datasets, but the goal of performing interpretability on math tasks is to discover this and many agree that this approach may bear fruit (and in some cases it already has).

Just to summarize the way the literature on modular addition evolved: modular addition is currently the dataset the community best understands what is learned (so far). The first work by Nanda et al focused on 1-hidden layer transformers and primarily looked at neuron activations and only looked at the trainable weights in the embedding matrix (not the weights after that leading to the first layer, or the weights after the layer). Following that, Gromov presented a paper about the weights in 1-hidden layer networks, and after that multiple papers provided explanations (under different assumptions) of why SGD converged to these weights. For deep networks, currently only the activations of what is learned are understood, and that work inspired this work to present the weights. We hope this pattern continues, following the work on the 1-layer case presenting weights, with the next works explaining why SGD converges to these weights.

This is the reason our paper kept mentioning our result should inspire future research to understand why SGD converges to these weights in deep non-linear networks. To be clear: we do not believe our method will generalize to natural data, and will only generalize (with additional work required) to other group tasks.

Question 2. See weakness 2.

We hope we sufficiently addressed your concerns and look forward to our continued discussion. If these answers are sufficient and clear, we hope you’ll reconsider your score. Also, for reviewer ohPo, we added more experiments on a different train/test split (60%/40%) and another activation function (swish) in Appendix A.4 and A.5. Both experiments matched the claims of the paper, and this puts us in a position where we believe our claims are strongly supported across a diverse array of experimental conditions and set-ups.

Refs:

Moisescu-Pareja et al. On the geometry and topology of representations: the manifolds of modular addition. In International Conference on Learning Representations, 2026.

Nanda et al. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023.

Gromov. Grokking modular arithmetic. arXiv preprint arXiv:2301.02679, 2023.

McCracken et al. Uncovering a universal abstract algorithm for modular addition in neural networks. NeurIPS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.