- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the weights learned by deep neural networks on modular addition, extending earlier work on networks with a single hidden layer. The authors use the phases of neurons’ two-dimensional Fourier transforms to obtain meaningful coordinates for comparing connections across hidden layers. They find that weights between neurons dominated by the same frequency often follow a cosine of their relative phase. In some deeper models, particularly those using GELU, neurons mix multiple frequencies, requiring a small sum of Fourier components to describe the weights. Experiments across architectures, depths, and training settings show that these compact approximations closely reconstruct the learned weights and generally preserve task accuracy. The contribution is an empirical characterization of trained networks; it does not explain why optimization learns these structures or establish a computational advantage from depth.
Critical strengths
- While previous work largely characterized the learned weights in networks with a single hidden layer, this paper examines connections between hidden layers in deeper models. It addresses an important gap by showing how sinusoidal weight structure extends to depth and how frequency mixing can complicate this structure.
- Using Fourier phases to compare neurons provides a meaningful way to study weight patterns despite arbitrary neuron ordering. The authors validate the resulting formulas across architectures, activation functions, depths, and random seeds, and strengthen their findings by showing that replacing learned weights with fitted approximations preserves task accuracy in the main experiments.
Critical weaknesses
- Fourier features in modular addition are well established, and Fourier-based analysis—including the 2D DFT—was already used by [1]. The paper should distinguish its specific contribution of fitting connections between hidden layers from these existing findings and techniques.
- The claim of being the first to describe learned weights “with non-linearities and depth” needs a more precise scope. Prior work already analyzes nonlinear transformers and characterizes weights in ReLU and quadratic networks, including [1][2][3][4][5]. The authors should explicitly compare these results and restrict their priority claim to the particular connections between hidden layers that they characterize.
- Prior work already establishes that networks collectively use multiple Fourier frequencies. The potentially new observation here concerns greater mixing of frequencies within individual neurons, rather than frequency diversity across the network. Although the paper discusses this mixing, it should more clearly explain and quantify what changes with depth and distinguish those effects from activation-function and training-setting differences.
[1] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress Measures for Grokking via Mechanistic Interpretability. ICLR, 2023.
[2] Jianliang He, Leda Wang, Siyu Chen, and Zhuoran Yang. On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking. arXiv:2602.16849, 2026.
[3] Andrey Gromov. Grokking Modular Arithmetic. arXiv:2301.02679, 2023.
[4] Depen Morwani, Benjamin L. Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham Kakade. Feature Emergence via Margin Maximization: Case Studies in Algebraic Tasks. ICLR, 2024.
[5] Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks. NeurIPS, 2023.
Questions for authors
- What the advantange of trainign with deep model then the canonical single layer MLP and transformer? Does it speed use the training or the emergence of representation?
- Despite teh representation, does the deep model help in generalization, i.e., grokking?
4 comments
\textbf{Prior work on weights.} As far weights are concerned, only 1-hidden layer models are understood and they are understood well. \citet{nanda2023progress} found that the learnable embedding matrix contained sinusoidal entries that could be read off the embedding vectors and avoided the permutation invariance of the weights in transformers by replacing the neurons with their fitted activations, replacing input and output weights together simultaneously with fitted second order sinusoidal functions and . This work inspired a field seeking to explain why gradient descent methods learn sinusoids in 1-layer networks, starting with \citet{gromov2023grokking}, who gave analytical equations for a toy model of the weights of 1-hidden layer multilayer perceptrons (MLPs). Later, work proving sinusoids would be learned came, with \citet{morwani2024feature} and \citet{mohamadi2023grokking} using the assumption of margin maximization to show that neurons must be sinusoids to maximize the margin. Then heuristic descriptions and proofs of stochastic gradient descent (SGD) converging to sinusoidal weights in 1-hidden layer networks began with \citet{ding2024survival, tian2025provable, tian2026composing, he2026mechanism} and recently the alternating gradient flow from small weight initialization of \citet{kunin2026alternating}.
\textcolor{blue}{Why are the weights between layers an open problem?} Part of the difficulty stems from neurons in hidden layers having permutation symmetry: in a layer with neurons there are ways to permute neurons with each permutation resulting in the exact same network, with neurons simply located in different indices of the layer across different permutations (and these reorderings can be done in all layers). When considering a 1-hidden layer network trained on , this permutation invariance within layers doesn't yet add difficulty because there is only 1 layer: the neurons can be permuted, but since they immediately connect to output logits, and the output logits have an ordering from to , it is easy to plot the output weights from a neuron and see they are sinusoidal, as found by prior work \citep{gromov2023grokking, morwani2024feature, yip2024modularadditionblackboxescompressing, wu2025towards}. \citet{moisescu2026geometry} indirectly inspected the weights via principal component analyses and diffusion maps, finding that almost all of the computation about the answer was finished by the preactivations of the second layer, and that deeper layers just seemed to rotate the neural representation manifolds in the latent weight space of the network. With all of this said, the equations describing \textit{what} the weights are in deeper networks have remained unknown, even with the forms of the activations of neurons and the forms of the manifolds corresponding to neural representation activations being known \citep{moisescu2026geometry}.
\textbf{This is where this paper builds on prior work.} We uncover the equations that model weights between hidden layers in multi-layer networks. Achieving this creates, to our knowledge, the first description of what is learned between layers in DNNs trained on a task requiring non-linear activations to generalize to the test data (the modular addition dataset is not linearly separable). \textcolor{blue}{Resultantly, our work provides theoreticians attempting to create a scientific theory of deep learning with a dataset where they know in advance the forms of the weights after training. This should make it easier for future work to prove why these weights are what SGD learns, providing the first full exact model of deep learning on a dataset. Thus, we see our result as the first step toward this goal. Furthermore, our result is the natural follow-up to the papers that were able to describe deep non-linear activations on modular addition \citep{mccracken2025uncoveringuniversalabstractalgorithm}, and even their manifolds \citep{moisescu2026geometry}, but neither could describe the weights forming these activations.}
A summary of our contributions to literature interpreting modular addition follows:
- We discover the expressions that describe weights in trained networks with different activation functions (ReLU and GELU), depths, and architectures (both transformers and multi-layer perceptrons) over a diverse range of training settings (learning rates and weight decays). We believe we are the first work to provide such a description of learned weights \textcolor{blue}{in deep non-linear networks}. Please note, prior work has achieved this in the case of deep \textbf{linear} networks \citep{saxe2013exact} and did this for the training dynamics of these deep linear networks (no non-linear activations). While our work only gives the form of the weights found after training, it is a critical step for future work investigating the training dynamics of deep nonlinear networks learning modular addition.
- The key to this result is beating the permutation invariance between neurons within layers. We do this via uncovering a novel insight: \textit{the 2d discrete Fourier transform (DFT) of neuron activations contains phase information that can be used to canonically order weights between hidden layers. This information allows for neurons to be ordered in a human-interpretable way.} In essence, we use 2d DFTs of neuron activations to acquire a human-interpretable coordinate system, yielding the ability to fully describe the inter-layer weights in deep networks with sinusoids, which require an ordering (since their domain is totally ordered from 0 to 2pi). - With the 2d DFT, we define a \textit{relative phase} that gets around the problems caused by permutation symmetry within layers by giving a canonical coordinate system based on the activations of neurons. Thus, our work is the natural sequel to both \citet{mccracken2025uncoveringuniversalabstractalgorithm} and \citet{moisescu2026geometry}, who together fully described the activations of neurons, the activation-geometry of neural representations, and even the activation-geometry of a global algorithm networks implement across neural representations to solve modular addition in 1-4 hidden layer networks. - Resultantly, our work supplements this story and serves as the logical next step: we present the form of the equations for the weights between hidden layers, and validate them exhaustively over different training conditions and architectures. As we have a full understanding of how to order neurons, we can even study the weights of an entire neural representation simultaneously.
Could you please explain how this fails to “distinguish its specific contribution of fitting connections between hidden layers from these existing findings and techniques.”?
Weakness 2.
The claim of being the first to describe learned weights “with non-linearities and depth” needs a more precise scope. Prior work already analyzes nonlinear transformers and characterizes weights in ReLU and quadratic networks, including [1][2][3][4][5]. The authors should explicitly compare these results and restrict their priority claim to the particular connections between hidden layers that they characterize.
We believe the record should show the context of the quote “with non-linearities and depth”.
We discover the expressions that describe weights in trained networks with different activation functions (ReLU and GELU), depths, and architectures (both transformers and multi-layer perceptrons) over a diverse range of training settings (learning rates and weight decays). We believe we are the first work to provide such a description of learned weights with non-linearities and depth.
We agree that using the word “and” in its mathematical sense, to include both non-linearities and depth at the same times could be confusing. We have amended this to
We believe we are the first work to provide such a description of learned weights \textcolor{blue}{in deep non-linear networks}.
Furthermore, we cite and summarize four of the five papers you mention, and all five papers are about 1-hidden layer networks. Again as our introduction above shows, we never claim novel insights about the weights in 1-hidden layer networks (before the first layer, or after the last layer). We state, five times in the introduction the word “between” in the context of between hidden layers, after first laying out a paragraph explaining the weights in 1-layer nets are solved, and then explaining why those papers and their methods don’t generalize to multilayer networks (because of permutation symmetry). We added two citations to He et al, which is a recent work we didn't know about. Please clarify if the above introduction is not clear enough about this.
Weakness 3.
Prior work already establishes that networks collectively use multiple Fourier frequencies. The potentially new observation here concerns greater mixing of frequencies within individual neurons, rather than frequency diversity across the network. Although the paper discusses this mixing, it should more clearly explain and quantify what changes with depth and distinguish those effects from activation-function and training-setting differences.
No, the mixing of frequencies was already described by McCracken et al and Moisescu-Pareja et al. Our main result is the fact we’re first to offer a description of what the learned weights are after training in networks with >1hidden layer, which is a substantial improvement on the existing literature.
The main paper quantifies what changes and what holds over an exhaustively large space of training settings, including both where training settings are conducive to learning and where they aren’t conducive to learning. We investigate two architectures (transformer and MLP), with three different non-linearities, over four depths, and 25 different combinations of (learning rate, weight decay) and report results over the entire grid. We quantify how our solution performs over this entire space, and report that in region where training parameters are conducive to learning, our solution is learned. The appendix A.4 and A.5 further investigates different train/test splits and the affect of the Swish activation function respectively. As reviewer ohPo pointed out in their critical strengths, these quantifications are broad and suggest a level of generality since our solution is learned across such a diverse array of architectures and training settings.
Question 1.
What the advantange of trainign with deep model then the canonical single layer MLP and transformer? Does it speed use the training or the emergence of representation?
The study of the training dynamics is stated to be out of the scope of our paper multiple times as we focus on what is learned, and speak at length that this is a major breakthrough as we do it for deep non-linear networks. We mention multiple times the study of how and why it is learned would be a massive breakthrough, giving deep learning its first exactly solved model, where everything about what, how and why deep neural networks learn, behave and act on a dataset is understood. With that said, prior work has shown that deeper networks learning modular addition achieve lower final cross-entropy loss and larger logit margins, see e.g. McCracken et al.
Question 2.
Despite teh representation, does the deep model help in generalization, i.e., grokking?
See question 1, where we cite prior work that shows deeper networks achieve lower final cross-entropy loss and larger logit margins.
Refs:
Moisescu-Pareja et al. On the geometry and topology of representations: the manifolds of modular addition. In International Conference on Learning Representations, 2026.
McCracken et al. Uncovering a universal abstract algorithm for modular addition in neural networks. NeurIPS 2025.
Thanks for your review. We are confident that we can fully address all your concerns because they are all questions about prior work that have been clarified with minor revisions, as there is a substantial difference between our results and prior work that can be summarized as follows. First, prior work on the weights only inspected 1-hidden layer networks and this is why after more than 20 publications on modular addition, the forms of the weights between layers in deep networks have not yet been published. Second, our work achieves a massive improvement and gives these weights between layers, by using the same method as all prior work (the discrete fourier transform (DFT)), but we find a novel insight that was missed by prior work: the DFT contains information allowing us to make a human-interpretable coordinate system and thus interpret the weights of deep networks for the first time, providing equations fitting all weights between layers with very high R2 and ablation accuracy.
Weakness 1.
We re-read our entire manuscript with your review in mind, and we can not find anywhere that we claim our novelty was the discovery of Fourier features, or that we were the first to use Fourier based analysis. To the latter part, we state that we discover how to use information in the 2d DFT in a new way that allows us to beat permutation symmetry within neurons in layers, and we even have an entire paragraph in the discussion and conclusion section speculating about why prior work missed this. No where do we claim, or want to claim that we are the first to use the 2D DFT, because that would distract from the impact of our main discovery: that we can and have modeled the weights between layers with very high R2 and ablation accuracy.
In our discussion and conclusion section we immediately state:
We realize from your feedback that it would be good to put a similar statement in the related work section, and thus we modified the last paragraph in related work as follows:
Furthermore, you state:
This was our goal with the long 2 page introduction to the paper. In it, we summarized prior work on activations, and prior work on weights, and then had a paragraph explaining exactly how our results are different to the 1-layer case: it lacks the permutation symmetry that deep networks have. Thus, we have made minor edits to absolutely clarify this and reduce all uncertainty and show them below in \textcolor{blue}{}: