- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies when neural networks learn modular solutions, using modular arithmetic mod n = p₁⋯pₖ as a testbed where the Chinese remainder theorem (CRT) provides a known subtask decomposition (compute mod each prime independently). The authors define modularity at three levels — behavioural (per-prime errors are independent), representational (each prime occupies its own subspace), and neural (each prime uses its own neurons) — and give ablation-based tests for each. One-layer transformers trained jointly on addition and multiplication (the ring Zₙ) consistently satisfy all three definitions across 20 seeds. Models trained on addition alone do not: they are either fully entangled or partially modular (9/20), exhibiting fused subtasks and asymmetric dependencies, which the authors visualize as subtask dependency graphs. An appendix shows that training on the multiplicative unit group alone does not yield modularity, suggesting the effect depends on the interaction of the two operations. Why the ring induces modularity remains open.
Critical strengths
- Both the subtask decomposition and the relevant subspaces (Fourier features) are known in advance, so modularity can be tested directly rather than discovered, avoiding search bias.
- Results are averaged over 20 seeds, replicated on several moduli, and checked on an MLP. The 0/20 fully-modular outcome for addition-only training is a strong negative result.
- Adding multiplication reliably turns modularity on. This is a useful foothold for future mechanistic work on why modularity emerges.
Critical weaknesses
- No mechanistic explanation. The title is honest about the descriptive scope, but the central question — why the ring is modular and addition is not — is addressed only by one elimination experiment (Appendix D).
- Toy scale. All models are one-layer, d=256. Implications for deeper models or real-world data are speculative.
- Evidence is output-side only. There is no analysis of how the input embeddings expose a mod p, so the claim that the model “implements the CRT” rests mainly on the readout and residual-stream ablations.
Questions for authors
- How do the embeddings represent a mod p in the ring? Does per-prime structure already appear at the embedding layer, or only after attention/MLP?
- Neural modularity is noticeably weaker for multiplication (kill mod 7 → 0.37 vs. 0.27 information-free). Is this due to the argmax neuron assignment, or do multiplication neurons genuinely mix primes?
6 comments
On multiplication being weaker
This question is also very interesting. The evidence points to multiplication genuinely being less modular. To test this, we can change how we check if an answer is correct by summing over the logits for a given output class instead of taking the argmax. This checks if the model is hiding anything when it takes the argmax. We find that the ablation results do not change meaningfully from the paper when this happens (if anything the results are slightly worse):
Table 3. Kill/keep ablations for multiplication in the ring mod , scored by the marginal over each prime: for each residue class we sum the model's probabilities over the answers in that class and take the largest. Mean over seeds at step ; baseline accuracy is . Rows are the ablation, columns are held-out accuracy mod each prime; the ablated prime is in bold. Chance: , , (majority residue) and , , (uniform).
| Readout: mod 7 | Readout: mod 11 | Readout: mod 13 | Residual: mod 7 | Residual: mod 11 | Residual: mod 13 | |
|---|---|---|---|---|---|---|
| Kill 7 | 0.14 | 1.00 | 1.00 | 0.23 | 1.00 | 1.00 |
| Kill 11 | 1.00 | 0.09 | 1.00 | 1.00 | 0.15 | 1.00 |
| Kill 13 | 1.00 | 1.00 | 0.08 | 1.00 | 1.00 | 0.11 |
| Keep 7 | 1.00 | 0.24 | 0.20 | 1.00 | 0.15 | 0.13 |
| Keep 11 | 0.29 | 1.00 | 0.15 | 0.28 | 1.00 | 0.13 |
| Keep 13 | 0.27 | 0.17 | 1.00 | 0.31 | 0.16 | 1.00 |
On toy scale and model size
Thank you very much for pointing this out, we have added a much more thorough analysis in the appendix, which confirms that our claims hold across a variety of widths, depths, and training parameters.
Table 4. Pure share of the residual stream for different widths and depths ().
| Model | ||
|---|---|---|
| 1 block, width 256 (appears in paper) | 1.00 | 1.00 |
| 1 block, width 512 | 1.00 | 0.99 |
| 1 block, width 1024 | 1.00 | 1.00 |
| 2 blocks, width 256 | 1.00 | 0.99 |
| 4 blocks, width 256 | 0.95 | 0.95 |
And then we have the same measurement for MLPs:
Table 5. Pure share of the residual stream for attention-free MLPs (), mean over seeds. Width is , training fraction , weight decay , learning rate .
| Model | ||
|---|---|---|
| 1 hidden layer, learned embedding (2 seeds) | 1.00 | 1.00 |
| 1 hidden layer, one-hot inputs (2 seeds) | 1.00 | 1.00 |
| 2 hidden layers, learned embedding (5 seeds) | 1.00 | 0.99 |
And we can test different hyperparameters and learning rates as well:
Table 6. Pure share of residual stream across different learning rates and weight decay, modulus . Only runs reaching accuracy included.
Addition ()
| lr \ wd | 0 | 0.01 | 0.1 | 0.5 | 1.0 |
|---|---|---|---|---|---|
| – | 1.00 | – | 0.94 | 1.00 | |
| 1.00 | 1.00 | 1.00 | 1.00 | – | |
| 1.00 | 1.00 | 1.00 | 1.00 | – | |
| – | 1.00 | 1.00 | – | – |
Multiplication ()
| lr \ wd | 0 | 0.01 | 0.1 | 0.5 | 1.0 |
|---|---|---|---|---|---|
| – | 0.94 | – | 0.96 | 0.96 | |
| 0.99 | 0.97 | 0.99 | 0.97 | – | |
| 1.00 | 0.99 | 1.00 | 0.98 | – | |
| – | 0.98 | 1.00 | – | – |
Initially we responded on the argmax assignment of the probabilities, rather than the argmax of the neuron selectivity scores. Here, we answer the neuron assignment question, which is now in the document as Appendix J.
Concerning the argmax of the neuron selectivity scores, it seems that the gap in modularity between addition and multiplication comes from the answers which are for some prime divisor (the zero divisors). Specifically, products which are for some prime are computed outside of the MLP, so ablating MLP neurons does not degrade those answers, and then it does not degrade those primes to chance.
First, we can check if the argmax assignment is causing issues by building our neurons sets differently. Instead of saying a neuron is in the set using the argmax, we can assign it to if it is highly selective to the prime , which we define as having a selectivity score above (selectivity for class is defined in the paper as , the sum over all the classes, and the between class variance over the total variance, so close to means that neuron is tuned specifically to that class alone).
When we do this, we find that the neuron ablations do not change meaningfully across the seeds, so argmax doesn't seem to be what is causing the lower modularity. However, if we only do the accuracy tests on answers which are nonzero , we find that the model is almost perfectly modular:
Multiplication: accuracy mod after ablating neurons (mean over 20 seeds)
All pairs
| Ablated | mod 7 | mod 11 | mod 13 |
|---|---|---|---|
| , argmax | 0.38 | 0.22 | 0.19 |
| , share | 0.39 | 0.23 | 0.20 |
| All neurons | 0.35 | 0.22 | 0.18 |
| Chance | 0.27 | 0.17 | 0.15 |
Answer
| Ablated | mod 7 | mod 11 | mod 13 |
|---|---|---|---|
| , argmax | 0.22 | 0.12 | 0.11 |
| , share | 0.22 | 0.12 | 0.10 |
| All neurons | 0.20 | 0.12 | 0.11 |
| Chance | 0.17 | 0.11 | 0.09 |
The full table is here (units mean only answers nonzero mod every prime):
Multiplication: neuron set ablations on all pairs and on pairs whose answer is a unit (mean over 20 seeds; chance on units: 0.17, 0.10, 0.08)
kill
| Pairs | mod 7 | mod 11 | mod 13 | |
|---|---|---|---|---|
| All | 7 | 0.38 | 1.00 | 1.00 |
| All | 11 | 1.00 | 0.22 | 1.00 |
| All | 13 | 1.00 | 1.00 | 0.19 |
| Units | 7 | 0.22 | 1.00 | 1.00 |
| Units | 11 | 1.00 | 0.12 | 1.00 |
| Units | 13 | 1.00 | 1.00 | 0.10 |
keep
| Pairs | mod 7 | mod 11 | mod 13 | |
|---|---|---|---|---|
| All | 7 | 1.00 | 0.21 | 0.17 |
| All | 11 | 0.36 | 1.00 | 0.18 |
| All | 13 | 0.36 | 0.22 | 1.00 |
| Units | 7 | 1.00 | 0.12 | 0.11 |
| Units | 11 | 0.21 | 1.00 | 0.10 |
| Units | 13 | 0.21 | 0.11 | 1.00 |
Next, if we set every MLP neuron to its mean value, we find that naturally the model outputs at chance, but shockingly not on the zero divisors, where it stays unscathed, so the zero divisors are being calculated somewhere in the attention heads or at the input layer:
Every MLP neuron ablated (set to mean): accuracy mod by whether the answer is mod (mean over 20 seeds)
| Op. | Answer | mod 7 | mod 11 | mod 13 |
|---|---|---|---|---|
| 0.95 | 0.93 | 0.88 | ||
| 0.21 | 0.11 | 0.11 | ||
| 0.15 | 0.14 | 0.09 | ||
| 0.14 | 0.10 | 0.09 | ||
| Chance | 0.17 | 0.10 | 0.09 |
So these products are not being computed in the MLP at all.
As another interesting experiment, we can ask if the neurons which are assigned to a specific prime under addition are the same as those assigned to under multiplication, or if a neuron can be selective for, say, under addition, but under multiplication. We find that there is almost no crossover though, on highly selective neurons.
Neurons by selective prime (share ) under (rows) and (columns) (mean count over 20 seeds)
| : 7 | : 11 | : 13 | : none | |
|---|---|---|---|---|
| : 7 | 137 | 0 | 0 | 3 |
| : 11 | 0 | 241 | 0 | 3 |
| : 13 | 0 | 0 | 330 | 34 |
| : none | 0 | 0 | 3 | 273 |
Thank you very much for the review, we appreciate that you think this may be a useful foothold on why modularity emerges.
On a mechanistic explanation
We agree that including an answer to why the ring is modular would strengthen the paper. Here is the argument, which we have added as a section to the paper:
With some experiments showing (3) to hold in the paper. The idea broadly comes down to the fact that both operations share an embedding and unembedding layer. By this we mean that given some amodn, there is an embedding and unembedding for that amodn, which is the same whether a appears in an addition equation or a multiplication one. But addition and multiplication have different structures, because addition uses additive Fourier frequencies for computations, and multiplication uses multiplicative characters. This "tying" together of the two operations in this manner causes them to influence each other in the way described above.
How do input embeddings represent tokens?
This is a very interesting question. Doing the same technique as before (taking the Fourier transform over the embeddings, and checking what proportion of each vector is in the subspaces spanned by the pure-prime frequencies), we can get the following, averaged over 20 seeds:
Table 1. Share of the input embedding's energy on pure, two-prime and joint Fourier frequencies, ring mod 1001. Mean ± standard deviation over 20 seeds at step 80,000 (all at 100% accuracy on both operations). The last row is the share for a matrix with no structure, such as the random initialization.
Here we can see that the model uses the pure-frequency subspaces at the embedding layer, so the modular structure is present at input embeddings as well.
One surprising thing is that if we train an MLP with one-hot (unlearned) embeddings, it will still be modular, for example in here:
Table 2. Pure share of the residual stream for attention-free MLPs (n=1001), mean over seeds. Width is 1024, training fraction 0.5, weight decay 0.1, learning rate 0.001.