acceptodds
Under review as a conference paper at ICLR 2027

Grokking Can Be Steered: Selecting Desirable Solutions By Choice Of Learning Rule

Abstract

During grokking, a network transitions from memorization to generalization, yet what shapes the neural implementation (circuit) of the generalizing solution and specifically the number of participating neurons (its size) remains unclear. We find that the learning rule, the combination of credit assignment algorithm and optimizer, controls this: On modular addition, every learning rule we study that groks recovers the same Fourier-based algorithm, yet implemented by circuits that differ in size. Why? Experimentally, we find that equalizing the magnitude of credit assigned to each neuron is responsible, e.g., expanding the effective circuit size, measured from the outgoing weights, from 60% to 99% of the layer's neurons at similar training and test accuracy. We vary which parameters are normalized together, from the whole layer to individual neurons, and find that only normalization per neuron fully distributes the circuit. Credit magnitudes are usually set by the built-in normalization of standard optimizers: Nero, which normalizes per neuron, distributes the computation, whereas Adam and seven others, including RMSProp, Muon, and Shampoo, use other normalization schemes and retain small, concentrated circuits. Distributed circuits tolerate neuron deletion better, while concentrated circuits transfer about twice as fast to a task that reuses the same features. Whether a learning rule groks at all is visible after 2% of training: The rules that memorize retain larger weight components in directions that leave training outputs locally unchanged. Learning rule choice thus steers circuit size, trading robustness against transfer speed without changing the learned algorithm.

Primary area:
interpretability and explainable AI
Keywords:
grokking, mechanistic interpretability, neural circuits, implicit bias, optimization, credit assignment, learning rules

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XWVyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors discovered that when training a fully-connected network on modular arithmetic tasks, networks that have grokked can exhibit a variety of distributed-ness. Some training algorithms (and some ways to modify the gradient) produce networks with less neurons participating in encoding the knowledge; these networks tend to perform better at transfer learning. Other training algorithms leads to virtually every neuron participating; and these networks perform better under neuron deletion.

Critical strengths

The main strength is the discovery that generalized networks can sit anywhere in a spectrum of distributed-ness of the learned knowledge, depending on how they are trained. Different ends of the spectrum have different advantages.

Critical weaknesses
  1. The authors didn't explore whether their discovery applies to other network architectures. Grokking was originally discovered in transformers, and understanding how transformers learn would be much more impactful than understanding how fully-connected networks learn.
  2. The paper is not very clearly written. The reviewer suggests that the authors spend more effort polishing it, maybe with the help of AI.
Questions for authors
  1. Would normalizing the activations (batch normalization and layer normalization) have similar effects as normalizing the updates?
  2. You have explored the effect of deleting neurons to minimize the accuracy. What if you delete neurons in typical pruning order, that are designed to retain as much performance as possible?
  3. Lines 262-263 would be a bit confusing if the reader has forgotten that you are talking about the first hidden layer, because in general, the input weight of one layer is the output weight of the preceeding layer. It's worth re-stressing that.
2 comments
AuthorsOpenReview ↗

Response to Reviewer XWVy

Thank you for the positive review. You summarized the main finding as networks that sit anywhere on a spectrum of distributedness depending on how they are trained, with different advantages at each end. The revision explains what sets the position on that spectrum: a neuron stays in use when its credit is large relative to the non-gradient term of its own update (Section 4.1, Table 2). Your questions sharpened both ends, and everything below is in the revised PDF.

1. Q1, normalization layers. Batch normalization, which normalizes each neuron, partly does: participation 0.81 against 0.61 without normalization, and 0.996 once its trainable affine parameters are removed. Layer normalization does the opposite: 0.36, with 164 of 256 neurons nearly unused (Appendix F.7). The cause is the trainable per-neuron gain, not the normalization:

  • A trainable gain without any normalization concentrates the circuit as much as layer normalization does (0.099 against 0.094).
  • The gain and the outgoing weights carry their own decay, which equalizing the incoming gradients does not reach (0.364). Rebalancing their decay per neuron restores the full layer (0.997 and 1.000) in all ten runs (five fresh seeds, two placements).

This is the third row of the paper's central table (Section 4.1, Table 2).

2. Q2, pruning in retention order. This adds a third property to the trade-off you described.

  • Magnitude pruning removes 62% of the concentrated circuit's neurons at test accuracy 0.9, against 20% of the distributed one, and greedy pruning gives the same order (Section 5, Table 3).
  • The extracted circuit depends on training. At width 2048, greedy pruning keeps 33 to 98 neurons of the reference, 138 to 229 under equalization and 14 to 22 with a decayed gain, in this order in every seed, also when pruning stops one or two points below each network's own accuracy (Appendix E.2).

The spectrum now has robustness to neuron loss at one end and compressibility with spare capacity at the other.

3. Other architectures. Appendix F.5 of the submission contained transformer experiments; the revision adds their mechanism and a language model.

  • In a transformer with a 4096-neuron MLP, equalization raises participation from 0.61 to 0.99 and norm-ranked deletion tolerance from 0.58 to 0.94 (Appendix F.5). The effect runs through Adam's ε (Table 2, second row). In the canonical one-block transformer the MLP is already distributed (0.955), and equalization leaves it there.
  • In a 30M-parameter GPT trained on TinyStories with Adam and L2 regularization, 24% of the MLP units end nearly unused; balancing each unit's decay leaves 1.3% at lower validation loss in all five seeds. AdamW leaves 0.9%, as the condition predicts (Appendix F.8).

4. Writing. The mechanism is now one paragraph that opens with the condition, with one table (Section 4.1); nearly unused neurons have a plain definition (outgoing norm below a tenth of the layer mean); and the abstract states the condition directly.

5. Q3. Section 4.1 now restates that the rescaling acts on the first hidden layer's incoming weights and that participation is measured on the same neurons' outgoing weights.

Do these results answer your questions on normalization and pruning? We would be glad to clarify anything further. If the revision strengthens your assessment, we would be grateful if your score reflected it.

Thank the authors for the reply! I agree that the new results have further strengthened the paper, but I don't think the paper has reached the bar for "clear acceptance." Therefore I maintain my rating of "weak acceptance."

Reviewer_aHPKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

A generalized solution might not be uniquely implemented by a network. For example, for the modular addition problem, two neural networks can represent the same Fourier-based generalized solution but one relies mainly on a subset of neurons while the other distributes the computation equally over the entire network layer. The central question studied in this paper is, whether changing the learning could change the implementation of the same learned algorithm. More specifically, is that implementation discrepancy required by the task or selected by the learning rule? If latter, which parts of the learning rule control the discrepancy? Does such a discrepancy affect robustness and adaptivity? This paper conducts thorough experiments to answer the above questions. The main conclusions are: (i) different learning rules can indeed affect how broadly neurons contribute to the implementation; (ii) widely-distributed/concentrated circuits (implementations) admit a tradeoff between robustness and adaptivity.

Critical strengths

According to my understanding, I particularly value the following two perspectives of the paper.

  1. The paper identifies a concrete method to distribute the computational contribution over neurons via controlled experiments. Specifically, ordinary backpropagation assigns unequal signals to neurons (measured by the magnitude of the gradient of the incoming weights), resulting in concentrated circuit. The authors found that balancing the magnitudes of signals over neurons (while preserving the gradient direction) before the optimizer leads to a more broadly-distributed contributions of neurons into the computation.

  2. The paper further demonstrates that similar solutions implemented by different circuits can have different downstream performances — widely-distributed circuits tolerate neuron deletion better (better robustness) while concentrated circuits transfer better to a new task (better adaptivity). While the robustness observation is fairly intuitive, the adaptivity perspective is less obvious and somewhat interesting. It might lead to useful training method design for specific tasks.

Critical weaknesses
  1. The paper's contributions are limited. I think many of the conclusions drawn in the paper are quite intuitive and well believed. For example, the magnitude of the gradient affects the computational contribution since weights with larger magnitudes contribute more to the gradient update and distributed circuits tolerate neuron deletion better since spreading computation could reduce the reliance on individual neurons. As for the grokking section 4.3, the conclusion is well-understood both theoretically and empirically by existing works. It seems that the authors simply made experiments to justify them. (Though, I do value that the authors have done sufficient and thorough controlled experiments.)

  2. I further have the following concerns. (i) I don't see that the question studied in this paper is that related to "grokking". Grokking emphasizes delayed generalization, while this paper seems to mainly focus on understanding different implementations of a similar generalized solution. Can you explain this a bit more? Moreover, the statement "Whether a learning rule groks at all is visible after 2% of training." in the abstract seems to be too strong to hold. (ii) I'm not convinced regarding the conclusion in Section 5.2. Specifically, I believe that even two different networks with the same participation level (i.e., different subsets of neurons participate) can have different adaptation speeds. Hence, I believe neuron participation level alone does not fully explain its adaptation performance.

Questions for authors

Can the authors explain how would the observations/conclusions in this work help understanding grokking or any other ML/AL problems (including representation, optimization, generalization, etc.)? I would like to know whether this work could lead to some meaningful theoretical/empirical research questions for future works.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aHPK

Thank you for the review, and for valuing the controlled experiments and the adaptation result. Your two concerns and your question led to changes in the revised PDF.

1. What the controls establish beyond the intuition. The intuitive reading, that a neuron with a larger gradient contributes more, does not hold under Adam. Equalizing every neuron's gradient norm while holding its ratio to the decay fixed leaves the circuit concentrated (0.574 against 0.606), and Lion and Adam-mini, which normalize the size of each neuron's update, stay concentrated as well. What decides is each neuron's credit relative to the non-gradient term of its update. Each route to unused neurons was known on its own; the shared condition and its matched controls are new:

  1. Coupled decay, Adam's ε and a decayed per-neuron gain follow one per-neuron condition, with matched controls for each (Section 4.1, Table 2).
  2. Equalizing credit in groups of eight or more neurons leaves the circuit as concentrated as the reference, within 0.02; only one factor per neuron distributes it fully (Section 4.2).
  3. The circuit that pruning extracts for the same algorithm depends on training. At width 2048, where the reference and equalization both start at test accuracy 0.998, greedy pruning keeps 33 to 98 neurons of the reference, 138 to 229 under equalization and 14 to 22 with a decayed per-neuron gain, in this order in every seed (Section 5, Appendix E.2).
  4. In a 30M-parameter language model trained with Adam and L2, balancing each unit's decay reduces nearly unused MLP units from 24% to 1.3% at lower validation loss; AdamW leaves 0.9%, as the condition predicts (Appendix F.8).

2. Relation to grokking. Our object is the circuit of the grokked solution: grokking produces a generalizing solution, and we ask how many neurons implement it and what sets that number.

  • The circuit concentrates during the grokking delay. In the reference, participation is 0.997 when the network first fits the training set, has fallen to 0.59 to 0.67 when test accuracy reaches 0.95, and changes by at most 0.05 afterwards (five seeds; Section 4.1, Appendix C.5). Concentration forms together with the generalizing computation.
  • The learning rule sets that number, and it can be changed after grokking. By the checks of Appendix B.3, every rule that groks recovers the same Fourier algorithm. Switching a grokked network to equalization takes participation from 0.60 to 0.99 at test accuracy above 0.98.
  • Section 4.3. We agree that this section confirms the existing weight-decay account. It is one paragraph that says so and is not among the contributions, and the "2% of training" statement is removed from the abstract and the main text after a within-rule test (Appendix H).

3. Participation and adaptation. You are right that participation alone does not determine adaptation, and the revision measures what drives the difference: the nearly unused neurons (Section 5.2, Appendix E.4).

  • Masking the 75 to 96 nearly unused neurons of the concentrated network, at unchanged width and test accuracy, slows its adaptation in every seed and closes 67% and 54% of the mean gap to the distributed network on (a − b) and (b − a) mod p.
  • Targets that change only the readout show no consistent advantage for the concentrated network.
  • The ordering persists across three ways of producing a distributed circuit: equalization, per-neuron decay and an outgoing-norm constraint.

4. Your question: what this helps understand, and which questions it opens.

  • Interpretability. The size of a circuit, including the one pruning extracts, depends on how the optimizer treats each neuron's credit relative to its decay or ε. Claims about circuit size should state the training configuration. Whether discovery methods such as ACDC return different circuits for the same algorithm under different training is open.
  • Optimization. Per-neuron balance is a design axis that recent optimizers address directly (Nero; Aurora, which prevents neuron death under Muon). The condition makes testable predictions for them, as the Adam-mini result shows.
  • Training design. You noted that the adaptation result might lead to training methods for specific tasks. The balance trades robustness to neuron loss against spare capacity for later tasks, which connects to work on dormant neurons in continual learning (Sokar et al., 2023; Dohare et al., 2024).

Do the timing result and the spare-neuron analysis address your two concerns? If anything remains unclear, we would be glad to answer in the discussion. You valued the concrete method and the controlled experiments, and the new results follow the same design; if they address your concerns, we would be grateful if you reconsidered your scores.

I thank the authors for your careful response! I'm still concerned about the following points.

  1. Regarding what the controls establish beyond the intuition, I don't think the authors directly answered my question about the limited contribution of the paper. Your claim "a neuron with a larger gradient contributes more, does not hold under Adam" is ambiguous. What does "contribute more" refer to?

  2. Regarding relation to grokking, are you saying that, a grokking training trajectory has a different effect on the circuit implementation of the learned generalization solution compared to a normal trained generalization solution (i.e., without overfitting)? In other words, I think you were not studying grokking but rather studying how does the learning rule affect the circuit implementation of a generalizing solution when grokking happens. In fact, if you're using weight decay, this observation of concentration is quite intuitive. However, I don't see why the story is different when grokking is not happening, like under the scenarios when there is no overfitting or other generalization patterns such as double descent.

  3. Regarding what this helps understand, I agree with the reviewer hLxZ that the paper does not provide enough insights towards the understanding of grokking.

I therefore maintain my score at this point.

Thank you for the follow-up. We ran the test your second point suggests, and it also bears on your third.

1. "Contribute", and what is new

A neuron's contribution is its share of the norm of the layer's outgoing weights (Equation 1). The activation-weighted and ablation measures give the same picture: under equalization they rise from 0.34 to 0.81 and from 0.42 to 0.80 (Appendix B.1). Our earlier sentence meant that under Adam the size of a neuron's gradient does not decide this share.

The intuition in your review has two parts: neurons with larger gradients grow more, and weight decay concentrates the circuit. Neither part explains the result under Adam:

  • We give every neuron the same gradient norm and scale its decay by the same factor. The circuit stays as concentrated as the reference (0.574 against 0.606, Table 2).
  • With decoupled decay, as in AdamW, and the paper's learning rate, Adam's participation rises as the decay gets stronger: 0.72 at λ = 0.3, 0.88 at λ = 1 and 1.00 at λ = 10, with mean test accuracy between 0.95 and 0.99 (new, three to five seeds per value). Weight decay concentrates the circuit only in its coupled form.

The circuit follows the ratio of each neuron's credit to the part of its update that does not come from the gradient. Rescaling only that part, with the gradients unchanged, gives 0.993 (Table 2).

We see this condition as the main contribution. It accounts for three routes to unused neurons that earlier work reported one at a time: coupled decay, Adam's ε and decayed per-neuron gains. The same condition predicts which optimizers concentrate the circuit, and it lets us move the circuit of one fixed algorithm from 60% to 99% of the layer. Circuit size set this way changes deletion tolerance, transfer and the circuit that pruning extracts (33 to 98 neurons against 138 to 229 at width 2048, Section 5).

2. Grokking, or generalization in general

We agree with this description of the paper. It asks how the learning rule sets the circuit of a generalizing solution. We use grokking on modular addition because there the algorithm is known and can be verified to be the same under every learning rule that groks, so circuit sizes can be compared at a fixed algorithm.

The story is the same without grokking. On sparse parity, which generalizes with no delay, equalization raises participation from 0.15 to 0.96 (Appendix F.3). On modular addition we shortened the delay on the same network (new, five seeds per setting). Test accuracy is still zero at the epoch the training set is first fitted, even with 95% of the pairs used for training. More training data and a larger learning rate cut the delay from about 1300 epochs to about 70:

SettingDelay of the reference (epochs)Participation, referenceParticipation, equalization
Paper's setting: 60% of pairs for training, lr 10⁻³1120 to 13500.6030.992
95% of pairs for training, lr 10⁻³410 to 5200.6400.983
95% of pairs for training, lr 10⁻²70 to 900.5380.987

In the last row the training set is fitted at epoch 30 and test accuracy passes 0.95 by epoch 120, below the paper's 500-epoch criterion for grokking. Equalization ends above the reference in all seeds and settings. Across the three delays the reference stays between 0.54 and 0.64 and equalization at 0.98 or above.

3. What this says about grokking

The sparse circuit seen during grokking in our reference comes from the learning rule. With equalization the network groks after a delay of 1410 to 1980 epochs, and its participation stays above 0.96 at every logged epoch.

Under coupled L2, weight decay does two things in these networks. It moves the network from memorization to generalization, as in the established account of grokking (Nanda et al., 2023; Varma et al., 2023). It also removes the neurons whose credit is small relative to their decay. Equalization keeps the first effect and removes the second. The timing in the reference agrees with this. In the paper's setting, 77% to 94% of the reference's concentration happens before test accuracy reaches 0.95. With 95% of the pairs for training, 59% to 75% of it happens after test accuracy reaches 0.95, at both learning rates.

This qualifies the view of grokking as a sparse generalizing subnetwork that takes over from a dense memorizing one (Merrill et al., 2023). It also means that a progress measure built on neuron sparsity would miss the transition under equalization. In the paper's setting equalized networks grok about 500 epochs later than the reference, and what sets the delay once concentration is removed is open. We will add the delay experiment to Appendix C.5.

Your first review asked how these observations help to understand grokking. Under coupled weight decay, generalization and circuit concentration are separate effects of the decay, and the learning rule decides whether the second one happens.

Reviewer_hLxZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains a small two-layer MLP on modular addition and observes that with Adam, about a third of the hidden neurons end up dead. It shows that rescaling each neuron’s gradient to a common size keeps all neurons active without changing the learned algorithm, and that the one standard optimizer with per-neuron normalization (Nero) does the same. It further reports that networks with dead neurons learn a related task faster but tolerate neuron pruning worse, and proposes a weight-based quantity that separates learning rules that grok from those that don’t.

Critical strengths
  • I found the presentation to be reasonably good and the paper to be well structured.
  • I find it interesting to read alternative viewpoints on specific aspects of how neural networks are working.
  • I believe the authors sincerely attempted to be objective and fair in their experiments, although I find some issues which i describe later.
Critical weaknesses

At a high level I'm not convinced of the significance of this work. Grokking is already quite a niche, toy topic. Although a toy problem can be instructive, it is the responsibility of the paper to demonstrate this relevance or broader relevance. Then this work attempts to connect grokking to a concepts of circuit size, credit assignment, and optimizer choice. Even the claimed contributions in the introduction don't appear particularly significant, even if they were fully realised. The experimental setting is toy and at tiny scale with no relation to practical usage. I believe the true mechanism of the results observed is actually due to coupled weight decay (at least in some cases), which i detail further below, rather than the specific claims of the authors. Having read the paper, I don't feel there is much to take away.

More specific weaknesses are listed below:

  • I'm struggle to understand how the papers intervention in eqn (2) acts as the authors intend. Optimizers are often approximately (e.g. Adam) or exactly (e.g. Lion) invariant to a constant rescale of the gradients. So an intervention of gradient rescaling before these optimizers is difficult to interpret. Therefore, as I understand in these cases the actual intervention is really only having any effect due to the choice of using coupled weight decay (I note this is not typically used in practice). Ultimately, the input to the optimizer because of this will be: s⋅g+λθ=s⋅(g+(λ/s)⋅θ)s\cdot g + \lambda \theta = s·(g + (\lambda/s)·\theta) (where ss is the authors rescale factor). Thus the net effect of the authors intervention for optimizers invariant to scale will be to rescale just the weight decay component. This seems to be a major missed detail in the paper and I note the impact even in Lion is quite big (i.e. raises participation from 0.535 to 0.984 in Table 11, App. C.3). Therefore the results seem to be largely driven not by some notion of "credit assignment" but instead by some rescaling of weight decay per parameter.
  • The authors claim that: "DFC groks, which shows that delayed generalization does not depend on a backpropagated loss gradient". As I understand this is false as each layer receives signal via the network’s Jacobian multiplied by uu which is coming from the squared error loss gradient, and therefore still backpropogates a loss gradient.
  • The smaller circuit the paper claims to be due to Adam seems to mostly be an oversized layer in which 1/3 of the neurons end up dead due to the model being overparameterized for this trivial task (i.e. 84 of 256 are dead IIUC from App. B.1). Once you have only 32 neurons, the network uses 30 of them (Table 20). So I'm not convinced this is primarily due to the optimizer in the way described in the paper.
  • Related to the point above, I think the robustness-vs-transfer trade-off might simply be just that these dead neurons are now available for a new task. Also the transfer benefit appears for only one task pair (addition to subtraction, not for multiplication)
  • The grokking predictor is tested on the 9 learning rules where according to the claims of the authors the rule already determines the outcome. So its possible that the predictor could just be predicting the rule itself and be unrelated to grokking. The authors would need a proper experimental design to prove their claim that they can predict grokking.
  • Minor: Maybe this is a broader issue in this literature than just this specific paper, but I find the language in this paper obscures clarity in a way that could be perceived as attempting appear more sophisticated than necessary. For example, framing in terms of “circuit size" and “concentration” seem a bit convoluted as a way to talk about the simple fact that if you have sufficient over capacity some neurons will not be used.
Questions for authors

See weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hLxZ (1/2)

Thank you for the detailed review. Your analysis of how gradient rescaling acts under Adam was correct, and following it led to the central result of the revision: one per-neuron condition behind three known routes to unused neurons. Everything below is in the revised PDF.

1. Mechanism: confirmed, and it generalizes.

  • Confirmed. Under Adam with coupled L2, equalization acts through each neuron's ratio of gradient to decay (Appendix C.6, Table 12). Rescaling only each neuron's decay gives the participation of equalization (0.993 in both); equalizing the gradient while holding that ratio fixed gives the reference (0.574 against 0.606). On the activation-weighted and ablation measures, decay-only rescaling reaches 0.98 and 0.92 and the ratio-held control 0.28 and 0.38, so the neurons change how they contribute to the output, not only the scale of their weights.
  • Generalized. We then tested the other terms of the update that do not scale with the gradient. Two more follow the same condition, each a known route to unused neurons (Section 4.1, Table 2, five seeds per cell):
Non-gradient termNetworkReferenceEqualizedOnly that term rescaledRatio held fixed
Coupled decayMLP, Adam with L20.6060.9930.9930.574
Adam's εTransformer, AdamW0.6090.9940.9810.818
Decay of the gainMLP, layer normalization0.3660.9960.9970.343

In the gain row, "Equalized" equalizes the gradients of the incoming rows, the gains and the outgoing weights, each with its own per-neuron factor; the incoming rows alone give 0.364. For ε, holding the ratio fixed removes about half of the effect (0.818), so the ratio explains a large part of the ε route but not all of it; the paper states this in Table 2 and the Limitations.

  • Appendix C.6 writes out each update. Coupled decay enters Adam's input. ε enters the denominator; once it exceeds the square root of the second-moment estimate, the step is approximately m̂/ε and the gradient competes with ελw. A gain and the outgoing weights carry their own decay.
  • The condition covers Lion, which you raised, and SGD (Appendix C.3). It also predicts Adam-mini: normalizing gradient and decay per neuron together leaves the ratio unchanged, and the circuit stays concentrated (0.586 against 0.594).
  • The separate reports are now cited (Yaguchi et al., 2018; Mehta et al., 2019; Kosson et al., 2024; Dufort-Labbé et al., 2025; Kolb et al., 2025). The paper contributes the shared condition, matched controls for each route, and its consequence for the circuit of a fixed algorithm.

2. Significance and practical scope. You note that coupled decay is not typically used in practice and that the setting is a toy. The condition states where nearly unused neurons arise and where they do not, and we tested it in a 30M-parameter GPT trained on 393M tokens of TinyStories (Appendix F.8, Table 31), with Adam and coupled L2 (as torch.optim.Adam applies weight_decay) and with AdamW:

Validation lossNearly unused MLP units
Adam with L21.37823.9%
Adam with L2, per-unit decay balance1.3671.3%
AdamW1.3300.9%
  • Under Adam with L2, balancing each unit's decay removes 95% of the nearly unused units and lowers the loss in all five seeds. Under AdamW the condition predicts few such units, and 0.9% remain.
  • AdamW can still leave neurons unused: when ε dominates Adam's denominator, as in the wide-MLP transformer, ε sets the scale against which the gradient is compared, as the decay does under coupled L2 (Table 2, second row; Appendix C.6). Where no term dominates, as for the MLP under AdamW, the reference is already broadly distributed (0.91).
  • For interpretability, the circuit that pruning extracts for one algorithm changes several-fold with training (point 3 in the second part of this reply).
AuthorsOpenReview ↗

Response to Reviewer hLxZ (2/2)

3. Overcapacity. We agree that spare capacity is what allows neurons to go unused. How much of it is used is set by the learning rule:

  • The width sweep shows both parts (Appendix F.2, Table 26). At width 32 the effective counts are 30 for the reference and 31 under equalization; at width 2048 they are 332 and 2046. The reference's own count keeps growing with width (101 at 128, 205 at 512, 332 at 2048), so it is not fixed by the task.
  • Participation measures more than unused neurons. From β = 0.25 on no neuron is nearly unused, yet participation keeps rising from 0.93 to 0.99, and adaptation to subtraction slows from 940 to 1540 epochs (Section 4.1, Appendix E.4, Table 22).
  • Switching an already grokked reference network to equalization takes participation from 0.60 to 0.99 at test accuracy above 0.98, and the ablation measure from 0.40 to 0.72 (Section 4.1, Appendix E.4).
  • The extracted circuit follows. At width 2048, where both networks start at test accuracy 0.998, greedy pruning (order chosen on training loss, read off the held-out curve) keeps 33 to 98 neurons of the reference and 138 to 229 under equalization; magnitude pruning keeps 143 against 1692. The order holds in every seed one and two points below each network's own accuracy (Section 5, Appendix E.2, Table 18).

4. Transfer and spare neurons. Your explanation is right, and the revision measures it (Section 5.2, Appendix E.4, Table 21).

  • Masking the 75 to 96 nearly unused neurons of the concentrated network, at unchanged width and test accuracy 0.996, slows its adaptation in every seed and closes 67% and 54% of the mean gap to the distributed network on (a − b) and (b − a) mod p.
  • Targets that change only the readout show no consistent advantage for the concentrated network, and multiplication, which does not share the Fourier features of addition, shows no transfer from any network.
  • The ordering holds for distributed circuits produced by equalization, by per-neuron decay alone and by an outgoing-norm constraint.
  • The same spare neurons make the concentrated circuit easier to prune: magnitude pruning removes 62% of its neurons at test accuracy 0.9, against 20%. Section 5 now presents the trade-off as robustness to neuron loss against compressibility and spare capacity.

5. DFC. Thank you for this correction. Section 3.1 now describes DFC as feedback control whose signal reaches each layer through the transposed Jacobian, and the claim is removed.

6. The grokking predictor. We ran the within-rule design you proposed (two rules, five seeds per setting; Appendix H, Table 32). Within a rule no common threshold separates the outcomes, so the prediction claim is removed from the abstract and the main text (Section 4.3).

7. Language. Nearly unused neurons are defined in plain terms (outgoing norm below a tenth of the layer mean; about 84 of 256 in the reference, Section 4) and counted next to participation. Participation remains the graded measure for the reason in point 3.

Where each concern is addressed in the revised PDF:

  • Mechanism: Section 4.1, Table 2; Appendices C.6, F.5 and F.7.
  • Practice and relevance: Appendix F.8, Table 2 (ε under AdamW), Section 5 with Appendix E.2.
  • Overcapacity: Section 4.1, Appendices E.4 and F.2.
  • Transfer: Section 5.2, Appendix E.4.
  • DFC and the predictor: Sections 3.1 and 4.3, Appendix H.

Do the per-neuron condition, with the update equations in Appendix C.6, and the language-model test address your concerns about the mechanism and its relevance? If a point remains open, we would like to resolve it in the discussion. If the controls address your concerns, we would be grateful if your scores reflected that.

I thank the authors for their response. Given that they accept my main weaknesses hold and these issues significantly undermine the paper and the required changes to the paper are substantial, I maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

Summary of the revision

We thank the three reviewers. Their reviews changed the paper in three ways: the mechanism is identified and tested, its relevance is tested in a language model, and two claims are corrected. Every item below is in the revised PDF. A version with all changes marked against the submission is included as supplementary material.

1. One per-neuron condition explains the effect (Section 4.1, Table 2, Appendix C.6). A neuron stays in use when its credit is large relative to the non-gradient term of its own update. Reviewer hLxZ identified this for the coupled weight decay. Matched controls confirm it and show the same condition for Adam's ε and for the decay of a trainable per-neuron gain, three routes to unused neurons that earlier work reported one at a time.

  • Rescaling only that term per neuron reproduces the participation of credit equalization (0.993, 0.981 and 0.997, against 0.993, 0.994 and 0.996).
  • Holding each neuron's ratio fixed keeps the circuit concentrated for the decay and the gain, and removes about half of the effect for ε.
  • The condition also accounts for Lion and Adam-mini, which normalize each neuron's update and still keep the circuit concentrated.

2. A language model (Appendix F.8). In a 30M-parameter GPT trained with Adam and coupled L2 regularization, 24% of the MLP units end nearly unused. Balancing each unit's decay leaves 1.3% and lowers validation loss in all five seeds. AdamW leaves 0.9%, as the condition predicts.

3. The circuit that pruning extracts depends on training (Section 5, Appendix E.2, Table 18). At width 2048, where the reference and the equalized network both start at test accuracy 0.998, greedy pruning keeps 33 to 98 neurons of the reference, 138 to 229 under equalization and 14 to 22 with a decayed per-neuron gain. The order holds in every seed, also when pruning stops one or two points below each network's own accuracy.

4. Grokking (Section 4.1, Appendix C.5). The circuit concentrates during the grokking delay and changes by at most 0.05 in participation afterwards.

5. Corrections. The DFC description (Section 3.1) and the early-prediction claim (abstract, Section 4.3, Appendix H), both after reviewer hLxZ's analysis.

Where the main concerns are addressed

ConcernRaised byLocation
How the intervention acts under AdamhLxZSection 4.1, Table 2, Appendix C.6
Relevance beyond the toy settinghLxZ, XWVyAppendices F.5 and F.8, Section 5
Overcapacity and unused neuronshLxZSection 4.1, Appendices E.4 and F.2
Transfer and spare neuronshLxZ, aHPKSection 5.2, Appendix E.4
Relation to grokking, the 2% claimaHPK, hLxZSections 4.1 and 4.3, Appendices C.5 and H
Normalization layers, pruning orderXWVyAppendix F.7, Section 5, Appendix E.2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.