- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors discovered that when training a fully-connected network on modular arithmetic tasks, networks that have grokked can exhibit a variety of distributed-ness. Some training algorithms (and some ways to modify the gradient) produce networks with less neurons participating in encoding the knowledge; these networks tend to perform better at transfer learning. Other training algorithms leads to virtually every neuron participating; and these networks perform better under neuron deletion.
Critical strengths
The main strength is the discovery that generalized networks can sit anywhere in a spectrum of distributed-ness of the learned knowledge, depending on how they are trained. Different ends of the spectrum have different advantages.
Critical weaknesses
- The authors didn't explore whether their discovery applies to other network architectures. Grokking was originally discovered in transformers, and understanding how transformers learn would be much more impactful than understanding how fully-connected networks learn.
- The paper is not very clearly written. The reviewer suggests that the authors spend more effort polishing it, maybe with the help of AI.
Questions for authors
- Would normalizing the activations (batch normalization and layer normalization) have similar effects as normalizing the updates?
- You have explored the effect of deleting neurons to minimize the accuracy. What if you delete neurons in typical pruning order, that are designed to retain as much performance as possible?
- Lines 262-263 would be a bit confusing if the reader has forgotten that you are talking about the first hidden layer, because in general, the input weight of one layer is the output weight of the preceeding layer. It's worth re-stressing that.
2 comments
Thank the authors for the reply! I agree that the new results have further strengthened the paper, but I don't think the paper has reached the bar for "clear acceptance." Therefore I maintain my rating of "weak acceptance."
Response to Reviewer XWVy
Thank you for the positive review. You summarized the main finding as networks that sit anywhere on a spectrum of distributedness depending on how they are trained, with different advantages at each end. The revision explains what sets the position on that spectrum: a neuron stays in use when its credit is large relative to the non-gradient term of its own update (Section 4.1, Table 2). Your questions sharpened both ends, and everything below is in the revised PDF.
1. Q1, normalization layers. Batch normalization, which normalizes each neuron, partly does: participation 0.81 against 0.61 without normalization, and 0.996 once its trainable affine parameters are removed. Layer normalization does the opposite: 0.36, with 164 of 256 neurons nearly unused (Appendix F.7). The cause is the trainable per-neuron gain, not the normalization:
This is the third row of the paper's central table (Section 4.1, Table 2).
2. Q2, pruning in retention order. This adds a third property to the trade-off you described.
The spectrum now has robustness to neuron loss at one end and compressibility with spare capacity at the other.
3. Other architectures. Appendix F.5 of the submission contained transformer experiments; the revision adds their mechanism and a language model.
4. Writing. The mechanism is now one paragraph that opens with the condition, with one table (Section 4.1); nearly unused neurons have a plain definition (outgoing norm below a tenth of the layer mean); and the abstract states the condition directly.
5. Q3. Section 4.1 now restates that the rescaling acts on the first hidden layer's incoming weights and that participation is measured on the same neurons' outgoing weights.
Do these results answer your questions on normalization and pruning? We would be glad to clarify anything further. If the revision strengthens your assessment, we would be grateful if your score reflected it.