- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how changing where weight decay is applied in a Transformer affects grokking. The basic finding is that allocating more of the decay to attention, MLP and LayerNorm parameters, rather than embeddings and the output head, generally makes generalization happen earlier in the settings studied.
Critical strengths
Interest in Grokking: Grokking is a consequential topic for understanding neural network generalization. For this, I appreciate the authors’ interest in the topic.
Critical weaknesses
1. Limited explanatory novelty: Prior work has already examined weight decay’s role in grokking [1,2,6], how training hyperparameters, including learning rate and batch size, affect grokking [1,3], and embedding optimization dynamics, including component-specific learning rates [4]. This paper examines where weight decay is applied, but it remains unclear how its findings advance these existing explanations of why generalization is delayed. The additional analyses do not establish why the proposed allocation accelerates grokking. Furthermore, the main experiments group attention, MLP and LayerNorm parameters together, as well as embeddings and the output head. Although Appendix I partially separates these groups, it remains unclear which component accounts for the stronger effect observed in the main experiments.
2. Restrictive experimental setting: Grokking has already been studied beyond algorithmic tasks, including image classification, sentiment analysis and molecular property prediction [2], alongside delayed generalization and delayed robustness in CNNs and ResNets on natural images [5]. The experiments here remain confined to small two-layer Transformers trained on algebraic tasks. Other architecture families, such as linear attention models and state space models, are not tested, so we do not know whether the findings extend to them.
References
[1] Power et al., "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", 2022.
[2] Liu et al., "Omnigrok: Grokking Beyond Algorithmic Data", ICLR 2023.
[3] Liu et al., "Towards Understanding Grokking: An Effective Theory of Representation Learning", NeurIPS 2022.
[4] AlQuabeh et al., "Mechanistic Insights into Grokking from the Embedding Layer", 2025.
[5] Humayun et al., "Deep Networks Always Grok and Here is Why", ICML 2024.
[6] Nanda et al., "Progress Measures for Grokking via Mechanistic Interpretability", ICLR 2023.
Questions for authors
I understood the work sufficiently.