- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper largely revisits existing work linking delayed generalization in small Transformers trained on modular arithmetic to weight decay and embedding reorganization. Its main addition is an intervention showing that deleting early content in selected embedding frequencies (and removing the components of AdamW updates that automatically restore it) speeds up generalization.
Critical strengths
Interest in Grokking: Grokking is a consequential topic for understanding neural network generalization. For this, I appreciate the authors’ interest in the topic.
Critical weaknesses
Severely limited novelty: The role of weight decay in grokking is already well established [1,2], while embedding dynamics and their contribution to delayed generalization have also been well studied [3,4]. Fourier-based computation for grokking in modular-addition Transformers is already well established by Nanda et al. [6], so showing that blocking selected embedding frequencies accelerates grokking in the same task family is, by itself, of limited novelty. Even the AdamW intervention is constructed specifically to prevent the manually deleted embedding pattern from being restored, leaving unclear what broader explanation of grokking this provides. Also, that AdamW experiment just tell us "optimisers work", which we already knew. Without identifying some downstream mechanism, or demonstrating relevance beyond modular arithmetic, I find the conceptual advance limited.
Narrow experimental scope: The experiments remain confined to small Transformers trained on modular arithmetic. Yet, grokking has already been studied beyond algorithmic tasks [2], including CNNs and ResNets trained on natural images [5]. It remains unclear whether the “embedding information deletion” effect transfers to these broader settings.
References
[1] Power et al., "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", 2022.
[2] Liu et al., "Omnigrok: Grokking Beyond Algorithmic Data", ICLR 2023.
[3] Liu et al., "Towards Understanding Grokking: An Effective Theory of Representation Learning", NeurIPS 2022.
[4] AlQuabeh et al., "Mechanistic Insights into Grokking from the Embedding Layer", 2025.
[5] Humayun et al., "Deep Networks Always Grok and Here is Why", ICML 2024.
[6] Nanda et al., "Progress Measures for Grokking via Mechanistic Interpretability", ICLR 2023.
Questions for authors
I understood the work sufficiently.