- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how representational capacity influences grokking. For a restricted two-layer complex-valued polynomial network on modular arithmetic, it characterizes the conditions under which target functions can or cannot be fit. The basic idea is that if a setting is sufficiently restricted, such that the targets are representable and the training samples are informative enough, then low training loss forces low error on unseen examples, and a prolonged grokking loss gap does not occur.
Critical strengths
Interest in Grokking: Grokking is a consequential topic for understanding neural network generalization. For this, I appreciate the authors’ interest in the topic.
Critical weaknesses
1. Failure to generalize is not the same as grokking: Grokking concerns delayed generalization after a model has already fit its training data. A substantial part of the paper focuses on representational impossibility (cases where the model cannot fit even its training data). Saying that “a network fails before grokking becomes possible” is like saying “acceleration is impossible whenever a car doesn't start”. It's honestly quite a tautological observation whose relevance to explaining grokking is unclear. Theorem 4 has an even odder argument that basically says “if we consider a sufficiently constrained setting with a representable target and sufficiently informative training examples, then low training error forces low error on unseen examples”. This is, of course, again quite limited as an explanation, as the definition of grokking is specific to when there is a delay in generalization. Therefore, simply telling us that a prolonged grokking loss gap cannot occur under some restrictive set of assumptions about the network and dataset setting, does not help identify the underlying cause of grokking, which occurs specifically not in these settings.
2. Severely limited novelty: Grokking has already been studied across a wide variety of settings, with prior work linking it to training hyperparameters [1,3], weight decay [1,2,6], representation learning [2,3], and embedding optimization dynamics [4]. Studies also extend beyond algorithmic tasks [2], including delayed generalization and delayed robustness in CNNs and ResNets on natural images [5]. Yet, this paper's main analysis uses a polynomial network with fixed input representations, which is an even more restrictive setting than Power et al. [1], who used Transformers with learned embeddings. It remains unclear what studying this restricted setting adds to our understanding of grokking, especially given the broader models and tasks already studied.
References
[1] Power et al., "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", 2022.
[2] Liu et al., "Omnigrok: Grokking Beyond Algorithmic Data", ICLR 2023.
[3] Liu et al., "Towards Understanding Grokking: An Effective Theory of Representation Learning", NeurIPS 2022.
[4] AlQuabeh et al., "Mechanistic Insights into Grokking from the Embedding Layer", 2025.
[5] Humayun et al., "Deep Networks Always Grok and Here is Why", ICML 2024.
[6] Nanda et al., "Progress Measures for Grokking via Mechanistic Interpretability", ICLR 2023.
Questions for authors
I understood the work sufficiently.