- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper explores how the coverage of the training data effects grokking in two one and two layer transformers models that have a width of 128, with a fix training recipe. They find that the coverage of the data has an impact on the delayed generalisation, with a critical threshold of approximately 24% with 100,000 training steps. Above this threshold the model groks below it the model does not.
Critical strengths
It should be commended the authors use of statistical analysis and experimental design, for example the use of factorial design, to deduce the findings.
Critical weaknesses
-
This paper misses a core citation [1], which shows that training datasets with higher coherence, increases the number samples needed to delay generalisation, whereas datasets with lower coherence results in faster generalisation, with fewer samples required. Where high coherence means that data is more similar or aligned and low coherence means the data is very different. Ultimately they showed that grokking can be mitigated through data selection. In the paper being reviewed the authors explore
unique-pair coverageasthe fraction of the $p^2$ ordered pairs seen, without regard for thecoherenceof the data, which may explain thesoft boundariesobserved. Furthermore, it is unclear to me what this paper is telling me beyond what is already known thatdata selection plays a role in grokkingas shown concretely in [1]. -
The evidence used to support the claims is only shown for modular addition and multiplication and on two types of transformer architectures, even though grokking has been exhibited on more complex tasks [3]. This hampers the bold claim of the paper stated by the title
Data Coverage Gates the Grokking Transition in Small Transformers. For such a statement, I would expect an exploration of a range of transformer architectures and training hyperparameters, to ensure this affect is not dominated by a singular training recipe or optimisation procedure. For example similar to Figure 6 in [3], which explores the landscape of grokking under hyperparameters. It is important to note that this limitation is acknowledgedour evidence is confined to small models on modular arithmetic: one- and two-layer transformers of a single width (dmodel = 128; ∼216K and ∼414K parameters) under one training recipe, so the behavior of c∗ at larger scale or on other task families is untested and we claim no transfer;but no rational behind the decision is provided. Given the computational cheap costs of exploring this problem, this limitation is too limiting. It is unclear if such analysis is generalisable or extends to other learning tasks, architectures, optimisation setups. I feel the community will not gain any meaningful insights from this work, nor would there be much interest due to the limited scope of exploration.
[1] Grokking Beyond the Euclidean Norm of Model Parameters Tikeng Notsawo Pascal Junior, Guillaume Dumas, Guillaume Rabusseau Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:28552-28618, 2025.
[2] Liu, Z., Michaud, E.J. and Tegmark, M., 2022. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117.
[3] Liu, Z., Kitouni, O., Nolte, N.S., Michaud, E., Tegmark, M. and Williams, M., 2022. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35, pp.34651-34663.
Questions for authors
See weaknesses but more concretely:
-
Could the authors situate there work in regards to [1], and explain the how it goes beyond there findings?
-
Could the authors provide further analysis across a range of transformer architectures and training hyperparameters, to ensure this affect is not dominated by a singular training recipe or optimisation procedure?
-
Could the authors extend the analysis to more complex learning tasks as found in [2]?
[1] Grokking Beyond the Euclidean Norm of Model Parameters Tikeng Notsawo Pascal Junior, Guillaume Dumas, Guillaume Rabusseau Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:28552-28618, 2025.
[2] Liu, Z., Michaud, E.J. and Tegmark, M., 2022. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117.