acceptodds
Withdrawn from ICLR 2027

ADAMW MAINTAINS WHAT DELAYS GROKKING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

20 views

Abstract

A network that groks fits its training data thousands of updates before it generalizes. What keeps it waiting? We find that the wait depends on which content the network's embedding holds once the training data are fit, and that the AdamW updates that keep rebuilding this content against weight decay lengthen it. In one- and two-layer Transformers learning modular arithmetic, the operand embedding decomposes into Fourier frequencies, so we can fork a run and change one thing: delete what a set of frequencies holds, put it back, or stop the optimizer from rebuilding it. Masking a random half of the frequencies during training halves the time to generalize, although the normally trained model relies on that half, so removing what a trained model relies on need not slow its learning. In one-layer models, putting back what those frequencies held before generalization, the network's own early content, into the same frequencies of the same model state delays generalization by over two thousand updates; a scrambled copy of equal energy, the network's later content, or another network's early content barely delays it. Weight decay alone would erase most of this early content within 2,000 updates, yet in ordinary training it keeps its size: AdamW's updates put back what decay removes. After the same deletion, blocking the rebuild brings generalization forward beyond deletion alone and beyond a control that shrinks each update to the norm blocking would leave. Over that control, the paired median gain is 850 updates at one layer and 800 at two, both on fresh seeds. Deleting the content and blocking its rebuild, which acts on a single direction of the embedding, recovers about 0.9 of the speed-up from masking. Part of the wait thus lies in early embedding content that AdamW keeps in place.

Primary area:
learning theory
Keywords:
grokking, delayed generalization, training dynamics, optimization, causal interventions during training, AdamW, weight decay, Fourier features, modular arithmetic

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_PS5COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper largely revisits existing work linking delayed generalization in small Transformers trained on modular arithmetic to weight decay and embedding reorganization. Its main addition is an intervention showing that deleting early content in selected embedding frequencies (and removing the components of AdamW updates that automatically restore it) speeds up generalization.

Critical strengths

Interest in Grokking: Grokking is a consequential topic for understanding neural network generalization. For this, I appreciate the authors’ interest in the topic.

Critical weaknesses

Severely limited novelty: The role of weight decay in grokking is already well established [1,2], while embedding dynamics and their contribution to delayed generalization have also been well studied [3,4]. Fourier-based computation for grokking in modular-addition Transformers is already well established by Nanda et al. [6], so showing that blocking selected embedding frequencies accelerates grokking in the same task family is, by itself, of limited novelty. Even the AdamW intervention is constructed specifically to prevent the manually deleted embedding pattern from being restored, leaving unclear what broader explanation of grokking this provides. Also, that AdamW experiment just tell us "optimisers work", which we already knew. Without identifying some downstream mechanism, or demonstrating relevance beyond modular arithmetic, I find the conceptual advance limited.

Narrow experimental scope: The experiments remain confined to small Transformers trained on modular arithmetic. Yet, grokking has already been studied beyond algorithmic tasks [2], including CNNs and ResNets trained on natural images [5]. It remains unclear whether the “embedding information deletion” effect transfers to these broader settings.

References

[1] Power et al., "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", 2022.

[2] Liu et al., "Omnigrok: Grokking Beyond Algorithmic Data", ICLR 2023.

[3] Liu et al., "Towards Understanding Grokking: An Effective Theory of Representation Learning", NeurIPS 2022.

[4] AlQuabeh et al., "Mechanistic Insights into Grokking from the Embedding Layer", 2025.

[5] Humayun et al., "Deep Networks Always Grok and Here is Why", ICML 2024.

[6] Nanda et al., "Progress Measures for Grokking via Mechanistic Interpretability", ICLR 2023.

Questions for authors

I understood the work sufficiently.

Reviewer_oULXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores why grokking occurs in modular arithmetic tasks (addition and multiplication) with mod (97 and 113) with one and two layer transformers when ADAMW is used. The paper suggests that the reason grokking occurs is due to ADAMW which maintains the early embedding content which prevents the model from generalising. If this early content is prevented from being maintained delayed generalisation is reduced.

Critical strengths

None

Critical weaknesses
  1. The main claim of the paper "ADAMW Maintains What Delays Grokking" neglects literature that shows that significantly delayed generalisation (grokking) can be exhibited without the use of ADAMW [1,2,3,4]. This means the main claim fails to explain grokking outside of the use of ADAMW and thus has limited use to the community beyond the cases explored, which are severely limited. Furthermore, it places an over empathises on a particular training setup ADAMW and the transformer architecture rather than aiming for a generalizable finding.

  2. The evidence used to support the claim "ADAMW Maintains What Delays Grokking" is only shown for modular addition and multiplication and on the transformer architecture, even though grokking has been exhibited on more complex tasks and different architectures [3,5]. A limitation is provided as to the use of modular addition and multiplication tasks: We work in modular arithmetic because these questions need a known basis in which to replace content. however this argument is weak as other literature [1,2,3,4] has explored grokking more generally without such a requirement.

[1] Kumar, T., Bordelon, B., Gershman, S. and Pehlevan, C., 2024, May. Grokking as the transition from lazy to rich training dynamics. In International Conference on Learning Representations (Vol. 2024, pp. 23010-23035).

[2] Saheb Pasand, A. and Dohmatob, E., 2026, April. Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking. In International Conference on Learning Representations (Vol. 2026, pp. 103473-103501).

[3] Miller, J., O'Neill, C. and Bui, T., 2023. Grokking beyond neural networks: An empirical exploration with model complexity. arXiv preprint arXiv:2310.17247.

[4] Grokking Beyond the Euclidean Norm of Model Parameters Tikeng Notsawo Pascal Junior, Guillaume Dumas, Guillaume Rabusseau Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:28552-28618, 2025.

[5] Liu, Z., Michaud, E.J. and Tegmark, M., 2022. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117.

Questions for authors

See weaknesses but more concretely:

  1. The core premise of the paper is that ADAMW is central to grokking, however literature has shown ADAMW is not required for grokking to occur [1,2,3,4]. Could the authors explain how the results account for existing literature, with regard to models and optimisers?

  2. Could the authors extend the analysis to more complex grokking tasks found in [5].

[1] Kumar, T., Bordelon, B., Gershman, S. and Pehlevan, C., 2024, May. Grokking as the transition from lazy to rich training dynamics. In International Conference on Learning Representations (Vol. 2024, pp. 23010-23035).

[2] Saheb Pasand, A. and Dohmatob, E., 2026, April. Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking. In International Conference on Learning Representations (Vol. 2026, pp. 103473-103501).

[3] Miller, J., O'Neill, C. and Bui, T., 2023. Grokking beyond neural networks: An empirical exploration with model complexity. arXiv preprint arXiv:2310.17247.

[4] Grokking Beyond the Euclidean Norm of Model Parameters Tikeng Notsawo Pascal Junior, Guillaume Dumas, Guillaume Rabusseau Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:28552-28618, 2025.

[5] Liu, Z., Michaud, E.J. and Tegmark, M., 2022. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117.

Reviewer_7g2cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper addresses why a grokking network generalizes long after it has fit the training data.
  • It proposes interventions on small Transformers that delete, restore or freeze parts of the learned representation, or block the optimizer updates that rebuild them.
  • The study finds that the representation learned early delays generalization, that AdamW keeps rebuilding it, and that removing it while blocking the rebuild speeds up generalization.
Critical strengths
  1. The experiments are carefully designed and transparently reported, including the tests that failed.
Critical weaknesses
  1. The claim is only partly supported. Holding the content in place barely delays generalization, and a radial-update control reproduces most of the effect, so it is unclear how this differs from ⊥Grad-style explanations.
  2. The paper is hard to read and narrow in scope. Terms are used without definition in the abstract and introduction, the argument depends on appendix sections, and the setting is limited to small models.
Questions for authors
  1. Can you restructure the main text so that the three main links can be followed without the appendix, with terms defined at first use in the abstract and introduction?
  2. What prediction distinguishes your account (AdamW maintains a particular fit) from ⊥Grad or radial-update explanations, and which of your results tests it?
Reviewer_jyJbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies grokking in a one-layer Transformer without LayerNorm, trained with full-batch AdamW on modular addition and multiplication. Each run is branched at update 2,000, after the model fits the training data, and the authors intervene on the part of the operand embedding EE that lies in a random half SS of its Fourier frequencies, which they call the early content DD. Masking SS roughly halves the time to generalize.

Critical strengths
  • The branching design is sound and reproducible. Each intervention starts from the same checkpoint, with the same optimizer state and random seed.
  • The reporting is transparent. The confirmations use fresh seeds with criteria fixed in advance.
Critical weaknesses
  • I think the paper is very hard to read. Much of the main text is run counts like "7/8 and 8/8".
  • I think the title claims more than the paper shows. No experiment changes the optimizer, so it is not clear what is specific to AdamW, and the authors leave the mechanism as "the next question".

Here are some more concerns:

  1. The maintenance result holds by construction. The ratio of update to weight decay along dd equals 1+(cT−c0)/(ηλTcˉ)1 + (c_T - c_0)/(\eta\lambda T\bar c), where ctc_t is the coordinate of EE along dd (Section 3.4). It is therefore near one for any weight that barely moves. The later content, which causes almost no delay, is kept too (Section 3.3).
  2. The overlap of dd with the radial direction (Section 4) is not an empirical finding. Since DD is a projection of EE, cos⁡(d,E)=∥D∥/∥E∥≈0.49\cos(d, E) = \lVert D\rVert/\lVert E\rVert \approx \sqrt{0.49} (App. B.36). Suppression is therefore ⊥\perpGrad (Prieto et al., 2025) applied to half of the embedding.
  3. The abstract's claim that "removing what a trained model relies on need not slow its learning" is contradicted by App. B.15: masking only the four frequencies the model relies on most slows generalization.
Questions for authors
  • What exactly is the early content? It is taken at update 2,100 in Section 3.3 and at update 2,000 in Section 3.4. Why these steps?
  • What is the "structured output" in Table 1?

Here are some more questions:

  1. How does the ratio of update to weight decay along dd compare with the same ratio along the complementary half's own content? This would show whether the maintenance is specific to the early content.
  2. How does suppression compare with plain ⊥\perpGrad on the embedding, without the 14–24× rescaling used in the radial control of App. B.31?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.