acceptodds
Under review as a conference paper at ICLR 2027

Group-Resolved Weight Decay Controls Grokking Arrival Across Checkpoint States

Abstract

Does grokking depend only on how much weight decay is applied, or also on where it is applied? We study two-layer Transformers using paired continuations from complete checkpoint states. We allocate decay between internal computation parameters (attention, MLP, LayerNorm; GC) and input/output parameters (embedding, head; EH), matching initial decay-only squared-norm contraction. On modular addition, shifting this matched total from EH toward GC advances sustained high test accuracy by steps on average, with the same direction in all 12 parents. Both groups accelerate arrival, but GC has the larger response in the tested dose units. At early checkpoints, high-GC continuations arrive in cases within steps, versus under low GC. Extending all 128 low- and intermediate-GC arms to steps yields arrival in every arm: the short-budget gap reflects delay for these runs, not permanent failure. The fixed allocation swap also advances arrival on permutation composition. Separate parameter-source interventions and a one-step response assay characterize complementary effects of inherited state, without establishing a mechanism for arrival. These results identify decay allocation as a control of grokking timing and show why finite-budget completion must be distinguished from eventual generalization.

Primary area:
optimization
Keywords:
grokking, weight decay, training dynamics, generalization, causal interventions, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jToXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how changing where weight decay is applied in a Transformer affects grokking. The basic finding is that allocating more of the decay to attention, MLP and LayerNorm parameters, rather than embeddings and the output head, generally makes generalization happen earlier in the settings studied.

Critical strengths

Interest in Grokking: Grokking is a consequential topic for understanding neural network generalization. For this, I appreciate the authors’ interest in the topic.

Critical weaknesses

1. Limited explanatory novelty: Prior work has already examined weight decay’s role in grokking [1,2,6], how training hyperparameters, including learning rate and batch size, affect grokking [1,3], and embedding optimization dynamics, including component-specific learning rates [4]. This paper examines where weight decay is applied, but it remains unclear how its findings advance these existing explanations of why generalization is delayed. The additional analyses do not establish why the proposed allocation accelerates grokking. Furthermore, the main experiments group attention, MLP and LayerNorm parameters together, as well as embeddings and the output head. Although Appendix I partially separates these groups, it remains unclear which component accounts for the stronger effect observed in the main experiments.

2. Restrictive experimental setting: Grokking has already been studied beyond algorithmic tasks, including image classification, sentiment analysis and molecular property prediction [2], alongside delayed generalization and delayed robustness in CNNs and ResNets on natural images [5]. The experiments here remain confined to small two-layer Transformers trained on algebraic tasks. Other architecture families, such as linear attention models and state space models, are not tested, so we do not know whether the findings extend to them.

References

[1] Power et al., "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", 2022.

[2] Liu et al., "Omnigrok: Grokking Beyond Algorithmic Data", ICLR 2023.

[3] Liu et al., "Towards Understanding Grokking: An Effective Theory of Representation Learning", NeurIPS 2022.

[4] AlQuabeh et al., "Mechanistic Insights into Grokking from the Embedding Layer", 2025.

[5] Humayun et al., "Deep Networks Always Grok and Here is Why", ICML 2024.

[6] Nanda et al., "Progress Measures for Grokking via Mechanistic Interpretability", ICLR 2023.

Questions for authors

I understood the work sufficiently.

Reviewer_GguGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether the timing of grokking depends on which parameter groups receive AdamW weight decay. A grokking study with small two-layer Transformers on Z97 modular addition, plus S5 permutation composition. Parameters are split into GC (attention, MLP, LayerNorm) and EH (embedding and output head). Training continues from a shared checkpoint, with weights, Adam moments, data order and RNG all restored, and with a different weight decay on each group. From fully restored checkpoints of a small Transformer, the authors allocate decay between internal computation parameters (GC) and embeddings/output head (EH), matching initial decay-only contraction in a specified norm-based coordinate. On modular addition, shifting decay toward GC advances the reported arrival time in paired continuations. A checkpoint-stage study shows that many low-GC continuations that do not generalize by 8,000 steps do generalize when extended to 32,000 steps. An allocation swap on S5 permutation composition gives a second toy-task check.

Main result: at equal total pressure, putting more decay on GC brings grokking about 2,100 steps earlier (12 of 12 seeds). Both groups help and GC helps more. The GC/EH ratio, however, ranges from about 2x to 92x depending on the unit used for dose. Second result: from early checkpoints, high-GC runs grok within 8k steps (63 of 64) and low-GC runs never do (0 of 64). When the low and mid-GC runs are extended to 32k steps, all 128 grok. The gap was a delay. The paper's lesson is to report the observation horizon and to separate delay from failure to generalize.

Critical strengths
  • The paired continuation design is well done. Restoring weights, optimizer state, batch sequence, and random state substantially reduces the confounding that would arise from comparing independent runs. The authors are also careful to identify the normalization used to match decay allocations.
  • The observation-horizon result is valuable. The big gap at an 8k-step cutoff does not persist as a difference in eventual completion when the lower-decay arms are extended. This is a useful reminder that a finite training budget can make a timing effect look like a failure to generalize.
Critical weaknesses
  • The scope of the main conclusion remains limited. The relative response of GC and EH depends on how decay dose is normalized, and matching initial contraction does not ensure equal cumulative regularization or equal functional perturbation. Results are mostly on small Transformers and algebraic tasks. I would want to see whether the allocation effect holds under alternative matching criteria, standard exclusions from weight decay, and a modest sweep over optimizer and model settings.
  • The supporting interventions do not yet provide a mechanism for the timing change. The parameter transplant and local Jacobian analysis are informative but use different experimental cohorts and are not causally connected to the observed arrival times. The event-definition correction and extensive protocol material are reported, but they make it unnecessarily hard to trace which arrival rule applies to each comparison. I recommend having a compact protocol table in the main text and a uniform-rule sensitivity check.
Questions for authors
  • Does the allocation advantage hold under a matching that does not depend on your coordinate? For example, match on the total realized norm change over the first N steps, or on the loss change after one decay step.
  • Does the effect hold across learning rates, training fractions, widths and depths, and with LayerNorm excluded from decay as is common practice? Even a small grid would help.
  • How do group-allocated policies compare with the best tuned uniform AdamW decay and with no decay, in time to generalization? Is there a setting where group allocation beats the best uniform coefficient?
  • What is your mechanistic hypothesis for why core decay speeds arrival (circuit cleanup in MLP weights, or logit-scale effects in the head)? Do any measurements over time (Fourier features, per-component norms) track the allocation effect?
  • Why does the S5 experiment use a different event rule (gated onset) from the primary factorial (onset)? Do any conclusions change if one rule is applied to all cohorts?
Reviewer_dhgBOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how weight decay allocation affects grokking in two-layer Transformers. The authors compare paired continuations from identical checkpoints, matching the initial first-order decrease in squared parameter norm due to decay. On addition modulo 97, shifting decay from the embeddings and output head toward internal parameters advances sustained high test accuracy by 2,100 steps on average across 12 seeds, with all seeds showing the same direction.

Critical strengths
  1. The experimental design is well controlled. By comparing continuations from the same checkpoint at matched total initial decay pressure, the authors convincingly show that where weight decay is applied affects when grokking occurs.
  2. The paper shows why the observation horizon matters. By extending training, the authors demonstrate that runs which initially appear unsuccessful can still generalize, making the short-budget results easier to interpret.
Critical weaknesses
  1. The experiments are limited to small, two-layer Transformers on synthetic algebraic tasks. It remains unclear whether the observed effect of weight decay allocation persists in larger models or on more complex language modeling tasks.
  2. The input embeddings and output head (EH) can account for a larger share of the parameters in some small models than in larger language models, where the core group (GC) often dominates the parameter count. The paper leaves open whether reallocating decay between GC and EH offers a measurable advantage over a tuned uniform AdamW baseline as the relative size of EH decreases.
Questions for authors
  1. Scaling Law Valiation. Have you tested the allocation swap across a series of progressively deeper and wider models, with training budgets set by a common scaling rule? Does the GC-heavy advantage persist across scales and on more complex language modeling tasks?
  2. How does the relative benefit of GC-heavy versus EH-heavy decay depend on the groups’ parameter counts and their squared norms at the fork? For example, are there configurations in which the EH-heavy policy reaches sustained high test accuracy earlier at the same total initial decay pressure?
  3. The experiments use AdamW with a fixed learning rate. Have you examined how the allocation effect interacts with the choice of optimizer or learning rate schedule? In particular, does the GC-heavy advantage persist when the schedule is tuned separately for each decay allocation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.