acceptodds
Under review as a conference paper at ICLR 2027

Mapping the Emergence of Regularization-Driven Dynamics in Grokking

Abstract

For overparameterized neural networks, many solutions can fit the training data equally well while behaving very differently on unseen samples. Grokking separates training fit from visible generalization, providing a window for studying how this selection develops during training. We sweep short, fixed-duration weight decay (WD) perturbations across the pre-generalization plateau and measure how they shift later generalization time. Across three grokking tasks, these shifts are unordered early in the plateau but later form a stable dose ordering before visible generalization, with stronger WD increases leading to earlier generalization and stronger WD decreases leading to later generalization. Test-loss barriers between perturbed and baseline generalization checkpoints collapse toward zero while the ordered timing effects persist. A similar response reorganization is observed under ℓ1 regularization in the grokking setting of Junior et al. (2025). Drawing on Waddington’s developmental landscape as an analogy, we call this combination of increasingly constrained solution selection and persistent dose-ordered timing shifts the canalization of grokking solution selection. Together, our response maps and loss-barrier measurements reveal a dynamical reorganization before visible generalization that is consistent with the theoretical picture of regularization-driven motion along a stable slow manifold (Boursier et al., 2025).

Primary area:
optimization
Keywords:
grokking, training dynamics, regularization, weight decay, delayed generalization, linear mode connectivity

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_5ofbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Motivated by existing theoretical and empirical works showing that training overparameterized models with weight decay induces grokking, this paper explores a straightforward question that how would strengthening/weakening weight decay during the plateau stage of grokking affect the generalization delay. The authors use the AdamW optimizer, which separate the effect of weight decay and the gradient computation. The authors implement large-scale intervention-sweep experiments on three concrete algorithmic tasks and observe that the temporary weight decay change early in the plateau stage has vague effects, while in a later stage, a large increase of weight decay advances generalization more and a large decrease delays generalization more. They also compare the perturbed and baseline generalization models and find that in a regime where regularization governs optimization, generalization solution selection becomes more restricted, thus matching the existing theoretical results. Overall, the contributions of this paper are mainly on the empirical side.

Critical strengths
  1. The empirical observation that towards the end of the test accuracy plateau, temporary interventions on the weight decay induce ordered generalization time change, is a bit interesting. Although no theoretical/analytical explanation has been provided in the paper, it might be of interests for future works.

  2. The experimental design as well as the intervention framework are clean. Also, the experiments are thorough and carefully controlled such that the results are convincing.

Critical weaknesses

I think the contribution of the paper is somewhat limited. Weight decay induces grokking has been extensively explored both theoretically and empirically. While existing works usually focus on small models such as linear models or shallow neural networks, this paper also just empirically tests three small selected algorithmic grokking tasks with clean plateau phenomenon (also discussed by the authors in Section 4). Does the conclusion observed in the three tasks generalize to more general learning scenarios? Could the intervention design be generalized to scenarios where no explicit plateau exists? I found it hard for me to come up with a broader significance of this work. I understand that an empirical paper could merit acceptance without any theorem proved or algorithm designed, but it should convey important messages to guide future works.

Questions for authors

From my understanding, one of the most interesting results in the paper is that, on the three small algorithmic tasks, weight decay interventions tend to have ordered effects when approaching the late stage of plateau. I think the results could be more meaningful if it is general and widely applicable. For example, would you like to test this observation on some general deep learning problems such as training overparameterized two-layer networks using SGD? (Since prior works have demonstrated grokking in two-layer ReLU networks trained on modular addition using full-batch gradient descent.)

Another minor question: how would the magnitude of the intervention duration τ\tau affect the generalization delay?

Reviewer_YAzAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The author(s) discovered a hidden transition during the pre-generalization stage. Before the transition, temporarily changing the weight-decay parameter for a short amount of time have random effect on the generalization clock, but after the transition such a perturbation have deterministic effect, indicating that the optimization path has chosen a canal.

Critical strengths

The discovery of a hidden transition during the pre-generalization stage. Even though the train accuracy and test accuracy appear flat, the authors discovered the precise timing that the optimization trajectory falls into a fixed canal.

Critical weaknesses

It is not yet demonstrated that the phenomenon is applicable to more neural-network structures. Grokking was initially discovered in small transformers. Whether the shortcut connections and normalization layers affect the results reported here is unknown. Later on studies on grokking often use shallower MLPs (usually 1 hidden layer). Will they show the same phenomenon that the authors discovered in their 4-hidden-layer MLPs?

Questions for authors

Have you tested whether the effect appears in small transformers and other network architectures?

Reviewer_S36WOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study grokking through an interesting experimental approach: they briefly increase or decrease weight decay during training, restore its original value, and measure when the model reaches a target test accuracy. For pulses applied later in training, stronger increases generally advance generalization and stronger decreases delay it. They compare this timing response with the loss barrier between perturbed and baseline models at the same test accuracy. The relation between ordering and connectivity differs across tasks. They interpret these observations as consistent with regularization-driven motion along a stable slow manifold.

Critical strengths
  1. The authors study an interesting question: what changes during the long period between fitting the training data and generalization? Temporarily changing weight decay at different stages gives a useful probe with which to study this period, when test accuracy alone provides little information.

  2. The main empirical result is that the effect of regularization becomes more predictable before test accuracy improves. In this later regime, stronger increases in weight decay generally lead to earlier generalization, while stronger decreases delay it. I find this interesting because it identifies a change in training dynamics during an apparently unchanged accuracy plateau.

Critical weaknesses
  1. The mechanism remains unresolved. The timing responses, aligned updates and low loss barriers are consistent with a slow-manifold interpretation, but they do not identify a manifold or establish regularization as the dominant cause of progress. The explanation of why grokking is rare in Task 1 is also tentative. These claims should remain qualified. I find the measured changes in timing and connectivity useful independently of this proposed explanation.

  2. The claim of stable ordering needs better quantification. Appendix E selects the first qualifying checkpoint, and its implementation treats constant responses as perfectly correlated. This does not establish persistence. A further analysis of the saved records found sustained signed ordering more consistently in the parity tasks than in modular addition. The paper should report how long the expected ordering and acceleration/delay signs persist, in how many runs, and how branches that never reach the target are counted. Tasks 1 and 2 also use selected grokking runs, and some scanned pulses extend beyond visible generalization. These limits matter for the scope of the central result.

Questions for authors
  1. Can you add a per-task summary of signed ordering and its persistence, with an explicit definition of visible generalization and counts of all attempted and successful branches? Please distinguish the first ordered checkpoint from an interval over which ordering persists, and clarify the treatment of constant responses.

  2. How do training and test loss evolve during and after the perturbations? Does the onset of predictable responses coincide with a change in the test-loss curve, such as its turning point? Please show whether the perturbations mainly shift the generalization transition or also change the height and shape of the loss curve.

  3. Please provide the full restart state for one central example. A fresh Task-1 seed-2 rerun during this review used the supplied code on and reproduced delayed generalization. However, neither branch following a 500-update pulse of Δλ=±0.25\Delta\lambda=\pm0.25 reached 95% test accuracy by update 40,000, unlike the saved results. The baseline also failed the final-accuracy selection cutoff. This leaves an open reproducibility question, which might be affected by seed choices, hardware etc.

  4. I would be curious to know whether the same phenomenology can be observed for smaller networks/linear networks on similar tasks or those where grokking is also known to occur. Such examples could lead to analytical/semi-analytical derivations and possible predictions that could add merit to the slow manifold explanation.

  5. Figure 2(a) is extremely hard to read, with overlapping labels and texts, please fix it.

  6. Please compare the interpretation with the following missing work. The linear models provide other causes of delayed accuracy, and Prieto et al. examine numerical effects on grokking.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.