acceptodds
Under review as a conference paper at ICLR 2027

Learning by Disruption: Accelerating Grokking with Monte Carlo Parameter Swapping

Abstract

Grokking is a striking phenomenon in neural network training, where a model can undergo a prolonged period of pure memorization before abrupt generalization. While previous works have attempted to interpret it through classical machine learning mechanisms like weight norm, recent research draws an analogy from statistical physics, framing grokking as a form of computational glass relaxation. This theory defines the initial memorization as a result of 'fast cooling' where the training loss is reduced so quickly that a glass state is formed, followed by a `slow relaxation' towards final generalization. Although providing a unifying framework for representative grokking theories, this perspective has remained largely at the theoretical macroscopic level without direct empirical validation on training dynamics. Here we introduce a three-component framework to directly characterize the training dynamics via parameter mobility (PM), and two representative measurements from glassy dynamics: replica correlation (RC) and fractal dimension (FD). We demonstrate that standard optimization presents clear signatures of glass dynamics and inherently traps the grokking network in a kinetically arrested memorization state with collapsed mobility, strong history dependence, and channel-like motions. This quantitative agreement motivates us to introduce State-Aware Monte Carlo Parameter Swapping (SAM-Swap), an optimization plug-in that can accelerate generalization, inspired by the swap Monte Carlo algorithm widely used in glass dynamics. Comparing SAM-Swap, weight decay, and Gaussian gradient noise, we find that accelerated generalization is consistently associated with random exploration in the parameter space, similar to diffusion in physics.

Primary area:
learning theory
Keywords:
grokking, neural network training, glass dynamics, computational glass relaxation, parameter mobility, replica correlation, fractal dimension, SAM‑Swap, optimization, generalization, random exploration, diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A4iyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the grokking phenomenon in neural network training, where the model first memorizes then generalizes. The authors connect the memorization to glass state formation concepts from physics, where the sudden collapse of the loss function is paralleled with the formation of the state of hot glass material and generaliztion of the model is paralleled with gradual cooling of the glass material. the authors also propose SAM-SWAP to improve generalization during training. Authors treat the model parameters as the state of the system under study, and the loss function as the energy of the system. The also define the equivalent notions for three metrics in the system: Parameter Movement (PM), Fractal Dimension (FD), and Replica Correlation (RC). These concepts (defined in the context of neural network training) help the authors study the behavior of the model being trained from a physical perspective.

Critical strengths
  • Formulation of learning neural networks as a physical phenomenon is interesting. The parallels drawn in the paper may have a positive impact on the current research in AI, especially at this age where AI works like a black box.
  • The intuitions of the paper are fair and easy to follow.
  • The introduction and abstract of the paper are nice and easy to follow.
  • The method "SAM-Swap" is inspired by the swap Monte Carlo method widely used to accelerate equilibration in glass-forming systems. It is simple and effective: swap the values of parameters at random; this will lead to increased Parameter Movement (PM), which means updates from one batch change the parameters mode. It reduces Replica Correlation (RC), which means training trajectories starting from the same phase during training become more similar. It also increases Fractal Dimension (FD), which means the training becomes more stable.
Critical weaknesses
  • The organization of the paper is not good. The authors start by defining some metrics, then jump into an experimental setup; Figure 1 is not explained well in advance, yet there are direct references to it in sections 3.1-3.4. Same goes about Figure 2.
  • In parts, the wording of the paper is non-trivial and hard to follow, especially for someone with no physics background. For example, what do "presistent and channel like" and "tortuous" exactly mean in this context? "FD characterizes intra-trajectory dynamics, indicating whether optimization paths remain persistent and channel-like or become tortuous".
  • Another example of complex explanations: "RC exhibits clear two-step relaxation coinciding with the memorization period and progressively slower decorrelation at longer waiting times, revealing strong history dependence and aging"
  • the last paragraph of the introduction is a restatement of the text before it; it could be summarized or removed.

My main concerns about the paper: 1- I'm not sure if the finding related to PM is new, models during training experience higher variations in early training iterations than later iterations. This is somewhat a known fact/intuition among ML practitioners. 2- I'm not sure if the experimental setup reflects a valid setup to be analyzed and make conclusions; the task selected seems to be toy scale, there are only 2 input variables. I expect the network architecture to be very small as well. All of these cast doubt on the validity of the proposed metrics. My main concern here is that the metrics that are studied are formulated based on intuitions that do make sense in low dimensions, e.g., the FD metric does make sense to me when I think of it in 2 dimensions. However, intuition in low-dimensional spaces often misleads us and makes us generalize concepts that do not hold in high dimensions. For that reason I believe the paper needs to study larger AI models.

Questions for authors

Have you studied larger neural networks with more practical applications? If not, I suggest showing your studies via a toy language model with several hundred million parameters.

3 comments

We thank the reviewer for their constructive feedback and positive assessment of our physics-inspired perspective and the proposed SAM-Swap method. We also appreciate the reviewer's concerns regarding manuscript organization, accessibility, and the applicability of the proposed dynamical framework across different model scales. Below, we address each of the reviewer's concerns in detail.

Response to Concerns Regarding Manuscript Organization and Clarity:

We thank the reviewer for these observations regarding presentation and clarity. The manuscript will be revised to improve its organization and accessibility. In particular, Figures 1 and 2 will be introduced and explained before being referenced in the analysis sections, with more explicit descriptions of the experimental conditions and motivations, allowing the subsequent discussion to be read more naturally. We will also streamline the Introduction by removing repetitive content from the final paragraph.

Several physics-inspired terms and descriptions will be clarified to improve accessibility for a broader machine learning audience. For example, phrases such as "channel-like," "tortuous," and "two-step relaxation" will be accompanied by more direct descriptions of the corresponding optimization behavior. Specifically, "persistent and channel-like" describes trajectories in which successive parameter updates tend to follow similar directions, whereas "tortuous" describes trajectories that frequently change direction. Additionally, two-step relaxation" refers to an initial decrease in replica correlation, followed by an extended plateau and a subsequent decrease. Discussions of aging and history dependence will also be rewritten using more intuitive language to clarify that replicas branched at later training stages decorrelate more slowly under the same perturbation protocol, while preserving the underlying technical meaning. We will also more explicitly distinguish the physical meanings of the three observables: PM quantifies the magnitude of parameter motion, RC measures inter-trajectory similarity, and FD characterizes trajectory geometry rather than training stability.

These revisions are intended to improve readability and exposition without affecting the methodology or conclusions of the study.

Response to Concerns Regarding the Novelty of Parameter Mobility (PM):

We agree with the reviewer that decreasing parameter mobility during training is a commonly recognized phenomenon in neural network optimization. However, we also would like to note that the main contribution of this work is not the observation that PM decreases during training. In fact, PM is included as a reference measure to characterize the slowdown of parameter motion and provide a complementary perspective for interpreting FD and RC. The central contribution is the combination of using three observables to characterize the optimized dynamics in delayed generalization. Specifically, FD and RC reveal changes in trajectory structure and replica-level dynamics that are not captured by parameter mobility alone. For example, a decrease in PM alone cannot distinguish whether parameter motion becomes more directionally persistent or whether independently perturbed replicas remain highly correlated. We will revise the manuscript to make this distinction explicit.

Response to Concerns Regarding Benchmark Scale and High-Dimensional Applicability:

The objective of this work is to study optimization dynamics in a controlled setting where grokking reliably emerges, rather than to establish performance trends in large-scale language models. Modular arithmetic has become a standard benchmark in the grokking literature, including the original work of Power et al. (2022) as well as subsequent studies such as Gromov (2023) and Mohamadi et al. (2024), because it provides a reproducible environment for investigating delayed generalization and optimization dynamics.

Importantly, we also would like to distinguish the dimensionality of the input space and that of the parameter space. Although the arithmetic task involves only two input variables, all three proposed observables characterize optimization dynamics in the high-dimensional parameter space of the neural network, rather than in the input space. Regarding the example of FD suggested by the reviewer, it is defined through the scaling relationship between net displacement and accumulated trajectory length, both of which are mathematically well-defined in parameter spaces of arbitrary dimensionality. A real example would be the ideal straight-line motion and ideal random-walk motion: they always exhibit distinct scaling behaviors regardless of the ambient dimensionality.

The manuscript therefore does not claim direct applicability to LLM-scale systems. Rather, it investigates whether glass-like optimization signatures can be identified and characterized within a canonical grokking setting. At the same time, we agree that demonstrating robustness beyond a single architecture configuration is important. To this end, we conducted additional width-scaling experiments with hidden widths ranging from 128 to 1024 and observed that Adam+Swap continues to substantially accelerate grokking across all tested model sizes. Further analysis also suggests that the qualitative behavior of FD remains largely unchanged across these scales, indicating that the observed dynamics are not tied to a specific architecture configuration or parameter-space dimensionality within the tested range.

While these experiments should not be viewed as a comparison to LLM-scale models, they provide evidence that the reported phenomena are robust to substantial changes in model capacity and parameter-space dimensionality within the studied setting. Detailed quantitative results will be included in the appendix of the revised manuscript. We also believe that extending the analysis to substantially larger architectures would provide an important test of the generality of these observations. We will clarify the scope of our claims and discuss this direction in the revised manuscript.

I thank the authors for their responses. my concern about the model size is partly resolved. regarding paper writing I appreciate the authors' efforts. overall, I believe the score I gave this paper originally is fair. I keep it unchanged.

Reviewer_y4MuOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper “Learning by disruption: accelerating grokking with monte carlo parameter swapping” nicely explores the grokking phenomenology in transformer encoders under the lens of statistical mechanics. The analysis isolates a separate phase structure where grokking emerges as a dynamic phase of learning where parameters evolve along a constrained nearly linear trajectory. The paper first introduces some novel observables to probe the performance of the neural network during training, then proposes to apply swapping technique used to explore the energy landscape in disordered systems to accelerate loss landscape exploration and catalyze generalization.

Critical strengths

The paper is well-written and easy to read. The dynamic characterization of training during grokking seems pretty novel and clear in the results. Also the usage of SWAP to accelerate generalization is new and a nice implementation on Monte Carlo techniques to neural network training.

Critical weaknesses

Previous literature about characterization of the training dynamics as a glassy landscape probing and the study of fractal dimensionality of trajectories is not well discussed.

Questions for authors

I will now present my comments and questions in form of Major and Minor Issues, with the goal of improving the quality of the presented work. I ask the Authors to address my points extensively.

Major issues:

Q1: Observation about aging is consistent with Baity-Jesi et al. (2018). How do the Authors compare their results with the provided reference, especially under the light of understanding grokking? Please cite the paper.

Q2: If DfD_f increases after grokking it means that coefficient α\alpha in Sec. 2.2. decreases and trajectories look more confined. So why do the Authors claim that this behaviour of DfD_f signals “an expansion of the explored parameter space” ? According to my comprehension trajectories are getting trapped into ergodic components that can live far apart, and this would rather emerge from the drop in qq. Also, how does this observation reconcile with previous works e.g. Simsekli et al. “Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks” (2021) and Birdal et al. (2021) ? The paper should include references about analysis of trajectory FD and generalization.

Q3: The paper nicely shows the phenomenology accompanying grokking, but it does not appear to me that it explains why grokking occurs. Can the Authors use their analysis to give insights about the reason why grokking manifests in such three phases? An initial memorization, a low-dimensional ballistic phase, the final emergence of a rugged landscape. In a nutshell, what is the reason for the ballistic phase?

Q4: Is there a particular layer of the transformer encoder that is more susceptible to swapping? Have the Authors tried to repeat the experiment by limiting only to the attention layer, or MLP ? This would help understand whether grokking is driven by a particular part of the architecture.

Q5: Imagine to limit to MLP or attention unit. What would it happen if, instead of swapping parameters across one sub-layer one swaps across different sub-layers of the same unit ?

Q6a: From fig. 2, why is Adam+Swap better than Adam+Noise in accelerating generalization?

Q6b: Always in fig. 2: what is the meaning of a higher dRdR for Adam+Noise and Adam+Swap ?

Q7: Assume one trains a transformer decoder via Adam+Noise and Adam+Swap. Given the same degree of accuracy, are there differences in the actual generation performance? How does such acceleration in generalization affect the sampling distribution in generative modeling?

Minor Issues:

Q8: I invite the Authors to number all formulas and equations in the main text: only Eq. (1) is numbered.

Q9: Authors should define vv in Algorithm 1.

8 comments

We thank the reviewer for their constructive feedback and positive assessment of the novelty, soundness, and presentation of our work. We particularly appreciate the reviewer’s insightful questions regarding the physical interpretation of grokking dynamics, their relationship to previous studies, and the potential extensions of our proposed perturbation mechanism. Below, we provide a point-by-point response to the reviewer’s questions.

Response to Concerns Regarding Prior Work on Aging Dynamics (Q1):

The aging-like behavior observed in RC is closely related to the phenomenology reported by Baity-Jesi et al. (2018), and we will add this reference and expand the discussion in the revised manuscript. In particular, both studies report pronounced waiting-time dependence and departures from time-translation invariance, consistent with slow relaxation dynamics in neural network optimization. The primary distinction lies in the learning regimes being investigated. While Baity-Jesi et al. (2018) identify an intermediate aging regime during conventional neural network training, our work extends the investigation of glass-like optimization dynamics to the generalization delay characteristic of grokking, where signatures of slow and history-dependent dynamics remain observable long after the training data have been fitted

Another important difference is the observables being studied. Baity-Jesi et al. (2018) focused primarily on displacement-based measures of aging, whereas our work characterizes the dynamics through RC and FD. RC probes the relative evolution of replicas, while FD is a scale-invariant local measure defined through the relationship between displacement and accumulated trajectory length. As a result, both quantities are substantially less sensitive to global rescaling of parameter updates than raw displacement observables.

Our observations also suggest a different interpretation of the underlying glassy dynamics. Previous work has linked glass-like behavior to the presence of flat directions in the loss landscape. In contrast, we find that continuously perturbed replicas remain highly correlated (RC ≈ 0.96) for an extended period before exhibiting a delayed two-step decorrelation process. If the dynamics were governed primarily by unconstrained motion along flat directions, one would expect perturbations to accumulate continuously and produce a more immediate decorrelation of replica trajectories. Instead, the prolonged RC plateau indicates that optimization remains strongly constrained despite the presence of perturbations.

While this observation does not rule out the potential co-existence of flat directions, it suggests that flatness alone should be insufficient to explain the observed glassy dynamics in the grokking regime. We therefore interpret the prolonged replica correlation as evidence for additional dynamical constraints imposed by the optimization landscape, which are gradually relaxed during the grokking process. We will revise the manuscript to more explicitly compare these perspectives and place our findings within the broader literature on glassy optimization dynamics in neural networks.

Response to Concerns Regarding the Interpretation of Fractal Dimension (Q2):

The reviewer is correct that an increase in DfD_f corresponds to a decrease in α\alpha, implying that net displacement grows more slowly relative to the accumulated path length. We therefore agree that the phrase "expansion of the explored parameter space" was imprecise and will revise it in the manuscript. A more accurate description is that larger DfD_f reflects increasingly diffusive or space-filling exploration within the accessible parameter space, rather than larger absolute displacements or exploration of a larger basin volume.

Importantly, the increase in (DfD_f) observed after grokking occurs only in the continuously perturbed replicas used to probe replica-correlation dynamics. During the grokking phase, DfD_f remains close to its ballistic limit Df≈1D_f \approx 1, indicating highly persistent low-dimensional motion. After grokking, DfD_f increases gradually, reflecting a progressive departure from this ballistic regime and the emergence of more diffusive trajectory dynamics. In this setting, a larger DfD_f indicates that perturbations cause trajectories to follow increasingly indirect paths, accumulating path length without a proportional increase in net displacement. We therefore interpret the effect as increased diffusivity within the accessible parameter space rather than expansion into a larger basin volume.

We will also expand the discussion of prior work on trajectory fractal dimension. In particular, the observations of Simsekli et al. (2021) and Birdal et al. (2021) are broadly consistent with our results. These studies associate lower trajectory fractal dimension with improved generalization. Similarly, our unperturbed models achieve strong generalization while maintaining relatively low DfD_f. The post-grokking increase in DfD_f appears only under continuous perturbation and therefore reflects the response of the system to an external driving process rather than the natural optimization dynamics of the baseline model. We will add these references and clarify this distinction in the revised manuscript.

Response to Questions Regarding the Mechanistic Origin of the Ballistic Phase (Q3):

While the present work is primarily empirical, our observations suggest a possible interpretation of the ballistic phase. We hypothesize that the ballistic regime reflects a period of constrained exploration following interpolation rather than a period of enhanced exploration.

The key observation is that RC remains near 0.96 for an extended period despite the continuous application of perturbations. If optimization were freely exploring nearby solutions, one would expect perturbations to accumulate and produce a gradual decorrelation of replicas. Instead, replicas remain strongly correlated throughout the ballistic phase. At the same time, FD remains near its ballistic limit Df≈1D_f \approx 1, indicating highly persistent and directed motion in parameter space.

Taken together, these observations suggest that optimization is confined to a restricted subset of accessible solutions during the ballistic phase. Under this interpretation, the ballistic regime emerges because the optimizer has not yet relaxed sufficiently to explore alternative solution pathways. Generalization then coincides with a gradual loss of this constraint, reflected by the subsequent decrease in RC and increase in FD. This interpretation is consistent with the perspective of Zhang et al. (2025), which views grokking as a slow relaxation process rather than a sudden algorithmic transition.

While these observations do not establish a definitive causal mechanism, they provide evidence that the ballistic phase corresponds to a prolonged period of constrained optimization dynamics preceding generalization.

Response to Questions Regarding Layer-Specific Perturbations (Q4):

We have conducted additional experiments in which Adam+Swap is applied only to the MLP layers. These experiments continue to exhibit substantial acceleration of generalization together with qualitatively similar RC and FD dynamics to those observed in the full-model perturbation setting. This suggests that the observed effect does not rely on perturbing all model parameters simultaneously and can arise even when perturbations are restricted to a specific architectural component.

In addition, we performed a layer-wise analysis of RC and FD. Across layers, the qualitative features remain remarkably consistent: RC exhibits the same plateau-and-decay behavior, while FD follows the same overall evolution observed at the full-model level. The primary differences are quantitative rather than qualitative, with individual layers exhibiting different magnitudes of decorrelation and different levels of fluctuation in FD. This suggests that the observed dynamics are not confined to a single layer, but instead appear as a shared feature of the optimization process across the network.

At present, we have not performed a comparable perturbation study targeting only the individual layers. Consequently, we cannot yet determine whether attention and MLP components contribute equally to the observed behavior. A systematic layer-wise perturbation analysis, including individual attention layers, MLP layers, and combinations thereof, would provide valuable insight into the architectural origins of grokking and will be discussed as an important direction for future work.

Response to Questions Regarding Cross-Sub-Layer Swaps (Q5):

We have not yet investigated swaps across different sub-layers within the same transformer block and therefore cannot draw reliable conclusions regarding their effect on optimization dynamics or generalization.

That said, such experiments could provide additional insight into the architectural origins of grokking by separating the effects of perturbations applied within a component from those introduced across components. Comparing intra-layer and cross-sub-layer swaps may help clarify whether the observed acceleration of generalization depends primarily on the perturbation itself or on the specific structure of the affected parameters. We view this as an interesting direction for future investigation and will discuss it in the revised manuscript.

Response to Questions Regarding Adam+Swap versus Adam+Noise(Q6a)

Our intention is not to claim that Adam+Swap is universally superior to Adam+Noise. Adam+Swap was introduced as a physically motivated perturbation mechanism inspired by the Swap Monte Carlo method used in glass simulations, which should be valid given our proposed theoretical framework, while Adam+Noise serves as an important baseline demonstrating that perturbations alone can accelerate generalization.

Nevertheless, we also want to note that Adam+Swap offers two conceptual advantages over continuous Gaussian perturbations. First, parameter swapping modifies the configuration of parameters while preserving their layer-wise distribution and weight norm at each intervention, whereas additive Gaussian noise does not impose these constraints. Second, SAM-Swap introduces perturbations selectively based on the observed trajectory FD, rather than continuously perturbing the optimization process. It therefore provides a state-aware mechanism for disrupting persistent optimization trajectories whenever the measured dynamics become sufficiently constrained.

The central question is therefore not whether one perturbation always outperforms the other, but whether a perturbation mechanism motivated by glass physics provides additional insight into the optimization dynamics associated with grokking. We will clarify this distinction in the revised manuscript.

Response to Questions Regarding the Interpretation of dRd_R(Q6b)

Regarding the higher dRd_R observed for both Adam+Noise and Adam+Swap, we interpret this as evidence of increased exploration of the configuration space. By continually perturbing the optimization trajectory, both methods encourage the model to access regions of parameter space that may be difficult to reach under standard Adam optimization. This interpretation is consistent with the accelerated generalization observed in both settings.

We do not interpret dRd_R in isolation. Rather, the increase in dRd_R is accompanied by corresponding changes in FD and RC, indicating that perturbations alter both the trajectory dynamics and the relative evolution of nearby solutions. Taken together, these observations suggest that perturbations facilitate exploration of alternative optimization pathways associated with improved generalization. We will clarify this interpretation in the revised manuscript.

Reviewer_eGvpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies grokking in a one-layer transformer trained on (x² + y) mod 67. It introduces three diagnostics of the training trajectory: Parameter mobility (PM): the displacement over a window of epochs. Replica correlation (RC): the Pearson correlation between replicas branched from the same state and perturbed by small Gaussian noise. Trajectory fractal dimension (FD). It reports collapsed mobility, a two-step decay of the replica correlation with waiting-time dependence, and D_f ≈ 1 during the delay. It interprets these as glass-like kinetic arrest. Inspired by swap Monte Carlo, it proposes SAM-Swap, which randomly exchanges parameter values within a layer whenever D_f drops below a threshold, and shows that this shortens the grokking time, as does large Gaussian noise.

Critical strengths

Connecting grokking to the out-of-equilibrium dynamics of glasses is an interesting direction. The replica-based and trajectory-based diagnostics are simple and cheap to compute. The SAM-SWAP algorithm is interesting and promising. The paper is generally clearly written and honest about some limitations, for instance that it cannot distinguish its picture from norm-minimization accounts.

Critical weaknesses

1)The study is too narrow. All results rest on a single task, a single architecture, a single training fraction (70%), and a single optimizer configuration chosen to minimize grokking time. None of the factors known to control grokking are varied: training fraction, weight decay, initialization scale, width or depth, other algorithmic tasks. Generality is mentioned in only one sentence of the conclusion. This is not enough to support claims about grokking in general.

2)“Memorization” is not defined, and the delay is in fact slow generalization. The memorization phase is identified with the window between 99% training accuracy and 99% test accuracy. In Fig. 1a, however, test accuracy jumps to about 0.2 as soon as training is fit, far above chance (1/67), and then rises steadily to about 0.6 by epoch 10³ before the final jump. The regime interpreted as kinetic arrest in a memorizing state is therefore largely a slow, progressive generalization. This undermines the central interpretation.

2)In the analogy with glassy dynamics, one should be careful about the amplitude and possible time-dependence of the stochastic noise, see eg Baity-Jesi et al. (ICML 2018). In the paper under review, time dependence of the effective noise (or temperature) is not taken into account or discussed. Training is full batch, so the only noise is the injected probe (σ = 10⁻⁵). The paper does not specify whether it is applied once at branching or at every step. If it is applied once, the dynamics is deterministic, i.e. at T = 0, and the analysis is consistent. If it is applied at every step, Adam’s preconditioner P_t = 1/(√v_t + ε) makes the effective temperature direction- and time-dependent, T_eff ∝ σ²(√v_t + ε)/η. It changes by orders of magnitude as the gradients collapse. This changing temperature must be accounted for before interpreting waiting-time dependence as aging, or the late decorrelation and rise of D_f as landscape properties. Baity-Jesi et al. (ICML 2018), which is not cited, discussed exactly this issue for SGD. After rescaling displacements by the instantaneous noise strength, they found that aging stops at late times and that the slowdown comes from flat directions rather than barrier crossing. An analogous analysis is needed here, for instance two-time mean-square displacements normalized by the effective noise, or full-batch GD with Langevin noise at fixed temperature. 3) The glass interpretation is not convincing. Glassy arrest means caging, with a plateau in the mean-square displacement. Here the “arrested” phase is ballistic (D_f ≈ 1), i.e. slow, persistent drift. That is more naturally explained by motion along flat directions, or by norm minimization under weight decay (Boursier et al., 2025), which the authors acknowledge they cannot rule out. Moreover, the baseline replica correlation stays between about 0.96 and 1 throughout, which is weak evidence for a two-step relaxation.

Questions for authors

none, see weaknesses.

4 comments

We thank the reviewer for the constructive and technically insightful comments. In response to the reviewer's concerns, we have conducted substantial additional experiments across training fractions, weight-decay strengths, model widths, and architectures. These results broaden the empirical support for both the effectiveness of SAM-Swap and the reported dynamical signatures. We also appreciate the reviewer's questions regarding the interpretation of aging and glass-like dynamics, and have clarified the distinction between our empirical observations and their physical interpretation. Below, we address each concern in detail.

Response to Concerns Regarding Generality Across Experimental Settings

The reviewer is correct that the original submission focused on a single task and a single reference configuration. To assess the robustness of the reported dynamics, we have since conducted additional experiments across training fractions, weight-decay strengths, model widths, and architectural variations. We ran a sweep across training fractions from 0.1 to 0.9. Across training fractions from 0.4 to 0.9, Adam+Swap consistently accelerates grokking, reducing the time-to-generalization from beyond the 10410^4-epoch training horizon to approximately 10210^2 epochs.For training fractions between 0.1 and 0.3, Adam+Swap still improves generalization relative to the baseline model, although neither configuration reaches full generalization within the 10410^4-epoch training budget. For training fractions between 0.5 and 0.7, we additionally measured RC and FD and found that the characteristic dynamics remain qualitatively unchanged despite substantial differences in the test-accuracy trajectories.

We also performed weight-decay sweeps from 0 to 10−510^{-5} to investigate the effectiveness of SAM-Swap under weak regularization. Consistent with prior work, weight decay strongly affects when grokking occurs. However, Adam+Swap remains effective throughout this range, including the case of zero weight decay. This suggests that the observed effect is not tied to a particular regularization configuration and can persist even when weight decay alone is insufficient to induce rapid grokking.

To evaluate whether the reported phenomena are specific to the chosen hidden dimension, we conducted additional MLP width-scaling experiments with hidden dimensions of 128, 256, 512, and 1024. Adam+Swap continues to substantially accelerate grokking across all tested widths, and preliminary analysis indicates qualitatively similar RC and FD dynamics. These results suggest that the observed behavior is not an artifact of the particular width used in the main manuscript.

While the primary goal of this work is to investigate the optimization dynamics of grokking in a well-established experimental setting using our three-component framework (PM, RC, and FD), we believe these additional experiments indicate that the acceleration achieved by SAM-Swap is robust across a substantially broader range of settings than explored in the original submission. The additional RC and FD measurements further support the robustness of the reported dynamical signatures. All these results will be included in the revised manuscript.

Response to Concerns Regarding the Interpretation of the Memorization Phase

The reviewer is right that test accuracy does not remain completely flat during the period between fitting the training data and achieving full generalization. We would like to clarify that the interval between reaching 99% training accuracy and 99% test accuracy is our operational definition of the generalization delay, rather than a regime of exclusive memorization. Several recent studies have argued that grokking should be viewed more broadly than abrupt transition from pure memorization to full generalization. For example, Huang et al. (2024) propose a unified framework in which different training regimes emerge depending on the relationship between training dataset size and model memorization capacity. Partial generalization may emerge in the early stage of training when the training data fraction is high and beyond the model memorization capacity in grokking systems (we also observed qualitatively similar behavior in our new experiments as detailed below). Humayun et al. (2024) demonstrate that delayed generalization can arise across a wide range of learning settings like training fractions. Nanda et al. (2023) also show that grokking can emerge from the gradual formation of generalizing circuits and the subsequent removal of memorizing components. From this perspective, the gradual rise in test accuracy observed in our experiments is compatible with delayed generalization characteristics of grokking. In addition, our interpretation of glass-like kinetic arrest implies strongly slowed and constrained optimization dynamics, rather than a stop of learning. Therefore, gradual improvement in test accuracy is not necessarily inconsistent with slow relaxation toward a generalizing state.

To directly address this concern, we repeated the experiment with a reduced training fraction of 0.5. In this setting, test accuracy remains below 20% throughout most of the pre-grokking regime and only exhibits a rapid increase near the eventual transition. Despite this substantially different test-accuracy trajectory, the qualitative behavior of RC and FD remains unchanged.

More broadly, additional experiments show that Adam+Swap continues to accelerate grokking across training fractions from 0.4 to 0.9, indicating that the effectiveness of SAM-Swap is not restricted to the particular training fraction used in the main manuscript.

These results suggest that the RC and FD signatures are not simply a consequence of slow, progressive generalization. While the two settings can lead to substantially different rates of test accuracy improvement before the final generalization, RC and FD dynamics remain qualitatively similar. This suggests that the reported dynamical signatures are not specific to the gradual generalization trajectory highlighted by the reviewer. We will revise the manuscript to more carefully distinguish between memorization, partial generalization, and full generalization, and clarify that the reported RC and FD signatures remain robust across both gradual and sharply delayed forms of grokking.

Response to Concerns Regarding Effective Temperature and Aging Interpretation

The reviewer raises an important point regarding the interpretation of aging phenomena under adaptive optimizers, and we will add a discussion of Baity-Jesi et al. (2018) in the revised manuscript. We first clarify that the Gaussian probe is independently added directly to the model parameters after each optimization step, rather than to the gradients. Therefore, the magnitude of the injected noise is not directly rescaled by Adam’s adaptive preconditioner. We also acknowledge that the effective temperature may be direction- and time-dependent because of the evolving deterministic mobility associated with the adaptive preconditioner, even though the injected perturbation variance is fixed. However, the observables considered in this work differ substantially from displacement-based analyses. RC measures the relative evolution of replicas, while FD is a scale-invariant local probe defined through the relationship between displacement and accumulated trajectory length. Unlike raw displacement observables, both quantities are invariant under appropriate uniform spatial rescaling, making them complementary to absolute displacement measurements such as PM. This is also the main motivation for us to introduce these two metrics into our framework. More importantly, if the observed dynamics were driven primarily by either a changing effective temperature or unconstrained diffusion along flat directions, one would expect a gradual increase in replica separation as a function of waiting time throughout training as perturbations continuously accumulate. Instead, most replicas remain highly correlated RC≈0.96RC \approx 0.96 for a period before undergoing a delayed two-step decorrelation process. This behavior suggests that while both adaptive noise scaling and flat directions may influence the timescale of exploration, they are unlikely to fully explain the observed replica dynamics. The RC plateau indicates that optimization remains significantly constrained despite continual perturbations, pointing to additional structure in the loss landscape beyond simple diffusion along flat directions.

Therefore, while we do not view our observations as contradictory to Baity-Jesi et al. (2018), we believe they point to a more nuanced picture in the grokking regime, as Baity-Jesi’s work studied general ML tasks (no delayed generalization). Our results provide a complementary characterization of slow relaxation in the context of grokking. Flat directions may contribute to slow relaxation, but the prolonged RC plateau indicates that they are unlikely to be the sole factor governing the observed glass-like dynamics. We will clarify these points in the revised manuscript.

Response to Concerns Regarding the Glassy Interpretation

Our interpretation does not pursue a one-to-one correspondence between neural-network optimization and structural glass formers. Rather, we use the term glassy in the dynamical sense, referring to the simultaneous presence of slow relaxation, waiting-time dependence, aging-like behavior (two-step replica relaxation depending on wait time), and persistent replica correlations. The central claim of the manuscript is therefore not that grokking reproduces every phenomenological feature of particle caging, but that it exhibits several signatures commonly associated with glassy dynamical systems.

We agree that persistent drift along flat directions or weight-decay-induced norm minimization, as proposed by Boursier et al. (2025), may contribute to the observed dynamics. However, as discussed in our Response to Concerns Regarding Effective Temperature and Aging Interpretation, simple stationary diffusion along flat directions alone cannot readily account for the observed waiting-time-dependent replica relaxation. Moreover, our additional experiments (in response to Response to Concerns Regarding Generality Across Experimental Settings) demonstrate that SAM-Swap remains effective even in the absence of weight decay, suggesting that its acceleration mechanism does not require weight-decay-induced norm minimization.

Importantly, the two-step relaxation is not limited to the initial plateau. As indicated in the subgraph of Fig. 1c, RC remains near ∼0.96\sim0.96 for an extended period before subsequently decorrelating to values as low as ∼0.2\sim0.2. These observations suggest that the dynamics are more structured than simple unconstrained motion along flat directions. We therefore view our results as consistent with a picture in which persistent drift may coexist with glass-like dynamical constraints, but additional constraints imposed by the optimization landscape should govern the long-timescale relaxation dynamics. The manuscript will be revised to make this interpretation more precise and to better distinguish the evidence from the interpretation.

I thank the authors for their detailed reply but I remain unconvinced that the results of this work offers substantial new solid insights on the training dynamics to be worth publishing in ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.