acceptodds
Under review as a conference paper at ICLR 2027

Controlled Demolition Is All You Need for Grokking

Abstract

There are a great many conjectures about how grokking is reached and what its properties are; many of them are supported by experiments and yet sit poorly with one another. I introduce three new methods for producing grokking — artificial ricochets (controlled demolition: the learning rate is escalated until the network breaks, and the network is then left to recover), an attention learning-rate over-drive, and an overdrive of the FFN input matrix together with row-wise weight normalization. All three run on the same training loop, the same architecture and the same initialization, and are compared there with the classical method, weight decay; this makes the paths to grokking commensurable and separates the intrinsic properties of the phenomenon from artifacts of the method that produces it. I show that the network’s work of generalizing can remain invisible to the usual metrics and is nonetheless brought out by applying another of these methods.

Primary area:
optimization
Keywords:
grokking, delayed generalization, representation learning, training dynamics, weight decay, learning rate interventions, weight normalization, training instability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_e6ghOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces several training interventions, such as learning-rate escalation and row normalization, and evaluates them on a transformer trained on modular division, a setting known to exhibit grokking. The stated aim is to compare different routes to grokking and distinguish properties of the phenomenon from effects of the training method. The main interpretations concern hidden formation of a generalizing representation and differences in weight-norm trajectories across interventions; they are qualitative and not sharply stated nor tested.

Critical strengths

The common experimental setup makes the interventions easy to compare, and the appendix documents substantial empirical exploration. This is the main strength of the work. However, the volume of experimentation does not compensate for the absence of a clear and convincingly established scientific conclusion.

Critical weaknesses

The paper modifies training in a system already known to grok, but does not establish what the modifications teach us about the phenomenon. Comparing training paths is a reasonable idea, yet the paper does not develop it into a decisive test of a clearly formulated hypothesis. The limited experimental scope makes this problem more serious. The principal interventions are studied in one task and model configuration, without establishing generality or robustness. In light of the small explanatory value of the results, it is hard to draw any conclusions from these experiments.

The main weakness is that the central mechanistic interpretation is not tested or quantified The claim that an efficient generalizing representation has already formed and merely needs to be revealed is not established nor properly defined. The intervention itself allows substantial further learning. Its outcome is therefore also consistent with partial progress, more favorable optimization conditions, or representation learning during recovery. The experiments do not distinguish these explanations or directly identify the representation claimed to be present. Hidden generalization is just an interpretation.

Likewise, the weight-norm observations are not quantitative and untested. In short, the paper does not turn its collection of observations into a scientific understanding.

Lastly, the idea that a "generalizing structure" can develop before a visible improvement in validation accuracy is already well established in the literature (eg hidden progress, mechanistic interpretability etc) which the author cites. But none of these was compared to, and the added value is unclear.

Questions for authors

What specific new scientific claim about grokking does this paper establish, and which result provides decisive evidence for it? In particular, which existing explanation can the proposed comparison distinguish from its alternatives? What evidence shows that the claimed generalizing representation is already present before the intervention, rather than being developed or substantially changed during the subsequent learning?

1 comment
AuthorsOpenReview ↗

Thank you for your thoughtful analysis of my paper.

New scientific claim about grokking: Controlled Demolition Is All You Need for Grokking. Artificially introducing perturbations into otherwise standard training without weight regularization is sufficient to accelerate grokking. Conversely, as shown in A Fix for Softmax and Cross‑Entropy (arXiv:2610.31642), which appeared after my submission, correcting errors in the training implementation substantially impedes grokking. I regard this as an extremely important observation, because most researchers seek to improve and stabilize training, whereas effective generalization, at least on some tasks, appears to require the opposite: making training less well behaved.

I must, however, accept the remainder of your criticism. Indeed, most of the other conclusions admit alternative interpretations, and it appears that these alternatives cannot be ruled out.

Nevertheless, many observations retain their scientific value even in the absence of rigorous proof. For example, the distinction between ricochets and slingshots as two entirely different phenomena is itself a useful observation that, to the best of my knowledge, has not previously been reported in the scientific literature. Correcting cross-entropy almost entirely eliminates slingshots but does not affect ricochets.

Thank you for your thoughtful analysis of my paper. In future submissions to other conferences, I will endeavor to focus on the central phenomenon, avoid diluting that focus, and refrain from including observations that admit multiple interpretations.

Reviewer_K74VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Grokking is the phenomenon where models generalize after a long period of (often complete) overfitting. Weight decay has been known to induce grokking, this paper looks at other methods: controlled demolition, and module wise lr scaling.

Critical strengths

Identifies three new approaches that speed up grokking relating to modifying learning rate which suggest a more intricate story behind grokking.

Critical weaknesses

Novel contributions are minor. Empirical results are narrow in scope focusing on one network and one dataset. Consequences of the results are not novel. Previous work has shown grokking without decreasing Euclidean norm. Approaches, while interesting, don’t meaningfully contribute to furthering our understanding of grokking, the author hasn’t proposed a new theory or a unified framework. The paper does not compare selective overdrive to a baseline that increases the learning rate uniformly across the network so its not clear how much of the improvement can be attributed to ordinary learning-rate tuning.

Questions for authors

Do these results suggest a new perspective or approach to understanding grokking beyond findings of prior work?

1 comment
AuthorsOpenReview ↗

Thank you for your careful consideration of the work; I will make the corresponding corrections to the text

You are partly correct that grokking without a reduction in weight norm was already known; however, after my submission, the paper A Fix for Softmax and Cross-Entropy (https://jonbarron.info/softmaxfix/) appeared, showing that cross-entropy implementations had contained a numerical error for many years.

With the corrected implementation, a network may generalize under favorable circumstances, but it does not fully achieve grokking. Even this modest result is obtained under conditions of reduced training stability due to an extremely small ε=10⁻¹⁶, compared with the ε=10⁻⁸ commonly used for this task. I have also verified that the slingshots described by Thilak almost entirely disappear. Meanwhile, the ricochets described in my paper remain effectively unchanged, supporting the novelty and validity of some of my observations.

Furthermore, the conclusion that grokking is inseparably linked to some form of disruption of normal network training receives additional support from the observation that correcting an arithmetic error in the implementation almost eliminates grokking, whereas introducing artificial errors, as in my “controlled demolition” method, accelerates it; even entirely random perturbations, as in “random punch,” improve generalization, and these findings retain their importance.

To me, this suggests a substantially different direction for future research on grokking. To the best of my knowledge, at least before my submission, no paper had proposed deliberately disrupting training as a route to grokking. My answer to your question is therefore unequivocally yes: this paper contributes a number of previously unreported findings.

Moreover, correcting cross-entropy does not alter the ricochets described in my paper, further supporting the distinction between these two phenomena and the need to study ricochets separately.

However, I fully accept your criticism that I did not compare my approach with simply increasing the learning rate of the entire network before submission. Following your comment, I performed this comparison, and it substantially affects the conclusions. Increasing the learning rate of the entire network produces an effect similar to increasing it only for the fully connected layers: a threefold increase accelerates grokking, but further increases destabilize the network. The conclusions concerning the last of the three methods therefore need to be revised. The first two methods remain relevant.

Thank you for your careful consideration of the work; I will make the corresponding corrections to the text and, in submissions to other conferences, retain only the “controlled demolition” method, with a stronger emphasis on clearly formulated conclusions about it. In the present paper, I will also document the discrepancies identified, should a language-model agent subsequently review the ICLR submission outcomes.

Reviewer_NXA1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces three new methods for producing grokking: artificial ricochets (controlled demolition) where the learning rate is escalated until the training diverges (which the paper weirdly calls 'the network breaks'), an attention learning-rate overdrive, and an overdrive of the FFN input matrix with normalization. All these terms are proposed yet not defined in the paper.

Critical strengths

I really cannot see the critical strengths of this paper and am quite negative about it.

Critical weaknesses

This paper contains numerous formatting issues (not in ICLR format, uncaptioned tables, unformatted or badly formatted text chunks), sloppy, imprecise, not technical language (yes it is heavily AI generated!), and does not even properly define the concepts that it uses: overdrive, demolition, artificial ricochets... Most of it reads like some notes taken on the side. Not even the references are proper. As such, it is a DDOS attack on our community and a waste of my valuable time. It should have been desk rejected but unfortunately we are slow and too liberal in implementing such policies. Hence, this is the only review I will provide to this paper.

Questions for authors

To ask questions, the paper should be understandable.

1 comment
AuthorsOpenReview ↗

I am very sorry that reading the paper proved too difficult a task for you.

You stated that the paper does not contain a definition of "overdrive". This is an ordinary English word whose definition can be found in a dictionary, for example here: https://dictionary.cambridge.org/us/dictionary/english/overdrive. The phrase "Attention LR overdrive", as used in the paper, is defined in Section "4.2. Attention LR overdrive", where the definition is in bold. Verbatim: the attention matrix should be given more opportunity to learn than the other layers. That is what I call attention LR overdrive: the optimized parameters are split into param_groups and the W_QKV matrix is set aside in a group of its own that receives 9 times the learning rate.

You stated that the paper does not contain a definition of "demolition". A translation of this English word can also be found in the same Cambridge dictionary. The phrases controlled and artificial demolition are defined in Section "4.1. Artificial ricochets — controlled demolition", immediately after the bold subheading "The scheme of an artificial demolition."

You stated that the paper does not contain a definition of "ricochet". The same Section 4.1 contains my definition, beginning with the subheading The phenomenon: a ricochet, where I describe in detail what it is and how it differs from the slingshots phenomenon described and studied by Thilak in '22.

You write: "yes it is heavily AI generated!" As I explicitly stated in the "AI Use Statement" section, all of the English text was translated from my language into English using AI; English is not my native language and, judging by your comments, it is not yours either.

In your review, you explicitly stated, "I learned little or nothing new from this paper." If you have actually read the paper, have "substantial direct research experience on this topic", and your assessment is not related to the possibility that the paper may have inadvertently refuted your own research, could you point to a scientific source from which you had previously learned that artificially induced disruptions in training can lead to accelerated grokking? I examined more than 140 papers on this topic in detail and about 600 at a cursory level, but encountered nothing of the kind in the scientific literature. If you indicated what your judgment was based on, you would greatly enrich not only my knowledge but also that of everyone fortunate enough to read this review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.