acceptodds
Under review as a conference paper at ICLR 2027

The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting

Abstract

While neural networks are typically evaluated by their training and test performance, these metrics do not reveal how robust a learned representation is. Recent studies have shown that solutions occupying larger volumes in parameter space, as quantified by Boltzmann entropy, often exhibit superior generalizability compared to those reached by conventional optimization, a phenomenon known as the high entropy advantage. Here we ask whether this advantage persists beyond generalization. Specifically, we investigate models' robustness, the ability to retain the learned knowledge when the model is subsequently trained to acquire new information. Using grokking in modular arithmetic as a controlled setting, we design a noise injection experiment to evaluate the robustness difference between AdamW-trained transformers and high-entropy model sampled from Wang-Landau Molecular Dynamics with identical saturated performance. By forcing both models to fully remember new data with random labels, we find that AdamW-trained models suffer from catastrophic forgetting, with original task test accuracy dropping from 100% to below 77%, whereas the high-entropy models maintain approximately 95% test accuracy. We term this hidden fragility behind apparent generalization the "grokked illusion." Through singular value decomposition of the neural network weights, we discover that high-entropy neural networks possess significantly higher effective rank in attention and MLP layers both before and after noise injection, indicating richer feature representations can serve as a buffer against catastrophic forgetting. Our findings demonstrate that perfect generalization does not imply equal robustness, offering a new perspective on what makes a trained model robust to interference.

Primary area:
learning theory
Keywords:
Boltzmann entropy, Wang-Landau Molecular Dynamics, Equilibrium, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mURSOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors investigate models’ robustness, the ability to retain the learned knowledge when the model is subsequently trained to acquire new information. Based on the analysis of the WLMD-trained model vs. the AdamW-trained model on a synthetic task, they found that the model with higher Boltzmann entropy can result in better robustness and less forgetting when learning new knowledge. Besides, the singular value analysis shows that high-entropy states exhibit higher effective rank.

Critical strengths
  1. The authors provide an interesting observation, the Grokked Illusion, on understanding robustness and catastrophic forgetting.
  2. The analysis of Boltzmann entropy and singular values provides a preliminary understanding of the Grokked Illusion.
Critical weaknesses
  1. The experiments are limited to the WLMD Model under a simple synthetic task (x+y)mod 67, while the author concludes that the observation in this setting is a general phenomenon.
  2. There are critical logical flaws in the induction from High Boltzmann entropy to high robustness and less catastrophic forgetting.
Questions for authors
  1. The author wants to demonstrate that the model with higher Boltzmann entropy can result in better robustness and less forgetting when learning new knowledge. However, the experiments are limited to the WLMD-trained model vs. the AdamW-trained model under a simple synthetic task (x+y)mod 67 without a control variable. The two models differ not only in the Boltzmann entropy, but also in the training process, optimization trajectory, and even training loss. For example, AdamW may lead to the formation of different internal representations due to more thorough optimization. These differences, in turn, can influence subsequent fine-tuning. Thus, the authors have not sufficiently demonstrated that the observed robustness gap stems from entropy, rather than from these differences. I believe it is necessary to include loss-matched baselines and to compare AdamW checkpoints from various stages of the training process.

  2. For each noise injection setting, 10 random seeds are used for the subsequent fine-tuning stage, rather than for training 10 independent WLMD models. Therefore, the stable results shown in Figure 1 merely indicate that given these specific checkpoints, the observed differences are stable with respect to the randomness of noise injection. It cannot be proven that WLMD-trained models are generally more robust than AdamW-based models. While the authors note that resampling WLMD requires approximately 4,000 GPU-hours, the high computational cost is not an excuse when they try to provide generalizable insight into robustness and forgetting.

  3. There is also a more fundamental conceptual issue in this paper. The authors define entropy as the volume of the parameter space corresponding to a given loss and accuracy. This is a property of a macrostate, rather than an attribute that can be naturally assigned to an individual checkpoint. The authors need to provide a clearer explanation of how they justify that two specific checkpoints being compared actually possess different levels of entropy.

  4. The authors argue that WLMD possesses a higher effective rank, which implies a richer representation capable of buffering the interference introduced by new knowledge. However, there is a significant logical leap here. A high weight rank does not necessarily equate to a richer task representation. The authors fail to demonstrate which high-rank directions actually encode knowledge relevant to the original tasks. Whether the gradients of new tasks conflict with the critical subspaces associated with the original tasks. Whether directly increasing the effective rank of an AdamW-based model actually improves retention. Consequently, this part of the work can only be characterized as a correlation analysis rather than a validation of the underlying mechanism.

  5. All experiments are under a simple synthetic task (x+y)mod 67. Before the new tasks, they also expanded the vocabulary from 68 to 136 and initialized the newly added parameters. So, here is the question: is the observed forgetting caused by new knowledge interfering with existing knowledge, or does it stem from the specific combination of vocabulary expansion and a shared output representation? At the very least, this should be verified using different datasets, model architectures, or tasks that do not require vocabulary expansion.

Reviewer_GJ1DOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors examine how starting from a high-entropy state rather than an AdamW-trained one affects catastrophic forgetting during subsequent fine-tuning. In their surgical experiment, they find that it dramatically reduces it.

Critical strengths
  • The applications of these kinds of models are wide and can maybe be used to improve pre-training pipelines---something ridiculous amounts of money are going towards right now.
  • The evaluation done here is methodical and quite surgical, lending significant empirical strength to the work. Working through it, I cannot find anything obviously wrong with it.
Critical weaknesses
  • There is already work showing that Adam is responsible for a lot of catastrophic forgetting compared to simpler algorithms like SGD (see, e.g., "Does the Adam Optimizer Exacerbate Catastrophic Forgetting?"). Since both models are fine-tuned with AdamW here, this is not an unfair comparison per se, but it does mean the size of the drop reported may be partly an Adam artifact, and the paper never checks whether the gap survives fine-tuning with SGD. It might be worth mentioning that, and ideally checking it.
  • The definition of forgetting used here is also quite problematic. Hetherington & Seidenberg found in 1989 that defining forgetting according to what is actually used when studying forgetting in humans causes the "catastrophic" nature of it to vanish. I think this is worth a footnote here to not misrepresent McClosky & Cohen.
  • Relatedly, if I'm reading this right, all of the original training data seems to be rehearsed in every fine-tuning update and stays at 100% training accuracy, so nothing is forgotten at the item level and instead what degrades is held-out generalization. The authors acknowledge this at the end of Section 2.2, but the title, abstract, and conclusions still present it as catastrophic forgetting, which is a bit of a deviation from the standard definition, no?
Questions for authors
  • Are the results presented here statistically significant?

Please also respond to the points raised in the weaknesses section.

1 comment

I appreciate the additional work done by the authors. If there was a mid-point between a 3 and a 4, I would change my score to that. But since there isn't, I'll maintain my score.

Reviewer_i1JfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether the high-entropy advantage extends beyond generalization to retention during subsequent learning. On modular addition, the authors compare an AdamW-trained Transformer with a WLMD-sampled equilibrium model, both initially achieving perfect test accuracy. Both models then learn new data while rehearsing all original training examples. The selected WLMD model preserves higher original-task test accuracy across random and structured new data, with the gap narrowing as the new task becomes more similar to the original one. Spectral analysis further reveals higher effective rank in the WLMD model's attention and MLP weights before and after subsequent learning, providing a structural correlate of its stronger retention.

Critical strengths

With the rapid development of AI, I greatly appreciate efforts to develop theoretical frameworks for understanding AI, and this paper represents one such effort. The paper identifies a quite interesting phenomena, moving from explaining generalization to ask whether equally accurate models retain their capabilities equally well when learning new information. The substantial retention gap provides an interesting observation about trained states that static accuracy alone does not reveal.

Critical weaknesses
  1. The authors aim to establish that high entropy leads to greater robustness against forgetting. However, the experiments mainly show that WLMD outperforms AdamW. These solutions may differ in many respects beyond entropy, including optimization trajectory, gradient structure, functionally sensitive directions, and local curvature of the loss landscape. Therefore, the observed advantage cannot be attributed solely to entropy. A more convincing test would be to obtain both high- and low-entropy solutions using the same optimization framework and directly compare their behavior.
  2. Given the statistical physics perspective and the stochastic nature of the sampling process, the evaluation appears insufficient to support general conclusions. The paper compares only one WLMD solution with one AdamW solution, making it difficult to rule out the possibility that the selected WLMD solution is unusually robust or the AdamW solution unusually fragile. Multiple independent runs and statistical analyses are needed to establish the robustness and generality of the findings.
  3. Although the models achieve similar classification accuracy, their training losses are not matched. In particular, AdamW achieves a lower loss than WLMD, potentially indicating different degrees of optimization and specialization to the original task. Such differences could substantially affect subsequent learning and forgetting, making it difficult to isolate the contribution of entropy.
  4. A minor concern is that subsequent learning continues to use all original training samples, which differs from standard continual learning protocols. The experiments therefore seem closer to evaluating how new data interfere with previously learned knowledge than to evaluating forgetting under typical sequential learning constraints.
  5. The novelty relative to Zhang et al. (NeurIPS 2025) is limited and needs further clarification. The previous work already established the high-entropy advantage in grokking and discussed how perturbations to parameters outside functional subnetworks can preserve generalization, linking larger solution volumes to functional robustness. The present study extends this intuition to learning new information but largely reuses the same framework and experimental setting. It remains unclear whether this extension provides fundamentally new physical or mechanistic insights beyond those already suggested by the previous work.
Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.