- Rating
- 1
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
In this paper, the authors investigate models’ robustness, the ability to retain the learned knowledge when the model is subsequently trained to acquire new information. Based on the analysis of the WLMD-trained model vs. the AdamW-trained model on a synthetic task, they found that the model with higher Boltzmann entropy can result in better robustness and less forgetting when learning new knowledge. Besides, the singular value analysis shows that high-entropy states exhibit higher effective rank.
Critical strengths
- The authors provide an interesting observation, the Grokked Illusion, on understanding robustness and catastrophic forgetting.
- The analysis of Boltzmann entropy and singular values provides a preliminary understanding of the Grokked Illusion.
Critical weaknesses
- The experiments are limited to the WLMD Model under a simple synthetic task (x+y)mod 67, while the author concludes that the observation in this setting is a general phenomenon.
- There are critical logical flaws in the induction from High Boltzmann entropy to high robustness and less catastrophic forgetting.
Questions for authors
-
The author wants to demonstrate that the model with higher Boltzmann entropy can result in better robustness and less forgetting when learning new knowledge. However, the experiments are limited to the WLMD-trained model vs. the AdamW-trained model under a simple synthetic task (x+y)mod 67 without a control variable. The two models differ not only in the Boltzmann entropy, but also in the training process, optimization trajectory, and even training loss. For example, AdamW may lead to the formation of different internal representations due to more thorough optimization. These differences, in turn, can influence subsequent fine-tuning. Thus, the authors have not sufficiently demonstrated that the observed robustness gap stems from entropy, rather than from these differences. I believe it is necessary to include loss-matched baselines and to compare AdamW checkpoints from various stages of the training process.
-
For each noise injection setting, 10 random seeds are used for the subsequent fine-tuning stage, rather than for training 10 independent WLMD models. Therefore, the stable results shown in Figure 1 merely indicate that given these specific checkpoints, the observed differences are stable with respect to the randomness of noise injection. It cannot be proven that WLMD-trained models are generally more robust than AdamW-based models. While the authors note that resampling WLMD requires approximately 4,000 GPU-hours, the high computational cost is not an excuse when they try to provide generalizable insight into robustness and forgetting.
-
There is also a more fundamental conceptual issue in this paper. The authors define entropy as the volume of the parameter space corresponding to a given loss and accuracy. This is a property of a macrostate, rather than an attribute that can be naturally assigned to an individual checkpoint. The authors need to provide a clearer explanation of how they justify that two specific checkpoints being compared actually possess different levels of entropy.
-
The authors argue that WLMD possesses a higher effective rank, which implies a richer representation capable of buffering the interference introduced by new knowledge. However, there is a significant logical leap here. A high weight rank does not necessarily equate to a richer task representation. The authors fail to demonstrate which high-rank directions actually encode knowledge relevant to the original tasks. Whether the gradients of new tasks conflict with the critical subspaces associated with the original tasks. Whether directly increasing the effective rank of an AdamW-based model actually improves retention. Consequently, this part of the work can only be characterized as a correlation analysis rather than a validation of the underlying mechanism.
-
All experiments are under a simple synthetic task (x+y)mod 67. Before the new tasks, they also expanded the vocabulary from 68 to 136 and initialized the newly added parameters. So, here is the question: is the observed forgetting caused by new knowledge interfering with existing knowledge, or does it stem from the specific combination of vocabulary expansion and a shared output representation? At the very least, this should be verified using different datasets, model architectures, or tasks that do not require vocabulary expansion.
I appreciate the additional work done by the authors. If there was a mid-point between a 3 and a 4, I would change my score to that. But since there isn't, I'll maintain my score.