- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tries to understand the relation between machine unlearning and the learning mechanisms. Specifically, it tries to analyze the damage of retained knowledge across the memorization-generalization spectrum. Through controlled experiments and mechanistic analyses, the authors validate the hypothesis that generalization suffers greater retained-knowledge damage, and verify that the conclusions hold in LLM unlearning scenarios. Finally, the authors discuss two practical insights that extend from the main findings.
Critical strengths
- The toy experiments are well-designed; I like how the paper uses a controlled scenario to analyze the memorization-generalization spectrum. The experimental design and findings clearly address the hypothesis mentioned in Section 3 (Line 158). As the paper states, this work is the first attempt to understand unlearning from this perspective.
- Practical insights discussed in the discussion are useful to the unlearning community. I think this practical insight helps (Line 465, copied below) future work align and evaluate unlearning methods on more realistic settings and with reasonable goals.
Thus, retraining without the forget set is a gold standard for removing its training contribution, but does not always achieve the behavioral goal of suppressing outputs.
Critical weaknesses
- While this paper provides interesting analyses and insights, the practical impact is unclear. For instance, how does the finding help identify which knowledge is unlearnable for a model? How do we estimate the best forget-retrain tradeoff in a given scenario, and how do we achieve it? The paper states its insights in Section 6, no actionable insights are provided or tested. This limits the significance of the paper's findings.
- [Minor] Following the first point, I think this paper could benefit from existing benchmarks and tested models to see if the paper's findings hold in the community's common evaluations. Since I am not familiar with the existing machine unlearning work, I am not sure if this request is feasible. However, I think at least an attempt or discussion is needed because it will test whether this paper's findings generalize to these common scenarios.
Questions for authors
Questions:
- For toy experiments, I wonder if the experimental findings will change if we use a less capable model. As the model's complexity decreases, its memorization capability decreases as well. Therefore, I am curious how sensitive the conclusion is when testing on different models.
Suggestions
- For the clearness of this presentation, I think it will be better to put the hypothesis (copied below) in the Introduction or at the beginning of Sec 3. It will help readers to understand why memorization and generalization matter for unlearning; it will also make the goal of the experiments clear before stating the experiment results.
We hypothesize that unlearning the latter must damage this circuit, on which retained examples also depend, and thus incurs greater retain damage.
5 comments
Q1. Would the conclusions change with a less capable model?
Model capacity can affect our two toy experiments differently, the modular addition experiment (Sec. 3) and BMA (Sec. 4), so we examine each separately.
-
Modular addition (Sec. 3).
- The binary comparison requires two models that fit the training set equally well but solve it differently, one by memorization and one by generalization. Such a pair is available only when generalization is delayed well after the training set is fit, i.e., when the model groks.
- Grokking requires enough capacity to memorize the training set first: the network fits it with a memorizing solution, which is faster to learn, and only later moves to the more efficient generalizing solution under regularization such as weight decay (Nanda et al., 2023; Doshi et al., 2024).
- A less capable model cannot memorize the training set, so we expect it to generalize directly without a memorizing stage. We tested this by training the Sec. 3 model at smaller widths (Table R2; see Figure 5 in App. B.5 of the revised paper for the full training curves).
Table R2. Modular addition at different widths (range over three seeds, 60k epochs). The largest gap between training and test accuracy marks the memorizing stage.
(parameters) Max. train − test accuracy Final train / test accuracy (%) 128 (227k), paper 0.96–0.98 100 / 100 64 (64k) 0.94–0.97 100 / 100 32 (20k) 0.62–0.80 100 / 100 16 (6.8k) 0.29–0.39 44–100 / 5–98 8 (2.7k) 0.19–0.21 68–81 / 51–68 - Models with fit the training set within a few hundred epochs and generalize thousands of epochs later. Models with show no such gap: their test accuracy rises together with training accuracy, or they fail to fit the training set. With less capacity, generalization follows fitting immediately, so the memorizing model required for the binary comparison cannot be obtained.
-
BMA (Sec. 4).
- BMA needs no such pair, since the bucket size sets the balance between memorization and generalization by design. We retrained all five BMA models at half the width (; all reach 100% training accuracy) and repeated RL at 5% forget over three seeds (Table R3; Figure 7 in App. C.4 of the revised paper).
Table R3. Retain damage at under RL (5% forget; mean ± 95% CI over three seeds).
Memorization floor (paper) 0 bits 0.874 ± 0.008 0.896 ± 0.014 2 bits 0.804 ± 0.037 0.723 ± 0.063 4 bits 0.717 ± 0.077 0.653 ± 0.017 6 bits 0.484 ± 0.031 0.639 ± 0.022 8 bits 0.291 ± 0.041 0.595 ± 0.013 - Retain damage again decreases monotonically with the memorization floor in the smaller model (Spearman ρ = −1 at both widths).
- The smaller model, however, shows more retain damage at high floors (0.60 vs. 0.29 at 8 bits). Since it memorizes close to its capacity (initial retain probability 0.963 at 8 bits), its memorized examples appear to interfere more with each other, in line with the reviewer's intuition that memorization capability shrinks with model size.
Suggestion. Stating the hypothesis earlier
- We agree that this hypothesis is the core of our work and deserves to be stated up front. In the revised paper, we now state it in the introduction (in blue): "We hypothesize that unlearning what a model has generalized must damage this shared mechanism, on which retained examples also depend, and thus incurs greater retain damage than unlearning what it has memorized."
References
Doshi, D., Das, A., He, T., & Gromov, A. (2024). To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=UHjE5v5MB7
Dorna, V., Mekala, A. R., Zhao, W., McCallum, A., Kolter, J. Z., Lipton, Z. C., & Maini, P. (2026). OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=Gy67Zh5X1i
Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., & Kolter, J. Z. (2024). TOFU: A Task of Fictitious Unlearning for LLMs. In First Conference on Language Modeling. https://openreview.net/forum?id=B41hNBoWLo
Morris, J. X., Sitawarin, C., Guo, C., Kokhlikyan, N., Suh, G. E., Rush, A. M., Chaudhuri, K., & Mahloujifar, S. (2026). How much can language models memorize? In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=bA6BgSbaUi
Wei, J. T.-Z., Godbole, A., Khan, M. A., Wang, R., Zhu, X., Flemings, J., Kashyap, N., Gummadi, K. P., Neiswanger, W., & Jia, R. (2025). Hubble: a Model Suite to Advance the Study of LLM Memorization. arXiv:2510.19811. https://arxiv.org/abs/2510.19811
We thank the reviewer for the thoughtful review, and for finding the toy experiments well designed and the practical insights useful to the unlearning community. Below, we first address the weaknesses and then answer the question and the suggestion.
On the practical impact of our findings
The reviewer notes that the lack of actionable insights limits the significance of our findings. Specifically, the reviewer asks how to identify which knowledge is unlearnable and to what extent, and how to achieve the best forget–retain trade-off accordingly.
While we focus largely on the science of how learning shapes unlearning, our insights point to a concrete practice for LLM unlearning in response to each question: (1) estimating how difficult a target is to unlearn before unlearning it, and (2) adjusting unlearning at the token level according to this difficulty.
Taken together, we believe our findings are not confined to controlled experiments but carry concrete implications for unlearning in practice: they indicate in advance how costly it is to unlearn a target, and suggest how unlearning methods can reduce this cost.
On testing our findings on existing benchmarks
Table R1. Retain damage when the forget per-token probability reaches 10% of its initial value. Retain-near is the other half of the same authors; retain-far is 100 QA pairs from other authors.