- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper points out that traditional forgetting metrics cannot measure whether a model's internal knowledge has been erased or whether the model is simply unable to utilize the knowledge.
It proposes the Restoration Saving metric, which measures the degree of forgetting by comparing how much the model recovers when relearning the same task under the same budget after "forgetting" has occurred, relative to learning a new task from scratch.
It also finds that under the same "behavioral failure" (at steps 800/1200, the model is unable to answer questions correctly in both cases), the corresponding recovery capabilities differ, emphasizing that in addition to measuring a model's "behavioral failure," it is also necessary to measure the model's "recovery capability."
Critical strengths
- This paper emphasizes that in measuring model forgetting, in addition to measuring behavioral failure, it is necessary to further measure the degree of model recovery.
Critical weaknesses
- The overall readability of the paper is poor.
- Q1 of the paper is an obviously valid conclusion, and related papers[1,2] have also demonstrated that LMs do not truly forget previously learned knowledge, so prior learning will indeed be helpful for subsequent recovery.
- In Q2, the tasks used in the paper are all subsets of SuQA. Tasks A and C may have similarities in format and other aspects, so transferability between tasks is relatively strong. If tasks with greater differences could be used for demonstration, the conclusion would be better, for example, classification tasks and generation tasks, or two QA tasks from clearly different domains.
[1]. Spurious forgetting in continual learning of language models (ICLR 2025)
[2]. Unveiling and addressing pseudo forgetting in large language models (ACL 2025)
Questions for authors
Please refer to weaknesses
2 comments
Continuation of our response.
W3: Does disjoint recovery depend on similarities between A and C?
A, B, and C are all randomly drawn from the same pool of initially unknown SQuAD questions, so they share similar formats and difficulty distributions. If the benefit came only from format similarity, we would expect B to benefit as well. Instead, restudying C restores A much more than B, producing a gap of 0.67 ± 0.26 at k = 8 (Section 4.2).
We also agree that testing more different restudy material would strengthen this result. Following your suggestion, we ran new experiments using restorers from other sources (Appendix D, Table 9).
The most informative result is that even plain WikiText continuation, without any QA format or target-fact supervision, restores A over B: the gap is 0.64 ± 0.09 at k = 8, positive in all seven seeds. Restudying Natural Questions, a different QA dataset, also produces a substantial gap of 0.48 ± 0.28. In both cases, B remains near zero.
These results show that disjoint recovery under DBpedia interference does not require the restudy material to share the same QA format or source dataset as the forgotten facts. They also helped us refine our interpretation: much of this recovery appears to reflect a return to factual-answering behavior, while prior acquisition determines which facts become accessible again.
We have added these experiments to Appendix D and clarified their implications in Sections 4.2 and 4.5.
Thank you again for the feedback. Your concerns helped us improve the presentation, clarify the contribution relative to prior work, and test whether disjoint recovery depends on similarities between the restudy and evaluation tasks.
We believe the revised paper now makes a more precise claim: prior learning can continue to affect recoverability after behavioral access is lost, and under classification interference this advantage can be revealed even through training on unrelated text. We hope these additional results and revisions address your concerns and would appreciate your reconsideration of the assessment.
Thank you for the thoughtful review. We appreciate your recognition of the paper's central motivation: behavioral forgetting alone does not tell us how much a model can still recover. Your feedback also helped us identify where the presentation and the distinction from prior work needed to be clearer. We address your concerns below.
W1: Readability
We agree that the experimental setup can be difficult to follow, especially with the different sets and restudy conditions. To improve clarity, we have added a notation summary defining A/B/C/D, direct versus disjoint restudy, and the budget k (Section 2.2), along with a consolidated table of experimental setups (Table 3). We have also clarified the role of each comparison and added uncertainty bands and seed-level information to the main figures.
We hope these changes make the experimental design and main findings easier to follow.
W2: Novelty relative to prior work
We agree that previous work, including Zheng et al. (2025) and Sun et al. (2025), has shown that apparent forgetting does not necessarily mean knowledge has been erased. Our goal is not to establish this observation again, but to measure how much prior learning still helps recovery after behavioral access is lost.
An important difference is the randomized control. We teach the model facts from A and C while withholding B, with all three randomly drawn from the same pool of initially unknown questions. This allows us to compare previously learned and never-learned facts under the same restudy budget, rather than measuring recovery alone.
This distinction matters because we found that even restudying facts the model never learned (D) can restore much of the access to A. A randomized withheld control is therefore important for separating the effect of prior learning from a general return to factual-answering behavior.
Our results also go beyond showing that recovery is possible. At 800 and 1200 interference steps, greedy access to A is zero in every seed, yet the direct recovery gap drops from 0.50 to 0.22 (Section 4.4). The new zero-step likelihood analysis further shows that the remaining answer-likelihood advantage decreases with interference depth (Fig. 4c). Thus, models with identical behavioral accuracy can still differ substantially in how easily previously learned facts are recovered.
We have revised Section 5 to clarify these distinctions and better acknowledge the contributions of earlier work.
Continued in our second response below.