acceptodds
Under review as a conference paper at ICLR 2027

Forgotten or Inaccessible? Restoration Savings Reveal What Language Models Retain After Fine-Tuning

11 views

Abstract

Catastrophic forgetting is commonly defined as a substantial drop in performance on previously learned tasks after training on new tasks or data. However, this behavioral failure does not establish that the underlying knowledge was erased. We test this distinction with restoration savings, which asks whether, given the same small amount of retraining, a forgotten ability is easier to recover than a comparable ability that was never learned. We teach a model a random subset of facts it did not know, induce forgetting by fine-tuning it on an unrelated task, and then retrain learned and never-learned facts under this same budget. Previously learned facts are recovered far more easily, even when they are not themselves retrained and when they are asked in new wordings. Together, these findings show that behavioral access can be lost while a recoverable trace of prior learning remains.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
catastrophic forgetting, llms, continual learning, language modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8Ba6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper points out that traditional forgetting metrics cannot measure whether a model's internal knowledge has been erased or whether the model is simply unable to utilize the knowledge.

It proposes the Restoration Saving metric, which measures the degree of forgetting by comparing how much the model recovers when relearning the same task under the same budget after "forgetting" has occurred, relative to learning a new task from scratch.

It also finds that under the same "behavioral failure" (at steps 800/1200, the model is unable to answer questions correctly in both cases), the corresponding recovery capabilities differ, emphasizing that in addition to measuring a model's "behavioral failure," it is also necessary to measure the model's "recovery capability."

Critical strengths
  • This paper emphasizes that in measuring model forgetting, in addition to measuring behavioral failure, it is necessary to further measure the degree of model recovery.
Critical weaknesses
  • The overall readability of the paper is poor.
  • Q1 of the paper is an obviously valid conclusion, and related papers[1,2] have also demonstrated that LMs do not truly forget previously learned knowledge, so prior learning will indeed be helpful for subsequent recovery.
  • In Q2, the tasks used in the paper are all subsets of SuQA. Tasks A and C may have similarities in format and other aspects, so transferability between tasks is relatively strong. If tasks with greater differences could be used for demonstration, the conclusion would be better, for example, classification tasks and generation tasks, or two QA tasks from clearly different domains.
[1]. Spurious forgetting in continual learning of language models (ICLR 2025)
[2]. Unveiling and addressing pseudo forgetting in large language models (ACL 2025)
Questions for authors

Please refer to weaknesses

2 comments

Thank you for the thoughtful review. We appreciate your recognition of the paper's central motivation: behavioral forgetting alone does not tell us how much a model can still recover. Your feedback also helped us identify where the presentation and the distinction from prior work needed to be clearer. We address your concerns below.


W1: Readability

We agree that the experimental setup can be difficult to follow, especially with the different sets and restudy conditions. To improve clarity, we have added a notation summary defining A/B/C/D, direct versus disjoint restudy, and the budget k (Section 2.2), along with a consolidated table of experimental setups (Table 3). We have also clarified the role of each comparison and added uncertainty bands and seed-level information to the main figures.

We hope these changes make the experimental design and main findings easier to follow.


W2: Novelty relative to prior work

We agree that previous work, including Zheng et al. (2025) and Sun et al. (2025), has shown that apparent forgetting does not necessarily mean knowledge has been erased. Our goal is not to establish this observation again, but to measure how much prior learning still helps recovery after behavioral access is lost.

An important difference is the randomized control. We teach the model facts from A and C while withholding B, with all three randomly drawn from the same pool of initially unknown questions. This allows us to compare previously learned and never-learned facts under the same restudy budget, rather than measuring recovery alone.

This distinction matters because we found that even restudying facts the model never learned (D) can restore much of the access to A. A randomized withheld control is therefore important for separating the effect of prior learning from a general return to factual-answering behavior.

Our results also go beyond showing that recovery is possible. At 800 and 1200 interference steps, greedy access to A is zero in every seed, yet the direct recovery gap drops from 0.50 to 0.22 (Section 4.4). The new zero-step likelihood analysis further shows that the remaining answer-likelihood advantage decreases with interference depth (Fig. 4c). Thus, models with identical behavioral accuracy can still differ substantially in how easily previously learned facts are recovered.

We have revised Section 5 to clarify these distinctions and better acknowledge the contributions of earlier work.

Continued in our second response below.

Continuation of our response.


W3: Does disjoint recovery depend on similarities between A and C?

A, B, and C are all randomly drawn from the same pool of initially unknown SQuAD questions, so they share similar formats and difficulty distributions. If the benefit came only from format similarity, we would expect B to benefit as well. Instead, restudying C restores A much more than B, producing a gap of 0.67 ± 0.26 at k = 8 (Section 4.2).

We also agree that testing more different restudy material would strengthen this result. Following your suggestion, we ran new experiments using restorers from other sources (Appendix D, Table 9).

The most informative result is that even plain WikiText continuation, without any QA format or target-fact supervision, restores A over B: the gap is 0.64 ± 0.09 at k = 8, positive in all seven seeds. Restudying Natural Questions, a different QA dataset, also produces a substantial gap of 0.48 ± 0.28. In both cases, B remains near zero.

These results show that disjoint recovery under DBpedia interference does not require the restudy material to share the same QA format or source dataset as the forgotten facts. They also helped us refine our interpretation: much of this recovery appears to reflect a return to factual-answering behavior, while prior acquisition determines which facts become accessible again.

We have added these experiments to Appendix D and clarified their implications in Sections 4.2 and 4.5.


Thank you again for the feedback. Your concerns helped us improve the presentation, clarify the contribution relative to prior work, and test whether disjoint recovery depends on similarities between the restudy and evaluation tasks.

We believe the revised paper now makes a more precise claim: prior learning can continue to affect recoverability after behavioral access is lost, and under classification interference this advantage can be revealed even through training on unrelated text. We hope these additional results and revisions address your concerns and would appreciate your reconsideration of the assessment.

Reviewer_j6JaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors test whether facts an LLM stops answering after fine-tuning on another task are erased or only harder to access. They split unknown SQuAD-v2 questions into taught sets (A, C) and an untaught set (B), suppress taught answers with DBpedia classification fine-tuning, and retrain briefly from the shared checkpoint.

Critical strengths
  • The topic is interesting.
  • Empirical validation seems solid because the main effect replicates on a second model family.
Critical weaknesses
  • Artificial forgetting setup. Severe forgetting only happens under DBpedia classification, while GSM8K and summarization cause only partial forgetting (Appendix A, Appendix I). After DBpedia tuning, the model often outputs class labels like Artist or Company instead of facts (Appendix L, on a separate dataset), and scaling down LoRA weights with zero retraining already restores about half the accuracy on A (Tab. 6). This looks much more like shallow format interference [2, 3] than real forgetting, yet the authors generalize to language models broadly in Section 4.5 and the conclusion.
  • Limited novelty over prior relearning work. Measuring retention via faster relearning goes back to connectionist work on savings [1], and fast recovery within a few steps is known [2]. The authors themselves note in Section 5 that disjoint recovery and no-acquisition controls are not new (citing [3]), and recent multimodal work [4] likewise showed that output-shift forgetting recovers after fine-tuning on a later unrelated task without rehearsal. None of [1, 2, 4] is cited, leaving the randomized control as the main new contribution.
  • Missing zero-step log-probability baseline in Fig. 4. Greedy accuracy reaches zero as soon as any wrong token ranks first, so zero greedy accuracy does not mean retraining is required. A already has much higher zero-step log-probability than B (Appendix C), and a two-step retraining probe does not reliably improve over zero-step likelihood (Tab. 14), yet the authors omit zero-step log-probability in Fig. 4.
  • Unexplained jump on the untaught set B in Fig. 2b. Since k steps cover 4k examples (Appendix B), fewer than half of B's questions are seen at k=16. Yet in Fig. 2b, B jumps from near zero to high accuracy at k=16 and later passes A, whereas never-learned facts in Fig. 2c rise much more slowly. They do not explain how unseen B questions become correct (shared SQuAD articles are an obvious possibility).
  • Hard-to-track experimental variations. Across Section 4 and the appendix, they keep changing dataset splits, step counts (Tab. 3), and scorers, and include an "A-only" branch in Fig. 1 never mentioned in the text, which is hard to keep track of due to limited human context window.
  • Missing error bars in most figures. Main runs use seven seeds, yet they omit error bars and seed points in Fig. 2 and Fig. 3a,c, even though disjoint restudy varies widely across seeds in Fig. 3b.

[1]. Is There "Catastrophic Interference" in Connectionist Networks? Phil A. Hetherington, et. al. CogSci 1989.

[2]. Mechanistically Analyzing the Effects of Fine-Tuning on Procedurally Defined Tasks. Samyak Jain, et. al. ICLR 2024.

[3]. Spurious Forgetting in Continual Learning of Language Models. Junhao Zheng, et. al. ICLR 2025.

[4]. How To Teach Large Multimodal Models New Skills. Zhen Zhu, et. al. ECCV 2026.

Questions for authors
  1. Why does untaught set B jump so high at k=16 in Fig. 2b when fewer than half of B has been seen, and were SQuAD questions split by article?
  2. Does this effect hold under a generative task that causes severe forgetting, and how does zero-step answer log-probability look in Fig. 4?
3 comments

Thank you for the thoughtful review. We appreciate your interest in the problem and your recognition of the empirical results, particularly the replication across model families. Your comments motivated several new experiments and helped us clarify the scope and novelty of our findings. We address your concerns below.


W1 & Q2: Artificial forgetting and generalization beyond classification

We agree that the DBpedia setting is largely driven by output-format interference and that our original claims were too broad. We checked this on the headline SQuAD set: after DBpedia interference, 75% of answers are class labels, falling to 6% after four restudy steps (Appendix D). Together with the LoRA scaling result, this supports your interpretation that much of the forgetting reflects output-format interference.

To test whether the effect extends beyond classification, we explored stronger generative interference settings (Appendix I). With Alpaca instruction tuning at a higher learning rate, access to A dropped from 0.91 to 0.003 across seven seeds, while retaining some instruction-following ability (token F1 = 0.25).

Previously learned facts remain easier to relearn, although the advantage is smaller and takes longer to emerge: the direct A−B gap reaches 0.25 ± 0.05 at k = 32, positive in all seven seeds. However, unlike DBpedia, restudying only C does not restore A (gap 0.02 ± 0.03 at k = 32).

This helped us clarify our findings: the benefit of prior learning extends beyond classification, but rapid recovery without directly relearning the forgotten facts appears specific to the DBpedia setting. We have revised the abstract, Section 4.5, conclusion, and Appendix A accordingly.


Novelty relative to prior work

We agree that relearning savings and rapid recovery after fine-tuning are not new, and thank you for highlighting the missing references. We have expanded Section 5 to discuss Hetherington and Seidenberg (1989), Jain et al. (2024), Zheng et al. (2025), and Zhu et al. (2026). We also explicitly acknowledge that disjoint recovery and no-acquisition controls have precedents, including Deeb and Roger (2024).

Our contribution is more specific: we randomly assign initially unknown facts to acquired and withheld sets within the same model, then compare their recovery from a shared post-interference checkpoint under matched restudy budgets. This lets us isolate the effect of prior acquisition rather than differences between facts the model already knew and those it did not.

We further test recovery under unseen question wordings and across interference depths. New restorer experiments (Appendix D) also show that plain WikiText continuation, without QA formatting or target-fact supervision, restores A over B (gap 0.64 ± 0.09). We agree this is consistent with earlier output-shift explanations and have clarified this connection in the revised related work.

Continued in our second response below.

Continuation of our response.


W3 & Q2: Zero-step likelihood and what restoration savings adds

We agree that Fig. 4 is better with the zero-step likelihood comparison. We have added the gold-answer log-probability advantage of A over B as a third panel (Fig. 4c).

The results support your observation. At 800 and 1200 interference steps, greedy access to A is zero, but its likelihood advantage over B drops from 2.0 to 1.0 nats/token. This initial advantage strongly predicts the subsequent recovery gap (Spearman ρ = 0.90 across 20 checkpoints).

We have therefore revised our interpretation: models with identical greedy accuracy can still differ in their remaining answer likelihood and recoverability. We also acknowledge that our prospective two-step probe does not reliably improve prediction beyond zero-step likelihood (Appendix J). We now present the two as complementary: likelihood measures the model's preference for the correct answer before restudy, while restoration savings measures what a fixed amount of retraining actually restores.


W4 & Q1: The jump in B and shared SQuAD articles

Thank you for pointing this out. We revisited Fig. 2b and found that the original figure did not accurately display the intermediate restudy budgets due to one seed. The corrected curve shows B improving gradually, from 0.07 at k = 8 to 0.15 at k = 16 and 0.32 at k = 32. The underlying numerical results were unaffected, and we have updated the figure.

Since Fig. 2b shows direct restudy, B is trained on its own questions. We checked which questions account for the improvement: at k = 16, access is 0.32 on already-restudied B questions but only 0.03 on unseen ones. Unseen questions remain near zero across these budgets (Appendix C), confirming that B is being learned from scratch.

You are also right that the original split was by question rather than article. To address this, we reran the experiment with entire SQuAD articles assigned to separate sets. The disjoint A−B gap remains 0.83 ± 0.05 at k = 8, positive in all seven seeds (Section 4.1; Appendix C, Table 6). Thus, the recovery advantage persists without shared source articles.


W5: Experimental clarity and error bars

We agree that the different experimental setups were difficult to follow. We have added a notation summary for A/B/C/D and the restudy conditions (Section 2.2), a consolidated experiment table (Table 3), and a clarification of the "A-only" branch in Fig. 1. We have also added uncertainty bands to Figures 2 and 3, and error bars and seed-level points to Fig. 4. The affected learning curves have been regenerated from the per-seed results.


Thank you again for the detailed feedback. Your questions helped us correct the figures, rule out article overlap as an explanation, and clarify the relationship between zero-step likelihood and restoration savings. The generative-interference experiments also helped us narrow our conclusions.

We believe the revised paper now makes a clearer claim: prior learning can make facts easier to recover even after behavioral access is lost, but the form of recovery depends on the interference setting. We hope these changes address your concerns and would appreciate your reconsideration of the assessment.

Thank you for the detailed response and the new experiments. And foremost, My main concerns remain, and the new results seem to confirm them:

  1. The DBpedia effect is confirmed to be output-format interference.

  2. Under severe generative interference, the recovery advantage of acquired over withheld facts is small. The direct gap is 0.06 at the budget k=8 and becomes the largest at 0.25, and disjoint restudy has no advantage (Table 18).

  3. Zero-step likelihood already predicts the k=8 gap (Spearman 0.90), so it is unclear what restoration savings provides additionally as a measure.

I appreciate the honest reporting, but the remaining contribution seems to me a randomized confirmation of known effects. Besides, I am not confident that all issues are brought up in the reviews. I feel the paper would benefit more from another round of reviews, therefore I am keeping my rating at 2.

Reviewer_cQkuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates lost access from lost knowledge after fine-tuning with a test called restoration savings. Questions the base model fails are randomly split into three sets; the model is taught two (A and C) and never sees the third (B), then fine-tuned on DBpedia classification until access to A drops from 0.94 to 0.04 (Sections 2.1 and 4.1). After eight restudy steps, which is four optimizer updates, A recovers to 0.85 while B stays at 0.06, a gap of 0.79 over seven seeds. The gap persists when only C is restudied, under a question wording unseen in training, under exact-match scoring and with full-parameter tuning. Equally forgotten models recover differently as interference grows.

Critical strengths
  • The method is simple and well controlled: shared post-interference checkpoint, fresh optimizer, same seeds and data order across arms.
  • Many controls are reported: exact-match scoring (Table 4), initial-likelihood matching, scrambled bindings, paraphrased restudy (Table 1), full-parameter tuning (Table 2 and Table 13), and a prompting-only baseline with best-of-16.
  • The headline budget was fixed before the later experiments, and all budgets are shown.
Critical weaknesses
  1. The conclusion is broader than the experiments. It is said the usual reading of a drop as erasure is wrong for language models, while severe forgetting appears only under DBpedia classification SFT. Summarization leaves access on A unchanged at 0.78, and on fresh seeds GSM8K access falls only from 0.86 to 0.70; neither task passed the severity gate (Appendix I). Some outputs become class labels such as Artist, Field, and Company, but those examples are from the mechanism set, not the SQuAD headline, and showing the answer format in context does not restore access, so the retention claim should be limited to this classification regime.

  2. The trace claim is only partly supported. Restudying never-learned D gives an A−B gap of 0.64, against 0.78 for restudying C, so much of the recovery is a return to factual answering. The likelihood advantage of acquired facts does not consistently grow during restudy, so the measure is a head start, not a faster learning rate. The authors say a positive gap does not show that an unchanged representation survives, while the abstract and L0475 say a recoverable trace of prior learning remains; false-answer and pseudo-word restorers give gaps of 0.11 and 0.06 against 0.55 for true answers, and the content-free restorer they say would settle the question was not run.

Questions for authors

N/A

3 comments

Thank you for the thoughtful and constructive review. We appreciate your recognition of the experimental design and restoration-savings framework.

We agree with your two main concerns regarding generalization beyond classification interference and the interpretation of the recovery gap. In response, we conducted additional experiments with severe generative interference, non-QA restorers, and counterbalanced restorer-history controls. These results have helped us narrow our claims and clarify the role of prior learning. We address each concern below.


W1: Generalization beyond classification fine-tuning

We agree that our original claim was too broad. The main difficulty we faced was finding an interference task that reliably causes severe forgetting without simply degrading the model. With summarization, access to A remained at 0.78 even after 1,600 steps. With GSM8K, it only dropped from 0.86 to 0.70, and with Alpaca instruction tuning at our original learning rate, it remained at 0.61. This is why we initially focused on DBpedia, where we could reliably drive access close to zero.

Following your feedback, we explored stronger generative interference settings (Appendix I). We found that increasing the learning rate for Alpaca instruction tuning reduces access from 0.91 to 0.003 across seven seeds, while the model still retains some instruction-following ability (token F1 = 0.25).

Importantly, the advantage from prior learning is still present in this setting, although it is weaker and takes longer to emerge. At k = 8, the A−B gap is only 0.06, but by k = 32 it reaches 0.25 ± 0.05, positive in all seven seeds. At the same time, we found an important difference from DBpedia: restudying only C no longer restores access to A (gap 0.02 ± 0.03 at k = 32).

This helped us clarify the scope of our findings. Previously learned facts can remain easier to relearn even after severe generative interference, but the rapid recovery from restudying other facts appears specific to the classification setting we studied.

We also followed up on your observation about output formats. You are right that the examples in the original paper came from the mechanism dataset rather than the headline SQuAD experiment. We checked the SQuAD outputs directly and found that, after DBpedia interference, 75% of answers are class labels. This falls to 6% after four restudy steps (Appendix D). Simply showing the model the expected answer format in context does not restore access, however, suggesting that the effect is not resolved by prompting alone.

We have updated the abstract, Section 4.5, conclusion, and Appendix A to make these distinctions clearer.

Continued in our second response below.

Continuation of our response.


W2: What does the recovery gap establish?

We agree that our original results did not fully separate the effect of prior learning from the return of general factual-answering behavior. Your observation that restudying never-learned facts (D) already restores much of the gap was particularly helpful. It motivated us to look more carefully at what kind of restudy is actually needed for recovery.

Following your suggestion, we tested restorers that do not contain the target facts, including plain text without any question-answer format (Appendix D, Table 9). The most striking result is that simply training on WikiText continuation restores A almost as well as restudying C: at k = 8, the A−B gap is 0.64 ± 0.09 with WikiText, compared with 0.67 ± 0.26 with C. Natural Questions also produces a substantial gap (0.48 ± 0.28), while training the constant answer "unknown" produces no gap.

These results support your interpretation that much of the recovery reflects a return to factual-answering behavior. They also help explain our earlier controls: false-answer and pseudo-word restorers produced much smaller gaps (0.11 and 0.06, compared with 0.55 for true answers). We initially could not tell whether this was because recovery required correct factual content or because these training objectives were less effective at restoring the model's behavior. The WikiText result shows that factual QA examples are not necessary.

At the same time, prior learning still makes a difference. In our randomized comparison, the same restudy updates restore previously learned facts A, while never-learned facts B remain largely inaccessible. We also ran a new counterbalanced experiment to test whether the restorer's own learning history matters (Appendix D, Table 10). Restudying previously acquired facts produces an additional A−B gap of 0.19 ± 0.08 over restudying never-learned facts, averaged across two prompt formats, positive in all five seeds. So while general behavioral restoration explains much of the effect, prior acquisition still contributes to which facts recover and, to a smaller extent, which restudy material helps most.

We also agree with your distinction between a head start and a faster learning rate. Before restudy, the model already assigns higher likelihood to acquired answers than withheld ones (a 3.71 nats/token advantage), and this difference does not consistently increase during restudy (Appendix C). Across interference depths, the initial likelihood advantage strongly predicts subsequent recovery (Spearman ρ = 0.90; Section 4.4). We now describe the effect as a head start in recoverability, rather than faster learning.

Finally, we agree that restoration savings does not establish that an unchanged knowledge representation survives interference. We have removed the "recoverable trace" wording from the abstract and clarified this limitation in Section 2.1 and Appendix A. Our claim is now narrower: even when a model can no longer produce previously learned answers, its learning history can still make those answers easier to recover. We leave the mechanism behind this advantage open.


Presentation

We also appreciate your feedback on the presentation. To make the experimental setup and restudy conditions easier to follow, we have added:

  • Notation summary: Explains A/B/C/D, direct versus disjoint restudy, and the restudy budget k (Section 2.2).
  • Consolidated experiment table: Summarizes experiments, setups, and numbers of seeds (Table 3).
  • Uncertainty and seed-level results: Adds uncertainty bands and seed-level information to Figures 2 and 3, with further details in the appendix.

We hope these changes make the experimental comparisons and main results easier to follow.


Thank you again for the constructive feedback. The new experiments clarify that the advantage from prior learning extends beyond classification interference, while recovery through disjoint restudy appears specific to the DBpedia setting. They also show that general behavioral restoration explains much of the recovery, although prior acquisition still matters for which facts become accessible again.

We have revised the paper to reflect these distinctions and avoid claims about unchanged knowledge representations. We hope these additions address your concerns.

Thanks for the new experiments and the softened "trace" language, which address my concerns well. Two points from other reviewers remain open: the unexplained rise of withheld set B in Figure 2b, possibly indicating SQuAD article-level leakage, and the adapter-scaling result in Table 6, where zero retraining already recovers half of A's accuracy, suggesting partly reversible format drift rather than genuine forgetting. I maintain my positive score for now.

Comments on the paper

1 comment

Dear Reviewers,

Please note that the authors have posted their responses. Please, read them carefully and comment whether the responses have resolved your concerns. When appropriate, please adjust your initial score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.