- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies spurious forgetting in language models, where learned facts become temporarily inaccessible during fine-tuning but can later be recovered without replay. The authors identify a three-stage trajectory (collapse, recovery, erosion) and explain it with a minimal associative memory model built on three ingredients: shared key structure, concentrated target values, and normalization. Fine-tuning applies a common shift that suppresses recall while preserving relative geometry; normalization later reverses this shift, while fact-specific changes accumulate and impair retrieval. The account is validated on a minimal memory model, synthetic-biography Transformers, and pretrained OLMo 2 1B.
Critical strengths
S1. An insightful mechanistic explanation of spurious forgetting.
The paper explains why forgetting and recovery occur, not merely that they do. The decomposition into a shared shift and fact-specific changes cleanly separates knowledge storage from accessibility, and the minimal model yields concrete emergence conditions and testable predictions. This, rather than any mitigation method, is the paper's main contribution.
S2. Convincing evidence across multiple abstraction levels.
The paper goes beyond correlation with causal interventions: removing the common logit shift largely eliminates the initial collapse in the controlled Transformer (Figure 4), and removing the leading singular component of the updates substantially restores old-fact accuracy in OLMo 2 1B (Figure 5, Appendix D). Comparisons at matched new-fact accuracy show the intervention does more than rewind fine-tuning.
Critical weaknesses
W1. Irreversible erosion vs. reversible interference is not fully established.
The decreasing effectiveness of removing the common shift over time does not by itself show that residual knowledge is irreversibly erased — fact-specific changes may cause recoverable interference requiring a more expressive intervention. The reversible component is convincingly established, but weaker evidence is given that the residual is permanent knowledge loss.
W2. Generality and causal specificity in pretrained LMs are uncertain.
The OLMo intervention removes the leading singular component from 112 weight matrices, changing much of the update; a random rank-one control and matched-accuracy comparisons help, but it is unclear how much of the gain specifically comes from removing a shared shift. The synthetic-vs-real recovery difference is attributed to subject–answer predictability, yet the settings differ in several other ways. Experiments also cover only one model size and use first-answer-token accuracy.
Questions for authors
Q1. (W1) Is the residual degradation genuine knowledge erosion or a more complex reversible access failure? Would a more expressive intervention or a controlled relearning experiment change this?
Q2. (W2) How does the OLMo intervention compare to an update-shrinkage baseline matched on update-norm reduction? Can subject–answer predictability be varied while holding other properties of the new facts fixed?
3 comments
Follow-up on our response
Dear Reviewer f4bQ,
Thank you again for your positive assessment and for the experiments you suggested, which we have added in the revised version. We wanted to check whether our response clarified your questions, since the discussion period is coming to an end.
We would be happy to answer anything that remains open, and we hope the response and the new experiments are helpful for your final assessment.
Follow-up on our response to Reviewer f4bQ
Dear Reviewer f4bQ,
Today is the last day of the discussion period, so we wanted to check one final time whether any question remains that we can clarify.
Thank you again for your review and your suggestions.
Official Comment by Authors
We thank you for the review, and your positive feedback and the experiments proposed. We hope the evidence below, detailed in our general response and in Appendix E of the revised PDF, clarifies your doubts, and we would be grateful if you would consider updating your score.
Q1. Residual degradation
Our paper mainly studies the reversible part of forgetting, the collapse and the recovery, where the old facts are hidden and not lost, cf. point 1 of our general response. For the residual degradation, we use "catastrophic" in the behavioural sense, for recall that further finetuning without replay does not bring back, as the abstract and Section 2 now state, and we do not claim that it is a permanent loss of knowledge. In the controlled Transformer, about a fifth of the old facts is brought back neither by removing the common shift nor by replaying other old facts late in finetuning (Appendix E.1), and this is the part we call erosion. Relearning these facts is likely to be easier than learning them the first time, as earlier work on savings observed (Hetherington & Seidenberg, 1989; Hinton & Plaut, 1987), and our paper does not study that question.
Q2. Shrinkage baseline and predictability
Update shrinkage matched on norm. Our edit recovers more old facts than shrinkage, as detailed in point 2 of our general response. Shrinking the whole update to the norm of the edited update recalls 59% of the old facts on real entities, against 79% for our edit. Shrinkage loses no new facts, so we also compare it with a weaker edit that removes only a quarter of the top direction and keeps the new facts about as well, and this one still recalls 65%. The shift shared by the old facts of a relation accounts for a fifth to two fifths of the edit's gain. We added this baseline and this analysis in Appendix E.2.
Predictability with the other properties fixed. Predictability can be varied alone, and doing so reproduces the difference the paper describes, as detailed in point 3 of our general response. With the relation, templates, answers and answer frequencies held fixed, we gave invented names a surname shared by all products of one manufacturer, so that the name hints at the answer. These names do not recover, whereas invented names that say nothing about their answer recover at each of three learning rates. This supports the explanation of the paper for real entities, and the experiment is new in Appendix E.3.
Model size (W2)
We repeated the finetuning experiment on synthetic individuals of Section 5 in OLMo 2 7B, a larger model of the same family, with the same protocol and the same learning rate. The collapse and the recovery are not specific to one model size, as described in point 5 of our general response and in Appendix E.5.