acceptodds
Under review as a conference paper at ICLR 2027

When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting

Abstract

Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
language modeling, interpretability, learning dynamics, factual recall

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_f4bQOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies spurious forgetting in language models, where learned facts become temporarily inaccessible during fine-tuning but can later be recovered without replay. The authors identify a three-stage trajectory (collapse, recovery, erosion) and explain it with a minimal associative memory model built on three ingredients: shared key structure, concentrated target values, and normalization. Fine-tuning applies a common shift that suppresses recall while preserving relative geometry; normalization later reverses this shift, while fact-specific changes accumulate and impair retrieval. The account is validated on a minimal memory model, synthetic-biography Transformers, and pretrained OLMo 2 1B.

Critical strengths

S1. An insightful mechanistic explanation of spurious forgetting.

The paper explains why forgetting and recovery occur, not merely that they do. The decomposition into a shared shift and fact-specific changes cleanly separates knowledge storage from accessibility, and the minimal model yields concrete emergence conditions and testable predictions. This, rather than any mitigation method, is the paper's main contribution.

S2. Convincing evidence across multiple abstraction levels.

The paper goes beyond correlation with causal interventions: removing the common logit shift largely eliminates the initial collapse in the controlled Transformer (Figure 4), and removing the leading singular component of the updates substantially restores old-fact accuracy in OLMo 2 1B (Figure 5, Appendix D). Comparisons at matched new-fact accuracy show the intervention does more than rewind fine-tuning.

Critical weaknesses

W1. Irreversible erosion vs. reversible interference is not fully established.

The decreasing effectiveness of removing the common shift over time does not by itself show that residual knowledge is irreversibly erased — fact-specific changes may cause recoverable interference requiring a more expressive intervention. The reversible component is convincingly established, but weaker evidence is given that the residual is permanent knowledge loss.

W2. Generality and causal specificity in pretrained LMs are uncertain.

The OLMo intervention removes the leading singular component from 112 weight matrices, changing much of the update; a random rank-one control and matched-accuracy comparisons help, but it is unclear how much of the gain specifically comes from removing a shared shift. The synthetic-vs-real recovery difference is attributed to subject–answer predictability, yet the settings differ in several other ways. Experiments also cover only one model size and use first-answer-token accuracy.

Questions for authors

Q1. (W1) Is the residual degradation genuine knowledge erosion or a more complex reversible access failure? Would a more expressive intervention or a controlled relearning experiment change this?

Q2. (W2) How does the OLMo intervention compare to an update-shrinkage baseline matched on update-norm reduction? Can subject–answer predictability be varied while holding other properties of the new facts fixed?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank you for the review, and your positive feedback and the experiments proposed. We hope the evidence below, detailed in our general response and in Appendix E of the revised PDF, clarifies your doubts, and we would be grateful if you would consider updating your score.

Q1. Residual degradation

Our paper mainly studies the reversible part of forgetting, the collapse and the recovery, where the old facts are hidden and not lost, cf. point 1 of our general response. For the residual degradation, we use "catastrophic" in the behavioural sense, for recall that further finetuning without replay does not bring back, as the abstract and Section 2 now state, and we do not claim that it is a permanent loss of knowledge. In the controlled Transformer, about a fifth of the old facts is brought back neither by removing the common shift nor by replaying other old facts late in finetuning (Appendix E.1), and this is the part we call erosion. Relearning these facts is likely to be easier than learning them the first time, as earlier work on savings observed (Hetherington & Seidenberg, 1989; Hinton & Plaut, 1987), and our paper does not study that question.

Q2. Shrinkage baseline and predictability

Update shrinkage matched on norm. Our edit recovers more old facts than shrinkage, as detailed in point 2 of our general response. Shrinking the whole update to the norm of the edited update recalls 59% of the old facts on real entities, against 79% for our edit. Shrinkage loses no new facts, so we also compare it with a weaker edit that removes only a quarter of the top direction and keeps the new facts about as well, and this one still recalls 65%. The shift shared by the old facts of a relation accounts for a fifth to two fifths of the edit's gain. We added this baseline and this analysis in Appendix E.2.

Predictability with the other properties fixed. Predictability can be varied alone, and doing so reproduces the difference the paper describes, as detailed in point 3 of our general response. With the relation, templates, answers and answer frequencies held fixed, we gave invented names a surname shared by all products of one manufacturer, so that the name hints at the answer. These names do not recover, whereas invented names that say nothing about their answer recover at each of three learning rates. This supports the explanation of the paper for real entities, and the experiment is new in Appendix E.3.

Model size (W2)

We repeated the finetuning experiment on synthetic individuals of Section 5 in OLMo 2 7B, a larger model of the same family, with the same protocol and the same learning rate. The collapse and the recovery are not specific to one model size, as described in point 5 of our general response and in Appendix E.5.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer f4bQ,

Thank you again for your positive assessment and for the experiments you suggested, which we have added in the revised version. We wanted to check whether our response clarified your questions, since the discussion period is coming to an end.

We would be happy to answer anything that remains open, and we hope the response and the new experiments are helpful for your final assessment.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer f4bQ

Dear Reviewer f4bQ,

Today is the last day of the discussion period, so we wanted to check one final time whether any question remains that we can clarify.

Thank you again for your review and your suggestions.

Reviewer_QHV9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mechanism of spurious forgetting. Through analysis of a two-layer network, the authors identify three ingredients that reproduce spurious forgetting: shared structure among input keys, new answers concentrated in one region of the output vocabulary, and a normalization layer. The authors further investigate the proposed mechanism in a Transformer (OLMo 2 1B) trained on synthetic data and observe that a similar forgetting phenomenon occurs.

Critical strengths
  1. The research question is clear and well-motivated. Understanding why spurious forgetting happens is an interesting question.
  2. The controlled experiment in Section 3 is well designed. It clearly identifies some key components that produce the spurious forgetting phenomenon.
Critical weaknesses
  1. I find the main paper insufficiently self-contained, which makes it difficult to follow. For example:
  • The assumptions in Section 3.2 are not clearly introduced, nor is the motivation for using gradient flow to analyze the dynamics.
  • The definition of "region of the output space" in Section 4.1 is not clearly defined in the main text.
  1. The paper identifies three conditions for spurious forgetting in Section 3. However, it is unclear why this is the mechanism that causes spurious forgetting. Other mechanisms may also produce spurious forgetting. The paper does not extract actionable insights from them that could help prevent forgetting. Therefore, I think the contribution is not significant in its current form.
Questions for authors

My questions are listed in the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank you for your review and your helpful critique. We hope the clarifications and the new evidence below, detailed in our general response and in the revised PDF, address your concerns, and we would be grateful if you would consider updating your score.

1. Self-containedness of the main text

We thank you for pointing this out, and the main text now introduces the assumptions of Section 3.2 and defines a region of the output space where the term first appears.

The assumptions of Section 3.2, standard in the study of learning dynamics, are the following:

  • Gradient flow. Gradient flow follows training in continuous time, as the limit of gradient descent with small steps. Such an approximation considerably simplifies the analysis, and is widely used to describe the learning dynamics of networks trained with gradient descent as in e.g. Saxe et al. (2014, 2019), Arora, Cohen & Hazan (2018), and Nichani et al. (2025).

  • High-dimensional keys. This assumption enables us to consider the dot products between different random inputs to be equal to their expected value, so that the parts of the keys that are specific to each fact are nearly orthogonal to one another. It is a standard assumption in the associative memory works we draw our inspiration from, such as Bietti et al. (2023) and Cabannes, Dohmatob & Bietti (2024). The transformers themselves are trained with AdamW on minibatches, and the behaviors our analysis predicts remain there.

A region of the output space is one half of the possible answers. The answers of each attribute are split at random into two halves, the old facts that we measure take their answers from one half, and all the new facts take theirs from the other. We introduced this split in Section 3.1 but used "halves", "subsets" and "regions" for it, and the revised text defines the term in Sections 3.1 and 4.1.

2. Why this mechanism

The mechanism is both sufficient and necessary for spurious forgetting in the settings we study.

  • Sufficient. A two-layer model with only the three ingredients reproduces the collapse, the recovery and the later decline, as shown in Section 3.
  • Necessary. Removing any one ingredient breaks this trajectory, and subtracting the shift common to the old facts removes the collapse in the trained Transformer, as shown in Sections 3 and 4.
  • Predictive. A shared shift should be reversible through a few facts, and replaying 250 old individuals indeed restores recall of the other 1,750 from 12% to 97%, in a new experiment described in point 1 of our general response.

Our explanation accounts for everything we observe, in theory and in experiments, and the collapse and the recovery disappear as soon as any one of the three ingredients is removed. We also tested two alternative explanations directly, and neither accounts for the collapse. The shift is carried mostly by the weights that write content into the model, with little contribution from those that decide where attention looks, so a change in attention routing does not explain it. A fixed training schedule does not explain it either, since the collapse ends at the same stage of learning the new facts whatever the learning rate or batch size, and not after a fixed number of steps. Both analyses are in Appendix C.

3. Actionable insights

The mechanism says that the old facts are hidden behind a shift they all share, which gives two actionable insights for preventing forgetting.

A small amount of replay is enough to keep the old facts accessible, since pushing the shared shift back through a few facts acts on all of them. Replaying 250 of the 2,000 old individuals throughout finetuning, about 3% of the data, leaves 72% of the other old facts recalled at the end, against 29% without replay. The same replay also brings back most of the never-replayed old facts in the pretrained model, OLMo 2.

The replayed data do not need to be the old data. New individuals whose answers lie in the same region as the old facts keep them accessible just as well, which matters when the original data are no longer available.

We added these experiments to Appendices E.1 and E.6 of the revised PDF. The phenomenon is not tied to one model size either, since the collapse and its recovery reappear in the larger OLMo 2 7B, as described in point 5 of our general response.

We thank you for this nudge and have additionally added a paragraph on practical consequences to the discussion section.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer QHV9,

We submitted our responses a few days ago but have not yet heard back from you. As the discussion period is coming to an end, we would appreciate it if you could revisit your score if our responses have addressed your concerns.

If you have any further questions, please feel free to let us know. We look forward to hearing your thoughts.

Thank you for the discussion, clarifications, and new content. My concerns have been addressed, and I have adjusted my score accordingly.

AuthorsOpenReview ↗

Response to Reviewer QHV9

Thank you for taking the time to read our response and for updating your score. Your suggestions made the paper stronger, and we appreciate your feedback.

Reviewer_hsPSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies factual recall that collapses, partially recovers, and then deteriorates during finetuning without replay. A two-layer associative memory explains this behavior through shared keys, concentrated new answers, and normalization. Shared updates initially suppress access to old facts. Normalization subsequently reverses this shift as the model learns individual new associations. The authors test this explanation using synthetic-biography transformers and OLMo 2 1B. Removing common logit changes restores recall in the controlled transformers. Removing leading singular components from weight updates improves OLMo retention, with reduced accuracy on new facts.

Critical strengths

The paper offers a concrete explanation for spontaneous recovery. The minimal model connects data structure, gradient dynamics, and normalization to the observed trajectory. Ablations remove shared key structure, answer concentration, or normalization and change the trajectory as predicted. This provides useful understanding beyond the established distinction between task alignment and retained knowledge. That distinction already appears in Zheng et al. and Kotha et al..

Several controlled experiments support the proposed mechanism. At the collapse trough, overall accuracy is 12%, while 86% of answers retain first place within their original answer region (Appendix C.1). With MLP blocks, subtracting the common logit change raises trough accuracy from 53% to 99% (Appendix C.4). Recovery also begins at similar new-fact accuracy across different learning rates and batch sizes (Appendix C.2). These results connect the explanation to measurable predictions and interventions.

Critical weaknesses

The OLMo intervention does not uniquely establish the proposed mechanism. Appendix D removes one singular component from each of 112 update matrices. The components depend on the difference between current and pretrained weights. Recovery through this operation therefore does not, by itself, establish that the edited associations remained accessible within the finetuned model. The operation can restore information from the pretrained checkpoint. The tradeoff also matters. For real entities at step 400, old-fact accuracy rises from 55% to 79%, while new-fact accuracy falls from 100% to 66%. The matched-accuracy comparison strengthens the result: old-fact accuracy improves from 72% to 79% at 66% new-fact accuracy. However, comparing against earlier checkpoints does not exclude other forms of partial parameter restoration. The random control is also incomplete. Subtracting an independent random matrix does not match the reduction in distance from pretrained weights caused by singular-component removal. These gaps limit the claim that this intervention specifically removes a shared access bias.

The evidence supports a narrower claim than preservation or erasure of complete facts. Both transformer settings measure only the first answer token (Appendix A). Different answers can share that token. The pretrained-model experiments additionally enforce disjoint first tokens between old and new answers. These choices isolate output competition effectively, but complete-answer evaluation remains necessary to establish factual recovery. Similarly, residual errors after common-shift removal do not establish irreversible erasure. The evidence supports slower deterioration that this particular correction does not recover. Calling all such deterioration catastrophic requires additional evidence or narrower wording.

Questions for authors
  1. Mechanism and control: Do the removed OLMo components measurably account for a common representation or logit shift? Does their retention advantage persist against interpolation toward pretrained weights at matched new-fact accuracy?
  2. Causal explanation: Can you isolate subject–answer predictability by permuting manufacturers among the same real entities while preserving answer frequencies? The current synthetic-versus-real comparison also changes relations, answer diversity, and dataset structure.
  3. Evaluation: Do collapse, recovery, and intervention gains persist under complete-answer exact match? How frequently do distinct answers share their first token?

\textbf{Minor mathematical correction}

Equation~(31) appears to double-count the contribution of fact bb. Under the stated leading-order key overlaps,

h˙b≃−1nBδb−αnB∑b′≠bδb′=−αδˉB−1−αnBδb,\dot{h}_b \simeq -\frac{1}{n_B}\delta_b -\frac{\alpha}{n_B}\sum_{b'\neq b}\delta_{b'} = -\alpha\bar{\delta}_B -\frac{1-\alpha}{n_B}\delta_b,

where δˉB=1nB∑b′δb′\bar{\delta}_B = \frac{1}{n_B}\sum_{b'}\delta_{b'} already includes δb\delta_b. This correction preserves the claimed 1/nB1/n_B scaling and does not overturn the proposed mechanism.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank you for your review, the helpful feedback and the controls you proposed. We hope the evidence below, detailed in our general response and in Appendix E of the revised PDF, clarifies your questions, and we would be grateful if you would consider updating your score.

1. Mechanism and control

The gain of the edit is not explained by a return to the pretrained weights, and the removed components carry a common shift that accounts for part of this gain, as detailed in point 2 of our general response. The edit removes the largest direction of the change ΔW\Delta W that finetuning made to each pretrained weight matrix W0W_0. We added the comparisons and the common-shift analysis asked for in Appendix E.2.

  • Interpolation toward the pretrained weights at matched new-fact accuracy. Interpolation moves all weights part of the way back, W0+λΔWW_0 + \lambda\Delta W with λ<1\lambda < 1. We choose λ\lambda so that it loses as many new facts as the edit, and on real entities it then recalls 73% of the old facts where the edit recalls 79%. The advantage of the edit therefore persists there. On synthetic individuals the two recall the same share of the old facts.
  • Control matched on distance from the pretrained weights. Removing other directions of the same size ends at the same distance and brings nothing back on real entities, with 52% of the old facts recalled against 55% with no edit.
  • Common shift. The average change that the edit makes to the old facts points along the one caused by finetuning, with a cosine similarity of .88 to .96 for the logits and .89 to .98 for the final hidden states, and the part of it shared by the old facts of a relation accounts for a fifth to two fifths of the edit's gain, as shown in Appendix E.2.
  • A test using nothing from the pretrained weights. In a new experiment, we replay sentences that state one eighth of the old facts of OLMo 2 at the bottom of the collapse. Recall of the never-replayed facts goes from 29% to 87% within forty steps, where the model without replay reaches only 45% by itself over the same steps. The facts that share neither their answer nor a word of their subject with a replayed fact come back too, from 19% to 76%, so what the model answers about them was still stored. The same test restores recall from 12% to 97% in the controlled Transformer. Both results are described in point 1 of our general response and in Appendices E.1 and E.6.

2. Causal explanation

The difference between real entities and synthetic individuals comes from their subjects, and not from relations, answer diversity or dataset structure, as described in point 3 of our general response. We ran the permutation you proposed, with answer frequencies preserved. Real products with permuted manufacturers do not recover, whereas invented names given the same answers do, so the relation, the answers and the structure of the data cannot explain the difference. We also gave invented names a surname shared by all products of one manufacturer, so that the name hints at the answer, and recall of the old facts then no longer recovers. This supports the explanation of the paper, in which the old facts stay hidden when the subject hints at its answer. We added both experiments in Appendix E.3.

3. Evaluation

The collapse, the recovery and the gain of the edit persist under complete-answer exact match, which stays within 3 points of first-token accuracy at every evaluated step in OLMo 2, as described in point 4 of our general response. This is because distinct answers rarely share a first token in this setup. No two old answers share one, 14 of the 264 synthetic answers share one with another new answer, and none of the 407 real-entity answers do. We added the details in Appendix E.4.

Wording of the later decline

The mechanism we study is that of the reversible part of forgetting, the collapse and the recovery. For the later decline, we use "catastrophic" in the behavioural sense in which the term was introduced (McCloskey & Cohen, 1989), for recall that further finetuning without replay does not bring back, as the abstract and Section 2 now state, and we do not claim that these facts are totally erased from the weights.

Equation 31

We thank you for this correction, and Equation 31 is corrected in the revised PDF. The fact-specific term keeps its 1/nB1/n_B scaling, and the conclusion is unchanged.

Reviewer_hsPSOpenReview ↗

Follow-up on the author response

Thank you for the detailed response and additional experiments. The reported interpolation and distance-matched controls substantially address my concern about partial restoration of pretrained parameters. The complete-answer evaluation also addresses my main evaluation concern.

The replay results provide particularly useful evidence that information survives the collapse. Recovery of facts sharing neither answers nor subject words with replayed examples strengthens this conclusion.

One point remains unclear about the subject–answer predictability explanation. If manufacturer labels were independently shuffled across real products, this should weaken the original subject–answer relationship. However, recovery still does not occur. Could you clarify the permutation procedure and explain why the proposed mechanism predicts this result? The surname experiment supports an effect of answer-predictive naming, but the explanation for real entities remains less clear.

The common-shift analysis also strengthens the mechanistic evidence. However, the reported shared component accounts for only 20–40% of the edit’s gain. Please ensure that the claims about the OLMo intervention reflect this partial contribution.

I appreciate the clarification that the later decline does not establish permanent knowledge loss. Please apply this distinction consistently, including replacing “erodes for good” if that phrase remains in the abstract.

Overall, the response substantially strengthens the paper. I maintain my weak acceptance recommendation. My remaining concerns primarily concern the interpretation and scope of the claims.

AuthorsOpenReview ↗

Response to the follow-up of Reviewer hsPS

We thank you for the follow-up, and we are glad the new experiments addressed your main concerns. We provide the required additional clarifications below.

Permutation. We keep the real products, prompts and templates and randomly reassign the manufacturers among the products, so that every manufacturer keeps its number of products and no product keeps its own. A new permutation is drawn for each seed, and invented names receive the same permuted answers. The two sets then share the relation, the templates, the answers and their frequencies, and differ only in their subjects. Invented names recover and real products do not, so the lack of recovery comes from the subjects, as stated in point 3 of our general response. The permutation is thus a test of where the difference comes from, and the test of the mechanism itself is the second experiment in Appendix E.3, where invented names whose surname predicts the answer no longer recover. We state it in Appendix E.3, that predictability matters among other associations of the subject. Which of these other associations are involved is the question this experiment opens, and we would consider it as future work.

Common shift. Appendix E.2 already states this partial contribution and we will state the same in Section 5.2.

"Erodes for good". We have removed this phrase. The abstract now reads "only then erodes, without recovering on its own".

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the three reviewers for their careful and constructive reviews. Reviewer QHV9 found the research question clear and well motivated, and the controlled experiment of Section 3 well designed. Reviewers hsPS and f4bQ found that the paper explains why forgetting and recovery occur, and not merely that they do, with causal evidence from a minimal model up to a pretrained language model. Reviewer f4bQ highlighted that this explanation, "rather than any mitigation method, is the paper's main contribution".

Yet, the reviewers rightly identified a few blind spots in our argument. We ran additional experiments to address them, which form the new Appendix E of the revised PDF, where changes to the main text are marked in red, and we summarize the main ones below. These experiments cover all the blind spots the reviewers identified, and their suggestions made our work stronger. We hope that the reviewers will reconsider their scores in light of this new evidence.

1. Are the old facts hidden or lost? (hsPS, f4bQ; QHV9 on what the mechanism implies)

We thank the reviewers for this question, and ran some additional experiments to directly test this. If one shift hides all old facts, pushing them back by replaying a few of them should restore recall. In the controlled Transformer, we replayed 250 of the 2,000 old individuals at the bottom of the collapse, and recall on the 1,750 left out goes from 12% to 97% within 20 steps. No information about the 1,750 was given, yet recall performance was almost fully restored; old facts are therefore hidden and not lost.

We ran the same test in the pretrained model. Replaying sentences that state one eighth of the old facts of OLMo 2 at the bottom of the collapse brings recall of the never-replayed facts from 29% to 87% within forty steps, where the model without replay reaches only 45% by itself over the same steps.

These results are consistent with our theoretical model, and we added them in Appendices E.1 and E.6.

2. Is the weight edit in the pretrained model a return to the pretrained weights? (hsPS, f4bQ)

No, the gain of the edit is not explained by undoing finetuning. We ran three new ablations on real entities, where the edit recalls 79% of the old facts against 55% without it:

  • Undoing the same amount of finetuning in other directions brings nothing back (52%).
  • Undoing it uniformly, until as many new facts are lost as with the edit, brings back fewer old facts (73%).
  • Shrinking the whole update to the same norm brings back far fewer (59%).

The details are in Appendix E.2.

3. Is the difference between real entities and invented individuals due to predictability? (hsPS, f4bQ)

The difference comes from the subjects, and it reappears when the name of an invented subject hints at its answer. We ran both experiments the reviewers asked for, with the relation, templates, answers and answer frequencies held fixed:

  • Permuting manufacturers among the real products (hsPS): recall of the old facts does not recover, whereas it recovers by 19 points when invented names receive the same answers.
  • Varying predictability alone (f4bQ): we gave the invented products of each manufacturer a shared surname, so that the name hints at the answer, as the name of a real product hints at its maker. Recall of the old facts then no longer recovers.

This is the explanation our paper gives for real entities, and we therefore find that it holds. We added both experiments in Appendix E.3.

4. First-token evaluation (hsPS, f4bQ)

Our results do not depend on scoring only the first token of an answer. When the complete answer must be correct, accuracy in OLMo 2 1B changes by less than 3 points at every evaluated step, and the collapse, the recovery and the gain of the edit are the same. We added this comparison in Appendix E.4.

5. A larger model size (f4bQ)

We repeated the finetuning experiment on synthetic individuals in OLMo 2 7B, a larger model of the same family, with the same protocol and the same learning rate. The collapse and the recovery reappear, with recall of the old facts falling from 87% to 19% before the new facts are learned and rising to 40% as they start to be learned. Spurious forgetting is not specific to one model size, and we added this experiment in Appendix E.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.