acceptodds
Under review as a conference paper at ICLR 2027

What Remains after Factual Forgetting? History-Conditioned Access

Abstract

We show that factual learning leaves a residual parameter state that shapes subsequent learning even after the learned answers are no longer expressed. Across twelve Pythia-410M training runs, we randomize which facts receive training before interference, then test updates that train only on other facts. Prior exposure increases the update-induced change in exact-answer accuracy by 5.03 percentage points over the full fact panel. Among current failures, previously acquired facts recover more often than never-exposed facts, including within-fact and margin-deficit-adjusted comparisons; history-dependent recovery and faster relearning recur in Pythia-1.4B and Qwen2.5-0.5B. A correct answer may start closer to overtaking its competitors, or an update may push it further toward that threshold. Starting gaps explain much of the recovery ordering under random parameter displacement, but factual updates also depend on learned associations: label correctness changes the recovery gap more for previously trained support facts than for fresh ones. We measure this sensitivity before training by comparing the gradient directions that raise the answer score and lower support loss. Exposure changes this pre-update QKV tangent geometry, improving prediction of the cue-by-label margin effect after ten real training steps at two Pythia scales. To test the influence of historical parameters directly, we transplant blocks from donors with different query–answer assignments into a common recipient. Across six recipient–donor triplets, answer preferences follow the donor's assignments, including ones it no longer expresses, and respond differently to later training. A separate predictor links geometry to transplantation: fitted only on exposure histories, it partly predicts how replacing blocks changes an update's effect on answer scores in three new triplets, reducing error by 9.69% relative to assuming that replacement leaves this response unchanged. Residual factual memory thus concerns both present accessibility and how past learning conditions future updates.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
catastrophic forgetting, continual learning, factual knowledge, language models, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_BMPROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how prior factual learning influences subsequent updates after the model no longer produces the learned answers. It randomizes factual exposure across training histories, applies interference, and measures both recovery under target-disjoint updates and savings under direct relearning.

In the primary twelve-chain Pythia-410M experiment, prior exposure increases update-induced exact-answer change by 5.03 percentage points over the full randomized panel. Among current failures, acquired-then-lost facts recover more frequently than never-exposed facts, although this comparison concerns realized histories rather than randomized acquisition. Related behavioral patterns recur in Pythia-1.4B and Qwen2.5-0.5B.

The paper further measures pre-update target–support gradient alignment, evaluates frozen predictors of finite update responses, and transplants native blocks between models trained on different query–answer assignments. Together, the experiments support persistent, history-dependent parameter influence on answer preferences and subsequent learning. They provide more limited evidence about the specific mechanism mediating complete-answer recovery.

Critical strengths
  1. The design identifies an effect of exposure while treating realized learning histories carefully. The primary experiment randomizes exposure across twelve training chains and reports a full-panel effect of 0.0503 on update-induced exact-answer change, with a 95% interval of [0.0258, 0.0749]. It separately reports recovery of 78/312 acquired-then-lost failures versus 2/1,145 never-exposed failures, without presenting this post-treatment comparison as a randomized acquisition effect. Within-fact comparisons, deficit adjustment, random-displacement controls, and relearning tests provide complementary evidence. This separation of estimands makes the central behavioral conclusion substantially more credible.

  2. The paper connects behavioral history effects to prospective measurements and controlled parameter interventions. Geometry-based prediction is evaluated on 27 fresh Pythia-410M histories and 12 untouched Pythia-1.4B histories using discovery-fitted predictors. The transplantation experiments hold the recipient complement fixed and vary donor bindings while preserving answer pools; cyclic contrasts rule out a restricted query-independent answer-shift explanation. These experiments establish history-sensitive parameter influence beyond a recovery curve alone. The explicit reporting of failed criteria, weak transfer endpoints, collateral changes, and audit costs further strengthens the evidence.

Critical weaknesses
  1. The connection between the measured geometry and complete factual recovery remains incomplete. The strongest geometry results concern continuous response targets, while native-block transplantation primarily shifts candidate-answer preferences and rarely restores complete donor-assigned answers. In Appendix H.3, the combined predictor reduces transplant-response MSE by 9.69% relative to zero effect, but the incremental contribution of angular alignment beyond scores and gradient norms is 3.36% pooled and negative in one of three confirmation triplets. The same predictor performs worse than zero on the secondary four-arm interaction target in Table 11. These results support a useful predictive descriptor of some update responses, but do not yet identify a common mechanism mediating exact recovery. The paper acknowledges much of this boundary; it remains the main limitation on the strength of its unified mechanistic account.

  2. The more specific interpretation of support selectivity is less robust than the general history effect. In the fresh-target panel of Appendix E, old-correct and fresh-correct support restore essentially the same fraction of exposed failures, so the prespecified two-part recovery criterion fails. The old-versus-fresh advantage also does not replicate in the 1.4B behavioral panel. For pretrained-known facts, recovery-level cue-by-label interactions vary across prompt forms, including a negative interaction for one alternate prompt. These findings do not negate history-conditioned response, but limit an interpretation based on a consistently privileged recovery route through previously learned cues. The distinction matters because, at 410M, the geometry interaction is concentrated in sensitivity to conflicting old-cue updates rather than an increased alignment with correct old-cue updates.

Questions for authors
  1. Geometry versus exact recovery: Do any existing held-out analyses show that the pre-update geometry adds predictive information about binary exact-answer recovery among current failures, beyond current scores and gradient norms, without observing an actual update step? The early-response predictor uses three observed steps, while the main pre-update predictors target continuous responses. If this link has not been established, please distinguish that boundary explicitly from the confirmed prediction results.

  2. Interpretation of support selectivity: How should the proposed account be interpreted jointly with the failed old-versus-fresh recovery comparison in Appendix E, the corresponding 1.4B result, and the concentration of the 410M geometry effect in old-cue wrong-label sensitivity? Do the results primarily identify sensitivity to reinforcement/conflict with historical associations, rather than a generally privileged reactivation route through old correct cues? Please make the distinction explicit in the main interpretation.

I would be happy to raise my score if some of the concerns are addressed.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer BMPR — answers (1/2)

Thank you for distinguishing a change in an answer’s score from recovery of the whole answer, and for suggesting how reinforcement and conflict might explain the cue results. For Q1, predicting which answers would return in full after ten steps of cue training showed no clear gain from adding local derivatives to the starting scores, gradient norms and other baseline features: log loss fell in six of eight runs, but the interval for the mean gain included zero (revised App. M.1, p. 44, lines 2322–2328). Original Fig. 3 established a gain in predicting how much more switching cue labels changed a target’s score margin on old cues than on fresh ones (original p. 6, lines 318–323, revised p. 7, lines 369–374).

However, we could use the cue gradient to choose where to measure the answer scores, and those readings improved our predictions of the same ten-step outcome. Along the direction of the first Adam step, the starting slopes tracked changes in token margins closely over a short displacement. Further along that direction, linear extrapolation missed 15 of the 49 recoveries observed at the displaced weights (revised App. M.2, lines 2374–2375). We therefore moved the weights temporarily by ten first-step lengths, read two scores for each answer there and restored the weights. We obtained both readings before observing any of the ten actual training steps. Adding them improved prediction over local features in all eight new runs and over the baseline in six, with a positive mean gain against both (revised §4.5, lines 410–419, Table 1 in our second reply).

To address your concern about the link to complete generation, we changed a trained update and found that removing a component identified before training changed which answers returned. For each target, we removed the component along its initial score gradient in the rank-one adapter space. Removing a component that favored the answer prevented 19 recoveries, while removing one that opposed it produced four additional recoveries. Neither matched random removal changed any target outcome. The initial gradient thus identifies a part of the trained update that can affect complete recovery, though this test cannot tell us how much of ordinary recovery depends on it (revised §4.5, lines 422–431).

We agree that the clearest effect of transplanting early layers is on which answer the recipient favors. When we changed the donor and kept the rest of the recipient fixed, its preference shifted toward the answer that donor had been taught. Even among answers the donor learned and lost, 36/40 gained in score over the original recipient. Those gains seldom carried the whole donor answer into generation (2/54 and 0/54 across the full panels). Wrong-label cue training subsequently weakened the transferred preference in all three donor–recipient triplets on the second panel (§5). Alignment modestly improved prediction of the transplant’s effect on the score response to an individual update (Table 10). For the four-arm comparison you cite, predicting zero gave lower error (Table 11).

In the fresh-target panel you cited in Q2, correct answers on old cues restored 14.7% of trained failures, against 8.6% for wrong answers on old cues. Yet correct answers on fresh cues also restored 14.7%, so only the label comparison met its criterion (original App. E, lines 1054–1058). Your interpretation led us to a further test: if old cues help by reinforcing earlier associations, switching them from correct to conflicting answers should reduce recovery more than making the same switch on new cues. We tested this in a separate panel, comparing all four updates on the same forgotten targets at matched strength, and the results support your reading. At 410M, old cues restored 15.4 percentage points more learned-then-lost answers than new cues under correct labels and 8.2 points fewer under wrong labels. We also found this reversal among previously trained, currently wrong answers in the twelve-run 1.4B panel (revised §4.2, lines 297–303, Table 2 in our second reply).

The other cases you identified help explain why the distinction matters. In the separate eight-run 1.4B behavioral panel, the recovery gap between learned-then-lost and never-trained answers was slightly smaller with old correct cues than with fresh correct cues. For facts the model already knew before training, the interaction in answer-content recovery even changed sign across question wordings (original F.1 and G.1, revised §4.2, lines 309–315). As you observed, at 410M the change in alignment on old cues was concentrated under wrong labels, with little change under correct labels. At 1.4B, correct labels contributed more to the old-minus-fresh alignment contrast (revised §4.4, lines 363–368). We now explain the cue results through reinforcement and conflict, rather than a general old-cue advantage (§§4.2 and 7, lines 309–315 and 519–522).

Do §4.5 and this reading address Q1 and Q2?

AuthorsOpenReview ↗

Reply to Reviewer BMPR — data (2/2)

In M.1, 60/2,180 tests on current failures ended in full recovery after ten cue-training steps. Adding starting derivatives to baseline lowered log loss from 0.07347 to 0.06850 (gain 0.00497 [−0.00392, 0.01241]). At a displacement of one initial Adam step on the cues, these slopes closely tracked changes in token margins against their original competitors (mean r = 0.927). Extrapolating to ten first-step lengths missed 15/49 recoveries under that displacement. In ten of the 15 the initially weakest token had a negative slope (M.2).

Table 1. We froze the predictors before testing on eight new runs. Four updates on 560 failed fact–run cases produced 80 full recoveries in 2,240 tests. Linear adds to Local two extrapolated scores, the weakest-token margin and mean log probability over the answer and stop tokens. Finite reads both at the displaced weights. Runs count equally in means and 95% intervals (revised Tables M4 and M6, p. 47).

PredictorLog lossFinite-probe gain over comparatorRuns improved
Baseline: target/paired-fact exposure, update arm, scores/margins, length, gradient/step norms0.087250.01485 [0.00200, 0.02771]6/8
Local: baseline + 20 directional-derivative features0.088290.01589 [0.01086, 0.02092]8/8
Linear: local + two linearly extrapolated readouts0.088660.01627 [0.01041, 0.02213]8/8
Finite: local + two finite readouts0.07239Reference—

Finite lowered log loss against Baseline by 17.02% overall and 17.94% on learned-then-lost cases (162 cases, 648 tests, 78 recoveries). Baseline had lower log loss in runs 1118 and 1123. Against Baseline, the 95% interval for the Brier-score gain included zero. We saw interim results and completed all eight planned runs.

In M.4 we removed the trained update’s component along the initial target-score gradient projected into rank-one QKV adapter space. Of 80 recoveries, 61 survived and 19 were lost. There were four additional recoveries. All 23 flips followed the component’s sign (22 learned then lost, one taught but initially missed). Equal-energy random removal gave no flips. The sign-weighted random-minus-target difference was +1.014 points [0.086, 1.943] under both norm and calibration-KL matching (six positive runs, two ties). Reversing the component gave +1.985 [0.861, 3.109] in a secondary test. For the 23 flips, no other initially correct answer was lost beyond those lost under the intact update. Target-directed removals outside these 23 cases did produce additional losses.

H.3 (Table 10) predicts how much the transplant changes the recipient’s score response to a cue update. Against zero transplant effect, the combined predictor lowered pooled MSE by 9.69%. Adding alignment to scores and norms lowered pooled MSE by 3.36%, with higher error in one triplet. For the secondary four-arm interaction, its error was 15.71% above zero prediction (Table 11).

Table 2. C and W are old-minus-fresh recovery differences under correct and wrong labels, in percentage points. I = C − W. Within each run and initialization we matched all four updates to the lowest KL on 32 calibration questions (5% tolerance). We averaged initializations within runs before taking means and 95% t intervals (revised Table C3, p. 23).

Panel and conditionCWI
410M exposed failures, unscaled10.21 [5.08, 15.33]−7.36 [−11.15, −3.57]17.57 [11.80, 23.33]
410M exposed failures, matched KL10.99 [6.18, 15.79]−5.73 [−9.12, −2.35]16.72 [11.20, 22.24]
410M learned-then-lost, unscaled14.21 [6.45, 21.98]−9.86 [−14.51, −5.22]24.08 [16.28, 31.87]
410M learned-then-lost, matched KL15.36 [7.84, 22.89]−8.24 [−12.47, −4.01]23.60 [15.49, 31.72]
1.4B exposed failures, unscaled2.27 [−2.08, 6.63]−4.30 [−6.77, −1.83]6.57 [1.61, 11.53]
1.4B exposed failures, matched KL6.19 [2.73, 9.65]−2.59 [−4.74, −0.43]8.77 [5.07, 12.48]

All 216 updates at 410M met tolerance. Old/correct, old/wrong, fresh/correct and fresh/wrong mean KL went from 0.260/0.542/0.365/0.431 to 0.259/0.260/0.259/0.261. Matched I was positive/zero/negative in 24/2/1 runs for 383 trained failures and 23/3/1 for 263 learned-then-lost cases. At 1.4B, I was positive in all twelve runs for 318 trained failures. Identifying learned-then-lost cases there would require the missing initial-training endpoint. Before matching, the 1.4B interval for C included zero.

In Appendix J we gave the same wrong-label update to models taught different query–answer pairings. On average it weakened preference for each model’s assigned answer. The between-model contrasts were 0.447 [0.333, 0.561] nats/token on old cues and 0.345 [0.214, 0.476] on fresh cues. The old-minus-fresh difference in wrong-minus-correct responses was 0.080 [−0.042, 0.201].

Reviewer_BMPROpenReview ↗

Response to Authors

Dear authors,

I would like to thank the authors for your response and effort. I think your response "we could use the cue gradient to choose where to measure the answer scores, and those readings improved our predictions of the same ten-step outcome" mostly addressed my question Q1. However, I am still unsatisfied with the answer in Q2, I think the interpretation is still unclear to me. Overall, I think the responses are fair enough for me to raise the score from 2 to 3. I think the interpretation mentioned by Q2 should be examined better in camera-ready version.

Thank you for taking the time to read our response and consider the new experiments.

For Q2, we agree with your interpretation that the effect of a familiar cue depends on whether later training reinforces or conflicts with its earlier cue–answer pairing. In the matched-strength tests, switching from correct to wrong answers turned an advantage for old cues over fresh cues into a disadvantage (Table 2 of our second reply). We read that reversal as evidence that later training can help recovery when it reinforces an earlier cue–answer pairing and hinder recovery when it conflicts with that pairing. Familiarity alone gives no consistent advantage: old and fresh cues gave nearly identical recovery rates under correct labels in Appendix E, and the recovery gap between learned-then-lost and never-trained facts was slightly smaller with old correct cues than with fresh correct cues in the separate eight-run 1.4B study. We will make this interpretation clearer in §§4.2 and 7 of the camera-ready version.

Reviewer_mJTuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

It is known that facts that are forgotten due to new learning can be recovered by finetuning on a small set of (even unrelated) examples. The paper studies whether the training history that a given fact undergoes impacts the recoverability of the fact, and how fast it is relearned. It is not easy to extract from the paper what training history means, but after an investigation, it turns out that it means what happened before the forgetting: (i) whether the model was trained and the knowledge acquired, (ii) trained without the knowledge being acquired, or (iii) untrained at all.

Critical strengths

Understanding learning, unlearning and forgetting is an important and unsolved problem in AI.

Critical weaknesses
  • This paper is very poorly written and very difficult to read. The paper's abstract/introduction/discussion, which are supposed to be the compass for reading the paper, are unreadable by a human (see examples below in the author comments). This is alone a big obstacle and, I believe, a largely sufficient reason to reject the paper. The methodology as well suffers the same problem which further complicates the task. The meaning of history-conditioned access never comes explicitly and clearly: the reader has to infer this from Figure 1 caption and a lost single sentence in the middle of Section 2.1 (that does not even expose what L, M, and N stand for).

  • The second weakness is the novelty and the incremental nature of the work. As the authors also acknowledge, the fact that lost knowledge is recoverable is not news. That this depends on the amount of exposure is both incremental and not surprising.

Questions for authors

On unclarity and the lack of compass to read the paper: Many terms are used without being defined, the same term (e.g. acquisition) can mean different things in different setting, and many simple phenomena are explained using fancy expressions.

Abstract, second sentence, you talk about facts that "receive training before interference, then test updates that train only on other facts" the reader here wonders: what's special about these facts? are they known? unknown? what training do you induce? What interfering intervention do you use? and what are the "other" facts? The abstract then immediately continues with "prior exposure": to what? that's too generic. How do you fine exposure here? then what's "excact-answer" accuracy? and what's the "full fact panel"? used without prior definition. The rest of the abstract follows the same typical human-unfriendly abstracts that lead with (too many) results without even explaining the setting. Please see the summary I wrote above for how i think the abstract should have been written...

The introduction does a better job at first, but it's also written as if the user knew the experimental set up and the results. Once the reader knows what the paper is doing, a (part) of the introduction becomes more accessible but not before. The work talks about varying exposure or history, but the nature of this exposure is never hinted, then enters into the details of randomly assigning half of target facts, and it does not explain if the facts are known before or not.

The methodology unfortunately suffers the same phenomenon as explained in the weaknesses above. The paper uses three terms for the same thing: Acquisition/exposure/training but then uses acquisition to qualify whether the fact was actually learned or not!

Line 101: "Acquisition trains on 96 assigned targets and 32 cue facts" vs Line 114: "Acquisition means alias-aware greedy exact correctness at the end of the acquisition phase"

The paper should have led straight with the distinctions between L, M and N.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer mJTu — answers

Thank you for the detailed and concrete examples. In trying to summarize every experiment, we made the abstract and introduction too dense and left out the setting needed to read them. Both sections now follow the order you suggested in your summary. The revised abstract opens as follows (p. 1, lines 11–17).

When a language model forgets a fact it previously learned, it is natural to ask whether anything from that learning remains in the model’s state and affects its ability to answer the same question correctly later. To find out, we designed a twelve-run experiment with 192 facts that pretrained Pythia-410M initially answered incorrectly. In each run, we trained the model on a random half of these facts and on 32 other facts that serve as cues. Training on 600 other facts then caused it to forget 29.8% of the answers it had learned.

As you pointed out, in compressing the setup we used “acquisition” for both initial training and its outcome and packed the definitions into one sentence, describing L as “exposed and acquired,” M as “exposed but not acquired,” and N as “never exposed” (original §2.1, lines 115–117), making them easy to miss. Revised §1 now explains what the model was taught and how it forgot before distinguishing the groups more explicitly: among the facts it now gets wrong, some were learned then lost (L), some were trained but not learned (M), and the rest were never included in that training (N) (p. 2, lines 55–61). We now explicitly use “exposure” for assignment to initial training. For acquisition, we replaced “alias-aware greedy exact correctness” with “the model gives the full correct answer at the end of that training” (revised §2.1, lines 122–124). Revised §2.3 covers aliases and the stop token (lines 171–172).

Our introduction described history-conditioned access as “how past learning shapes the return of an answer under subsequent training,” but left readers to connect it to the tests given later (original p. 1, lines 47–52). Revised §1 now gives the tests first, so readers see how recovery probability and relearning speed depend on earlier training (p. 2, lines 64–69).

We agree that recovery of lost answers after training on other examples is an established finding, as Hu et al. and Deeb and Roger showed. From that starting point we asked how training history changes a fact’s response to a later update, whether we can measure this beforehand, and whether a parameter intervention can change it (original p. 1, lines 43–48). We extended Zhang et al.’s include–exclude design through interference. Even on the same questions, learned-then-lost answers recovered more often than never-trained ones. Previously trained failures had shorter recorded times to the relearning threshold (original §3, lines 193–195, 205–208).

Yoon et al. used alignment between the average gradients of datasets to explain recovery from syntactically similar data. Siddiqui et al. linked relearning resistance to weight-space distance, and Song et al. predicted recovery across unlearned checkpoints. We instead predict each answer’s response to cue labels. In original Fig. 3 (§4.2, p. 6, lines 318–323), alignment between the target-score gradient and the negative cue-loss gradient helped predict how much more switching cue answers would change the target’s score margin on previously taught cues than on new ones. In revised §4.4 (p. 7, lines 369–375), alignment improved this prediction even within learned-then-lost facts.

While starting distance helped explain recovery under random changes (original §4.1, lines 260–265, and revised §4.1, lines 269–272), switching cue answers from wrong to correct widened the L–N recovery gap more on old cues than on new ones (Fig. 2b in both versions, with estimates in original §4.1, lines 291–294, and revised §4.2, lines 304–306). In response to other reviewers’ questions, we matched update strength. Even among learned-then-lost facts, switching to wrong answers reduced full recovery more on old cues than on new ones (revised §4.2, lines 297–299). Section 7 explains how cue answers can reinforce or conflict with prior learning. Since labels also change the update, we gave two models taught different pairings the same wrong-label update. It tended to weaken each model’s preference for the answer it had been taught (revised §4.3, lines 343–350).

As for the amount of exposure, L and M were taught equally often, but at the end of initial training the model answered L correctly and missed M. In original Fig. 4, both donors saw the same questions and answers equally often, but we varied their pairings. Giving the same recipient early layers from one donor or the other shifted its preference toward the answer paired with that question in the donor’s training. Exposure count alone leaves open which answer the recipient would favor (original §5, lines 369–383, revised §5, lines 472–484).

Reviewer_mJTuOpenReview ↗

A new paper, which needs a new review

Dear authors, I wish i've read this version of your paper in my initial review during your initial submission. I think the changes made go in the right direction, and i would encourage the authors to apply them on the entire paper (including the limitations section and the appendices which still use the old language).

However, the main text has been nearly entirely rewritten, and the revision adds new sections and new appendices with experimental results. This is effectively a new paper to evaluate, which i believe deserves a careful reassessment as a new submission rather than a prompt revision phase.

Before resubmitting, I would encourage the authors to deepen it. Now that the paper is readable, content-wise, the headline findings are still unsurprising and lack deep insights. As the authors rightfully now also mention: the main recovery and relearning results which make the headline of this work are expected given earlier work. As it stands, the remaining additions are incremental and are more suitable for a topical workshop rather than a conference like ICLR. What could bring it to the level of a conferencem in order of importance: (i) more (and finer grained) mechanistic analysis (shedding insights on what happens inside the model beyond the current coarse "transplant", (ii) other update mechanisms: the recovery update and gradient measurements are limited to Q K V adapters which is thin, especially given existing evidence that fatual associations are located in MLP layers, and finally, (iii) scaling the analysis.

I am sorry i cannot be more positive and will have to keep my clear reject recommendation.

AuthorsOpenReview ↗

Several Clarifications

Thank you for reading the revised manuscript and acknowledging the improvement in presentation. However, we would like to clarify several points relevant to your assessment.

The phrase “As one might expect from earlier work” in §1 of the second revision refers to the learned-then-lost advantage in recovery and relearning. The original introduction already stated that “Recovery after later learning is established in distributed networks and language models” (original p. 1, §1), and the original related-work section discussed recovery through unrelated learning and partial retraining (§7, p. 9). From that starting point, the work asked how training history changes a fact’s response to a specified update, whether that response can be predicted before training, and whether changing historical parameters changes it. The cue-by-label comparisons, pre-update predictions and transplantation results addressing these questions were already in the original submission (original §§4–5, Figures 2–4; §§4.2, 4.4 and 5 of the second revision).

On update mechanisms, the original submission also tested MLP-only and all-module LoRA recovery updates, alongside QKV adapters at several ranks (Appendix D.1, Table 3: original p. 19; p. 23 of the second revision). Both the MLP-only and all-module configurations gave a positive recovery contrast in all twelve runs. The restriction to QKV coordinates applies to the gradient measurements, rather than to all recovery experiments.

The revisions addressed the presentation concerns, and our main findings and the original experiments and data supporting them remain the same. In response to reviewers’ questions, we matched update strength and added a selection analysis and new experiments on complete-answer recovery. Our mechanistic evidence also includes predicting later responses from pre-update geometry and, in §4.5 of the second revision, testing how removing a component of the update changes which answers return. Finer-grained localization and larger-scale studies would extend these findings, but the current experiments have established how earlier learning affects subsequent responses and how parameter interventions change that influence.

Reviewer_Xc1hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether earlier factual learning continues to influence a language model after it stops producing the correct answers. Through controlled training experiments, the authors find that previously taught facts recover and relearn more readily than withheld facts, and that recovery depends on the content of later training. They also show that gradient measurements taken before training help predict subsequent changes in answer margins. Finally, transplanting parameter blocks between models trained on different question–answer pairings shifts answer preferences and responses to later training, providing evidence that earlier learning leaves effects in the parameters even after correct answers disappear.

Critical strengths

The main strength is the controlled design: the authors randomly assign facts to earlier training and compare their responses to later updates. They test both recovery without directly teaching the target answer and relearning when that answer is supplied again. Together with replication in two other model configurations, these experiments show that current answer accuracy alone does not fully capture the effects of earlier learning. This design also supports subsequent experiments on predicting training responses from gradient measurements and testing how replacing parameter blocks changes answer preferences and later learning.

Critical weaknesses

1. Poor organization and insufficient explanation of the experimental evidence.

The paper is very difficult to follow. Section 3 presents many numerical results without clearly explaining how they build the argument that earlier learning continues to matter after the model stops answering correctly.

The authors train the model on a randomly selected subset of facts and then apply additional training that causes some answers to become incorrect. Among these incorrect answers, they distinguish facts learned and then forgotten (L), facts included in the initial training but still answered incorrectly afterward (M), and facts withheld from that training (N). They then test whether answers return after training on other facts or after teaching the target answers again. Recovery rates differ substantially across L/M/N, and previously taught facts also return faster when their answers are taught again. However, the paper provides little explanation of how these two tests complement each other and jointly support the first contribution.

The authors report two further comparisons that could strengthen their interpretation. First, they compare the same facts across training runs, addressing whether the recovery advantage simply reflects differences in question difficulty. Second, they account for how close the model already was to choosing the correct answer, testing whether this explains the advantage. The authors briefly explain what these comparisons establish and leave unresolved, but that explanation is split between short clauses in §3.1 and a discussion in Appendix A.2.2. This makes it difficult to judge how much each comparison strengthens the argument and which alternative explanations remain.

Further investigation appears separately in §4.1 and Appendix D.2, where the authors test random parameter changes and different training examples. These are meaningful experiments, but their connection to the initial findings in Section 3 is insufficiently developed. Readers must move between the main text and appendices to assemble the setup, results, explanations, and limitations. A consolidated discussion would make the strength of the evidence much easier to assess.

2. The recovery comparisons do not isolate the effect of successfully learning a fact.

The remaining question is whether earlier learning made these facts easier to recover, or whether cases that were already easier to recover were more likely to enter L. Entering L requires the model to have learned the fact and subsequently forgotten it. This may select cases that are systematically more likely to recover after further training.

Comparing the same facts across runs reduces the possibility that L simply contains easier questions, but the same question can respond differently to training in different runs. Similarly, accounting for how close the model was to choosing the correct answer does not establish that two cases will respond equally to further training. Neither comparison therefore fully separates the benefit caused by earlier learning from differences associated with which cases enter L.

The authors acknowledge that these analyses do not change how the acquired-and-lost groups were formed. Their randomized experiment establishes that including a fact in earlier training affects its response to later training. However, it does not determine how much of the larger L–N recovery gap was caused specifically by successfully learning the fact.

Questions for authors

Please see weaknesses

6 comments
AuthorsOpenReview ↗

Reply to Reviewer Xc1h — answers (1/2)

Thank you for your time and care in tracing the comparisons across the paper and appendix and for pointing out why selection could still explain part of the recovery gap. Using the randomly withheld facts in the original twelve runs, we estimate that selection could account for at most 1.75 points of the observed 24.8-point gap (Table 1 in our data reply). To test the selection explanation directly, we also followed the same question with and without its initial teaching.

Because half the targets were randomly withheld from initial training, we can use them to estimate how many learned-then-lost answers (L) might have come back without that teaching. Only six withheld answers were correct after cue training, even counting those already correct beforehand. Even assigning all six successes to cases that would have entered L if taught gives an estimated rate without initial teaching of at most 6/312, compared with 78/312 under teaching (our Table 1, App. A.2.2). By counting every withheld success, the bound allows cases that started equally close to the right answer to respond differently to cue training.

As you point out, the same question can be easier to recover in one run than in another. We therefore started two copies from the same weights with the same training examples in the same order. Both copies saw the target question during initial training, but only one was trained to produce its answer. In 133 pairs, only the copy taught the answer got it right at the end of initial training, and both copies got it wrong after interference. Across four kinds of cue training, it returned in 72/532 tests in the taught copy and 0/532 in its partner. For these cases, withholding just the target answer eliminated the recoveries we saw when it was taught (App. L).

It is natural to ask whether an answer taught equally often is more likely to return when the model initially learned it. We first compared the same 37 questions across the original runs. Recovery averaged 21.7% when the model learned and lost the answer and 15.4% when it was taught but missed at the end of training. Comparing copies taught the target with the same seed also favored initial success on average, though the interval included zero. Those copies still withheld different other answers. We therefore reran the twelve original assignments with identical questions, answers and teaching counts, moving early repetitions closer together in one copy to vary initial learning. Both schedules gave about 93% correct after initial training, and neither had a clear overall recovery advantage. In 51 pairs, however, only one copy initially learned the answer and both later got it wrong. The answer returned in 10/51 cases on the initially successful side and 3/51 on the other (+13.73 points [0.02, 27.43], revised App. L.1, lines 2275–2282). Thus, even with equal teaching, the answer returned more often on the side that had learned it. Since practice order could also affect recovery directly, this comparison cannot isolate the share of the original gap due to initial success alone.

To address the organization problem you pointed out, we revised §§3–4. We summarize the revised argument here.

Training on other facts tells us whether a currently missed answer can return without being taught again, and direct reteaching from the same post-interference model measures how quickly it relearns the answer. Across all 1,548 failures, the mean recorded time to threshold over ten steps was 1.46 steps shorter for previously taught failures than for never-trained failures, or 0.85 after accounting for starting scores.

Across all 192 targets, random assignment to initial training increased the net accuracy gain from cue training by 5.03 points [2.58, 7.49]. Among current failures, comparing the same questions across runs removed stable differences in question difficulty and left a 30.3-point L versus never-trained recovery gap. Separately, accounting for how close the model already was to giving the right answer left an 18.9-point gap. As you note, answers that start equally close can still move differently when training resumes. Section 4 tests why.

An answer that starts closer can return even under random changes to the weights. Such changes favored learned-then-lost over never-trained answers by 8.9 points (§4.1, App. D.2). We then held the starting model fixed and varied what equally strong updates taught on the cues. Switching correct to wrong answers reduced recovery more for previously taught cues than for new ones, by 23.6 points [15.5, 31.7] among learned-then-lost cases at 410M (§4.2). Since changing the labels also changes the update, we next gave two models the same wrong-label update. The shift in preference depended on which answer each had been taught for the same question (§4.3).

We would be glad to know whether the bound and the paired experiment address your question about selection into L.

AuthorsOpenReview ↗

Reply to Reviewer Xc1h — data (2/2)

Table 1. Each original run tested 192 targets after ten steps on previously taught cues with correct answers. For questions wrong after interference, the table compares answers learned and then lost (L) with those never taught (N). Among the 91 cases taught but missed both at the end of initial training and after interference (M), seven recovered (original §3.1, p. 4, revised §3.1, pp. 4–5, and App. A.2.2, p. 17).

QuantityCount or rate
Fact–run records2,304
Trained / untrained1,152 / 1,152
Learned-then-lost fact–run cases (L)312
L recovery78/312 = 25.00%
Never-trained facts that are wrong (N): recovery2/1,145 = 0.17%
Untrained answers correct after the update6: four already correct, two newly recovered

Since training was assigned at random, the taught half tells us how often cases would enter L. The withheld half tells us how many answers could be correct after cue training without initial teaching. If between zero and six withheld successes belonged to cases that would have entered L, their estimated rate without teaching was 0–1.92%. For these cases, the estimated benefit of assignment to teaching was 23.08–25.00 points. Selection could explain at most the difference between 1.92% and the observed 0.17% for N. These estimates pool question–run counts. Resampling the twelve runs 100,000 times gave sampling intervals. The lower endpoint had a 95% interval of [17.31, 30.30] points and the upper endpoint [19.28, 31.88] (App. A.2.2).

Table 2. In each of eight runs on a separate 94-fact panel, one copy was trained to produce the answers to all 47 assigned targets (ON). For each target its paired copy saw the same examples but omitted that target’s answer and stop token from the training loss (OFF). Starting weights, order, other answers and loss normalization were held fixed. After interference each copy trained on the same four cue recipes from its own weights. We included targets learned in ON and wrong in both copies after interference. OFF also answered all 133 incorrectly at the end of initial training (revised Table L1, p. 42).

SeedPaired targetsON recoveriesOFF recoveries
1125207/800/80
11261614/640/64
1127113/440/44
1128115/440/44
1129247/960/96
11301710/680/68
11311614/640/64
11321812/720/72
Total13372/532 (13.53%)0/532 (0.00%)

At least one update restored the taught answer in 39 pairs. Neither copy recovered it in the other 94 pairs. Across the four updates the difference was 13.53 points [8.27, 18.80], with uncertainty measured across seeds. Counting only the cue condition used on the original panel, 26/133 taught answers recovered here (App. L).

For the comparison in which both sides received the target answer, we first used the 37 original questions with both L and M cases across runs (95 L, 78 M). With each question weighted equally, L led by 6.31 points. Thirteen questions favored L, seven favored M and 17 tied. A post hoc analysis of the eight runs also fixed the seed. Copies taught that target equally often differed in which other answer was omitted. In 143 question–seed pairs from 28 questions, all copies compared were wrong after interference and some had initially succeeded while others had missed. Averaging over copies and cue conditions within each side gave 10.06% versus 7.97% recovery (difference 2.09 points, approximate 95% interval [−1.33, 5.51], each pair weighted equally). Five seeds favored success and three favored a miss.

For the teaching-order experiment (App. L.1), each of the twelve original assignments was run twice from the same seed. One copy used the original order. For the other we clustered early repetitions of each target, keeping cue positions and the final two epochs fixed. Of 1,152 taught cases in each schedule, 1,072 and 1,075 were initially correct (original minus clustered, −0.26 points [−3.95, 3.43]). After interference, cue training restored 92 versus 86 answers (+0.52 points across all taught cases [−2.32, 3.36]). The original order left more answers available to recover. Among answers wrong after interference the rates were 92/403 and 86/362.

In 51 pairs, both copies failed after interference but only one had initially learned the answer. That copy used the original order in 28 pairs and the clustered order in 23. Nine answers returned only in the initially successful copy, two only in its partner, one in both and 39 in neither. Recovery was therefore 10/51 versus 3/51 (+13.73 points [0.02, 27.43]). The 95% intervals use the twelve paired runs. Both schedules were freshly run. The original-order reruns reproduced the L, M, N recovery ordering (85/337, 7/66, 4/1,146), though the final correct/incorrect outcome differed from the archive in 264 of 2,304 cases.

I thank the authors for the additional upper-bound analysis and paired experiment, which largely address my concern about selection. I also appreciate the added explanations in the revised manuscript. However, my concern about the presentation remains.

After rereading Section 3, I still find Section 3.1 difficult to follow. The revision clarifies individual comparisons, but the dense numerical details make it hard to distinguish the main findings from supporting analyses and follow the overall argument. I think each experiment’s main results would be easier to read in a dedicated table or figure, with further numerical details moved to the appendix. Readers could then compare conditions and check the results without following long sequences of numbers in the prose. The main text could then focus on why each experiment is needed, what the findings establish, and how they support the argument.

I will therefore retain my current score. I hope the authors will further improve the presentation in the final version if the paper is accepted, or in a future submission.

Reviewer_N1dZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses a question of knowledge re-appearance in large language models. This includes a question of how remembered, but later forgotten facts can again be remembered when the model is tuned on other facts present at training time. In their experiments, the authors randomize which of 192 facts are trained before interference in 12 separate Pythia-410M training runs. The authors then apply LoRA fine-tuning on other facts to establish that previously trained facts recover more often and relearn faster than never-trained ones. Interestingly, random-perturbation control experiments attribute much of this to lost facts being closer to threshold. Additional cue-history and label interaction, a pre-update gradient-alignment measure, and a layer-transplant experiment are offered as evidence of an additional, content-dependent component.

Critical strengths
  • The paper features a careful experimental design and a very transparent reporting of experimental results. Core findings are made credible by considering well-designed randomized exposure with peer balancing, within-fact contrasts, frozen predictors evaluated on fresh histories, and matched random controls. In additional to positive results, the paper also reports its failed tests openly.
  • The authors separate two possibilities for fact recovery: proximity to the decision boundary and sensitivity to the content of the update. They show that under random parameter perturbations, previously learned facts also recover more often, which is aligned with the proximity argument (additional simple parameter-free reconstruction based only on starting margin deficits also supports this). The cue and label variations then isolate a residual that depends on update content. This decomposition is perhaps the paper's most useful conceptual contribution.
Critical weaknesses
  • The evidence linking the effects beyond proximity to forgotten facts specifically is not particularly strong. The geometry measure does not show clear difference between learned-then-forgotten facts and never-trained facts. The cue and label interaction is barely significant in the main experiment and inconclusive when restricted to currently failed facts. The authors also note that the transplants shift scores but almost never affect outputs. Considering small models, simple rank-1 probes, and curated facts, the paper's main claim (forgotten facts retain a residual memory that shapes later learning) is only partially supported.
  • The paper is very dense and generally difficult to read. Core results are introduced very compactly and key caveats seem to be mainly in the appendices.
Questions for authors
  • Can you compare factual updates with nonce-QA and random-token updates at matched update strength? My concern is that the effect can come from the update magnitude and not the fact contents.
  • Why should the geometry measure be understood as a signature of forgotten facts, when it does not differ between learned-then-forgotten and never-trained facts?
  • The cue-label interaction is inconclusive for currently failed facts. Can you provide a better distinguishing test for them (since these are the facts the paper's main question is about)?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer N1dZ — answers (1/2)

Thank you for drawing out the distinction between starting close to the right answer and responding to the content of a later update. You are right that our original test left open whether switching cue answers affected recovery more on old cues than on new ones when the model got a previously taught target wrong (Q3, original Appendix B). We found that it did for facts the model had learned and lost, even when we matched update strength.

We tested 263 such cases across 27 Pythia-410M runs. All four updates in a run started from the same model, so the model started equally close to giving the correct answer on each target in all four conditions. We trained on both previously taught and new cues, pairing each type with correct answers in one update and wrong answers in another, and scaled the four updates to change outputs on separate questions by the same amount within a 5% tolerance. Our results showed that switching from correct to wrong answers on previously taught cues cut full-answer recovery from 30.1% to 8.9%. On new cues the same switch took recovery from 14.7% to 17.1%. With starting distance held fixed and update strength matched, this 23.6-point difference [15.5, 31.7] shows that a forgotten answer’s chance of returning depended on which answers we taught on which cues (§4.2, Table 1 in our second reply).

Changing a cue answer also changes the update the model receives. To see what earlier learning contributes when even that update is held fixed, we applied the same wrong-label update to two models taught different answers to the same questions. On average it weakened each model’s preference for the answer that model had been taught relative to the alternative. Among questions both models had learned and lost, this difference was clear for new cues. With previously taught cues the mean also favored weakening, though the interval included a small reversal (§4.3, Table 3 in our second reply).

For Q1, we compared factual training with the two alternatives you suggested, invented-word questions and random tokens. With equal token counts, factual training changed the model’s outputs least in every run, so we scaled the synthetic updates down to that strength. Factual training still led on mean net accuracy change by 1.3 points over invented-word questions and 0.6 over random tokens, compared with 3.5 and 2.0 points before matching. The smaller synthetic updates spoiled fewer previously correct answers, showing how update magnitude accounted for part of the original lead. Among learned-then-lost targets, we found that factual training still restored more answers on average (9.2% versus 4.1% and 4.7%), though intervals for both recovery and net accuracy differences allow a small advantage for synthetic training (§4.1, Table 2 in our second reply).

As you observed in Q2, the mean change in gradient alignment did not clearly distinguish learned-then-lost facts from never-trained facts the model also got wrong. What it helped us predict was how a forgotten answer would respond to the choice of cue labels. We tested the original predictor on learned-then-lost facts alone. Before cue training, we compared the gradient of the target answer’s score with the negative gradient of the training loss on the cues. Adding the change in their alignment from the pretrained model to the model after interference improved prediction of how much more switching labels would change the target’s score margin on previously taught cues than on new ones (§4.4, Table 4 in our second reply).

As you pointed out, the recipient rarely generated the donor’s complete answer after a transplant (2/54 and 0/54 across the two panels). Yet when we switched donors, the same recipient shifted its preference toward the answer paired with that question in the new donor’s training. Even among the 40 answers the donors had learned and lost, 36 scored higher with the corresponding donor’s layers than with the recipient’s own. We then asked how later training affected that transferred preference. Wrong cue answers weakened it in all three donor–recipient triplets on the second panel. On the first, the effect of switching labels on old versus new cues varied across triplets (original §5, lines 369–383, revised §5, lines 472–488).

We selected the easiest quarter of candidate facts to obtain enough learned answers to study forgetting, and how far the findings extend to harder facts remains open. The recovery and relearning advantages recurred at 1.4B and in Qwen2.5-0.5B, and at 410M we observed the recovery advantage across adapter ranks 1, 4 and 8 and several module choices. To make the evidence easier to follow, §3.1 now explains why we need each comparison, §4.1 gives the uncertainty alongside the material comparison, and §5 reports the complete donor answers beside the shifts in preference.

We would appreciate your view on whether the matched test on learned-then-lost facts addresses Q3.

AuthorsOpenReview ↗

Reply to Reviewer N1dZ — data (2/2)

Table 1. For Q3 we matched the four updates to the smallest change in output probabilities (KL) on 32 separate questions, separately for each initialization and run. We averaged the two initializations within runs, then weighted runs equally. Entries are differences in full-answer recovery rates, in points, with 95% intervals. L denotes cases learned then lost (revised Tables C3–C4).

CasesOld minus fresh, correct answersOld minus fresh, wrong answersDifference between the two
410M, all 1,678 cases now wrong2.40 [1.34, 3.47]−1.34 [−2.16, −0.53]3.75 [2.40, 5.09]
410M, 383 trained cases now wrong10.99 [6.18, 15.79]−5.73 [−9.12, −2.35]16.72 [11.20, 22.24]
410M, 263 L cases15.36 [7.84, 22.89]−8.24 [−12.47, −4.01]23.60 [15.49, 31.72]
1.4B, 318 trained cases now wrong6.19 [2.73, 9.65]−2.59 [−4.74, −0.43]8.77 [5.07, 12.48]

For the 263 L cases at 410M, the four recovery rates (old/correct, old/wrong, new/correct, new/wrong) were 30.29/10.98/16.08/20.84% before matching and 30.06/8.90/14.70/17.14% after. The corresponding mean KLs were 0.260/0.542/0.365/0.431 before and 0.259/0.260/0.259/0.261 after. All 216 updates met the 5% tolerance. The last column was positive in 23/27 runs for L at 410M and 12/12 runs for trained failures at 1.4B, where the initial-training answers needed to identify L were unavailable.

Table 2. For Q1 we compared factual training, invented-word questions and random tokens. With token counts equal, their mean KLs were 0.110, 0.262 and 0.206. The table gives recovery for 157 learned-then-lost cases at matched KL. Across all 94 targets, all three mean net changes were negative. We averaged the two materials of each synthetic type within runs and weighted the eight runs equally (Tables K1–K2).

UpdateRecoveryFactual minus this update, pp
Factual9.2%—
Nonce QA4.1%5.0 [−1.1, 11.2]
Random tokens4.7%4.4 [−0.6, 9.4]

Table 3. Two models were taught different pairings of the same twelve questions and answers. We replayed one wrong-label update on both. In each we scored the answer assigned to model A relative to the one assigned to B using mean log probability through the stop token. Positive values mean a larger shift toward A’s answer in B than in A, averaged over update sources (Tables J1, J4).

AnalysisValue
All 12 questions, old/wrong0.447 [0.333, 0.561]
All 12 questions, new/wrong0.345 [0.214, 0.476]
Both models learned then lost, old/wrong0.373 [−0.070, 0.815]
Both models learned then lost, new/wrong0.357 [0.176, 0.539]

The both-lost subset was post hoc. It includes 23 question–run pairs (46 answers) in seven run pairs. All eight run pairs enter the intervals. Accounting for general preferences for particular answers gave 0.272 [0.073, 0.472] on new cues. Across replays, 17/84 learned-then-lost answers returned at least once, including 8/46 in the subset.

Table 4. For Q2 we compared the target-score gradient with the negative cue-loss gradient. We took correct minus wrong alignment for old and new cues separately, then old minus new. ΔA is the change in this quantity from the pretrained model to the model after interference. The predictor estimates the corresponding contrast in ten-step margin changes (§4.4, paper Tables 2 and C2).

ComparisonEstimate
Effect of assigned initial training, 410M0.0191 [0.0099, 0.0283]
Effect of assigned initial training, 1.4B0.0268 [0.0223, 0.0314]
410M mean ΔA, L minus untrained failures0.0055 [−0.0173, 0.0283]
Reduction in mean squared error for the ten-step margin contrast, 263 L cases0.548 [0.256, 0.840]; 23/27 runs improve

To predict full answers we also scored targets after a temporary displacement of ten first-step lengths along the cue-training direction (M.3). Four updates on 560 failures gave 80 recoveries in 2,240 tests. Adding two such scores to the local features reduced log loss from 0.08829 to 0.07239, a gain of 0.01589 [0.01086, 0.02092], positive in eight of eight runs. Against the baseline of prior training, update type, scores and gradient sizes (0.08725), the gain was 0.01485 [0.00200, 0.02771], positive in six runs. Within L the reduction against baseline was 17.94%. We fixed the predictors before testing, saw interim results and completed all eight runs.

To test whether a trained update’s direction matters for generation, we removed the part of each trained update along the initial target-score gradient (M.4). This prevented 19 recoveries and produced four new ones. All 23 changes followed the predicted sign (22 L, one taught but initially missed). Removing an equally large random component, matched on weight or output change, produced no flips. The mean sign-weighted difference against either control was +1.014 points [0.086, 1.943], positive in six runs with two ties.

I would like to thank the authors for their very detailed response and for numerous new experiments. The matched-strength label-switch test on learned-then-lost facts (my third question) is convincing. This directly addresses one of my concerns. The response to my first question is not as conclusive and partly confirms my concern. The reframing of the gradient-alignment measure in response to my second question is reasonable. Overall, I think that the new evidence slightly strengthens the paper and I will keep my "weak acceptance" recommendation.

Thank you for reading our response and the new experiments so carefully, and for your specific feedback on all three questions. We appreciate your assessment.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Overview of our responses and revisions

In our work, we found that training on other facts restored learned-then-lost answers more often than never-taught ones. When we retaught missed answers, the mean recorded time to the relearning threshold was also shorter for previously taught answers than for never-taught ones. While starting closer to the correct answer explained much of the difference in recovery across groups under random weight changes, recovery after factual training also depended on which answers were paired with the cues. Gradients measured before training helped predict this sensitivity in answer scores. Transplanting early layers shifted preferences toward the donor’s pairings and changed how those preferences responded to further training (original p. 2, lines 73–89).

We thank the reviewers for showing us where the paper was hard to follow. We revised the main text throughout to make it less dense and easier to follow. The abstract and §1 now give the setting first, §2 distinguishes being taught an answer from learning it, and §§3–5 explain why each comparison is needed. The Discussion (§7) explains when old cues help or hinder recovery. We also revised the descriptions of the original experiments in Appendices A–I and added Table O1 (pp. 14–15) to show which runs and facts each comparison uses. Our main findings remain the same and rest on the original experiments. Also, Figures 1–11, Tables 1–14 and Appendices A–I retain their numbering. The reviewers’ questions prompted new tests of selection, update strength and complete recovery.

N1dZ asked whether the content of later training mattered on genuinely forgotten facts. At matched update strength, switching from correct to wrong answers reduced complete recovery 23.6 percentage points more on previously taught cues than on new cues in 263 learned-then-lost cases [15.5, 31.7]. Old cues led with correct answers and fell behind with wrong answers, as BMPR’s reinforcement-and-conflict interpretation predicts. We now give that interpretation in §7. In N1dZ’s separate comparison across materials, factual training still restored more lost answers on average than either synthetic alternative at matched strength, though both intervals allowed the ordering to reverse (revised p. 6, lines 277–283 and 297–303, p. 10, lines 519–522).

Xc1h asked whether the model had simply learned the facts that would have come back anyway. From the original randomized data, we estimate that selection could explain at most 1.75 points of the 24.8-point pooled recovery gap. In new paired runs where the model learned the answer when taught and both copies failed after interference, the target returned in 72/532 tests with initial teaching and 0/532 when we withheld just its answer. In a separate experiment where both copies were taught equally often, neither practice order gave a clear overall recovery advantage. Among the 51 pairs where only one copy learned the answer and both later failed, it returned in 10 cases on the side that had learned it and in 3 on the other. Because practice order could also affect later recovery directly, this comparison cannot isolate how much of the original recovery gap was due to learning the answer during initial training. (revised p. 4, lines 199–209, p. 5, lines 216–219, and p. 43, lines 2275–2283).

On mJTu’s question about exposure, both donors in original Fig. 4 saw the same questions and answers equally often, yet changing their pairings changed which answer the same recipient favored (p. 7, lines 332–345 and 369–376). In our reply to mJTu, we quote the original definitions, show how we introduce them in the revision, and compare our findings with prior work.

BMPR asked whether gradients could predict complete answers. In the original Fig. 3, gradients measured before training helped predict how switching cue answers affected a target’s score margin on old cues relative to new ones (p. 6, lines 318–323). For complete recovery, adding local derivatives at the starting weights gave no clear gain. However, using the cue gradient to choose where to move the weights temporarily and read the scores improved prediction over local features in all eight new runs. In a separate intervention, removing the component of a trained update along the initial target-score gradient prevented 19 recoveries and allowed four others (revised p. 8, lines 402–419 and 422–431).

AuthorsOpenReview ↗

Changes by section in the revised paper

Original paperChanges in the revision
Abstract, §§1–2The abstract and §1 now describe what the model was taught and what it forgot before introducing the groups in plain language. Section 2 distinguishes being taught an answer from learning it.
§3.1, Fig. 1Section 3.1 now uses the original randomized data to bound how much selection could explain the recovery gap (A.2.2). New paired runs compare recovery of the same answer when it was taught and when it was withheld (L). A separate test in L.1 varies practice order while keeping the number of repetitions fixed.
§§3.2–3.3, Fig. 1Sections 3.2–3.3 retain the relearning results and the comparisons across models. Section 3.2 now briefly describes how the first three training steps help predict recovery at step ten. The detailed prediction results are in A.1, D.1 and E.
§4.1, Fig. 2(a–b)Sections 4.1–4.2 retain the original comparisons and add two controls for update strength. One compares factual training with invented question–answer pairs and random tokens (K). The other compares correct and wrong answers on old and new cues (new Fig. 2c, C.3). Section 4.3 then applies the same update to two models taught different answers to the same questions (J).
§§2.4 and 4.2, Fig. 3Section 4.4 explains how much correct and wrong cue answers each contribute to the gradient result, then tests whether alignment helps predict score changes within learned-then-lost facts (C.1–C.2). New §4.5 asks whether measurements before training can predict which complete answers will return and whether removing part of a trained update changes which answers return (M).
§5, Fig. 4Section 5 now reports how rarely the recipient generated the donor’s complete answer (original H.1–H.2) alongside the preference shifts shown in Fig. 4. It also brings a result from original H.3 into the main text. The predictor improved on predicting zero transplant effect for individual updates, while predicting zero gave lower error for the comparison across all four updates (Tables 10–11).
§6, Extensions and a potential applicationSection 7 summarizes recovery of facts the pretrained model already knew and recovery after interference from reasoning and classification tasks. The detailed results are in G.1 and G.2. Appendix I gives the results and cost of checking individual facts after an update.
§§7–9Related work, discussion and limitations are now §§6–8. Appendix N gives further comparisons with prior work.
AuthorsOpenReview ↗

Further revisions to Sections 3–5

We have further revised the manuscript following Xc1h's suggestions on presenting the main findings and supporting analyses, reducing numerical details in the prose of Sections 3–5 and presenting the main comparisons in tables and figures.

Section 3.1 now states the two main recovery findings first, followed by the comparisons that examine possible explanations. Figure 1 and Table 3.1 (pp. 3 and 5) present the recovery comparisons, including the selection bound and paired ON/OFF experiment; the table states what each comparison establishes. Table 3.2 (p. 5) presents the relearning results across the three models. In Section 4, Table 4.1 (p. 6) compares the interaction between cue history and label correctness before and after matching update strength.

Across Sections 3–5, we have shortened numerical reporting in the prose and added direct references to the corresponding tables, figures and appendices. The text retains the main qualitative findings and focuses on why each experiment is needed, what it establishes and how it contributes to the argument.

These revisions concern presentation. The experiments, results and conclusions remain unchanged, and all existing figures and tables retain their numbering.

AuthorsOpenReview ↗

Changes by section in the second revision

The table below summarizes the changes from the original submission to Revised 1 and the further changes from Revised 1 to Revised 2. Section numbers and page references refer to Revised 2 unless otherwise stated.

SectionRevised 1: changes from the original submissionRevised 2: further changes from Revised 1
Abstract and §§1–2Rewrote the abstract and introduction to give the setting before the groups and findings. Clarified in §2 the distinction between being taught an answer and learning it.Retained the rewritten opening and setup. Specified in the Figure 1 caption that recovery uses old cues with correct answers; and in §2.3, clarified that the answer margin uses the primary answer’s tokens and removed three closing sentences about analysis protocols.
§3.1: Recovery and supporting comparisonsUsed the original randomized data to bound how much of the pooled recovery gap selection could explain. Added separate paired ON/OFF runs to test how including the target answer in the initial-training loss affected later recovery. Appendix L.1 also varied practice order while keeping teaching counts fixed.Identifies the two main recovery findings first, then explains how the subsequent comparisons examine selection, initial teaching, question difficulty and starting distance. Table 3.1 collects the main recovery results, selection bound and paired experiment, with a column explaining what each comparison establishes (pp. 4–5).
§§3.2–3.3: Relearning and other modelsExpanded the explanation of how recovery and relearning complement each other, retained the cross-model results, and directed detailed prediction results using the first three training steps to the appendices.Added Table 3.2 to present the relearning results across all three models together, shortened numerical reporting in the prose, and retained the explanation connecting the two tests (p. 5).
§4: Starting distance and responses to updatesSeparated tests of starting distance, cue content and responses to the same update. Added strength-matched comparisons of factual versus synthetic training and of the four cue-by-label updates, plus same-update replay. Expanded the analysis of correct- and wrong-label contributions to the full interaction and tested the existing margin predictor within current failures. Added pre-update complete-answer prediction and directional intervention.Reworked the opening to preview the finding from each of the three tests motivated by §3, with references to §§4.1–4.3. Added Table 4.1 for the cue-history × label interaction before and after matching strength. Shortened numerical reporting while retaining the explanations of what the comparisons establish and why the next test is needed (pp. 5–9).
§5: Parameter transplantationPlaced rare complete donor-answer generation alongside the score changes, and brought limitations of the existing transplant-prediction results into the main text.Reworked the opening to state the transplant experiment’s purpose and its two main findings: changing donors shifts answer preferences and changes responses to later training. Shortened numerical sequences in the results paragraphs and added direct references to appendix figures and §H.2 (pp. 9–10).
§§6–8: Related work, discussion and limitationsCondensed the original §6 extensions into summaries in §7 and references to Appendices G and I. Reorganized related work, discussion and limitations as §§6–8, with fuller related-work comparisons in Appendix N.Shortened wording in related work (§6) and limitations (§8). Retained the discussion and conclusions in §7 (p. 10).

The second revision changes the presentation of the evidence; its experiments, results and conclusions remain unchanged from the first revision. We have kept the existing figure and table numbers so that references in our earlier replies remain easy to follow. We will continue to refine the presentation, including making the table numbering consistent and easier to follow, and updating the cross-references.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.