- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how prior factual learning influences subsequent updates after the model no longer produces the learned answers. It randomizes factual exposure across training histories, applies interference, and measures both recovery under target-disjoint updates and savings under direct relearning.
In the primary twelve-chain Pythia-410M experiment, prior exposure increases update-induced exact-answer change by 5.03 percentage points over the full randomized panel. Among current failures, acquired-then-lost facts recover more frequently than never-exposed facts, although this comparison concerns realized histories rather than randomized acquisition. Related behavioral patterns recur in Pythia-1.4B and Qwen2.5-0.5B.
The paper further measures pre-update target–support gradient alignment, evaluates frozen predictors of finite update responses, and transplants native blocks between models trained on different query–answer assignments. Together, the experiments support persistent, history-dependent parameter influence on answer preferences and subsequent learning. They provide more limited evidence about the specific mechanism mediating complete-answer recovery.
Critical strengths
-
The design identifies an effect of exposure while treating realized learning histories carefully. The primary experiment randomizes exposure across twelve training chains and reports a full-panel effect of 0.0503 on update-induced exact-answer change, with a 95% interval of [0.0258, 0.0749]. It separately reports recovery of 78/312 acquired-then-lost failures versus 2/1,145 never-exposed failures, without presenting this post-treatment comparison as a randomized acquisition effect. Within-fact comparisons, deficit adjustment, random-displacement controls, and relearning tests provide complementary evidence. This separation of estimands makes the central behavioral conclusion substantially more credible.
-
The paper connects behavioral history effects to prospective measurements and controlled parameter interventions. Geometry-based prediction is evaluated on 27 fresh Pythia-410M histories and 12 untouched Pythia-1.4B histories using discovery-fitted predictors. The transplantation experiments hold the recipient complement fixed and vary donor bindings while preserving answer pools; cyclic contrasts rule out a restricted query-independent answer-shift explanation. These experiments establish history-sensitive parameter influence beyond a recovery curve alone. The explicit reporting of failed criteria, weak transfer endpoints, collateral changes, and audit costs further strengthens the evidence.
Critical weaknesses
-
The connection between the measured geometry and complete factual recovery remains incomplete. The strongest geometry results concern continuous response targets, while native-block transplantation primarily shifts candidate-answer preferences and rarely restores complete donor-assigned answers. In Appendix H.3, the combined predictor reduces transplant-response MSE by 9.69% relative to zero effect, but the incremental contribution of angular alignment beyond scores and gradient norms is 3.36% pooled and negative in one of three confirmation triplets. The same predictor performs worse than zero on the secondary four-arm interaction target in Table 11. These results support a useful predictive descriptor of some update responses, but do not yet identify a common mechanism mediating exact recovery. The paper acknowledges much of this boundary; it remains the main limitation on the strength of its unified mechanistic account.
-
The more specific interpretation of support selectivity is less robust than the general history effect. In the fresh-target panel of Appendix E, old-correct and fresh-correct support restore essentially the same fraction of exposed failures, so the prespecified two-part recovery criterion fails. The old-versus-fresh advantage also does not replicate in the 1.4B behavioral panel. For pretrained-known facts, recovery-level cue-by-label interactions vary across prompt forms, including a negative interaction for one alternate prompt. These findings do not negate history-conditioned response, but limit an interpretation based on a consistently privileged recovery route through previously learned cues. The distinction matters because, at 410M, the geometry interaction is concentrated in sensitivity to conflicting old-cue updates rather than an increased alignment with correct old-cue updates.
Questions for authors
-
Geometry versus exact recovery: Do any existing held-out analyses show that the pre-update geometry adds predictive information about binary exact-answer recovery among current failures, beyond current scores and gradient norms, without observing an actual update step? The early-response predictor uses three observed steps, while the main pre-update predictors target continuous responses. If this link has not been established, please distinguish that boundary explicitly from the confirmed prediction results.
-
Interpretation of support selectivity: How should the proposed account be interpreted jointly with the failed old-versus-fresh recovery comparison in Appendix E, the corresponding 1.4B result, and the concentration of the 410M geometry effect in old-cue wrong-label sensitivity? Do the results primarily identify sensitivity to reinforcement/conflict with historical associations, rather than a generally privileged reactivation route through old correct cues? Please make the distinction explicit in the main interpretation.
I would be happy to raise my score if some of the concerns are addressed.
4 comments
Reply to Reviewer BMPR — data (2/2)
In M.1, 60/2,180 tests on current failures ended in full recovery after ten cue-training steps. Adding starting derivatives to baseline lowered log loss from 0.07347 to 0.06850 (gain 0.00497 [−0.00392, 0.01241]). At a displacement of one initial Adam step on the cues, these slopes closely tracked changes in token margins against their original competitors (mean r = 0.927). Extrapolating to ten first-step lengths missed 15/49 recoveries under that displacement. In ten of the 15 the initially weakest token had a negative slope (M.2).
Table 1. We froze the predictors before testing on eight new runs. Four updates on 560 failed fact–run cases produced 80 full recoveries in 2,240 tests. Linear adds to Local two extrapolated scores, the weakest-token margin and mean log probability over the answer and stop tokens. Finite reads both at the displaced weights. Runs count equally in means and 95% intervals (revised Tables M4 and M6, p. 47).
| Predictor | Log loss | Finite-probe gain over comparator | Runs improved |
|---|---|---|---|
| Baseline: target/paired-fact exposure, update arm, scores/margins, length, gradient/step norms | 0.08725 | 0.01485 [0.00200, 0.02771] | 6/8 |
| Local: baseline + 20 directional-derivative features | 0.08829 | 0.01589 [0.01086, 0.02092] | 8/8 |
| Linear: local + two linearly extrapolated readouts | 0.08866 | 0.01627 [0.01041, 0.02213] | 8/8 |
| Finite: local + two finite readouts | 0.07239 | Reference | — |
Finite lowered log loss against Baseline by 17.02% overall and 17.94% on learned-then-lost cases (162 cases, 648 tests, 78 recoveries). Baseline had lower log loss in runs 1118 and 1123. Against Baseline, the 95% interval for the Brier-score gain included zero. We saw interim results and completed all eight planned runs.
In M.4 we removed the trained update’s component along the initial target-score gradient projected into rank-one QKV adapter space. Of 80 recoveries, 61 survived and 19 were lost. There were four additional recoveries. All 23 flips followed the component’s sign (22 learned then lost, one taught but initially missed). Equal-energy random removal gave no flips. The sign-weighted random-minus-target difference was +1.014 points [0.086, 1.943] under both norm and calibration-KL matching (six positive runs, two ties). Reversing the component gave +1.985 [0.861, 3.109] in a secondary test. For the 23 flips, no other initially correct answer was lost beyond those lost under the intact update. Target-directed removals outside these 23 cases did produce additional losses.
H.3 (Table 10) predicts how much the transplant changes the recipient’s score response to a cue update. Against zero transplant effect, the combined predictor lowered pooled MSE by 9.69%. Adding alignment to scores and norms lowered pooled MSE by 3.36%, with higher error in one triplet. For the secondary four-arm interaction, its error was 15.71% above zero prediction (Table 11).
Table 2. C and W are old-minus-fresh recovery differences under correct and wrong labels, in percentage points. I = C − W. Within each run and initialization we matched all four updates to the lowest KL on 32 calibration questions (5% tolerance). We averaged initializations within runs before taking means and 95% t intervals (revised Table C3, p. 23).
| Panel and condition | C | W | I |
|---|---|---|---|
| 410M exposed failures, unscaled | 10.21 [5.08, 15.33] | −7.36 [−11.15, −3.57] | 17.57 [11.80, 23.33] |
| 410M exposed failures, matched KL | 10.99 [6.18, 15.79] | −5.73 [−9.12, −2.35] | 16.72 [11.20, 22.24] |
| 410M learned-then-lost, unscaled | 14.21 [6.45, 21.98] | −9.86 [−14.51, −5.22] | 24.08 [16.28, 31.87] |
| 410M learned-then-lost, matched KL | 15.36 [7.84, 22.89] | −8.24 [−12.47, −4.01] | 23.60 [15.49, 31.72] |
| 1.4B exposed failures, unscaled | 2.27 [−2.08, 6.63] | −4.30 [−6.77, −1.83] | 6.57 [1.61, 11.53] |
| 1.4B exposed failures, matched KL | 6.19 [2.73, 9.65] | −2.59 [−4.74, −0.43] | 8.77 [5.07, 12.48] |
All 216 updates at 410M met tolerance. Old/correct, old/wrong, fresh/correct and fresh/wrong mean KL went from 0.260/0.542/0.365/0.431 to 0.259/0.260/0.259/0.261. Matched I was positive/zero/negative in 24/2/1 runs for 383 trained failures and 23/3/1 for 263 learned-then-lost cases. At 1.4B, I was positive in all twelve runs for 318 trained failures. Identifying learned-then-lost cases there would require the missing initial-training endpoint. Before matching, the 1.4B interval for C included zero.
In Appendix J we gave the same wrong-label update to models taught different query–answer pairings. On average it weakened preference for each model’s assigned answer. The between-model contrasts were 0.447 [0.333, 0.561] nats/token on old cues and 0.345 [0.214, 0.476] on fresh cues. The old-minus-fresh difference in wrong-minus-correct responses was 0.080 [−0.042, 0.201].
Response to Authors
Dear authors,
I would like to thank the authors for your response and effort. I think your response "we could use the cue gradient to choose where to measure the answer scores, and those readings improved our predictions of the same ten-step outcome" mostly addressed my question Q1. However, I am still unsatisfied with the answer in Q2, I think the interpretation is still unclear to me. Overall, I think the responses are fair enough for me to raise the score from 2 to 3. I think the interpretation mentioned by Q2 should be examined better in camera-ready version.
Thank you for taking the time to read our response and consider the new experiments.
For Q2, we agree with your interpretation that the effect of a familiar cue depends on whether later training reinforces or conflicts with its earlier cue–answer pairing. In the matched-strength tests, switching from correct to wrong answers turned an advantage for old cues over fresh cues into a disadvantage (Table 2 of our second reply). We read that reversal as evidence that later training can help recovery when it reinforces an earlier cue–answer pairing and hinder recovery when it conflicts with that pairing. Familiarity alone gives no consistent advantage: old and fresh cues gave nearly identical recovery rates under correct labels in Appendix E, and the recovery gap between learned-then-lost and never-trained facts was slightly smaller with old correct cues than with fresh correct cues in the separate eight-run 1.4B study. We will make this interpretation clearer in §§4.2 and 7 of the camera-ready version.
Reply to Reviewer BMPR — answers (1/2)
Thank you for distinguishing a change in an answer’s score from recovery of the whole answer, and for suggesting how reinforcement and conflict might explain the cue results. For Q1, predicting which answers would return in full after ten steps of cue training showed no clear gain from adding local derivatives to the starting scores, gradient norms and other baseline features: log loss fell in six of eight runs, but the interval for the mean gain included zero (revised App. M.1, p. 44, lines 2322–2328). Original Fig. 3 established a gain in predicting how much more switching cue labels changed a target’s score margin on old cues than on fresh ones (original p. 6, lines 318–323, revised p. 7, lines 369–374).
However, we could use the cue gradient to choose where to measure the answer scores, and those readings improved our predictions of the same ten-step outcome. Along the direction of the first Adam step, the starting slopes tracked changes in token margins closely over a short displacement. Further along that direction, linear extrapolation missed 15 of the 49 recoveries observed at the displaced weights (revised App. M.2, lines 2374–2375). We therefore moved the weights temporarily by ten first-step lengths, read two scores for each answer there and restored the weights. We obtained both readings before observing any of the ten actual training steps. Adding them improved prediction over local features in all eight new runs and over the baseline in six, with a positive mean gain against both (revised §4.5, lines 410–419, Table 1 in our second reply).
To address your concern about the link to complete generation, we changed a trained update and found that removing a component identified before training changed which answers returned. For each target, we removed the component along its initial score gradient in the rank-one adapter space. Removing a component that favored the answer prevented 19 recoveries, while removing one that opposed it produced four additional recoveries. Neither matched random removal changed any target outcome. The initial gradient thus identifies a part of the trained update that can affect complete recovery, though this test cannot tell us how much of ordinary recovery depends on it (revised §4.5, lines 422–431).
We agree that the clearest effect of transplanting early layers is on which answer the recipient favors. When we changed the donor and kept the rest of the recipient fixed, its preference shifted toward the answer that donor had been taught. Even among answers the donor learned and lost, 36/40 gained in score over the original recipient. Those gains seldom carried the whole donor answer into generation (2/54 and 0/54 across the full panels). Wrong-label cue training subsequently weakened the transferred preference in all three donor–recipient triplets on the second panel (§5). Alignment modestly improved prediction of the transplant’s effect on the score response to an individual update (Table 10). For the four-arm comparison you cite, predicting zero gave lower error (Table 11).
In the fresh-target panel you cited in Q2, correct answers on old cues restored 14.7% of trained failures, against 8.6% for wrong answers on old cues. Yet correct answers on fresh cues also restored 14.7%, so only the label comparison met its criterion (original App. E, lines 1054–1058). Your interpretation led us to a further test: if old cues help by reinforcing earlier associations, switching them from correct to conflicting answers should reduce recovery more than making the same switch on new cues. We tested this in a separate panel, comparing all four updates on the same forgotten targets at matched strength, and the results support your reading. At 410M, old cues restored 15.4 percentage points more learned-then-lost answers than new cues under correct labels and 8.2 points fewer under wrong labels. We also found this reversal among previously trained, currently wrong answers in the twelve-run 1.4B panel (revised §4.2, lines 297–303, Table 2 in our second reply).
The other cases you identified help explain why the distinction matters. In the separate eight-run 1.4B behavioral panel, the recovery gap between learned-then-lost and never-trained answers was slightly smaller with old correct cues than with fresh correct cues. For facts the model already knew before training, the interaction in answer-content recovery even changed sign across question wordings (original F.1 and G.1, revised §4.2, lines 309–315). As you observed, at 410M the change in alignment on old cues was concentrated under wrong labels, with little change under correct labels. At 1.4B, correct labels contributed more to the old-minus-fresh alignment contrast (revised §4.4, lines 363–368). We now explain the cue results through reinforcement and conflict, rather than a general old-cue advantage (§§4.2 and 7, lines 309–315 and 519–522).
Do §4.5 and this reading address Q1 and Q2?