- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
- The authors have proposed a framework for understanding why on-policy learning reduces forgetting.
- They describe the mechanism for separating the influence of actions learned as part of a previous task A while solving a new task B.
- Given the mechanism, they propose ways of manipulating the effect of updates which would've reinforced the rehearsed actions of Task A while learning Task B, define how to find how much an update can reinforce the actions of the old task A ('selection'), provide theoretical results about when the updates can preserve the old task capability ('function'), and a test of whether immediate effects survive later on-policy training (i.e. 'persistence').
- They run experiments on OLMo checkpoints by doing training Math->Code and another training Base->Code, and on Control RL on Meta-World-based manipulation.
Critical strengths
- Though the ideas presented in the paper have been tangentially discussed elsewhere, this brings a novel presentation and perspective to the problem of identifying effect of historical learning and measuring it downstream.
- Instead of merely measuring projections, they propose a more nuanced re-weighting of the updates for analysis which they are able to reason about using Theorem 1.
- They are able to connect all three legs links together.
- The experiments are well designed and thought out, albeit limited to only one LLM family.
Critical weaknesses
- Some seminal papers from CL literature which deal with similar ideas should be referenced and discussed (mostly around Eqn 2, Section 2; which is without citations):
Farajtabar, M., Azizan, N., Mott, A.; Li, A.. (2020). Orthogonal Gradient Descent for Continual Learning. (orthogonal gradients projections with the explicit intention of avoiding forgetting) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi (2023) Editing Models with Task Arithmetic. (differences not in logits, but in weights)
-
The scale of experiments is smaller than desirable, e.g. Table 1, the differences of 1 problem being solved makes a difference of 3.3p.p.; so the different cells differ by at most 2 problems. This makes it difficult to judge the effect of FULL/SHUFFLED/TARGET on forgetting Math performance.
-
In a broader sense, the paper suppresses the reinforcement of history-favored tokens, and posits decay in historical expression AND old-task performance. There is evidence for decay of the historical expression, but the evidence on old-task performance is only suggestive, at least in the LLM setting.
Questions for authors
-
The statement of the theorems should be in the main body of the paper, even if the proofs are relegated to the appendix. That will make reading the material significantly easier.
-
I can make guesses about why the particular datasets and the model checkpoints were chosen, but it would be good to have them explicitly mentioned in the paper. In particular, if it is the paucity of the data which makes the scale of the experiments, it would be useful to call it out and perhaps work with larger datasets to showcase the differences.
-
TARGET deviates quite significantly from SHUFFLED in terms of KL ratio. Is there anything fundamental preventing a KL-matched version of the TARGET policy for LLMs similar to how a KL-matched policy was created for the Gaussian policy experiments?
- KL-matched credit updates, e.g. a per-arm step size so each update hits the same realised KL? - a KL-matched logit lesion along d^* in the LLM, mirroring the Gaussian lesions?
Minor nits on notation (main body alone): J is overloaded, and θ^{+} is unused.
7 comments
[W2]:
The scale of experiments is smaller than desirable, e.g. Table 1, the differences of 1 problem being solved makes a difference of 3.3p.p.; so the different cells differ by at most 2 problems. This makes it difficult to judge the effect of FULL/SHUFFLED/TARGET on forgetting Math performance.
(Table 1 of the original submission is Table 19 in the revised manuscript.)
[A2]: We expanded Math evaluation to AIME24+25 (60 questions × 128 responses). We also added an intervention that starts from the published Math checkpoint with freshly initialized optimizer state and continues for 100 Code updates. TARGET attenuates positive credit at history-associated positions; SHUFFLED redistributes the mask within strata on its own rollout. We average correctness within each question, then across questions. This measures solution reliability alongside the binary event of solving a problem at least once. (Table 1, Figure 3, and Appendix A.9.)
We measure KL on the same fixed Code prefixes in the direction initial Math policy→current policy.
| Condition | Initial-reference Code KL (fp32) | Math average correctness | HumanEval+ average correctness |
|---|---|---|---|
| Initial Math model | 0 | 27.98% | 72.41% |
| TARGET@100 | 0.09286 | 25.77% | 61.59% |
| SHUFFLED@100 | 0.11818 | 27.59% | 71.42% |
| FULL@100 | 0.0084 | 28.49% | 75.91% |
The Math evaluation uses temperature 0.6, top-p 0.95, and a 16,384-token budget. HumanEval+ uses 164 questions × 8 responses at temperature 1.
On Math, SHUFFLED and FULL exceed TARGET by 1.82 pp [0.59, 3.14] and 2.72 pp [1.16, 4.43], respectively. On HumanEval+, SHUFFLED exceeds TARGET by 9.83 pp [6.78, 12.96]. A SHUFFLED run with a 2.165× learning rate (SHUFLR2) has 33% higher endpoint KL than TARGET, yet still performs better on both Math (26.65% vs. 25.77%) and HumanEval+ (65.5% vs. 61.6%). This directly counters the explanation that TARGET's lower scores simply result from greater policy drift.
At the planned update-100 endpoint, TARGET has lower initial-reference Code KL than SHUFFLED despite lower Math and Code scores.
LLM confidence intervals resample paired questions.
Historical-expression readouts at the same endpoint are:
| Probe | Common start | TARGET@100 | SHUFFLED@100 |
|---|---|---|---|
| Code | 1 | −0.34 | 0.70 |
| AIME | 1 | 0.59 | 1.18 |
| MATH-L5 | 1 | 0.66 | 1.19 |
In the additional control-RL experiment, FULL, TARGET, and SHUFFLED start from independent copies of the same actor, critic, Adam state, and observation normalizer in each of 15 pick-and-place training histories. Each branch generates its own experience and retains its optimizer through 64 updates. (Figure 4(e,f) and Appendix A.6.)
| Group | Historical expression on the shared B probe at update 16 | Old-task success at update 64 |
|---|---|---|
| FULL | 0.957 | 51.50% |
| TARGET | 0.764 | 39.67% |
| SHUFFLED | 0.956 | 60.13% |
At update 16, the planned SHUFFLED−TARGET expression difference is +0.1924 [0.1344, 0.2527], positive in 15/15 histories. At update 64, SHUFFLED exceeds TARGET in old-task success by 20.46 pp [11.87, 28.35] (positive in 12/15 histories). FULL exceeds TARGET by 11.83 pp [0.72, 22.92].
Intervals resample complete paired training histories.
The submitted 16-seed pushing study tests function through statewise KL-matched directional lesions. The full-dose final-policy new-task contrast (orthogonal control−historical-direction lesion) is 8.39 pp [4.81, 12.32]. Both old- and new-task contrasts are positive in 13/16 seeds, supporting a functional role for the historical direction in both old and new tasks. (Figure 4(a,b) and Tables 8–9.)
[W3]:
- KL-matched credit updates, e.g. a per-arm step size so each update hits the same realised KL?
- a KL-matched logit lesion along
d*in the LLM, mirroring the Gaussian lesions?
[A3]: We examine this question through both single updates and sustained training. The historical-expression difference persists after matching the actual KL of single updates. In the additional 0–100-update Code RL experiment, TARGET also has lower historical expression and lower Math and Code performance than SHUFFLED.
Single-update credit interventions: the expression difference persists at matched KL. We numerically search for a scale along each group's actual update direction in eight independent Table 3 batches starting from the same update-400 state. With a fixed gradient and AdamW state and zero weight decay, scaling the one-step displacement is equivalent to adjusting the learning rate.
KL is matched on the fixed Code reference set. Expression differences are SHUFFLED−TARGET, with 95% confidence intervals from bootstrap resampling of the eight batches. All differences are positive in every batch (8/8 for each comparison and probe); the matched results and calibration checks are in Tables 4–5.
| Comparison | Expression probe | Mean expression difference [95% CI] |
|---|---|---|
| Original update | Code | 0.00501 [0.00469, 0.00536] |
| SHUFFLED enlarged to TARGET's KL | Code | 0.00391 [0.00363, 0.00417] |
| TARGET reduced to SHUFFLED's KL | Code | 0.00180 [0.00160, 0.00199] |
| Original update | AIME | 0.00080 [0.00076, 0.00083] |
| SHUFFLED enlarged to TARGET's KL | AIME | 0.00298 [0.00280, 0.00316] |
| TARGET reduced to SHUFFLED's KL | AIME | 0.00136 [0.00130, 0.00142] |
| Original update | MATH-L5 | 0.00051 [0.00047, 0.00055] |
| SHUFFLED enlarged to TARGET's KL | MATH-L5 | 0.00274 [0.00255, 0.00295] |
| TARGET reduced to SHUFFLED's KL | MATH-L5 | 0.00126 [0.00119, 0.00132] |
The table below reports the measured KL error for all four matching operations. Relative error is the absolute difference between achieved and target KL, divided by target KL; each row reports the maximum over eight batches.
| Matching operation | Batches within 1% KL tolerance | Maximum relative KL error |
|---|---|---|
| SHUFFLED enlarged to TARGET's KL | 8/8 | 0.601% |
| TARGET reduced to SHUFFLED's KL | 8/8 | 0.667% |
| FULL enlarged to TARGET's KL | 8/8 | 0.953% |
| TARGET reduced to FULL's KL | 8/8 | 0.595% |
| Total | 32/32 | 0.953% |
Credit interventions over updates 0–100: expression and task performance change together. The additional experiment in A2 starts FULL, TARGET, and SHUFFLED from the same published Math checkpoint and continues for 100 Code RL updates. Each group generates training data from its own current policy throughout. TARGET attenuates history-associated positive credit, while SHUFFLED redistributes the attenuation mask within strata on its own rollout. At update 100, TARGET has lower historical expression on all three probes. Its AIME24+25 and HumanEval+ correctness are lower than SHUFFLED's by 1.82 pp [0.59, 3.14] and 9.83 pp [6.78, 12.96], respectively. A2 provides the full means, evaluation protocols, and confidence-interval procedure.
This experiment directly tests how sustained changes to historical-credit allocation affect expression, old-task retention, and new-task performance. At the same endpoint, TARGET also has smaller initial-reference Code KL than SHUFFLED. We further compare the saved update-75 checkpoints from this run to examine expression and Code performance at similar initial-reference Code KL.
Additional comparison of existing update-75 checkpoints. The saved TARGET and SHUFFLED checkpoints have similar initial-reference KL at update 75. We measure this KL on the same fixed Code prefixes in the direction initial Math policy→current policy. We analyze their existing expression measurements and HumanEval+ evaluations (164 questions with eight responses per question, temperature 1). (Table 21.)
| Update-75 measure | TARGET | SHUFFLED |
|---|---|---|
| Initial-reference Code KL (fp32) | 0.10506 | 0.10835 |
| Code expression E | −0.329 | 0.800 |
| AIME expression E | 0.638 | 1.206 |
| MATH-L5 expression E | 0.702 | 1.221 |
| HumanEval+ average correctness | 63.11% | 73.40% |
The relative KL difference is 3.13%. TARGET has lower expression on all three probes and lower HumanEval+ correctness by 10.29 pp [7.16, 13.57] (paired question-bootstrap interval). The update-75 results come from a supplementary analysis of saved checkpoints after training. Update 100 remains the primary endpoint, with its Math evaluations reported separately.
We thank Reviewer 7aRm for the constructive and insightful feedback. We address your specific points directly below.
We have completed additional experiments and analyses since submission. The revised manuscript includes these results, experimental protocols, and statistical details. References below use its numbering unless explicitly identified as original-submission numbering.
[W1]:
Some seminal papers from CL literature which deal with similar ideas should be referenced and discussed (mostly around Eqn 2, Section 2; which is without citations):
(These remain Eq. (2) and Section 2 in the revised manuscript.)
[A1]: We have added both references and compared them with SFP in Section 2, near Eq. (2). Orthogonal Gradient Descent [1] projects training gradients to protect earlier-task outputs. Task arithmetic [2] uses parameter differences to represent tasks and edit models. Both are precedents for analyzing learned directions.
SFP tracks historical expression using a context-dependent output-logit difference and a fixed categorical Fisher metric. It analyzes how credit changes that expression, tests its function through task performance, and follows the effects through continued on-policy training. It thus applies direction-based analysis to the data-generation and update cycle of implicit rehearsal. Appendix C now distinguishes implicit rehearsal from the policy-drift explanations in Retaining by Doing [3] and RL's Razor [4].