acceptodds
Under review as a conference paper at ICLR 2027

From Re-expression to Retention: A Framework for Understanding Forgetting

Abstract

Why can on-policy learning reduce forgetting while a model adapts to new tasks? Prior studies identify on-policy data as a key factor and propose mode-seeking or KL-based explanations, but the mechanism linking new-task updates to the preservation of old capabilities remains insufficiently understood. We propose **implicit rehearsal** as a mechanism for this retention benefit: the current policy carries previously learned behaviors into newly generated responses and actions, allowing them to participate in new-task training. When these behaviors help solve the new task, feedback can reinforce their use, and the updated policy can reuse them in subsequent trajectories. This recurring process provides a route to preserving skills shared by old and new tasks without replaying old-task examples. We introduce the **Selection, Function, and Persistence** (SFP) framework to formalize this mechanism. Selection analyzes which historical contributions new-task updates reinforce; Function connects their preservation to old-task capability; Persistence tracks how their effects evolve as the policy and its training data change together. We derive conditions under which reinforcing historical behavior improves retention after further training. Complementary experiments in language models and control reinforcement learning provide evidence for the framework's components.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Reinforcement Learning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_7aRmOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary
  • The authors have proposed a framework for understanding why on-policy learning reduces forgetting.
  • They describe the mechanism for separating the influence of actions learned as part of a previous task A while solving a new task B.
  • Given the mechanism, they propose ways of manipulating the effect of updates which would've reinforced the rehearsed actions of Task A while learning Task B, define how to find how much an update can reinforce the actions of the old task A ('selection'), provide theoretical results about when the updates can preserve the old task capability ('function'), and a test of whether immediate effects survive later on-policy training (i.e. 'persistence').
  • They run experiments on OLMo checkpoints by doing training Math->Code and another training Base->Code, and on Control RL on Meta-World-based manipulation.
Critical strengths
  • Though the ideas presented in the paper have been tangentially discussed elsewhere, this brings a novel presentation and perspective to the problem of identifying effect of historical learning and measuring it downstream.
  • Instead of merely measuring projections, they propose a more nuanced re-weighting of the updates for analysis which they are able to reason about using Theorem 1.
  • They are able to connect all three legs links together.
  • The experiments are well designed and thought out, albeit limited to only one LLM family.
Critical weaknesses
  • Some seminal papers from CL literature which deal with similar ideas should be referenced and discussed (mostly around Eqn 2, Section 2; which is without citations):

Farajtabar, M., Azizan, N., Mott, A.; Li, A.. (2020). Orthogonal Gradient Descent for Continual Learning. (orthogonal gradients projections with the explicit intention of avoiding forgetting) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi (2023) Editing Models with Task Arithmetic. (differences not in logits, but in weights)

  • The scale of experiments is smaller than desirable, e.g. Table 1, the differences of 1 problem being solved makes a difference of 3.3p.p.; so the different cells differ by at most 2 problems. This makes it difficult to judge the effect of FULL/SHUFFLED/TARGET on forgetting Math performance.

  • In a broader sense, the paper suppresses the reinforcement of history-favored tokens, and posits decay in historical expression AND old-task performance. There is evidence for decay of the historical expression, but the evidence on old-task performance is only suggestive, at least in the LLM setting.

Questions for authors
  • The statement of the theorems should be in the main body of the paper, even if the proofs are relegated to the appendix. That will make reading the material significantly easier.

  • I can make guesses about why the particular datasets and the model checkpoints were chosen, but it would be good to have them explicitly mentioned in the paper. In particular, if it is the paucity of the data which makes the scale of the experiments, it would be useful to call it out and perhaps work with larger datasets to showcase the differences.

  • TARGET deviates quite significantly from SHUFFLED in terms of KL ratio. Is there anything fundamental preventing a KL-matched version of the TARGET policy for LLMs similar to how a KL-matched policy was created for the Gaussian policy experiments?

    - KL-matched credit updates, e.g. a per-arm step size so each update hits the same realised KL?
    - a KL-matched logit lesion along d^* in the LLM, mirroring the Gaussian lesions?
    

Minor nits on notation (main body alone): J is overloaded, and θ^{+} is unused.

7 comments

We thank Reviewer 7aRm for the constructive and insightful feedback. We address your specific points directly below.

We have completed additional experiments and analyses since submission. The revised manuscript includes these results, experimental protocols, and statistical details. References below use its numbering unless explicitly identified as original-submission numbering.

[W1]:

Some seminal papers from CL literature which deal with similar ideas should be referenced and discussed (mostly around Eqn 2, Section 2; which is without citations):

(These remain Eq. (2) and Section 2 in the revised manuscript.)

[A1]: We have added both references and compared them with SFP in Section 2, near Eq. (2). Orthogonal Gradient Descent [1] projects training gradients to protect earlier-task outputs. Task arithmetic [2] uses parameter differences to represent tasks and edit models. Both are precedents for analyzing learned directions.

SFP tracks historical expression using a context-dependent output-logit difference and a fixed categorical Fisher metric. It analyzes how credit changes that expression, tests its function through task performance, and follows the effects through continued on-policy training. It thus applies direction-based analysis to the data-generation and update cycle of implicit rehearsal. Appendix C now distinguishes implicit rehearsal from the policy-drift explanations in Retaining by Doing [3] and RL's Razor [4].

[W2]:

The scale of experiments is smaller than desirable, e.g. Table 1, the differences of 1 problem being solved makes a difference of 3.3p.p.; so the different cells differ by at most 2 problems. This makes it difficult to judge the effect of FULL/SHUFFLED/TARGET on forgetting Math performance.

(Table 1 of the original submission is Table 19 in the revised manuscript.)

[A2]: We expanded Math evaluation to AIME24+25 (60 questions × 128 responses). We also added an intervention that starts from the published Math checkpoint with freshly initialized optimizer state and continues for 100 Code updates. TARGET attenuates positive credit at history-associated positions; SHUFFLED redistributes the mask within strata on its own rollout. We average correctness within each question, then across questions. This measures solution reliability alongside the binary event of solving a problem at least once. (Table 1, Figure 3, and Appendix A.9.)

We measure KL on the same fixed Code prefixes in the direction initial Math policy→current policy.

ConditionInitial-reference Code KL (fp32)Math average correctnessHumanEval+ average correctness
Initial Math model027.98%72.41%
TARGET@1000.0928625.77%61.59%
SHUFFLED@1000.1181827.59%71.42%
FULL@1000.008428.49%75.91%

The Math evaluation uses temperature 0.6, top-p 0.95, and a 16,384-token budget. HumanEval+ uses 164 questions × 8 responses at temperature 1.

On Math, SHUFFLED and FULL exceed TARGET by 1.82 pp [0.59, 3.14] and 2.72 pp [1.16, 4.43], respectively. On HumanEval+, SHUFFLED exceeds TARGET by 9.83 pp [6.78, 12.96]. A SHUFFLED run with a 2.165× learning rate (SHUFLR2) has 33% higher endpoint KL than TARGET, yet still performs better on both Math (26.65% vs. 25.77%) and HumanEval+ (65.5% vs. 61.6%). This directly counters the explanation that TARGET's lower scores simply result from greater policy drift.

At the planned update-100 endpoint, TARGET has lower initial-reference Code KL than SHUFFLED despite lower Math and Code scores.

LLM confidence intervals resample paired questions.

Historical-expression readouts at the same endpoint are:

ProbeCommon startTARGET@100SHUFFLED@100
Code1−0.340.70
AIME10.591.18
MATH-L510.661.19

In the additional control-RL experiment, FULL, TARGET, and SHUFFLED start from independent copies of the same actor, critic, Adam state, and observation normalizer in each of 15 pick-and-place training histories. Each branch generates its own experience and retains its optimizer through 64 updates. (Figure 4(e,f) and Appendix A.6.)

GroupHistorical expression on the shared B probe at update 16Old-task success at update 64
FULL0.95751.50%
TARGET0.76439.67%
SHUFFLED0.95660.13%

At update 16, the planned SHUFFLED−TARGET expression difference is +0.1924 [0.1344, 0.2527], positive in 15/15 histories. At update 64, SHUFFLED exceeds TARGET in old-task success by 20.46 pp [11.87, 28.35] (positive in 12/15 histories). FULL exceeds TARGET by 11.83 pp [0.72, 22.92].

Intervals resample complete paired training histories.

The submitted 16-seed pushing study tests function through statewise KL-matched directional lesions. The full-dose final-policy new-task contrast (orthogonal control−historical-direction lesion) is 8.39 pp [4.81, 12.32]. Both old- and new-task contrasts are positive in 13/16 seeds, supporting a functional role for the historical direction in both old and new tasks. (Figure 4(a,b) and Tables 8–9.)

[W3]:

  • KL-matched credit updates, e.g. a per-arm step size so each update hits the same realised KL?
  • a KL-matched logit lesion along d* in the LLM, mirroring the Gaussian lesions?

[A3]: We examine this question through both single updates and sustained training. The historical-expression difference persists after matching the actual KL of single updates. In the additional 0–100-update Code RL experiment, TARGET also has lower historical expression and lower Math and Code performance than SHUFFLED.

Single-update credit interventions: the expression difference persists at matched KL. We numerically search for a scale along each group's actual update direction in eight independent Table 3 batches starting from the same update-400 state. With a fixed gradient and AdamW state and zero weight decay, scaling the one-step displacement is equivalent to adjusting the learning rate.

KL is matched on the fixed Code reference set. Expression differences are SHUFFLED−TARGET, with 95% confidence intervals from bootstrap resampling of the eight batches. All differences are positive in every batch (8/8 for each comparison and probe); the matched results and calibration checks are in Tables 4–5.

ComparisonExpression probeMean expression difference [95% CI]
Original updateCode0.00501 [0.00469, 0.00536]
SHUFFLED enlarged to TARGET's KLCode0.00391 [0.00363, 0.00417]
TARGET reduced to SHUFFLED's KLCode0.00180 [0.00160, 0.00199]
Original updateAIME0.00080 [0.00076, 0.00083]
SHUFFLED enlarged to TARGET's KLAIME0.00298 [0.00280, 0.00316]
TARGET reduced to SHUFFLED's KLAIME0.00136 [0.00130, 0.00142]
Original updateMATH-L50.00051 [0.00047, 0.00055]
SHUFFLED enlarged to TARGET's KLMATH-L50.00274 [0.00255, 0.00295]
TARGET reduced to SHUFFLED's KLMATH-L50.00126 [0.00119, 0.00132]

The table below reports the measured KL error for all four matching operations. Relative error is the absolute difference between achieved and target KL, divided by target KL; each row reports the maximum over eight batches.

Matching operationBatches within 1% KL toleranceMaximum relative KL error
SHUFFLED enlarged to TARGET's KL8/80.601%
TARGET reduced to SHUFFLED's KL8/80.667%
FULL enlarged to TARGET's KL8/80.953%
TARGET reduced to FULL's KL8/80.595%
Total32/320.953%

Credit interventions over updates 0–100: expression and task performance change together. The additional experiment in A2 starts FULL, TARGET, and SHUFFLED from the same published Math checkpoint and continues for 100 Code RL updates. Each group generates training data from its own current policy throughout. TARGET attenuates history-associated positive credit, while SHUFFLED redistributes the attenuation mask within strata on its own rollout. At update 100, TARGET has lower historical expression on all three probes. Its AIME24+25 and HumanEval+ correctness are lower than SHUFFLED's by 1.82 pp [0.59, 3.14] and 9.83 pp [6.78, 12.96], respectively. A2 provides the full means, evaluation protocols, and confidence-interval procedure.

This experiment directly tests how sustained changes to historical-credit allocation affect expression, old-task retention, and new-task performance. At the same endpoint, TARGET also has smaller initial-reference Code KL than SHUFFLED. We further compare the saved update-75 checkpoints from this run to examine expression and Code performance at similar initial-reference Code KL.

Additional comparison of existing update-75 checkpoints. The saved TARGET and SHUFFLED checkpoints have similar initial-reference KL at update 75. We measure this KL on the same fixed Code prefixes in the direction initial Math policy→current policy. We analyze their existing expression measurements and HumanEval+ evaluations (164 questions with eight responses per question, temperature 1). (Table 21.)

Update-75 measureTARGETSHUFFLED
Initial-reference Code KL (fp32)0.105060.10835
Code expression E−0.3290.800
AIME expression E0.6381.206
MATH-L5 expression E0.7021.221
HumanEval+ average correctness63.11%73.40%

The relative KL difference is 3.13%. TARGET has lower expression on all three probes and lower HumanEval+ correctness by 10.29 pp [7.16, 13.57] (paired question-bootstrap interval). The update-75 results come from a supplementary analysis of saved checkpoints after training. Update 100 remains the primary endpoint, with its Math evaluations reported separately.

Reviewer_Pk9GOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why on-policy learning can mitigate catastrophic forgetting. The authors propose implicit rehearsal, where previously learned behaviors are reused and reinforced during new-task training. They introduce the Selection Function Persistence (SFP) framework to explain how historical behaviors receive training credit, contribute to old-task capabilities, and persist through subsequent updates. The framework is supported by theoretical analysis and experiments on language models and control RL tasks.

Critical strengths

The paper studies an interesting question of why on-policy data can help mitigate forgetting. It shows that new-task trajectories can contain behaviors learned from previous tasks, enabling implicit rehearsal of previously learned capabilities.

Critical weaknesses
  1. Retaining by Doing has already shown that on-policy data plays an important role in reducing forgetting, while RL's Razor provides a KL-based explanation. This paper argues that reinforcing historical behaviors helps preserve old-task capabilities. However, in Table 3, TARGET produces 3.55–6.35 times larger policy KL than SHUFFLED. Therefore, it is unclear whether the decrease in historical expression is caused by suppressing historical credit or simply by larger policy changes.
  2. The paper identifies history-favored tokens by comparing token probabilities between Math2Code and Base2Code models. However, a higher token probability does not necessarily mean that the token represents a skill learned from Math training. The paper does not directly verify that these identified tokens are responsible for the model's Math performance.
  3. Although the LLM experiments show that suppressing historical credit reduces historical expression and leads to lower subsequent Math performance, it remains unclear whether the reduction in Math performance is directly caused by the loss of historical expression or by other changes in the policy updates.
Questions for authors
  1. Can the authors compare TARGET and SHUFFLED under matched policy KL? Does suppressing historical credit still reduce historical expression and old-task performance?
  2. How can the authors verify that the identified history-favored tokens actually represent skills learned from the old task? Can they directly demonstrate that these tokens contribute to Math performance?
  3. The LLM experiments show that suppressing historical credit reduces both historical expression and subsequent Math performance. How can the authors demonstrate that the decrease in Math performance is caused by the loss of historical expression, rather than other changes introduced by the intervention?
9 comments

We thank Reviewer Pk9G for the constructive and insightful feedback. We address your specific points directly below.

We have completed additional experiments and analyses since submission. The revised manuscript includes these results, experimental protocols, and statistical details. References below use its numbering unless explicitly identified as original-submission numbering.

[W1]:

Can the authors compare TARGET and SHUFFLED under matched policy KL? Does suppressing historical credit still reduce historical expression and old-task performance?

[A1]: Yes for expression under matched single-update KL; the sustained-training results also show lower old-task performance despite lower endpoint KL. In all 48 matched comparisons, TARGET leaves less historical expression than SHUFFLED. At update 75, TARGET and SHUFFLED have similar initial-reference Code KL (fp32; 3.13% relative difference), yet TARGET has lower expression on all three probes and lower HumanEval+ correctness (63.11% vs. 73.40%). Over 100 updates, TARGET ends with lower KL yet lower Math and Code performance (Math 25.77% vs. 27.59%; HumanEval+ 61.6% vs. 71.4%). A SHUFFLED run with 33% higher endpoint KL than TARGET also exceeds it (Math 26.65%; HumanEval+ 65.5%). (Table 1, Figure 3, and Appendix A.9.)

We compare TARGET and SHUFFLED on the same eight single-update batches in Table 3, using the fixed reference set. For each group, we search for a scale along its actual update direction. With a fixed gradient and AdamW state and zero weight decay, this is equivalent to adjusting the learning rate.

KL is matched on the fixed Code reference set. Expression differences are SHUFFLED−TARGET, with 95% confidence intervals from bootstrap resampling of the eight batches. All differences are positive in every batch (8/8 for each comparison and probe); the matched results and calibration checks are in Tables 4–5.

ComparisonExpression probeMean expression difference [95% CI]
Original updateCode0.00501 [0.00469, 0.00536]
SHUFFLED enlarged to TARGET's KLCode0.00391 [0.00363, 0.00417]
TARGET reduced to SHUFFLED's KLCode0.00180 [0.00160, 0.00199]
Original updateAIME0.00080 [0.00076, 0.00083]
SHUFFLED enlarged to TARGET's KLAIME0.00298 [0.00280, 0.00316]
TARGET reduced to SHUFFLED's KLAIME0.00136 [0.00130, 0.00142]
Original updateMATH-L50.00051 [0.00047, 0.00055]
SHUFFLED enlarged to TARGET's KLMATH-L50.00274 [0.00255, 0.00295]
TARGET reduced to SHUFFLED's KLMATH-L50.00126 [0.00119, 0.00132]

The table below reports the measured KL error for all four matching operations. Relative error is the absolute difference between achieved and target KL, divided by target KL; each row reports the maximum over eight batches.

Matching operationBatches within 1% KL toleranceMaximum relative KL error
SHUFFLED enlarged to TARGET's KL8/80.601%
TARGET reduced to SHUFFLED's KL8/80.667%
FULL enlarged to TARGET's KL8/80.953%
TARGET reduced to FULL's KL8/80.595%
Total32/320.953%

We search the matching scale separately at each fixed starting state. Extending this step-size search to continued training is feasible in principle, but it requires additional KL evaluations and parameter adjustments at every update, making a full rerun computationally costly. For continued training, we also compare the saved update-75 checkpoints with similar initial-reference Code KL.

Additional comparison of existing update-75 checkpoints. The saved TARGET and SHUFFLED checkpoints have similar initial-reference KL at update 75. We measure this KL on the same fixed Code prefixes in the direction initial Math policy→current policy. We analyze their existing expression measurements and HumanEval+ evaluations (164 questions with eight responses per question, temperature 1). (Table 21.)

Update-75 measureTARGETSHUFFLED
Initial-reference Code KL (fp32)0.105060.10835
Code expression E−0.3290.800
AIME expression E0.6381.206
MATH-L5 expression E0.7021.221
HumanEval+ average correctness63.11%73.40%

The relative KL difference is 3.13%. TARGET has lower expression on all three probes and lower HumanEval+ correctness by 10.29 pp [7.16, 13.57] (paired question-bootstrap interval). The update-75 results come from a supplementary analysis of saved checkpoints after training. Update 100 remains the primary endpoint, with its Math evaluations reported separately.

At update 100, TARGET also has lower task performance despite a smaller initial-reference Code KL (A2). Together, these results show that a larger measured policy KL alone does not explain TARGET's lower expression and task performance.

[W2]:

How can the authors verify that the identified history-favored tokens actually represent skills learned from the old task? Can they directly demonstrate that these tokens contribute to Math performance?

[A2]: Yes, and we test it by intervention rather than by inspecting tokens. By construction, the history score marks where Math training changed the policy's preferences (Math-path vs. Base-path model at the same prefix). Attenuating Code credit on exactly these positions lowers Math correctness relative to attenuating the same amount of credit at shuffled positions (25.77% vs. 27.59%), and lowers Math-prefix expression (AIME probe 0.59 vs. 1.18). The control-RL lesions test the function directly: perturbing along the historical direction harms task success more than KL-matched orthogonal perturbations.

The new step-0 experiment starts from the published Math checkpoint with freshly initialized optimizer state and continues for 100 Code RL updates. TARGET attenuates positive credit at history-associated positions; SHUFFLED redistributes the mask within strata on its own rollout. Historical-expression readouts at the same endpoint are:

ProbeCommon startTARGET@100SHUFFLED@100
Code1−0.340.70
AIME10.591.18
MATH-L510.661.19

Independent Math evaluation uses AIME24+25 (60 questions × 128 responses, temperature 0.6, top-p 0.95, 16,384-token budget). HumanEval+ uses 164 questions × 8 responses at temperature 1. Scores average response correctness within questions, then equally across questions.

We measure KL on the same fixed Code prefixes in the direction initial Math policy→current policy.

ConditionInitial-reference Code KL (fp32)Math average correctnessHumanEval+ average correctness
Initial Math model027.98%72.41%
TARGET@1000.0928625.77%61.59%
SHUFFLED@1000.1181827.59%71.42%
FULL@1000.008428.49%75.91%

On Math, SHUFFLED and FULL exceed TARGET by 1.82 pp [0.59, 3.14] and 2.72 pp [1.16, 4.43], respectively. On HumanEval+, SHUFFLED exceeds TARGET by 9.83 pp [6.78, 12.96]. A SHUFFLED run with a 2.165× learning rate (SHUFLR2) has 33% higher endpoint KL than TARGET, yet still performs better on both Math (26.65% vs. 25.77%) and HumanEval+ (65.5% vs. 61.6%). This directly counters the explanation that TARGET's lower scores simply result from greater policy drift.

At the planned update-100 endpoint, TARGET has lower initial-reference Code KL than SHUFFLED despite lower Math and Code scores.

Intervals resample paired questions.

In the submitted 16-seed control experiment, the full-dose final-policy new-task contrast (orthogonal control−historical-direction lesion) at matched statewise Gaussian KL is 8.39 pp [4.81, 12.32]. Both old- and new-task contrasts are positive in 13/16 seeds, supporting a functional role for the historical direction in both old and new tasks. (Figure 4(a,b) and Tables 8–9.)

[W3]:

How can the authors demonstrate that the decrease in Math performance is caused by the loss of historical expression, rather than other changes introduced by the intervention?

[A3]: The SHUFFLED control tests whether the placement of attenuated credit matters beyond the attenuation operation itself. TARGET and SHUFFLED use the same attenuation form and maximum dose, but place it at history-associated versus shuffled positions. TARGET also has lower endpoint KL than SHUFFLED at update 100 despite lower Math correctness, so a larger endpoint drift alone does not explain the decrease. The comparison links credit placement to both lower expression and lower Math performance; it does not isolate expression as the sole mediator of the performance change.

In the 100-update Code RL experiment in A2, FULL, TARGET, and SHUFFLED start from the same published Math checkpoint and continue generating training data from their respective current policies. TARGET attenuates positive credit according to the historical mask. SHUFFLED retains the same attenuation form and maximum dose while redistributing the mask within strata on its own rollout. FULL provides the standard-continuation reference.

A2 shows that TARGET has lower historical expression than SHUFFLED on both the AIME and MATH-L5 probes. In the independent AIME24+25 evaluation, SHUFFLED and FULL exceed TARGET in Math correctness by 1.82 pp [0.59, 3.14] and 2.72 pp [1.16, 4.43], respectively. The Math-expression differences thus accompany Math-performance differences at the same training endpoint.

To examine the policy-drift explanation, the single-update experiment at checkpoint 400 in A1 matches output KL on the fixed Code reference set: the SHUFFLED−TARGET expression difference remains positive in 8/8 batches on both Math probes. At the 100-update endpoint, TARGET has lower Math performance despite a smaller initial-reference Code KL than SHUFFLED (A2). In the 100-update LLM continuation, sustained targeted attenuation of history-associated positive credit reduces Math expression and final Math performance relative to shuffled attenuation. A SHUFFLED run with a 2.165× learning rate (SHUFLR2) has 33% higher endpoint KL than TARGET, yet still performs better on both Math (26.65% vs. 25.77%) and HumanEval+ (65.5% vs. 61.6%). This directly counters the explanation that TARGET's lower scores simply result from greater policy drift.

Reviewer_wKuZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why on-policy learning can reduce catastrophic forgetting. The main hypothesis is implicit rehearsal: behaviors learned from previous tasks are expressed during new-task training, and when these behaviors are useful for the new task, they receive positive credit and are reinforced.

The authors propose the Selection, Function, and Persistence (SFP) framework to formalize this idea, and provide theoretical analysis along with experiments on LLM math-to-code training and control RL tasks. The experiments show that new-task credit can preserve historical behavior, and that some of these behaviors are useful for both old and new tasks.

Critical strengths

I think the idea of implicit rehearsal is interesting, and provides a useful perspective on why on-policy learning can preserve previously learned capabilities. In particular, the distinction between historical behavior being expressed and actually being useful for the old task is important.

I like that the authors try to study this mechanism through controlled interventions, rather than just measuring forgetting. The experiments showing that attenuating credit on history-favored tokens reduces their subsequent expression are particularly interesting.

Critical weaknesses

My main concern is that the experiments don't fully establish the proposed mechanism. In the LLM experiments, reducing historical expression also changes the overall policy update, and the authors report substantially different KL between the interventions. So it is unclear how much of the observed effect comes from selectively removing historical credit versus simply changing the magnitude of the update.

The connection between historical expression and actual forgetting is still somewhat weak. The LLM experiments show a large difference in historical expression, but the corresponding Math performance differences are much smaller. Also, the main Math-to-Code experiment only has one training run per initialization. I think stronger evidence connecting the proposed mechanism to actual retention would make the paper much more convincing.

Questions for authors

Can the authors compare the targeted attenuation against controls with matched policy KL or update norm? This would help clarify whether the effect is really due to the selection of historical credit.

Can the authors provide more evidence that preserving historical expression actually improves old-task performance? In particular, it would be useful to see whether the effect holds across multiple training seeds and task pairs.

5 comments

We thank Reviewer wKuZ for the constructive and insightful feedback. We address your specific points directly below.

We have completed additional experiments and analyses since submission. The revised manuscript includes these results, experimental protocols, and statistical details. References below use its numbering unless explicitly identified as original-submission numbering.

[W1]:

Can the authors compare the targeted attenuation against controls with matched policy KL or update norm?

[A1]: The historical-expression difference persists in both directions of matching on realized policy KL, as you suggested. Using the eight single-update batches starting from the same training state in Table 3, we hold each group's actual update direction fixed and scale it until its output KL matches the comparison group within 1% on the fixed reference set. In this one-step AdamW configuration, the gradient and optimizer state are fixed and weight decay is zero, so scaling is equivalent to adjusting the learning rate.

KL is matched on the fixed Code reference set. Expression differences are SHUFFLED−TARGET, with 95% confidence intervals from bootstrap resampling of the eight batches. All differences are positive in every batch (8/8 for each comparison and probe); the matched results and calibration checks are in Tables 4–5.

ComparisonExpression probeMean expression difference [95% CI]
Original updateCode0.00501 [0.00469, 0.00536]
SHUFFLED enlarged to TARGET's KLCode0.00391 [0.00363, 0.00417]
TARGET reduced to SHUFFLED's KLCode0.00180 [0.00160, 0.00199]
Original updateAIME0.00080 [0.00076, 0.00083]
SHUFFLED enlarged to TARGET's KLAIME0.00298 [0.00280, 0.00316]
TARGET reduced to SHUFFLED's KLAIME0.00136 [0.00130, 0.00142]
Original updateMATH-L50.00051 [0.00047, 0.00055]
SHUFFLED enlarged to TARGET's KLMATH-L50.00274 [0.00255, 0.00295]
TARGET reduced to SHUFFLED's KLMATH-L50.00126 [0.00119, 0.00132]

The table below reports the measured KL error for all four matching operations. Relative error is the absolute difference between achieved and target KL, divided by target KL; each row reports the maximum over eight batches.

Matching operationBatches within 1% KL toleranceMaximum relative KL error
SHUFFLED enlarged to TARGET's KL8/80.601%
TARGET reduced to SHUFFLED's KL8/80.667%
FULL enlarged to TARGET's KL8/80.953%
TARGET reduced to FULL's KL8/80.595%
Total32/320.953%

We search the matching scale separately at each fixed starting state. Extending this step-size search to continued training is feasible in principle, but it requires additional KL evaluations and parameter adjustments at every update, making a full rerun computationally costly.

Update-75 comparison from the new 100-update experiment. We also ran a 100-update Code RL experiment starting from the published Math checkpoint, comparing FULL, TARGET, and SHUFFLED. A2 presents the full setup and update-100 results. Here we first compare the saved update-75 checkpoints, where TARGET and SHUFFLED have similar initial-reference KL. We measure this KL on the same fixed Code prefixes in the direction initial Math policy→current policy. We analyze their existing expression measurements and HumanEval+ evaluations (164 questions with eight responses per question, temperature 1). (Table 1, Figure 3, and Appendix A.9.) (Table 21.)

Update-75 measureTARGETSHUFFLED
Initial-reference Code KL (fp32)0.105060.10835
Code expression E−0.3290.800
AIME expression E0.6381.206
MATH-L5 expression E0.7021.221
HumanEval+ average correctness63.11%73.40%

The relative KL difference is 3.13%. TARGET has lower expression on all three probes and lower HumanEval+ correctness by 10.29 pp [7.16, 13.57] (paired question-bootstrap interval). The update-75 results come from a supplementary analysis of saved checkpoints after training. Update 100 remains the primary endpoint, with its Math evaluations reported separately.

At update 100, TARGET also has lower task performance despite a smaller initial-reference Code KL (A2). Together, these results show that a larger measured policy KL alone does not explain TARGET's lower expression and task performance.

[W2]:

Can the authors provide more evidence that preserving historical expression actually improves old-task performance? In particular, it would be useful to see whether the effect holds across multiple training seeds and task pairs.

[A2]: In the additional control-RL experiment, sustained targeted attenuation of historical credit yields lower mean old-task success for TARGET than for FULL and SHUFFLED.

We study 15 paired training histories. FULL, TARGET, and SHUFFLED start from independent copies of the same actor, critic, Adam state, and observation normalizer in each pick-and-place history, then continue for 64 on-policy updates. Each branch keeps its own optimizer and generates its own data. (Figure 4(e,f) and Appendix A.6.)

GroupHistorical expression on the shared B probe at update 16Old-task success at update 64
FULL0.95751.50%
TARGET0.76439.67%
SHUFFLED0.95660.13%

At update 16, the planned SHUFFLED−TARGET expression difference is +0.1924 [0.1344, 0.2527], positive in 15/15 histories. At update 64, SHUFFLED exceeds TARGET in old-task success by 20.46 pp [11.87, 28.35] (positive in 12/15 histories). FULL exceeds TARGET by 11.83 pp [0.72, 22.92].

Intervals resample complete paired training histories.

In the submitted 16-seed pushing study, statewise KL-matched directional lesions give a full-dose final-policy new-task contrast (orthogonal control−historical-direction lesion) of 8.39 pp [4.81, 12.32]. Both old- and new-task contrasts are positive in 13/16 seeds, supporting a functional role for the historical direction in both old and new tasks. (Figure 4(a,b) and Tables 8–9.)

For LLMs, we intervene from step 0 of Math→Code. We start from the published Math checkpoint with freshly initialized optimizer state and continue for 100 Code updates. TARGET attenuates positive credit at history-associated positions; SHUFFLED redistributes the mask within strata on its own rollout. Evaluation uses AIME24+25, with 60 questions × 128 responses.

We measure KL on the same fixed Code prefixes in the direction initial Math policy→current policy.

ConditionInitial-reference Code KL (fp32)Math average correctnessHumanEval+ average correctness
Initial Math model027.98%72.41%
TARGET@1000.0928625.77%61.59%
SHUFFLED@1000.1181827.59%71.42%
FULL@1000.008428.49%75.91%

We average response correctness within each question, then equally across questions. Math evaluation uses temperature 0.6, top-p 0.95, and a 16,384-token budget. HumanEval+ uses 164 questions × 8 responses at temperature 1.

On Math, SHUFFLED and FULL exceed TARGET by 1.82 pp [0.59, 3.14] and 2.72 pp [1.16, 4.43], respectively. On HumanEval+, SHUFFLED exceeds TARGET by 9.83 pp [6.78, 12.96]. A SHUFFLED run with a 2.165× learning rate (SHUFLR2) has 33% higher endpoint KL than TARGET, yet still performs better on both Math (26.65% vs. 25.77%) and HumanEval+ (65.5% vs. 61.6%). This directly counters the explanation that TARGET's lower scores simply result from greater policy drift.

At the planned update-100 endpoint, TARGET has lower initial-reference Code KL than SHUFFLED despite lower Math and Code scores.

The LLM intervals resample paired questions.

Historical-expression readouts at the same endpoint are:

ProbeCommon startTARGET@100SHUFFLED@100
Code1−0.340.70
AIME10.591.18
MATH-L510.661.19

In the LLM step-0 experiment, each primary condition is a single training run (24 GPUs per run). An identical-configuration rerun of SHUFFLED gives similar results at update 100 (HumanEval+ 70.6% vs. 71.4%; Code expression 0.68 vs. 0.70); the differences between runs are much smaller than their gaps to TARGET (about 9–10 pp and 1.0, respectively). Replication across seeds and task pairs is provided by the control-RL studies (16 pushing seeds; 15 pick-and-place histories).

Additional experimental evidence

The following completed analyses are also included in the revised manuscript.

Local prediction on actual updates. We check the first-order expansion in Appendix B.4 (Eq. (52)) against the saved Table 3 updates. We compare ∇EᵀΔθ at the starting point with the measured expression change. Each of the eight batches executes one FULL, TARGET, and SHUFFLED update from the same Math→Code@400 state, giving 24 displacements. The readout follows Eq. (2), with 100 fixed reference responses per probe; the prediction results are shown in Figure 6.

Expression probeCorrect change directionMedian relative errorMaximum relative error
Code24/240.707%11.200%
MATH-L524/240.317%1.869%
AIME24/240.503%2.499%

Relative error is the absolute prediction error divided by the absolute measured change. The direction agreement across probes supports the local expression approximation at the actual small-update scale.

Independent reconstruction and position-level decomposition. We independently reconstruct the historical-expression readout in Eq. (2) and decompose it algebraically using the fixed probes in Appendix A.1 (Table 6). Historical directions, Fisher weights, and the Base→Code reference remain fixed at update 400. We recompute all three readouts for TARGET and SHUFFLED at updates 430, 450, 470, and 500, giving 24 aggregate values. We also decompose TARGET's 430-to-500 change across positions and vocabulary entries.

Each probe contains 100 saved reference responses. We average within each response, then equally across responses; the 100 AIME responses cover 60 distinct problems.

Fixed probeMaximum aggregate reconstruction errorTARGET ΔE (430→500)Reference responses with decreasing expressionTop 1% position contribution share
Code1.58 × 10⁻¹⁰−0.2900100/10013.99%
AIME1.24 × 10⁻¹⁰−0.2422100/10010.20%
MATH-L56.44 × 10⁻¹¹−0.2093100/10011.37%

We rank the top 1% by absolute weighted position contribution; their share uses the sum of absolute contributions across all positions as the denominator. Expression declines across all saved reference responses and many positions.

Reviewer_EvCZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes implicit rehearsal as a mechanism for understanding why on-policy learning can preserve previously learned capabilities. The idea is that earlier behaviors reappear in new-task trajectories and receive further reinforcement when they help solve the new task. They introduce the SFP to distinguish which historical contributions receive credit, whether they support old-task performance, and how their effects evolve through subsequent training. They derive conditional theoretical results and investigate these components through Math-to-Code language-model experiments and control-RL interventions.

Critical strengths
  1. The framework separates history-dependent outputs from preserving old-task capability, and distinguishes immediate support from its effects after continued training. These distinctions help clarify what can reasonably be inferred from output similarity, lesions, and local update interventions. The analysis also explicitly accounts for situations where historical expression persists without guaranteeing retained capability.

  2. The experiments go beyond comparing training trajectories. They include credit interventions from the same training state, targeted-shuffled attenuation, and historical lesions with orthogonal controls matched in statewise policy KL. The LLM interventions consistently affect measured historical expression, while the control experiments support shared functional dependence. These provide useful evidence for individual components of the proposed mechanism.

Critical weaknesses
  1. The strongest experimental results concern changes in historical expression, but their connection to old-task capability remains unclear. For example, in Fig. 18, TARGET substantially reduces historical expression on Code prefixes without consistently reducing Math performance relative to SHUFFLED. The later pass@128 difference in Table 1 corresponds to only two out of 30 AIME problems. Although the control experiments support individual components of the proposed mechanism, the evidence that these components jointly explain lasting retention remains limited.

  2. The SFP framework provides useful distinctions, but its theoretical results mainly establish conditions under which historical expression could contribute to retention. It remains unclear what substantive new conclusions or predictions follow beyond existing explanations based on on-policy data and policy drift. The experiments examine different components of the framework separately, without demonstrating how these conditions predict retention after further training. This limits the framework’s additional explanatory value.

Questions for authors
  1. Which existing results most directly support the connection between historical expression and improved Math retention? In particular, how should the mixed Math performance results in Figure 18 be understood alongside the later difference reported in Table 1?

  2. In experiments, TARGET and SHUFFLED produce substantially different amounts of policy drift despite removing similar amounts of credit. The extended comparison with FULL also involves different optimizer continuation settings. What evidence helps distinguish the proposed historical-credit mechanism from these differences, and how do they limit the conclusions drawn from the comparisons?

  3. Could the authors identify a specific insight or prediction provided by SFP that existing on-policy retention and policy-drift explanations do not provide? Which theoretical or experimental result most directly supports this contribution?

I would be happy to raise my score if these concerns are satisfactorily addressed.

5 comments
AuthorsOpenReview ↗

Rebuttal from authors

We thank Reviewer EvCZ for the constructive and insightful feedback. We address your specific points directly below.

We have completed additional experiments and analyses since submission. The revised manuscript includes these results, experimental protocols, and statistical details. References below use its numbering unless explicitly identified as original-submission numbering.

[W1]:

Which existing results most directly support the connection between historical expression and improved Math retention? In particular, how should the mixed Math performance results in Figure 18 be understood alongside the later difference reported in Table 1?

(In the revised manuscript, these are Figure 21 and Table 19, respectively.)

[A1]: In Figure 21, the ten-update intervention sharply reduces TARGET's Code-prefix expression while leaving Math-prefix expression near its start. SFP does not equate reduced Code expression with Math loss (Eq. (7)), consistent with your observation that TARGET's Math performance in Figure 21 is not consistently lower than SHUFFLED's. In Table 19, SHUFFLED's pass@128 exceeds TARGET's by 6.7 pp (two problems) at update 430. To directly test the role of behaviors learned during Math training in retaining Math capability, we attenuate their positive credit from Code-training step 0 and expand evaluation to 60 AIME problems × 128 responses: after 100 updates, TARGET/SHUFFLED have AIME-probe expression of 0.59/1.18 and Math correctness of 25.77%/27.59%. (Table 1, Figure 3, and Appendix A.9.)

Additional control-RL experiment: continued on-policy training. We study all 15 enrolled training histories in spatially split pick-and-place. Within each history, FULL, TARGET, and SHUFFLED start from the same actor, critic, Adam state, and observation normalizer. Each branch generates its own experience and keeps its evolving optimizer state for 64 updates. (Figure 4(e,f) and Appendix A.6.)

GroupHistorical expression on the shared B probe at update 16Old-task success at update 64
FULL0.95751.50%
TARGET0.76439.67%
SHUFFLED0.95660.13%

At update 16, the planned SHUFFLED−TARGET expression difference is +0.1924 [0.1344, 0.2527], positive in 15/15 histories. At update 64, SHUFFLED exceeds TARGET in old-task success by 20.46 pp [11.87, 28.35] (positive in 12/15 histories). FULL exceeds TARGET by 11.83 pp [0.72, 22.92].

Intervals resample complete paired training histories. Every branch also improves mean new-task success.

Additional LLM experiment: historical expression and task performance after sustained intervention. We start from the published Math checkpoint with freshly initialized optimizer state and continue for 100 Code RL updates. TARGET attenuates positive credit at history-associated positions; SHUFFLED redistributes the attenuation mask within strata on its own rollout.

Historical-expression readouts are:

ProbeCommon startTARGET@100SHUFFLED@100
Code1−0.340.70
AIME10.591.18
MATH-L510.661.19

We independently evaluate Math and Code performance at this endpoint. Math evaluation uses AIME24+25: 60 questions × 128 responses, temperature 0.6, top-p 0.95, and a 16,384-token budget. We average correctness within each question, then equally across questions. HumanEval+ uses 164 questions × 8 responses at temperature 1.

We measure KL on the same fixed Code prefixes in the direction initial Math policy→current policy.

ConditionInitial-reference Code KL (fp32)Math average correctnessHumanEval+ average correctness
Initial Math model027.98%72.41%
TARGET@1000.0928625.77%61.59%
SHUFFLED@1000.1181827.59%71.42%
FULL@1000.008428.49%75.91%

On Math, SHUFFLED and FULL exceed TARGET by 1.82 pp [0.59, 3.14] and 2.72 pp [1.16, 4.43], respectively. On HumanEval+, SHUFFLED exceeds TARGET by 9.83 pp [6.78, 12.96]. A SHUFFLED run with a 2.165× learning rate (SHUFLR2) has 33% higher endpoint KL than TARGET, yet still performs better on both Math (26.65% vs. 25.77%) and HumanEval+ (65.5% vs. 61.6%). This directly counters the explanation that TARGET's lower scores simply result from greater policy drift.

At the planned update-100 endpoint, TARGET has lower initial-reference Code KL than SHUFFLED despite lower Math and Code scores.

Intervals resample paired questions. Average correctness measures how reliably each problem is solved, supplementing coverage.

[W2]:

In experiments, TARGET and SHUFFLED produce substantially different amounts of policy drift despite removing similar amounts of credit. The extended comparison with FULL also involves different optimizer continuation settings.

[A2]: The expression difference persists under bidirectional realized-KL matching in the eight single-update batches reported in Table 3, all starting from the same training state. For each group, we scale its actual update,

θ_r(k) = θ_pre + k Δθ_r

to match the KL from the pre-update policy to the policy with parameters θ_r(k) on the same fixed Code reference set. In this one-step configuration, the gradient and AdamW state are fixed and weight decay is zero, so scaling is equivalent to adjusting each group's learning rate. The expression direction, metric, and probe prefixes remain fixed.

KL is matched on the fixed Code reference set. Expression differences are SHUFFLED−TARGET, with 95% confidence intervals from bootstrap resampling of the eight batches. All differences are positive in every batch (8/8 for each comparison and probe); the matched results and calibration checks are in Tables 4–5.

ComparisonExpression probeMean expression difference [95% CI]
Original updateCode0.00501 [0.00469, 0.00536]
SHUFFLED enlarged to TARGET's KLCode0.00391 [0.00363, 0.00417]
TARGET reduced to SHUFFLED's KLCode0.00180 [0.00160, 0.00199]
Original updateAIME0.00080 [0.00076, 0.00083]
SHUFFLED enlarged to TARGET's KLAIME0.00298 [0.00280, 0.00316]
TARGET reduced to SHUFFLED's KLAIME0.00136 [0.00130, 0.00142]
Original updateMATH-L50.00051 [0.00047, 0.00055]
SHUFFLED enlarged to TARGET's KLMATH-L50.00274 [0.00255, 0.00295]
TARGET reduced to SHUFFLED's KLMATH-L50.00126 [0.00119, 0.00132]

The table below reports the measured KL error for all four matching operations. Relative error is the absolute difference between achieved and target KL, divided by target KL; each row reports the maximum over eight batches.

Matching operationBatches within 1% KL toleranceMaximum relative KL error
SHUFFLED enlarged to TARGET's KL8/80.601%
TARGET reduced to SHUFFLED's KL8/80.667%
FULL enlarged to TARGET's KL8/80.953%
TARGET reduced to FULL's KL8/80.595%
Total32/320.953%

We search the matching scale separately at each fixed starting state. Extending this step-size search to continued training is feasible in principle, but it requires additional KL evaluations and parameter adjustments at every update, making a full rerun computationally costly. For continued training, we also compare the saved update-75 checkpoints with similar initial-reference Code KL.

Additional comparison of existing update-75 checkpoints. The saved TARGET and SHUFFLED checkpoints have similar initial-reference KL at update 75. We measure this KL on the same fixed Code prefixes in the direction initial Math policy→current policy. We analyze their existing expression measurements and HumanEval+ evaluations (164 questions with eight responses per question, temperature 1). (Table 21.)

Update-75 measureTARGETSHUFFLED
Initial-reference Code KL (fp32)0.105060.10835
Code expression E−0.3290.800
AIME expression E0.6381.206
MATH-L5 expression E0.7021.221
HumanEval+ average correctness63.11%73.40%

The relative KL difference is 3.13%. TARGET has lower expression on all three probes and lower HumanEval+ correctness by 10.29 pp [7.16, 13.57] (paired question-bootstrap interval). The update-75 results come from a supplementary analysis of saved checkpoints after training. Update 100 remains the primary endpoint, with its Math evaluations reported separately.

At update 100, TARGET also has lower task performance despite a smaller initial-reference Code KL (A1). Together, these results show that a larger measured policy KL alone does not explain TARGET's lower expression and task performance.

We control optimizer initialization in the continuation experiments. The step-0 LLM conditions share the published starting weights, freshly initialized optimizer state, and prompt schedule; FULL is the main-path reference. In control RL, each branch receives an independent copy of the same initial optimizer state and keeps its optimizer throughout training.

[W3]:

Could the authors identify a specific insight or prediction provided by SFP that existing on-policy retention and policy-drift explanations do not provide?

[A3]: Yes. An explanation based only on scalar policy-drift magnitude cannot distinguish equal-KL updates by which historical contributions lose credit. SFP makes this distinction through credit–influence alignment (Eq. (6)). In our experiment, at matched realized KL, in both matching directions, on all three probes, and in all eight batches (48/48 comparisons), TARGET leaves less historical expression than SHUFFLED. The control-RL lesions show the same directional distinction at the level of function: at equal statewise KL, perturbing along the historical direction harms new-task success more than orthogonal perturbations (8.39 pp). These results cannot be explained by drift magnitude alone. More generally, SFP separates three testable questions:

  1. Selection: The selection identity relates expression changes to the within-stratum alignment of attenuation scores and token influences on the readout. The bidirectional KL experiment confirms the predicted expression ordering at matched local output KL.
  2. Function: In the submitted pushing experiment, we compare lesions along the historical direction with statewise KL-matched orthogonal perturbations to test shared functional use. The new LLM experiment separately measures Math and Code performance.
  3. Persistence: In the additional 15-history experiment, we measure expression at update 16 and old-task success at update 64 as each branch generates its own training experience. The planned comparisons show positive differences in both early expression and terminal capability.

Original 16-seed pushing experiment. With statewise KL-matched directional lesions, the full-dose final-policy new-task contrast (orthogonal control−historical-direction lesion) is 8.39 pp [4.81, 12.32]. Both old- and new-task contrasts are positive in 13/16 seeds, supporting a functional role for the historical direction in both old and new tasks. (Figure 4(a,b) and Tables 8–9.)

Local prediction on actual updates. We check the first-order expansion in Appendix B.4 (Eq. (52)) against the saved Table 3 updates. We compare ∇EᵀΔθ at the starting point with the measured expression change. Each of the eight batches executes one FULL, TARGET, and SHUFFLED update from the same Math→Code@400 state, giving 24 displacements. The readout follows Eq. (2), with 100 fixed reference responses per probe; the prediction results are shown in Figure 6.

Expression probeCorrect change directionMedian relative errorMaximum relative error
Code24/240.707%11.200%
MATH-L524/240.317%1.869%
AIME24/240.503%2.499%

Relative error is the absolute prediction error divided by the absolute measured change. The direction agreement across probes supports the local expression approximation at the actual small-update scale.

Additional experimental evidence

The following completed analyses are also included in the revised manuscript.

Independent reconstruction and position-level decomposition. We independently reconstruct the historical-expression readout in Eq. (2) and decompose it algebraically using the fixed probes in Appendix A.1 (Table 6). Historical directions, Fisher weights, and the Base→Code reference remain fixed at update 400. We recompute all three readouts for TARGET and SHUFFLED at updates 430, 450, 470, and 500, giving 24 aggregate values. We also decompose TARGET's 430-to-500 change across positions and vocabulary entries.

Each probe contains 100 saved reference responses. We average within each response, then equally across responses; the 100 AIME responses cover 60 distinct problems.

Fixed probeMaximum aggregate reconstruction errorTARGET ΔE (430→500)Reference responses with decreasing expressionTop 1% position contribution share
Code1.58 × 10⁻¹⁰−0.2900100/10013.99%
AIME1.24 × 10⁻¹⁰−0.2422100/10010.20%
MATH-L56.44 × 10⁻¹¹−0.2093100/10011.37%

We rank the top 1% by absolute weighted position contribution; their share uses the sum of absolute contributions across all positions as the denominator. Expression declines across all saved reference responses and many positions.

Comments on the paper

4 comments

We thank all reviewers for their constructive feedback. We provide a clean revised PDF and summarize the changes below for the reviewers and the Area Chair. The revised main text fits within the 10-page limit. For a detailed comparison with the original submission, please see the PDF diff provided through the revision process described in the ICLR 2027 Author Guidelines.

  1. Theory and positioning. We clarified SFP’s definitions, assumptions, and notation, and distinguished credit selection, task function, and persistence. Theorems 1–3 are now stated in Sections 3.1–3.3, with direct links to their appendix proofs and extensions. We explain how implicit rehearsal distinguishes updates with equal policy drift by their effects on historical expression, and we added comparisons with Orthogonal Gradient Descent and task arithmetic. Appendix C now compares implicit rehearsal with the bimodal analysis in Retaining by Doing. We give an example in which two policies both solve the new task and have the same reverse KL to the target, yet only one reuses the behavior needed for the old task.

  2. KL controls and interpretation. We added single-update comparisons that match realized KL at both the TARGET and SHUFFLED scales, and we report the accuracy of the calibration (Figure 2(c), Tables 4–5). We distinguish these controls from the sustained training experiments and specify the reference policy and KL direction for each comparison. We also clarify why reduced historical expression on Code prefixes alone does not determine Math capability.

  3. Sustained LLM interventions. We added interventions starting from the Math initialization and continuing through 100 Code RL updates, with independent evaluations on 60 AIME problems and 164 HumanEval+ problems (Table 1, Figure 3, Appendix A.9). Supplementary analyses include a comparison of saved update-75 checkpoints with similar KL, a SHUFFLED run with a fixed higher learning rate (SHUFLR2), and a SHUFFLED rerun. These experiments measure how sustained credit attenuation affects Math retention and new-task Code performance, and how those outcomes relate to endpoint policy drift.

  4. Control-RL continuation and functional evidence. We added 64-update continuations across 15 paired training histories to examine how early differences in historical expression relate to later old-task success (Figure 4(e,f), Appendix A.6). We report individual-history outcomes, bounds on cumulative attenuation, new-task performance, and endpoint drift diagnostics. We also clarified the definitions of the existing directional lesions and how we account for opposing local credit effects, distinguishing immediate effects from terminal retention.

  5. Numerical validation. We checked local first-order predictions against measured expression changes on actual updates (Figure 6), independently reconstructed the historical-expression readouts and decomposed their changes across positions (Table 6), and verified the KL matching between historical-direction logit interventions and their orthogonal controls (Appendix A.1).

  6. Protocols and organization. We clarified the choice of models and tasks, training and evaluation protocols, excerpt selection, statistical units, uncertainty estimates, and which saved artifacts support remeasurement. We streamlined repeated exposition and the conclusion, updated figures and evidence summaries, and adjusted figure and table placement to reduce unnecessary blank space. The original short-branch AIME coverage table now appears in Appendix A.8 as Table 19 with its data unchanged. The new 100-update results appear in the main text.

Our individual reviewer responses explain the evidence in more detail. Section, equation, figure, and table references use the revised numbering. When retaining a reviewer’s reference to the original submission, we give the revised location in parentheses.

AuthorsOpenReview ↗

Overall (1/3)

We thank the reviewers for their constructive feedback. Two questions ran through the reviews. When a model learns a new task, does it matter which of its earlier behaviors receive credit? If so, does the effect persist long enough to influence performance on the old task? To address these questions, we added KL-matched single-update controls, a new intervention spanning Code RL steps 0–100, and a control-RL experiment following 15 paired training histories. All figure, table, and equation numbers below refer to the revised manuscript.

Core contributions

  • Implicit rehearsal. In on-policy learning, the current policy generates the next batch of training trajectories. A behavior learned on an earlier task can therefore reappear while the model solves a new one. If that behavior helps, feedback on the new task can reinforce it, making the updated policy more likely to use it again. In this way, training can preserve skills shared by the two tasks without replaying old-task examples.

  • SFP explains when new-task training favors behavior that the old task needs, and when that reinforcement helps preserve capability.

    • RL’s Razor [4] finds that, among policies that perform similarly well on a new task, on-policy RL tends to favor solutions closer in KL to the initial policy. In our Math-to-Code experiment, attenuating positive credit at positions favored by prior Math training produced smaller endpoint KL on fixed Code prefixes, yet Math accuracy fell further. Shuffling the attenuation mask produced larger endpoint KL, but Math accuracy stayed close to its starting value. The measured drift alone therefore did not explain the difference in retention between these interventions.

    • Retaining by Doing [3] shows that, in a bimodal policy, reverse-KL training can move one component toward a new target while leaving the component associated with prior knowledge largely intact. We study a complementary case in which an old skill can be reused on the new task. Suppose either A or B solves the new task, but only A solves the old one. If the target assigns equal probability to A and B, always choosing A and always choosing B give the same reverse KL. Both policies succeed on the new task, yet only the first retains the old skill. In this example, retention depends on which behavior the selected mode reuses. SFP explains when new-task feedback favors the shared behavior A, when it favors B instead, and whether the resulting effect on old-task performance persists.

    Our Selection, Function, and Persistence (SFP) framework takes the next step by asking what makes training favor A in the first place. The two-action model in Section 3 gives a precise answer (p. 4, Equation 4). When A earns more new-task reward than B, an on-policy gradient update increases the probability of A and improves old-task success. When B earns more, the update decreases the probability of A, even if A can still solve the new task. When the rewards are equal, the expected reward gradient is zero. SFP develops this analysis for actual training by identifying which historical behaviors receive credit, testing whether they support old-task performance, and tracing whether that support survives subsequent updates. Linking the conditions for selection to function and eventual retention is the framework’s central contribution (Sections 3.1–3.3 and Appendix C).

  • Testing the mechanism through intervention. Earlier work on learned directions includes Orthogonal Gradient Descent [1], which projects gradients to protect old-task outputs, and task arithmetic [2], which uses parameter differences to edit models. We measure how prior training changed the model’s output preferences in a given context, then intervene on the credit those behaviors receive. In the language-model experiments, we change where positive credit is attenuated during Code training and independently evaluate Math capability. In control RL, directional perturbations test whether the old and new tasks depend on the same behavior, and continued training tests whether early differences carry through to final old-task performance (Sections 2 and 4).

AuthorsOpenReview ↗

Overall (2/3)

Key concerns and responses

1. Could policy drift explain the effect of targeted attenuation? (EvCZ, wKuZ, Pk9G, 7aRm)

TARGET attenuates positive credit at token positions favored by prior Math training. SHUFFLED permutes the attenuation mask within strata. To test whether the difference could be explained by update magnitude alone, we matched realized single-update KL on the same fixed Code prefixes at two scales. We first enlarged SHUFFLED’s update to match TARGET’s KL, then reduced TARGET’s update to match SHUFFLED’s KL.

At both scales, SHUFFLED retained more historical expression on all three probes (Code, AIME, and MATH-L5) in every one of the eight batches. This gives 48 of 48 comparisons in the same direction. The maximum relative KL-matching errors were 0.601% and 0.667%, respectively. At matched local policy drift, the placement of credit attenuation still affected how much historical behavior the updated policy expressed (p. 7, with expression differences in Table 4 and matching accuracy in Table 5).

2. Does targeting historical credit affect retained Math capability? (EvCZ, wKuZ, Pk9G, 7aRm)

We ran a new experiment from the same published Math checkpoint, with a freshly initialized optimizer in each condition. Each intervention began at Code RL step 0 and continued through step 100. We then evaluated 60 AIME problems with 128 responses per problem.

Both conditions started at 27.98% average Math correctness. At step 100, TARGET scored 25.77% and SHUFFLED scored 27.59%, a gap of 1.82 percentage points (95% paired-question bootstrap CI 0.59–3.14). Despite the larger drop in Math accuracy, TARGET had smaller endpoint KL from the initial Math policy to the current policy on fixed Code prefixes (0.09286 for TARGET and 0.11818 for SHUFFLED).

At the same endpoint, AIME-probe expression moved from a common starting value of 1 to 0.59 under TARGET and 1.18 under SHUFFLED. SHUFFLED also scored 9.83 percentage points higher on HumanEval+. These results connect credit placement during Code training to historical expression and to independently measured performance on both tasks (Table 1 and Figure 3).

3. Do early differences in expression carry through to later old-task performance? (EvCZ, wKuZ)

In pick-and-place, we created paired branches from 15 training histories. Within each history, the branches started from the same policy and optimizer state, then generated their own experience for 64 updates. At update 16, SHUFFLED retained more historical expression than TARGET in all 15 histories. By update 64, SHUFFLED had higher old-task success in 12 of 15 histories, with a mean advantage of 20.46 percentage points (95% CI 11.87–28.35). Mean new-task success improved in all three conditions over the same period (Figure 4e–f).

We also independently reran the LLM SHUFFLED condition with the same configuration. At step 100, HumanEval+ correctness was 70.6% in the rerun and 71.4% in the original run. Code-probe expression was 0.68 and 0.70, respectively (Appendix A.9).

4. What testable prediction does SFP add? (EvCZ, 7aRm)

Given the same starting policy and the same batch of new-task trajectories, Theorem 1 predicts how the placement of credit attenuation affects historical expression. If stronger attenuation falls on positions that contribute more to historical expression within a stratum, targeted attenuation yields lower expected first-order expression than shuffling the mask (Equation 6). The KL-matched comparisons above test this ordering. We also checked the first-order prediction against the optimizer’s actual updates. It predicted the direction of expression change correctly for all 24 updates on each of the three probes (Figure 6).

For continued training, SFP gives a sufficient condition for better final retention. If the predicted old-task benefits accumulated across updates exceed the combined bounds on functional prediction error and the change introduced by subsequent learning, the intervention yields better final old-task performance than the comparison condition (Sections 3.1–3.3 and Equation 10).

AuthorsOpenReview ↗

Overall (3/3)

Reviewer discussion

ReviewerCurrent ratingDiscussion outcome
EvCZRaised to Weak Accept (3)Said the new experiments addressed most empirical concerns and strengthened the evidence for the mechanism. Retained reservations about the framework’s additional theoretical and predictive contribution.
wKuZMaintained Weak Accept (3)Found the rebuttal convincing, particularly the KL-matched controls and control-RL experiments. Encouraged additional independent LLM runs.
7aRmRaised from Weak Accept (3) to Clear Accept (4)Said the KL-matched single-update controls and other new experiments addressed most of their concerns. They also found the paper easier to follow with the main theorems stated in the main text.
Pk9GMaintained Weak Reject (2)Pk9G acknowledged that our initial rebuttal addressed several of their original concerns and that the experiments showed that credit placement affects retention. The reviewer maintained the score and raised further concerns about whether implicit rehearsal explains why on-policy RL forgets less, SFP’s broader methodological value, and the evidence for its theoretical predictions. We responded to these concerns and have received no further reply from Pk9G as of this update.

Overall

On-policy sampling gives previously learned behavior an opportunity to reappear. SFP explains when that opportunity can translate into retention by tracing which behavior the new task reinforces, what that behavior contributes to the old task, and whether the effect persists. In our experiments, targeting historical credit reduced subsequent expression even at matched single-update KL. Sustaining that intervention led to poorer old-task performance than shuffled attenuation in both Math-to-Code learning and control RL. Together, these findings support implicit rehearsal as one mechanism through which learning a new task can preserve an earlier capability.

References

[1] Orthogonal Gradient Descent for Continual Learning. AISTATS 2020.

[2] Editing Models with Task Arithmetic. ICLR 2023.

[3] Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting. ICML 2026.

[4] RL’s Razor: Why Online Reinforcement Learning Forgets Less. ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.