acceptodds
Under review as a conference paper at ICLR 2027

Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff

Abstract

Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training. SFT is expected to provide a useful behavioral prior for RL to further enhance model capabilities. However, checkpoints with excessive SFT often show limited improvement during RL. We attribute this failure to the loss of model plasticity: the reduced ability of an SFT-initialized policy to be effectively reshaped by subsequent RL. To better understand this phenomenon, we conduct detailed analysis from multiple perspectives, including parameter changes, output spaces, and RL optimization dynamics. Our results show that models from excessive SFT tend to produce over-confident token distributions and exhibit sharp parameter landscapes, which make them harder to optimize in the RL stage. To enable a more robust SFT-to-RL handoff, we propose Rejuvenation, a simple yet effective method that restores plasticity while preserving useful SFT-acquired priors. Rejuvenation leverages base-anchored model fusion to reduce excessive SFT-induced drift with targeted neuron reset to mitigate model rigidity. Experimental results on both math reasoning tasks and agentic tasks demonstrate that our approach recovers RL performance from checkpoints at different SFT stages, widening the viable SFT-to-RL handoff window.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Supervised Fine-Tuning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GZnqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzed how overtraining in the SFT stage can lead to loss of plasticity that makes it difficult to RL during the following stage of posttraining. The authors proposed Rejuvenation, which is composed of weight-level base-anchoring + neuron-level selective reset, to remedy the overtraining and the resulting loss of plasticity. The results demonstrate that rejuvenation is effective in salvaging overtrained SFT checkpoints for RL.

Critical strengths
  1. The paper provides ample interesting analysis measured in many aspects such as weight/gradient norm, diversity-related metrics (loss, entropy, pass@1 vs. pass@k accuracy, etc).
  2. I appreciate the simplicity of the remedy proposed, namely the base-anchoring involves a simple linear interpolation with base model weight.
  3. I also appreciate the per-neuron contribution, which is also finegrained enough and effective.
Critical weaknesses
  1. The result section could benefit from more ablation studies, namely there are two stages of the rejuvenation, which stage is responsible for the main recovery and how so?
  2. Can we over rejuvenate? From Fig6 it seems like even 2 epochs of SFT can benefit from rejuvenation? It seems to me that there might be some benefit to the rejuvenation fusion that lead to better performance compared to just RL without SFT (i.e. setting alpha to 0 for the base-anchored linear fusion stage). The authors should definitely show these extreme cases of alpha as well as omega for the interpolation parameters (e.g. in Table 5), and in Fig6 consider showing RL without SFT or rejuvenation.
  3. The interesting thing here is the choice of the linear interpolation, and how it works. More analysis needs to go into how the interpolation qualitatively changed the output because intuitively we don't expect that linearly interpolating weights give us a linear interpolation in the behavior outcome, which is dependent on the output logits. Presumably, the highly confident token logits resulted from over SFTing can be "sticky," meaning tolerate some range of weight perturbation before a flip in outcome occurs.
  4. Overall, I think the key insight here is that SFT leads to overconfident token distrib, which is amplified via RL. I think this point is worth being highlighted more as motivating design principle.
  5. Figure captions can be more descriptive.
Questions for authors

Questions are contained in the weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for the constructive comments. Our responses to all weaknesses and questions are as follows.


Weaknesses

W1: The result section could benefit from more ablation studies, namely there are two stages of the rejuvenation, which stage is responsible for the main recovery and how so?

A: Our component ablation (Table 4) shows that global fusion accounts for most of the recovery (+2.1 points), while neuron reset adds +0.5 points:

Initialization before RLID Avg.OOD Avg.Overall Avg.
OverSFT17.512.214.9
OverSFT + fusion19.314.817.0
OverSFT + fusion + reset19.715.317.5

The two stages play complementary roles. Global fusion uniformly scales down SFT-induced parameter shifts (θOverSFT−θbase\theta_{\text{OverSFT}} - \theta_{\text{base}}, Section 3.3), dampening sharp weight spikes and smoothing the optimization landscape (Figures 3 and 4). Attribution-guided reset then restores a small set of rigid, high-contribution neurons (mostly in late attention key/value and MLP projections, Figure 11) to their base values (Section 3.4), reopening token entropy at over-confident reasoning steps (Figures 9 and 10) for an extra +0.5-point gain, concentrated on OOD tasks (14.8 to 15.3).


W2: Can we over rejuvenate? From Fig6 it seems like even 2 epochs of SFT can benefit from rejuvenation? It seems to me that there might be some benefit to the rejuvenation fusion that lead to better performance compared to just RL without SFT (i.e. setting alpha to 0 for the base-anchored linear fusion stage). The authors should definitely show these extreme cases of alpha as well as omega for the interpolation parameters (e.g. in Table 5), and in Fig6 consider showing RL without SFT or rejuvenation.

A: 1. Over-rejuvenation trade-off. Yes. Pulling weights too close to the base model erases useful SFT priors: lowering α\alpha from 0.5 to 0.4 drops ID Avg. from 19.3 to 18.5, whereas raising α\alpha to 0.6 hurts OOD Avg. (14.8 to 11.9) (Table 5). Similarly, increasing the reset ratio pp from 1% to 2% lowers overall accuracy from 17.5 to 17.3 (Table 6).

2. Why SFT is needed before RL (Base+RL, α=0\alpha = 0) and why 2-epoch SFT still benefits. We adopt EvoLM-4B as our base model, which is a clean pre-trained model with publicly available training data and code and without benchmark contamination (Appendix B). Because it has not been heavily post-trained on math data, its mathematical capability prior to SFT is relatively weak. Consequently, skipping SFT and running RL directly from the base model (Base+RL, α=0\alpha = 0) yields little to no improvement due to sparse rewards, making SFT necessary to establish basic mathematical reasoning and formatting priors before RL. At 2 SFT epochs (ModSFT), the model has acquired these priors without severe plasticity loss, and Rejuvenation yields a modest additional gain (17.6 vs. 17.0 on EvoLM-4B; 54.3 vs. 53.8 on Qwen3-8B, Figure 6) by slightly widening exploration room.

3. Extreme cases of α\alpha and ω(i)\omega^{(i)}. In Eq. (1), α=0\alpha = 0 gives Base+RL and α=1\alpha = 1 gives OverSFT+RL. In Eqs. (4)–(5), ω(i)=I(i∈S)∈{0,1}\omega^{(i)} = \mathbb{I}(i \in \mathcal{S}) \in \{0, 1\} is a binary mask (not a continuous weight), where ω(i)=1\omega^{(i)} = 1 restores neuron ii to θbase\theta_{\text{base}}. We have added this discussion and the Base+RL reference to the revised manuscript.

We have added discussion on these points in the revised version.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

W3: The interesting thing here is the choice of the linear interpolation, and how it works. More analysis needs to go into how the interpolation qualitatively changed the output because intuitively we don't expect that linearly interpolating weights give us a linear interpolation in the behavior outcome, which is dependent on the output logits. Presumably, the highly confident token logits resulted from over SFTing can be "sticky," meaning tolerate some range of weight perturbation before a flip in outcome occurs.

A: We agree that linear weight interpolation induces nonlinear changes in output logits and policy behavior, so its effect on sampling diversity should be explained through logit margins rather than assumed from Eq. (1) alone. While Hindsight Merging [1] and LM-Cocktail [2] motivate weight interpolation (Sections 2 and 3.3), they do not analyze how it reshapes token-level distributions at the SFT-to-RL handoff.

Your "sticky token" intuition captures why moderate interpolation works well here. As shown in Figure 7 and Table 1, OverSFT pushes top-1 logits far above runner-up candidates (~45 vs. ~20 in ModSFT), collapsing token entropy from 0.184 to 0.024 and narrowing the Pass@K–Pass@1 gap from 15.3 to 11.2. Because this logit margin is so large, scaling the SFT weight delta (θOverSFT−θbase\theta_{\text{OverSFT}} - \theta_{\text{base}}) by α=0.5\alpha = 0.5 compresses the top-1/top-2 gap without flipping the greedy argmax token at most positions. As a result, basic formatting and step-by-step reasoning priors remain intact ("sticky"), while under softmax with temperature-1.0 RL rollouts (Appendix B), the reduced margin shifts probability mass back to runner-up tokens so RL can sample alternative reasoning continuations.

This qualitative shift is reflected in our quantitative ablations: fusion alone (α=0.5\alpha = 0.5) raises pre-RL token entropy from 0.024 (OverSFT, Table 1) to 0.146 and recovers post-RL overall accuracy from 14.9 to 17.0 (Table 4). Moreover, Table 5 illustrates the boundary of this stickiness: at α=0.6\alpha = 0.6, logits remain overly rigid, preserving ID accuracy (19.7) at the cost of OOD exploration (11.9); at α=0.4\alpha = 0.4, excessive shrinkage erodes SFT priors and lowers ID accuracy (18.5). We have expanded Section 3.3 in the revised manuscript to clarify how weight scaling narrows logit margins to balance prior retention with exploration.


W4: Overall, I think the key insight here is that SFT leads to overconfident token distrib, which is amplified via RL. I think this point is worth being highlighted more as motivating design principle.

A: We have elevated this insight to a core motivating principle in the Introduction and Section 3 of the revised manuscript. Extending SFT from ModSFT to OverSFT raises MATH-500 Pass@1 from 15.2 to 15.8, while token entropy drops from 0.184 to 0.024 and the Pass@K–Pass@1 gap narrows from 15.3 to 11.2 (Table 1). Our token-level visualizations (Figures 9 and 10) show that basic arithmetic steps in GSM8K have low entropy in both ModSFT and OverSFT, reflecting genuine computation skills. In contrast, OverSFT becomes artificially over-confident on tokens that should not be memorized, such as person names in GSM8K word problems (e.g., Eric in Figure 9) and rigid reasoning connectives.

Our design directly addresses this distinction: global fusion scales down overall SFT drift to retain distributed task skills, while local reset relaxes sparse high-attribution neurons driving spurious memorization. We also frame further confidence concentration during RL as an observed failure mode in our setting rather than a universal property of RL.


W5: Figure captions can be more descriptive.

A: Thanks for your suggestion. We have expanded the figure captions in the revised manuscript.


Thanks again for your valuable feedback. We would be happy to address any further questions.

References

[1] Hindsight Merging: Diverse Data Generation with Language Models. UAI 2025.

[2] LM-Cocktail: Resilient Tuning of Language Models via Model Merging. ACL 2024 Findings.

Thank you for the detailed responses, my concerns have been largely resolved. I will keep my positive rating!

AuthorsOpenReview ↗

Thanks for your reply

Thank you very much for your follow-up and for maintaining your positive rating! We truly appreciate your time and constructive suggestions throughout the review process, which have helped us strengthen the paper.

Reviewer_pSdpOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates the loss of model plasticity during the transition from SFT to RL. The authors observe that excessive SFT leads to concentrated token distributions, larger parameter deviations, and reduced improvements during subsequent RL. To address this issue, they propose Rejuvenation, a post-hoc recovery method combining base-anchored model fusion with attribution-guided neuron reset. Experiments on mathematical reasoning and agentic tasks suggest that the proposed method can recover the performance of over-trained SFT checkpoints and reduce sensitivity to SFT duration.

Critical strengths

The paper addresses a practically relevant challenge in the SFT-to-RL pipeline. Recovering an already over-trained checkpoint without repeating SFT or searching over multiple RL initializations. The analysis considers multiple aspects of SFT over-training, including parameter drift, output entropy, and RL optimization dynamics. The experiments also cover two model families and different downstream task settings.

Critical weaknesses

The causal role of attribution-guided neuron reset is insufficiently established.

The neuron reset component is presented as a targeted mechanism for removing neurons responsible for over-confident predictions. However, Table 4 shows that base-anchored fusion alone recovers the overall score from 14.9 to 17, while attribution-guided reset only provides an additional .5 point improvement.

More importantly, the current experiments do not establish that selecting neurons through logit attribution is necessary for this improvement. Direct logit attribution measures contributions to selected logits, but does not necessarily identify neurons causally responsible for entropy collapse or impaired RL optimization. For example, neurons that changes much during the training proces and causes the over-confidence bahavior are not necessarily the one responsible for the high-confidence prediction. They may instead contribute to overconfidence by suppressing some competing tokens, thereby sharpening the output distribution. The distinction matters because resetting influential neurons could improve exploration through a generic perturbation effect, without specifically correcting the proposed failure mechanism, especially given the marginal improvement.


Another concern about the neuron reset process is whether this operation will harm LLMs' performance on knowledge-intensive tasks, which worth testing. In the critical neuron identification process, the method compares entropy gap between trained model with base model. However, some factual knowledge could obtained from this process and also produce a high confidence correlation, resetting them will affect the factual knowledge obtained from the SFT process. One suggestion is that the author can mix current SFT dataset with some domain specific knowledge data, which the knowledge is not originally in the base model, and compares whether the process will also eliminate the knowledge obtained from the SFT process (i.e., the domain-specific knowledge)

Questions for authors

some informal nouns presents, need to be revised, for example "lm_head, k_proj, ..." if AI tools are being used to generate some figures, need to be declare as well, current AI statement looks insufficient or not fully disclosed the usage.

6 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for the constructive comments. Our responses to all weaknesses and questions are as follows.


Weaknesses

W1 (Original):

The causal role of attribution-guided neuron reset is insufficiently established.

The neuron reset component is presented as a targeted mechanism for removing neurons responsible for over-confident predictions. However, Table 4 shows that base-anchored fusion alone recovers the overall score from 14.9 to 17, while attribution-guided reset only provides an additional .5 point improvement.

More importantly, the current experiments do not establish that selecting neurons through logit attribution is necessary for this improvement. Direct logit attribution measures contributions to selected logits, but does not necessarily identify neurons causally responsible for entropy collapse or impaired RL optimization. The distinction matters because resetting influential neurons could improve exploration through a generic perturbation effect, without specifically correcting the proposed failure mechanism.

The paper would benefit from controlled comparisons against random neuron reset, magnitude-matched reset, and alternative neuron selection strategies under the same reset budget.

A: Direct logit attribution (DLA) serves as a ranking heuristic rather than standalone causal proof (Section 3.4.1), and global fusion provides the primary gain (+2.1 points) while neuron reset adds +0.5 points (17.0 to 17.5, concentrated on OOD tasks from 14.8 to 15.3). Mechanistically, OverSFT produces sharp logit spikes driven by a small set of neurons (Figures 7 and 8), concentrating probability on greedy continuations at reasoning-organizing tokens such as Since, First, and conclude (Figure 9). Resetting these high-attribution neurons relaxes the over-confident logits and substantially restores token entropy at the affected positions (Figure 10), reopening exploration and partially restoring RL plasticity.

To verify whether attribution-guided selection outperforms generic parameter perturbation under the exact same fused checkpoint (α=0.5\alpha = 0.5) and reset budget, we have added controlled comparisons in the revised manuscript against layer/module-matched random reset, weight-delta magnitude-matched reset (largest ∣θfuse−θbase∣|\theta_{\text{fuse}} - \theta_{\text{base}}|), and lowest-attribution reset, while keeping our discussion of DLA strictly aligned with these controls:

Reset strategy on fused checkpoint (α=0.5\alpha = 0.5)Pre-RL EntropyID Avg. after RLOOD Avg. after RLOverall Avg. after RL
Fusion only0.14619.314.817.0
Fusion + layer/module-matched random reset0.15319.014.216.6
Fusion + magnitude-matched reset0.15819.215.017.1
Fusion + lowest-attribution reset0.15218.714.316.5
Fusion + attribution-guided reset (ours)0.16519.715.317.5

These controlled results support three conclusions:

  1. Strongest pre-RL entropy recovery. Attribution-guided reset achieves the highest pre-RL entropy (0.165 vs. 0.146 for fusion only, compared with 0.152–0.158 for alternative strategies), confirming that DLA isolates the neurons most responsible for prediction collapse.
  2. Untargeted perturbation degrades RL performance. Both layer/module-matched random reset (16.6) and lowest-attribution reset (16.5) underperform fusion only (17.0) on both ID and OOD benchmarks, showing that generic parameter noise disrupts useful SFT priors rather than aiding RL optimization.
  3. Parameter-shift magnitude cannot separate priors from rigidity. Magnitude-matched reset yields virtually no overall gain over fusion only (17.1 vs. 17.0, with ID Avg. dipping from 19.3 to 19.2), because large weight deltas conflate beneficial task adaptations with over-confident logit drivers. Attribution-guided reset is the only strategy that improves both ID (19.7) and OOD (15.3) performance to reach 17.5.
AuthorsOpenReview ↗

Rebuttal (Part 2/2)

W2: Another concern about the neuron reset process is whether this operation will harm LLMs' performance on knowledge-intensive tasks, which worth testing. In the critical neuron identification process, the method compares entropy gap between trained model with base model. However, some factual knowledge could obtained from this process and also produce a high confidence correlation, resetting them will affect the factual knowledge obtained from the SFT process. One suggestion is that the author can mix current SFT dataset with some domain specific knowledge data, which the knowledge is not originally in the base model, and compares whether the process will also eliminate the knowledge obtained from the SFT process (i.e., the domain-specific knowledge)

A: Thank you for this insightful comment. We notice that this is a newly posted weakness added very close to the rebuttal deadline, so running a new SFT-to-RL pipeline with mixed domain-specific knowledge data is not feasible within the remaining time. We will include this controlled domain-knowledge mixing experiment in the final version of the paper.

In the meantime, our token-level visualizations and existing evaluations on knowledge-intensive benchmarks provide direct evidence addressing this concern:

  1. Reset targets spurious memorization rather than learned task skills. In our token-level entropy analysis (Figures 9 and 10), routine response tokens, such as basic arithmetic operations in GSM8K, exhibit low entropy in both ModSFT and OverSFT, indicating that SFT has properly acquired mathematical computation capabilities. Comparing ModSFT and OverSFT shows that the pronounced entropy gap primarily arises at positions that should not be memorized, such as character names in GSM8K word problems (e.g., Eric in Figure 9), where OverSFT forces the model to memorize irrelevant surface tokens. Attribution-guided reset specifically targets these over-memorized positions rather than general reasoning or factual operations.
  2. Sparse reset preserves SFT-acquired knowledge. Neuron reset modifies only a tiny fraction of parameters (Table 6) while leaving the vast majority of the fused weights (θfuse\theta_{\text{fuse}}) intact, and restores selected neurons to the pre-trained base model (θbase\theta_{\text{base}}) rather than zeroing them out. Distributed task knowledge learned during SFT is therefore preserved in θfuse\theta_{\text{fuse}}.
  3. Performance on knowledge-intensive benchmarks improves rather than degrades. Our OOD evaluation suite comprises three knowledge-intensive benchmarks: GPQA-Diamond, ARC-Challenge, and MMLU-Pro. Rather than harming factual or scientific knowledge, Rejuvenation+RL improves the OOD average over OverSFT+RL from 12.2 to 15.3 on EvoLM-4B (Table 2; GPQA 9.9 to 12.9, ARC 21.3 to 26.2, MMLU-Pro 5.6 to 6.9) and from 59.9 to 60.5 on Qwen3-8B (Table 3; ARC 75.9 to 78.2, MMLU-Pro 61.7 to 62.4).

In the revised manuscript, we have added a discussion of this scenario, and we will add the full empirical evaluation mixing domain-specific factual data with reasoning SFT in the final version.


Questions

Q1: some informal nouns presents, need to be revised, for example "lm_head, k_proj, ..."

A: Thanks for pointing this out. We originally used these identifiers (lm_head, k_proj, etc.) following standard Hugging Face module naming conventions [1, 2]. We agree that standard architectural terms are clearer in formal writing. In the revised manuscript, we have replaced these code identifiers in the main text with standard terms (unembedding head, attention query/key/value/output projections, and MLP gate/up/down projections), keeping the code names only in parentheses where needed for reproducibility.

Q2: if AI tools are being used to generate some figures, need to be declare as well, current AI statement looks insufficient or not fully disclosed the usage.

A: Thanks. All quantitative figures (Figures 2–14) were plotted with Python, while the conceptual illustration in Figure 1 was created with AI image-generation assistance and manually refined by the authors. In the revised manuscript, we have updated the AI Use Statement to explicitly disclose this usage.


Thanks again for your valuable feedback. We would be happy to address any further questions.

References

[1] Who Reasons in the Large Language Models? NeurIPS 2025.

[2] Forget the Data and Fine-Tuning! Just Fold the Network to Compress. ICLR 2025.

I appreciate the additional explanations and experiments, which addresses my main concern about the efficacy of the neuron reset process. To this end, I will raise my score accordingly.

Reviewer_7WgTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the loss of model plasticity caused by excessive SFT, which limits subsequent RL improvement. The authors propose Rejuvenation, combining base-anchored model fusion and attribution-guided neuron reset to restore plasticity, and demonstrate its effectiveness on mathematical reasoning and agentic tasks.

Critical strengths
  • The paper addresses an important practical challenge in the SFT-to-RL pipeline, particularly when selecting an appropriate SFT checkpoint is difficult or expensive.
  • The authors provide empirical analyses of parameter drift, output entropy, and RL optimization dynamics, offering useful insights into the effects of excessive SFT.
  • The proposed method is simple, computationally efficient, and shows consistent improvements across different SFT checkpoints and tasks.
Critical weaknesses
  • Despite extensive prior work on the SFT-to-RL trade-off, the paper only includes DFT as a competing SFT method, without direct comparisons to approaches such as AESL (Getting Your LLMs Ready for Reinforcement Learning with Lightweight SFT) and PEAR (Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning) under the same downstream RL setting. Moreover, the proposed base-anchored fusion closely resembles existing model-merging approaches (Hindsight Merging: Diverse Data Generation with Language Models, LM-Cocktail: Resilient Tuning of Language Models via Model Merging), but the paper lacks discussion and comparisons of alternative merging strategies, such as merging with intermediate SFT checkpoints or using layer-wise/adaptive merging weights.

  • The contribution of attribution-guided neuron reset appears limited, as fusion alone improves the overall score from 14.9 to 17.0, while neuron reset only further improves it to 17.5 (Table 4). Comparisons with simpler strategies, such as random neuron reset, are missing.

  • The paper does not convincingly demonstrate that the improvements come from restoring model plasticity rather than simply changing the RL initialization. More direct evidence of improved trainability, such as controlled RL learning curves or comparisons with alternative initializations, is needed.

  • The mathematical SFT experiments use a constant learning rate of 3e-6 for up to 32 epochs without learning-rate decay or weight decay, which may exaggerate the observed plasticity loss. Moreover, using the same RL learning rate across different checkpoints without a sensitivity analysis makes it unclear whether OverSFT's poor performance is partly due to suboptimal optimization hyperparameters.

Questions for authors

Please refer to comments above

4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for the constructive comments. Our responses to all weaknesses and questions are as follows.


Weaknesses

W1

A: Comparison with AESL and PEAR. AESL [1] and PEAR [2] modify the SFT loss prior to training, whereas Rejuvenation repairs an existing over-trained checkpoint post-hoc in 3 minutes on a single H800 GPU for EvoLM-4B and under 5 minutes for Qwen3-8B (Appendix C). Evaluating AESL and PEAR (with validation-based checkpoint selection) on EvoLM-4B under the identical 100k math SFT dataset and GRPO budget yields:

MethodID Avg. after RLOOD Avg. after RLOverall Avg. after RL
ModSFT20.014.117.0
OverSFT17.512.214.9
DFT18.312.115.2
AESL19.312.315.8
PEAR19.011.915.5
Rejuvenation19.715.317.5

Rejuvenation outperforms both AESL (17.5 vs. 15.8) and PEAR (17.5 vs. 15.5), particularly on OOD generalization (15.3 vs. 12.3 and 11.9). Unlike loss-reweighting methods that still accumulate parameter drift over extended SFT, pulling weights back toward the pre-trained base anchor and resetting rigid neurons preserves generalizable base representations while retaining ID priors.

Comparison with model-merging variants. We have cited and discussed Hindsight Merging [3], LM-Cocktail [4], and Model Breadcrumbs [5] in Sections 2 and 3.3 of the revised manuscript. Whereas these works focus on static inference performance (diversity, forgetting, or multi-task merging), we address restoring RL trainability and optimization dynamics, pairing global fusion with local attribution-guided neuron reset (Section 3.4). We further compare base-anchored fusion against intermediate ModSFT fusion (α=0.5\alpha = 0.5) and Model Breadcrumbs under the identical RL recipe:

Recovery method before RLID Avg.OOD Avg.Overall Avg.
Base-anchored fusion only (α=0.5\alpha = 0.5)19.314.817.0
Fusion with ModSFT checkpoint + attribution-guided reset19.815.017.4
Model Breadcrumbs + attribution-guided reset19.214.016.6
Base-anchored fusion + attribution-guided reset19.715.317.5

These comparisons highlight two findings: (1) Base vs. intermediate checkpoint fusion: Base-anchored fusion matches ModSFT fusion overall (17.5 vs. 17.4) with stronger OOD generalization (15.3 vs. 15.0), without requiring intermediate checkpoint storage or selection. (2) Global shrinkage vs. sparse masking: Model Breadcrumbs [5] underperforms (16.6 vs. 17.5, notably 14.0 vs. 15.3 on OOD), confirming that uniformly scaling down SFT drift toward the base anchor is essential to smooth the optimization landscape before local reset.


W2

A: Global fusion indeed provides the primary recovery (+2.1 points, from 14.9 to 17.0), while neuron reset adds a targeted +0.5-point gain to 17.5 (Table 4). Mechanistically, OverSFT induces extreme next-token logit separation driven by a small subset of high-attribution neurons (Figures 7 and 8), collapsing predictions at key reasoning-organizing tokens such as Since, First, and conclude (Figure 9). Resetting these neurons relaxes the over-confident logits and restores token entropy at the affected positions (Figure 10), reopening alternative continuations for exploration and partially restoring RL plasticity. Consequently, the +0.5-point gain is concentrated on OOD generalization (14.8 to 15.3, including ARC from 22.7 to 26.2) and harder competition math (AMC23 from 6.3 to 8.0). In the revised manuscript, we frame neuron reset as a targeted refinement on top of global fusion.

To verify that attribution-guided selection outperforms simpler reset baselines, we have added comparisons against layer/module-matched random reset under the exact same fused checkpoint (α=0.5\alpha = 0.5), reset budget, and RL recipe:

Method before RLID Avg.OOD Avg.Overall Avg.
Fusion only19.314.817.0
Fusion + layer/module-matched random reset19.014.216.6
Fusion + attribution-guided reset19.715.317.5

Layer/module-matched random reset degrades performance relative to fusion only (16.6 vs. 17.0 overall, dropping on both ID from 19.3 to 19.0 and OOD from 14.8 to 14.2), confirming that untargeted perturbation disrupts useful SFT priors. In contrast, attribution-guided reset improves both ID (19.7) and OOD (15.3) accuracy to reach 17.5.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

W3

A: We define model plasticity operationally as an SFT checkpoint's capacity to convert reward-driven RL updates into task gains (Section 1). Three lines of evidence in our paper and rebuttal distinguish restored plasticity from simply starting at a stronger initialization or adding generic weight noise:

  1. Lower starting score, larger RL gain: Rejuvenation lowers pre-RL overall accuracy from 10.0 (OverSFT) to 5.4, yet achieves a +12.1 RL gain to reach 17.5, compared with +4.9 (reaching 14.9) for OverSFT under the identical RL recipe (Table 2). Conversely, starting RL from Base (with even lower initial accuracy) yields negligible gain due to sparse rewards, showing that a lower starting score alone does not drive RL improvement.
  2. RL optimization dynamics: Figure 4 shows that OverSFT exhibits higher RL gradient norms than ModSFT (0.33 vs. 0.225), yet its actual parameter update magnitude drops (6.17×10−96.17 \times 10^{-9} vs. 6.90×10−96.90 \times 10^{-9}), leaving SFT weight spikes intact during RL (Figures 3 and 12–14).
  3. Controlled comparisons across alternative initializations: Under the identical RL recipe, we compare multiple initializations across the paper and rebuttal, including Base, UnderSFT (0.5–1 epochs, Figure 5), ModSFT, DFT, AESL, PEAR, and untargeted perturbations (e.g., layer/module-matched random reset in W2, which drops post-RL accuracy from 17.0 to 16.6). Only initializations that reduce SFT weight drift while preserving task priors achieve strong post-RL gains.

W4

A: 1. Plasticity loss is not an artifact of disabled weight decay or 32 epochs. In the math setup, we disabled LR decay and weight decay only to isolate the effect of SFT duration (Appendix B). Two results in the paper already show that this degradation occurs under shorter training and standard regularization: (a) in math SFT (Figure 6a), post-RL performance already drops at epochs 4 and 8 (16.4 and 16.5 vs. 17.0 at epoch 2), well before 32 epochs; and (b) in our agentic SFT experiments on Qwen3-8B (Appendix B), standard weight decay (0.1) is enabled, yet OverSFT still lowers post-RL performance from 53.8 (ModSFT) to 52.5 (Table 3, Figure 6b).

2. Fixed RL hyperparameters ensure a fair comparison. Across all SFT checkpoints and baselines, we follow the default RL optimization settings in verl (math) and slime (agentic tasks), including the standard RL learning rate of 1e-6, without per-checkpoint hyperparameter tuning (Section 4.1, Appendix B). Keeping the RL recipe fixed ensures a fair comparison under an identical optimization budget and reflects our practical goal of repairing an SFT checkpoint post-hoc rather than retuning RL hyperparameters for each checkpoint.

We have clarified both points in Section 4.1 and Appendix B of the revised manuscript.


Thanks again for your valuable feedback. We would be happy to address any further questions.

References

[1] Getting Your LLMs Ready for Reinforcement Learning with Lightweight SFT. ICLR 2026.

[2] Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning. ICML 2026.

[3] Hindsight Merging: Diverse Data Generation with Language Models. UAI 2025.

[4] LM-Cocktail: Resilient Tuning of Language Models via Model Merging. ACL 2024 Findings.

[5] Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks. ECCV 2024.

Thank you for the responses and additional experiments, which address my concern about missing baselines and provide stronger evidence for attribution-guided neuron reset. I would like to raise my rating.

AuthorsOpenReview ↗

Thanks for your reply

Thank you very much for your follow-up and for raising your rating! We truly appreciate your time and constructive feedback throughout the review process, which helped us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.