- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper analyzed how overtraining in the SFT stage can lead to loss of plasticity that makes it difficult to RL during the following stage of posttraining. The authors proposed Rejuvenation, which is composed of weight-level base-anchoring + neuron-level selective reset, to remedy the overtraining and the resulting loss of plasticity. The results demonstrate that rejuvenation is effective in salvaging overtrained SFT checkpoints for RL.
Critical strengths
- The paper provides ample interesting analysis measured in many aspects such as weight/gradient norm, diversity-related metrics (loss, entropy, pass@1 vs. pass@k accuracy, etc).
- I appreciate the simplicity of the remedy proposed, namely the base-anchoring involves a simple linear interpolation with base model weight.
- I also appreciate the per-neuron contribution, which is also finegrained enough and effective.
Critical weaknesses
- The result section could benefit from more ablation studies, namely there are two stages of the rejuvenation, which stage is responsible for the main recovery and how so?
- Can we over rejuvenate? From Fig6 it seems like even 2 epochs of SFT can benefit from rejuvenation? It seems to me that there might be some benefit to the rejuvenation fusion that lead to better performance compared to just RL without SFT (i.e. setting alpha to 0 for the base-anchored linear fusion stage). The authors should definitely show these extreme cases of alpha as well as omega for the interpolation parameters (e.g. in Table 5), and in Fig6 consider showing RL without SFT or rejuvenation.
- The interesting thing here is the choice of the linear interpolation, and how it works. More analysis needs to go into how the interpolation qualitatively changed the output because intuitively we don't expect that linearly interpolating weights give us a linear interpolation in the behavior outcome, which is dependent on the output logits. Presumably, the highly confident token logits resulted from over SFTing can be "sticky," meaning tolerate some range of weight perturbation before a flip in outcome occurs.
- Overall, I think the key insight here is that SFT leads to overconfident token distrib, which is amplified via RL. I think this point is worth being highlighted more as motivating design principle.
- Figure captions can be more descriptive.
Questions for authors
Questions are contained in the weaknesses.
4 comments
Rebuttal (Part 2/2)
W3: The interesting thing here is the choice of the linear interpolation, and how it works. More analysis needs to go into how the interpolation qualitatively changed the output because intuitively we don't expect that linearly interpolating weights give us a linear interpolation in the behavior outcome, which is dependent on the output logits. Presumably, the highly confident token logits resulted from over SFTing can be "sticky," meaning tolerate some range of weight perturbation before a flip in outcome occurs.
A: We agree that linear weight interpolation induces nonlinear changes in output logits and policy behavior, so its effect on sampling diversity should be explained through logit margins rather than assumed from Eq. (1) alone. While Hindsight Merging [1] and LM-Cocktail [2] motivate weight interpolation (Sections 2 and 3.3), they do not analyze how it reshapes token-level distributions at the SFT-to-RL handoff.
Your "sticky token" intuition captures why moderate interpolation works well here. As shown in Figure 7 and Table 1, OverSFT pushes top-1 logits far above runner-up candidates (~45 vs. ~20 in ModSFT), collapsing token entropy from 0.184 to 0.024 and narrowing the Pass@K–Pass@1 gap from 15.3 to 11.2. Because this logit margin is so large, scaling the SFT weight delta () by compresses the top-1/top-2 gap without flipping the greedy argmax token at most positions. As a result, basic formatting and step-by-step reasoning priors remain intact ("sticky"), while under softmax with temperature-1.0 RL rollouts (Appendix B), the reduced margin shifts probability mass back to runner-up tokens so RL can sample alternative reasoning continuations.
This qualitative shift is reflected in our quantitative ablations: fusion alone () raises pre-RL token entropy from 0.024 (OverSFT, Table 1) to 0.146 and recovers post-RL overall accuracy from 14.9 to 17.0 (Table 4). Moreover, Table 5 illustrates the boundary of this stickiness: at , logits remain overly rigid, preserving ID accuracy (19.7) at the cost of OOD exploration (11.9); at , excessive shrinkage erodes SFT priors and lowers ID accuracy (18.5). We have expanded Section 3.3 in the revised manuscript to clarify how weight scaling narrows logit margins to balance prior retention with exploration.
W4: Overall, I think the key insight here is that SFT leads to overconfident token distrib, which is amplified via RL. I think this point is worth being highlighted more as motivating design principle.
A: We have elevated this insight to a core motivating principle in the Introduction and Section 3 of the revised manuscript. Extending SFT from ModSFT to OverSFT raises MATH-500 Pass@1 from 15.2 to 15.8, while token entropy drops from 0.184 to 0.024 and the Pass@K–Pass@1 gap narrows from 15.3 to 11.2 (Table 1). Our token-level visualizations (Figures 9 and 10) show that basic arithmetic steps in GSM8K have low entropy in both ModSFT and OverSFT, reflecting genuine computation skills. In contrast, OverSFT becomes artificially over-confident on tokens that should not be memorized, such as person names in GSM8K word problems (e.g., Eric in Figure 9) and rigid reasoning connectives.
Our design directly addresses this distinction: global fusion scales down overall SFT drift to retain distributed task skills, while local reset relaxes sparse high-attribution neurons driving spurious memorization. We also frame further confidence concentration during RL as an observed failure mode in our setting rather than a universal property of RL.
W5: Figure captions can be more descriptive.
A: Thanks for your suggestion. We have expanded the figure captions in the revised manuscript.
Thanks again for your valuable feedback. We would be happy to address any further questions.
References
[1] Hindsight Merging: Diverse Data Generation with Language Models. UAI 2025.
[2] LM-Cocktail: Resilient Tuning of Language Models via Model Merging. ACL 2024 Findings.
Thank you for the detailed responses, my concerns have been largely resolved. I will keep my positive rating!
Thanks for your reply
Thank you very much for your follow-up and for maintaining your positive rating! We truly appreciate your time and constructive suggestions throughout the review process, which have helped us strengthen the paper.
Rebuttal (Part 1/2)
We thank the reviewer for the constructive comments. Our responses to all weaknesses and questions are as follows.
Weaknesses
W1: The result section could benefit from more ablation studies, namely there are two stages of the rejuvenation, which stage is responsible for the main recovery and how so?
A: Our component ablation (Table 4) shows that global fusion accounts for most of the recovery (+2.1 points), while neuron reset adds +0.5 points:
The two stages play complementary roles. Global fusion uniformly scales down SFT-induced parameter shifts (θOverSFT−θbase, Section 3.3), dampening sharp weight spikes and smoothing the optimization landscape (Figures 3 and 4). Attribution-guided reset then restores a small set of rigid, high-contribution neurons (mostly in late attention key/value and MLP projections, Figure 11) to their base values (Section 3.4), reopening token entropy at over-confident reasoning steps (Figures 9 and 10) for an extra +0.5-point gain, concentrated on OOD tasks (14.8 to 15.3).
W2: Can we over rejuvenate? From Fig6 it seems like even 2 epochs of SFT can benefit from rejuvenation? It seems to me that there might be some benefit to the rejuvenation fusion that lead to better performance compared to just RL without SFT (i.e. setting alpha to 0 for the base-anchored linear fusion stage). The authors should definitely show these extreme cases of alpha as well as omega for the interpolation parameters (e.g. in Table 5), and in Fig6 consider showing RL without SFT or rejuvenation.
A: 1. Over-rejuvenation trade-off. Yes. Pulling weights too close to the base model erases useful SFT priors: lowering α from 0.5 to 0.4 drops ID Avg. from 19.3 to 18.5, whereas raising α to 0.6 hurts OOD Avg. (14.8 to 11.9) (Table 5). Similarly, increasing the reset ratio p from 1% to 2% lowers overall accuracy from 17.5 to 17.3 (Table 6).
2. Why SFT is needed before RL (
Base+RL, α=0) and why 2-epoch SFT still benefits. We adopt EvoLM-4B as our base model, which is a clean pre-trained model with publicly available training data and code and without benchmark contamination (Appendix B). Because it has not been heavily post-trained on math data, its mathematical capability prior to SFT is relatively weak. Consequently, skipping SFT and running RL directly from the base model (Base+RL, α=0) yields little to no improvement due to sparse rewards, making SFT necessary to establish basic mathematical reasoning and formatting priors before RL. At 2 SFT epochs (ModSFT), the model has acquired these priors without severe plasticity loss, and Rejuvenation yields a modest additional gain (17.6 vs. 17.0 on EvoLM-4B; 54.3 vs. 53.8 on Qwen3-8B, Figure 6) by slightly widening exploration room.3. Extreme cases of α and ω(i). In Eq. (1), α=0 gives
Base+RLand α=1 givesOverSFT+RL. In Eqs. (4)–(5), ω(i)=I(i∈S)∈{0,1} is a binary mask (not a continuous weight), where ω(i)=1 restores neuron i to θbase. We have added this discussion and theBase+RLreference to the revised manuscript.We have added discussion on these points in the revised version.