acceptodds
Under review as a conference paper at ICLR 2027

Surgical Post-Training: Improving Reasoning While Mitigating Forgetting

Abstract

Injecting new reasoning knowledge into Large Language Models (LLMs) via post-training often induces catastrophic forgetting. Recent studies emphasize the importance of on-policy data in mitigating forgetting, but suggest that explicit KL regularization does not explain the observed retention benefits. We study the implicit regularization induced by reference-relative reward objectives: gradient analysis reveals adaptive attenuation of updates, while controlled comparisons on the same rectified data demonstrate improved capability retention. This motivates our Surgical Post-Training (SPoT), a post-training framework designed to optimize reasoning efficiently while preserving prior knowledge. SPoT consists of (1) a data rectification pipeline employing an Oracle to surgically correct erroneous steps via minimal edits, generating proximal on-policy data; and (2) a reward-based binary cross-entropy objective that learns from both corrected and erroneous responses. Empirically, with only 4k rectified math pairs, SPoT improves Qwen3-8B's accuracy by 6.3 percentage points on average across 9 in-domain and out-of-domain tasks, requiring merely 16 minutes of model training on 8× H800 GPUs. We further validate SPoT through Connect4 post-training, where it improves both game and mathematical reasoning while retaining instruction-following capabilities. Moreover, SPoT provides a superior initialization for subsequent reinforcement learning, enabling further gains in reasoning performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
reasoning, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2QQPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Surgical Post-Training (SPOT) to improve LLM reasoning while reducing catastrophic forgetting. Its main contribution combines minimal-edit data rectification with a reward-based objective. (1) Minimal-edit data rectification: Converts model failures into erroneous–corrected response pairs close to the student’s distribution. (2) Reward-based objective: Uses reference-relative BCE/BCO to reinforce corrected traces, suppress errors, and adaptively attenuate updates.

Critical strengths
  1. Good empirical analysis and clear motivation: The analysis of the Elastic Tether, pull-up effect, and DPO’s learning dynamics clearly motivates each component. It provides useful insight into how training objectives affect capability retention during post-training.
  2. Simple and efficient approach: SPOT achieves substantial improvements with a simple design, limited training data, and short training time.
  3. The paper is well organized and easy to follow,
Critical weaknesses
  1. SPOT combines existing BCE/BCO objectives with teacher-guided correction. Much of the analysis also builds on findings from prior work, such as pull-up effects and DPO learning dynamics. While the empirical results on reasoning improvement and capability retention are valuable, I find the methodological contribution somewhat incremental.
  2. Experiments only cover models up to 8B and two training domains. OOD reasoning is evaluated only on GPQA-D and Connect4, while general capabilities are mainly assessed using IFEval. This makes it unclear how well the findings generalize to larger models and more diverse tasks.
Questions for authors
  1. Appendix F.1 reports a 56.94% parsing-failure rate for Qwen3-8B in a diagnostic evaluation. Could the authors report corresponding rates for the base and trained models on the main Connect4 benchmark? This would help assess whether improved output formatting contributes to the reported gains.
3 comments

We thank the reviewer for the positive recognition on our analysis and motivation, as well as the constructive feedback. Below, we address each point in detail:

1. Contribution Beyond Combining Existing Objectives

As noted in Sec. 4.3, our contribution lies not in inventing the BCE/BCO loss forms themselves, but in uncovering how objective design and data proximity interact:

  1. Finding the objective's independent role in knowledge retention (Sec. 3, App. D.2): While prior work (Chen et al., 2025; Shenfeld et al., 2025) attributes capability retention primarily to on-policy data, but not loss design like KL-regularized effect, we come to a different conclusion. We show that data proximity alone is insufficient and that the training objective plays an independent role. Comparing SFT+ and Reward-SFT on the exact same positive-only rectified data (y+y^+), SFT+ still suffers monotonic IFEval decay (83.0 → 80.0) as does self-generated RFT (81.5), whereas switching only the loss to Reward-SFT preserves IFEval (83.7).

  2. Text-only black-box OPD and superior RL initialization. Unlike logit-based OPD, SPoT requires only text edits. For subsequent RL exploration, SPoT injects new knowledge while retaining diverse valid paths.


2. Generalization to Larger Models and More Diverse Tasks

We conducted additional evaluations across a larger model family, coding, and broader general-capability benchmarks:

  • Larger model scale (Phi4-14B): Using gpt6-luna with only 2k rectified pairs, SPoT improves Phi4-14B from 43.96 → 45.87 (+1.91 average across 9 benchmarks, e.g., +2.70 on AIME24, +4.00 on AMC23, and +2.52 on GPQA-Diamond) while preserving IFEval (61.74 → 61.28).
BenchmarkPhi4-14BPhi4-14B (SPoT)Δ
AIME2420.0022.70+2.70
AIME2519.3320.33+1.00
AMC2360.0064.00+4.00
MATH-50080.8782.00+1.13
Minerva38.8541.10+2.25
OlympiadBench47.8547.850.00
GPQA-Diamond50.7653.28+2.52
IFEval61.7461.28−0.46
Connect416.2520.30+4.05
Average43.9645.87+1.91
  • Coding (MBPP+) & general capabilities (TruthfulQA, MMLU-Pro): We additionally eval on coding task. On Qwen3-8B, math-trained SPoT improves MBPP+ from 63.23% → 64.81% (+1.58%).
  • Moreover, evaluating all post-training baselines under a unified protocol on TruthfulQA and MMLU-Pro shows that SPoT achieves the highest TruthfulQA scores and best preserves MMLU-Pro (−0.70, within std), whereas all baselines drop noticeably on MMLU-Pro (−1.27 to −4.27):
Benchmark (Qwen3-8B)BaseSFT+DPOSmaug / DPOPKTOSPoT
TruthfulQA MC136.4736.1136.3536.4736.6036.84
TruthfulQA MC254.5354.3754.3954.3754.3655.07
MMLU-Pro66.20 ± 1.6063.40 ± 2.0961.93 ± 0.9062.53 ± 1.9264.93 ± 2.1665.50 ± 1.70

We will include these results in the revision.


3. Connect4 Output Parsing-Failure Rates and Source of Gains

On Connect4, the parsing-failure rate drops from 56.94% (Base) to 35.2% (SPoT) (−21.74%). Decomposing the net gain, 79.8% comes from improved format adherence and 20.2% comes from correcting the answer content on already-parseable outputs.

This should not be viewed as inflated by format fitting: the Table 1 SPoT model was trained exclusively on math data with zero Connect4 format supervision. Maintaining output formatting during unfamiliar OOD reasoning is itself a core generalization gain—whereas SFT/SFT+ lose −3.4/−3.0 on IFEval. Together with the 20.2% gain on already-parseable answers, both OOD instruction adherence and reasoning contribute to the improvement. We will add this breakdown to Appendix F.1.


We hope these additional experiments and clarifications address your concerns. We would be glad to answer any further questions.

Dear Reviewer,

With the discussion deadline approaching, we would greatly appreciate your feedback on our rebuttal. Please let us know if any concerns remain or if further clarification would be helpful.

Thank you for your time and consideration. We look forward to hearing from you.

AuthorsOpenReview ↗

Providing SPoT checkpoints/ Follow up

Dear Reviewer,

As the discussion deadline approaches, we would like to highlight the additional evidence and resources provided in our rebuttal. We have added MBPP+ results, complete evaluations on MMLU-Pro and TruthfulQA, and experiments with Phi-4 (14B), extending our evaluation to a larger model from a different model family.

We have also released the SPoT checkpoints, (Qwen3-8B and Llama-3.1-8B-Instruct) and the 500-position Connect4 evaluation set to support reproducibility, available on the anonymous download page. Alongside our evaluation suite, and code shown in the supplementary materials to facilitate reproducibility and further research. These resources enable the reviewers and community to directly evaluate our checkpoints on additional out-of-distribution tasks across other domains.

We hope these additional results and efforts can be taken into account in the final assessment. We would greatly appreciate any feedback on whether our responses address your concerns.

Reviewer_hYDVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Surgical Post-Training (SPoT) to improve reasoning while preserving existing capabilities. An oracle minimally corrects student responses, and the resulting pairs are filtered by token overlap. Experiments show reasoning gains with better instruction-following retention than SFT baselines.

Critical strengths
  1. SPoT offers a simple recipe that improves reasoning while retaining instruction following. This useful trade-off is the main practical contribution supporting acceptance.
Critical weaknesses
  1. It is hard to build the insight connecting the "surgical process" with "Mitigating Forgetting". Authors should further justify how the precise data processing could decrease forgetting.
  2. The gradient analysis explains attenuation, but does not directly show that updates concentrate on erroneous portions. The proposed mechanism for preserving unrelated capabilities remains insufficiently validated.
  3. It is limited to putting IF as the only general capability. The experiments on TruthfulQA and MMLU-Pro comparing SPoT are shown in the appendix, but only with the original model. Authors should also evaluate SFT and DPO on these benchmarks to show whether SPoT preserves existing capabilities better than these methods.
Questions for authors

See weakness.

3 comments

We thank the reviewer for the constructive feedback. Below, we address each point in detail:

1. Connection Between Surgical Data Processing and Mitigating Forgetting

Surgical data processing mitigates forgetting through two mechanisms:

  1. On-policy proximity (Sec. 2, App. E.3): Minimal edits and LCS filtering (RLCS≤0.6R_{\text{LCS}} \le 0.6) keep y+y^+ close to πθ\pi_\theta—verified by lower student perplexity (mean PPL: 1.60 vs. 1.95 for RLCS>0.6R_{\text{LCS}} > 0.6)—thereby reducing off-policy distributional shift (Chen et al., 2025; Shenfeld et al., 2025).
  2. Localized updates via gradient cancellation (Sec. 4.5): Because (y−,y+)(y^-, y^+) share a long prefix, opposing gradients from L+\mathcal{L}^+ and L−\mathcal{L}^- largely cancel out on shared correct tokens, concentrating updates on the localized error span rather than inducing global weight drift.

Empirically, isolating data processing under the same SFT loss (SFT → SFT+) cuts Llama-3.1 IFEval forgetting from −11.5 to −5.0 (Table 2).

2. Update on Errorneous portion

Beyond sample-wise attenuation, Section 4.5 discusses shared-prefix gradient cancellation: positive and negative loss terms contribute opposing gradients on shared content while retaining separate supervision on divergent continuations. For a pair y±=p⊕s±y^\pm=p\oplus s^\pm, the shared-prefix contribution is

g_{\mathrm{prefix}} &=\beta(w_- - w_+)\, \nabla_\theta\log\pi_\theta(p\mid x),\\ w_+&=1-\sigma(r_\theta(x,y^+)-\delta),\\ w_-&=\sigma(r_\theta(x,y^-)-\delta). \end{aligned}$$ Here, $\delta$ is held constant during differentiation, with $\delta=0$ for BCE. When the weights are balanced, the shared-prefix contribution cancels. This directly supports our conditional wording, **“can reduce the net update on the shared prefix.”** Together with Oracle-guided localized corrections, this motivates the surgical interpretation. We will make this analysis explicit in the revision. ### 3. Preserving Exsiting Abilities Alongside our OOD evaluations in the main paper (**GPQA-Diamond**, **Connect4**) and the newly added **MBPP+** coding benchmark (where SPoT improves Qwen3-8B from 63.23% to **64.81%**), we evaluated **Base**, **SFT+**, **DPO**, **Smaug / DPOP**, **KTO**, and **SPoT** under a unified standardized evaluation protocol on **TruthfulQA (MC1/MC2)** and **MMLU-Pro** (which we will also use to update Appendix Table 9): | Benchmark (Qwen3-8B) | Base | SFT+ | DPO | Smaug | KTO | SPoT | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | TruthfulQA MC1 | 36.47 | 36.11 | 36.35 | 36.47 | 36.60 | **36.84** | | TruthfulQA MC2 | 54.53 | 54.37 | 54.39 | 54.37 | 54.36 | **55.07** | | MMLU-Pro | **66.20 ± 1.60** | 63.40 ± 2.09 | 61.93 ± 0.90 | 62.53 ± 1.92 | 64.93 ± 2.16 | 65.50 ± 1.70 | The results confirm that **SPoT preserves general capabilities substantially better than all compared baselines**: * On **TruthfulQA**, SPoT achieves the highest scores on both MC1 (**36.84**, +0.37 over Base) and MC2 (**55.07**, +0.54 over Base), whereas the baselines slightly degrade or stay flat. * On **MMLU-Pro**, **DPO** (−4.27), **Smaug** (−3.67), **SFT+** (−2.80), and **KTO** (−1.27) all experience clear drops relative to Base, whereas **SPoT** stays closest to the Base model. --- We hope these additional experiments and clarifications address your concerns. We would be glad to answer any further questions.

Dear Reviewer,

With the discussion deadline approaching, we would greatly appreciate your feedback on our rebuttal. Please let us know if any concerns remain or if further clarification would be helpful.

Thank you for your time and consideration. We look forward to hearing from you.

AuthorsOpenReview ↗

Providing SPoT checkpoints/ Follow up

Dear Reviewer,

As the discussion deadline approaches, we would like to highlight the additional evidence and resources provided in our rebuttal. We have added MBPP+ results, complete evaluations on MMLU-Pro and TruthfulQA, and experiments with Phi-4 (14B), extending our evaluation to a larger model from a different model family.

We have also released the SPoT checkpoints, (Qwen3-8B and Llama-3.1-8B-Instruct) and the 500-position Connect4 evaluation set to support reproducibility, available on the anonymous download page. Alongside our evaluation suite, and code shown in the supplementary materials to facilitate reproducibility and further research. These resources enable the reviewers and community to directly evaluate our checkpoints on additional out-of-distribution tasks across other domains.

We hope these additional results and efforts can be taken into account in the final assessment. We would greatly appreciate any feedback on whether our responses address your concerns.

Reviewer_humvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that the training objective also affects retention on the same proximal data, with DPO and Reward-SFT preserving instruction-following performance better than SFT. The paper presents SPoT, a post-training framework that combines a surgical rectification pipeline with a binary optimization objective to efficiently inject reasoning knowledge while mitigating forgetting in our evaluations. The paper also shows that SPOT establishes a superior initialization for subsequent reinforcement learning. To evaluate OOD reasoning without data contamination, the paper dynamically constructs a Connect4 dataset for reliable evaluation.

Critical strengths

(1) The paper is handling a fundamental and important challenge. The paper is well motivated. The observations are interesting.

(2) The illustration is intuitive. The developed approach has clear intuitions.

Critical weaknesses

(1) The evaluation could be more comprehensive. It is not clear why only Qwen3-8B and Llama-3.1-8B-Instruct are selected and used in the evaluation. It would be great to present more results using other reasoning models (e.g., some models among gpt-oss, Phi-4-reasoning or DeepSeek models). It is not clear why the baselines used in the evaluation of Qwen3-8B (Table 1) and the evaluation of Llama-3.1-8B-Instruct (Table 2) are different. It would be great to evaluate on the models with various sizes. The paper title and some contributions mentioned in the introduction seem to be generic but the evaluations mainly focus on math reasoning and game reasoning. Some other types of reasoning datasets (e.g., coding, agentic) might be considered.

(2) (minor) Some case study results could be added to show how the reasoning is improved while mitigating forgetting.

Questions for authors

It would be great to add some discussions or evaluations.

It is not clear why only Qwen3-8B and Llama-3.1-8B-Instruct are selected and used in the evaluation. Would the approach be feasible when using other reasoning models (e.g., some models among gpt-oss, Phi-4-reasoning or DeepSeek models)?

It is not clear why the baselines used in the evaluation of Qwen3-8B (Table 1) and the evaluation of Llama-3.1-8B-Instruct (Table 2) are different.

Would the approach be feasible when using the models with various sizes?

Would the approach be feasible when using other types of reasoning datasets (e.g., coding, agentic)?

Some case study results could be added to show how the reasoning is improved while mitigating forgetting.

2 comments

We thank the reviewer for recognizing our motivation, intuition, and the importance of the problem we address, as well as for the constructive suggestions. Below, we respond to each point in detail.

1. Generalization Across Architectures & Scales

We selected Qwen3-8B and Llama-3.1-8B-Instruct as our primary testbeds because they represent two distinct, widely adopted open-source model families. Evaluating SPoT across both families demonstrates that our method does not depend on model-specific architectures or pre-training recipes. In addition, we already report results on a smaller model (Qwen3-1.7B) in the Appendix.

To further verify scalability on a larger model from a third family, we conducted an additional evaluation on Phi4-14B. Using gpt6-luna as the teacher model, we generated 2k training samples for SPoT. Even under this limited 2k-sample budget due to rebuttal time constraints, SPoT achieves consistent improvements across reasoning benchmarks (+1.91 on average) while preserving instruction-following performance on IFEval. We expect that scaling up the training data would yield further gains.

BenchmarkPhi4-14BPhi4-14B (SPoT)Δ
AIME2420.0022.70+2.70
AIME2519.3320.33+1.00
AMC2360.0064.00+4.00
MATH-50080.8782.00+1.13
Minerva38.8541.10+2.25
OlympiadBench47.8547.850.00
GPQA-Diamond50.7653.28+2.52
IFEval61.7461.28−0.46
Connect416.2520.30+4.05
Average43.9645.87+1.91

2. Consistency of Baselines Between Table 1 and Table 2

We agree that aligning the baseline coverage across Table 1 and Table 2 makes the comparison clearer. We have completed the preference-optimization baselines (DPO, Smaug, and KTO) on Llama-3.1-8B-Instruct, alongside the Base and SPoT results from Table 2:

BenchmarkBaseDPOSmaugKTOSPoT
AIME243.34.04.04.34.0
AIME252.70.30.70.72.1
AMC2319.521.322.327.027.0
MATH-50047.048.449.748.548.6
Minerva22.422.922.923.926.1
OlympiadBench16.916.517.717.316.3
GPQA-Diamond28.530.831.830.830.0
Connect45.07.66.64.07.0
IFEval73.670.970.665.873.2
Average24.324.725.124.726.0

Overall, SPoT achieves the highest overall average (26.0) across all 9 benchmarks. In particular, whereas DPO, Smaug, and KTO suffer noticeable drops on IFEval (73.6 → 70.9 / 70.6 / 65.8), SPoT maintains instruction-following performance (73.2) while delivering strong gains on reasoning tasks.


3. Evaluation on Other Domains

In addition to the general-capability evaluations on MMLU-Pro and TruthfulQA in the Appendix, we conducted an extra evaluation on MBPP+ to assess code generation performance:

BenchmarkQwen3-8BQwen3-8B (SPoT)Δ
MBPP+63.23%64.81%+1.58%

4. Qualitative Case Studies

To concretely illustrate the behavioral changes induced by SPoT, we provide three paired case studies comparing Qwen3-8B and Qwen3-8B (SPoT) under identical prompting and decoding settings:

  1. Improved arithmetic accuracy with preserved reasoning structure (MATH-500): On a two-stage divisor-sum problem, both models correctly derive the intermediate value (220) and enumerate its proper divisors. While the base model miscalculates the final sum (234), SPoT retains the same decomposition and performs accurate step-by-step accumulation to obtain the correct answer (284).
  2. Better edge-case handling in code generation (HumanEval/97): When tasked with multiplying the units digits of two integers, the base model applies % 10 directly and fails on negative inputs (returning 4 instead of 54 for (-6, -9)). SPoT preserves the clean implementation structure while adding abs() prior to modulo extraction, passing all 985 test cases.
  3. Preserved algorithmic capability (HumanEval/40): On the 3-Sum problem, both models generate structurally identical solutions (identical ASTs) using sorting and a two-pointer search, and both pass all 988 test cases—confirming that existing coding competence is retained after SPoT training.

Full paired examples are available at: Interactive HTML Examples


We hope these additional experiments, clarifications, and case studies address your concerns. We would be glad to answer any further questions.

Thank you very much for the detailed responses. I think my questions have been partially addressed. I would increase the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.