acceptodds
Under review as a conference paper at ICLR 2027

Fine-Tuning at What Cost? Learning-Forgetting Trade-offs Across Training Algorithms

Abstract

Fine-tuning Large Language Models (LLMs) on specialized tasks often comes at the cost of degraded performance on general capabilities—a phenomenon known as catastrophic forgetting. While numerous fine-tuning methods have been proposed, systematic comparisons of their learning-forgetting trade-offs remain limited. In this work, we present the first comprehensive empirical study comparing various fine-tuning methods spanning supervised fine-tuning (SFT) variants, reinforcement learning (RL) variants, and on-policy distillation (OPD) variants. We evaluate learning gains on target tasks against forgetting measured on a diverse benchmark suite. Using Pareto frontier analysis, we characterize the achievable trade-offs for each method across learning rates and training durations. Our results reveal that no single algorithm universally dominates the learning-forgetting Pareto front, and the trend varies across models and datasets. What holds across different algorithms is that learning rates affect both learning and forgetting, while training steps primarily affect learning but not forgetting after the initial steps. Through this study, we suggest that algorithm and hyperparameter selection interact with model architecture and target dataset in ways that preclude universal recommendations, and that the learning-forgetting trade-off is better understood through Pareto front analysis than single-point comparisons.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, fine-tuning, forgetting, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_F5eXOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines learning as the performance change on the target task and forgetting as the equally weighted mean performance change across five non-target tasks. For each algorithm, checkpoints from different learning rates and training stages form a point set that is compared through its nondominated frontier and hypervolume relative to a fixed nadir point. Experiments cover Qwen2.5-1.5B-Instruct and LLaMA3.1-8B-Instruct, with Countdown and MMLU as target tasks. Additional studies examine KL regularization, LoRA rank, TrSFT clipping, degree of off-policy training, data scale, ten-epoch training, and recovery from forgetting.

Critical strengths
  1. The Pareto perspective addresses the right evaluation problem. Relative performance changes with learning rate and stopping time, so a single checkpoint can yield a misleading ranking. The four model–task settings demonstrate this clearly.
  2. The diagnostic experiments are thorough. The methods use the same filtered query sets, and the paper examines teacher traces, rollout reuse, data scale, extended training, earlier checkpoints, and benchmark-level sources of forgetting. These analyses make early-stage forgetting a credible and important empirical finding.
Critical weaknesses
  1. The frontiers and rankings lack statistical support. Several hypervolume gaps are only at the third or second decimal place, yet the paper reports no repeated runs, bootstrapped frontiers, hypervolume confidence intervals, or rank-stability probabilities. The adaptive stopping rule over learning rates also affects the estimated frontiers.
  2. The cost definition is not aligned with algorithmic budgets. Averaging five benchmarks equally can obscure domination by a single benchmark. RL, OPD, SFT, and LoRA also differ in rollout generation, batch size, teacher inference, and trainable parameter count. Without frontiers under matched FLOPs, tokens, or wall-clock time, the study cannot establish which method is preferable at equal cost.
Questions for authors
  1. If each main setting were run with 3–5 independent seeds and jointly bootstrapped over evaluation examples and training seeds, would the Pareto dominance relations, hypervolume rankings, and the claim that 83.35% of forgetting occurs before the first checkpoint remain stable?
  2. How do the frontiers for SFT, GRPO, REINFORCE, SDFT, and LoRA change under matched training FLOPs, generated tokens, or wall-clock time? Please include teacher and rollout costs, and report leave-one-benchmark-out or weighting-sensitivity analyses.
4 comments
AuthorsOpenReview ↗

Author Response 1/3

Thank you for your review. We address each point below.

The frontiers and rankings lack statistical support. Several hypervolume gaps are only at the third or second decimal place, yet the paper reports no repeated runs, bootstrapped frontiers, hypervolume confidence intervals, or rank-stability probabilities.

The reported rankings describe the evaluated checkpoint sets; the paper does not establish that every small HV difference is statistically significant. Our main evidence is the different frontier shapes and attainable learning ranges in Figures 1–3, supported by trajectories in Figures 5 and 16–23.

While repeated runs are prohibitively expensive, we do provide results that address sensitivity to the tested data sizes and training durations. Approximately 6 learning rates × 10 algorithms × 2 models × 2 datasets require 240 training runs; 5 checkpoints evaluated on 6 benchmarks yield roughly 7,200 evaluations. Replication exceeds our compute budget. Existing checks vary SFT demonstrations (Appendix F.1.1), increase Countdown queries from 4,161 to 10,000 and MMLU queries from 9,645 to 12,000 for GRPO and REINFORCE (F.2.5), and extend Qwen/Countdown training from two to ten epochs for SFT, GRPO, and SDFT, including constant-learning-rate variants (F.2.6). More data has mixed effects without overturning the LLaMA comparison; longer training adds little forgetting, and two-epoch peak learning is close to the ten-epoch peak.

If each main setting were run with 3–5 independent seeds and jointly bootstrapped over evaluation examples and training seeds, would the Pareto dominance relations, hypervolume rankings, and the claim that 83.35% of forgetting occurs before the first checkpoint remain stable?

Repeating training and evaluation runs will be prohibitively expensive due to the number of algorithms, hyperparameters, models, datasets, and intermediate checkpoints, and evaluation benchmarks. However, we did have additional evidence for the early forgetting phenomenon.

Appendix F.3 reports 83.75%, 78.07%, 83.72%, and 87.87% first-checkpoint forgetting shares, averaging 83.35%. Figures 24–25 show most forgetting already occurs by approximately 10 steps for SFT/SDFT and 16 for GRPO on Qwen/Countdown. Figures 14–15 extend training to ten epochs with little additional forgetting. Appendix F.3 identifies exceptions to the timing observation.

The adaptive stopping rule over learning rates also affects the estimated frontiers.

The learning-rate stopping rule is validated by our pilot study on SFT. Across both Countdown backbones, we also tested 1e-7, 5e-7, 1e-3, and 5e-3. The latter two exceed the stopping rate of 5e-4; their checkpoints learn less and forget more than those at 5e-4 and are dominated by existing frontier points.

Nevertheless, it is not possible to fully recover the true Pareto front, and our empirical fronts are approximations.

AuthorsOpenReview ↗

Author Resposne 2/3

The cost definition is not aligned with algorithmic budgets. RL, OPD, SFT, and LoRA also differ in rollout generation, batch size, teacher inference, and trainable parameter count. Without frontiers under matched FLOPs, tokens, or wall-clock time, the study cannot establish which method is preferable at equal cost.

The cost studied in the paper is degradation of non-target capabilities. For this study, we do not need a claim about equal-compute optimality to establish the observed learning–forgetting trade-offs.

Matching compute would require accounting for SFT demonstration generation, RL rollouts, distillation teacher/student inference, and full-tuning versus LoRA update costs. It also requires choosing how to amortize demonstrations and whether to charge hyperparameter search. Therefore, equal compute implies that training data has to differ across algorithms. As such, with compute cost introduced as an additional aspect, is it not possible to have a fair comparison in every sense.

AuthorsOpenReview ↗

Author Response 3/3

Averaging five benchmarks equally can obscure domination by a single benchmark.

Appendix I, which decomposes forgetting by benchmark both along selected Pareto-front points and along training trajectories, directly addresses aggregation. The composition of forgetting therefore depends on the target task, with a particularly substantial safety contribution when MMLU is the target. Table 26 attributes 36.53% of Countdown-target forgetting to MMLU and 20.05% to MATH; for MMLU targets, WildJailbreak contributes 65.91% and IFEval 18.77%. Tables 27–28 give per-setting breakdowns. We will highlight these findings.

We do not establish that method rankings are invariant to changing the benchmark weights, but neither is such invariance required for the reported macro-average comparison. Equal weights define the evaluated objective, without implying equal realized contributions or preservation of every capability.

Please include teacher and rollout costs, and report leave-one-benchmark-out or weighting-sensitivity analyses.

For benchmark sensitivity, we report the each non-target benchmark's contributions to forgetting in Appendix I. We additionally evaluate benchmark omission on its 12 Countdown trajectories (two models × three algorithms × two learning rates). After each omission, we recompute the four-benchmark mean forgetting magnitude, d(t) = max(0, −ΔF(t)), at every saved checkpoint, calculate each run's d(first) / max_t d(t), and average across runs. Runs with zero maximum forgetting have undefined ratios and are excluded.

TargetOmitted non-target benchmarkMean first-checkpoint share of maximum within-run forgettingRuns included
CountdownMMLU78.6%12
CountdownIFEval79.8%12
CountdownMATH83.7%11
CountdownWildGuardTest81.1%12
CountdownWildJailbreak83.2%12

The early forgetting conclusion holds even under leave-one-benchmark-out settings. The first checkpoint accounts for 78.6%–83.7% of maximum within-run forgetting on average across these Countdown runs. Only MATH omission excludes a zero-forgetting run.

Dear Reviewer,

With the author response window closing soon, we would deeply appreciate it if you could go through our responses and reconsider your rating if they address your concerns.

Thank you again for your time and effort in reviewing our paper.

Best regards,

The Paper Authors.

Reviewer_PihjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares ten fine-tuning algorithms from three families in terms of the trade-off between learning and forgetting. The SFT family contains SFT, LoRA, DFT, TrSFT and MSE, the RL family contains REINFORCE and GRPO, and the OPD family contains OPD, SDFT and the authors' own SDFT-teacher-SFT (s.t.s.). Learning is the change ΔL in target test accuracy, and forgetting is the change ΔF in mean accuracy over five non-target benchmarks. The paper reports some findings.

  • No single algorithm or family ranks first in all four settings.
  • Plain SFT holds up well against newer methods, and SDFT is worse than SFT in three of the four settings.
  • On-policy training is not always better. Some off-policy GRPO variants beat the original.
  • Forgetting happens before the first checkpoint at 0.4 epoch, and a few OPD steps recover most of the forgetting.
Critical strengths
  • The comparison is broad and uses one data protocol for all methods. The ten algorithms span the three main post-training paradigms, and all of them are evaluated on the same non-target set.
  • The paper sweeps the full grid of learning rates and steps for each algorithm and compares Pareto fronts, which is a sounder protocol than the usual practice of fixing hyperparameters and comparing final accuracy.
  • The findings on early forgetting and fast recovery are useful. These results speak directly to whether early stopping can avoid forgetting and whether forgetting is cheap to repair.
Critical weaknesses
  • The reported HV values do not match the definition, and HV carries a hidden weighting. Even under the stated definition, HV implies an exchange rate that changes across settings. Near the initial point, the marginal weight of ΔL is the mean non-target accuracy and the marginal weight of ΔF is the target accuracy. In Qwen/Countdown the two weights are about 0.44 and 0.16, so learning counts about 2.8 times more, and with MMLU as the target the weighting roughly reverses.
  • Part of the evidence is in tension with the main claims. The abstract and Section 5.1 stress that no algorithm or family always leads. In Table 11, however, the RL family has an average rank of 1.5 and comes first in three of four settings. The only exception, LLaMA/Countdown, is caused by the likely failed REINFORCE run that ranks ninth there. Observation 2 itself also says RL forgets less than SFT at the same amount of learning.
  • Only two models (1.5B and 8B) are tested, and both targets are short-answer tasks with automatic scoring. Code, long-form generation and domain knowledge injection, which are closer to practice, are missing.
Questions for authors
  • Under the nadir defined in Section 3, the best HV values on Qwen/Countdown, Qwen/MMLU and LLaMA/MMLU exceed the bounding box given by Table 1 and the largest ΔL in the figures. Is HV normalized, do the reference point or the axes differ from the text, and how is (0,0) added?
  • Which teacher does OPD use, given that Section 5.1 calls it a hand-picked stronger teacher? In Figure 6 learning also drops during recovery, for example from about 0.05 to about 0.03 on Qwen/MMLU. Could the authors measure this loss of learning and compare OPD recovery with replay or model merging at the same number of steps?
5 comments
AuthorsOpenReview ↗

Author Response 1/3

Thank you for your review. We address each point below.

The reported HV values do not match the definition, and HV carries a hidden weighting. Even under the stated definition, HV implies an exchange rate that changes across settings. Near the initial point, the marginal weight of ΔL is the mean non-target accuracy and the marginal weight of ΔF is the target accuracy. In Qwen/Countdown the two weights are about 0.44 and 0.16, so learning counts about 2.8 times more, and with MMLU as the target the weighting roughly reverses.

For the definition stated in Section 3, the marginal-weight argument must account for the untrained model at (0,0). Let a be initial target accuracy and b initial mean non-target accuracy. A checkpoint (l,f), with l ≥ 0 and −b ≤ f ≤ 0, has rectangle area (a+l)(b+f), but including the origin gives

HV({(0,0), (l,f)}; (−a,−b)) = ab + l(b+f).

Thus, ∂HV/∂l = b+f and ∂HV/∂f = l, rather than a+l. Near the origin, b/a does not describe the specified calculation. With multiple points, marginal contributions depend on neighboring frontier segments.

Our principal evidence is the two-dimensional Pareto front: whether one checkpoint dominates another does not depend on the HV reference point. Figures 1–3 directly show where RL retains capabilities better and where SFT reaches higher learning gains.

Within each model–target setting, the same reference is specified for every algorithm; raw HV differences across settings are not a common utility scale. HV remains reference dependent for crossing fronts. Subtracting the shared baseline area ab preserves within-setting rankings but does not remove that dependence.

Zitzler et al. "The hypervolume indicator revisited: On the design of Pareto-compliant indicators via weighted integration". Evolutionary Multi-Criterion Optimization, 2007.

Part of the evidence is in tension with the main claims. The abstract and Section 5.1 stress that no algorithm or family always leads. In Table 11, however, the RL family has an average rank of 1.5 and comes first in three of four settings. The only exception, LLaMA/Countdown, is caused by the likely failed REINFORCE run that ranks ninth there. Observation 2 itself also says RL forgets less than SFT at the same amount of learning.

Being best on average, or winning three settings, is fully consistent with failing to dominate universally. Table 11's RL family rank of 1.5 and three first-place finishes establish an average advantage. Our claim concerns leading across all settings and operating points. Section 5.1 calls GRPO highly competitive, and Observation 2 acknowledges RL's retention advantage at matched learning.

Even removing ninth-ranked REINFORCE would leave GRPO's global rank of 6 worse than the SFT family's mean global rank of 4.2 in that setting. Table 10 places GRPO behind SDFT-teacher-SFT, SFT, MSE, TrSFT, and DFT on LLaMA/Countdown. Figure 3 also shows SFT's higher learning cap, independently of family averaging or the HV reference.

Observation 2 explains the distinction: an algorithm that forgets less over a shared learning range does not dominate checkpoints with learning gains beyond its own attainable range. Appendix F.1.1 varies SFT demonstrations, while F.2.5 increases RL data from 4,161 to 10,000 Countdown queries and 9,645 to 12,000 MMLU queries without overturning the LLaMA comparison. The evidence extends beyond one weak REINFORCE run.

We will clarify the distinction between average family rank, best individual method, and whole-front dominance.

AuthorsOpenReview ↗

Author Response 2/3

Only two models (1.5B and 8B) are tested.

The paper does not infer a universal ranking from two models; it demonstrates that rankings and attainable trade-offs already vary within the tested settings. The backbones differ in model family and initial capabilities (Table 1).

Both targets are short-answer tasks with automatic scoring. Code, long-form generation and domain knowledge injection, which are closer to practice, are missing.

Countdown is one of our two primary target tasks and provides a complementary, high-headroom learning scenario. Initial accuracies are 16.00% for Qwen and 15.50% for LLaMA, versus 55.04% and 58.57% on MMLU. Its compositional arithmetic reasoning complements knowledge QA (Figures 1–6), although low initial accuracy alone does not establish a wholly novel capability.

SFT demonstrations contain reasoning plus the answer (Section 4), and prompts request reasoning for both targets (Appendix E.4). Automatic scoring checks final answers, providing consistent evaluation and verifiable RL rewards; training is not answer-only. The five non-target benchmarks additionally assess knowledge or arithmetic, advanced mathematics, instruction following, and safety.

Code, unrestricted long-form generation, and domain knowledge injection would broaden coverage, but they are not prerequisites for the conclusions established by the current experiments. We will scope conclusions to the tested tasks and backbones, without extrapolating rankings to untested settings.

Hendrycks et al. "Measuring massive multitask language understanding". International Conference on Learning Representations, 2021.

AuthorsOpenReview ↗

Author Response 3/3

Which teacher does OPD use, given that Section 5.1 calls it a hand-picked stronger teacher?

OPD recovery uses the original model before SFT, q0, as its teacher: Qwen2.5-1.5B-Instruct or LLaMA3.1-8B-Instruct. Students start from the SFT checkpoints in Table 7. LLaMA3.3-70B-Instruct supplies SFT demonstrations, not the recovery teacher. We will correct Section 5.1's inaccurate “hand-picked stronger teacher” description.

In Figure 6 learning also drops during recovery, for example from about 0.05 to about 0.03 on Qwen/MMLU. Could the authors measure this loss of learning?

Figure 6 already measures both recovery and the accompanying change in target learning: each recovery checkpoint has coordinates (ΔL,ΔF). The reviewer's approximate Qwen/MMLU readings, 0.05 to 0.03, imply a 2-percentage-point accuracy loss, retaining 60% of the original gain. Section 5.2 reports forgetting falling from approximately 5 to 2.5 points after 41 OPD steps at learning rate 5e−7. We will report the target-learning cost alongside recovery; these approximate readings are not exact checkpoint values.

The experiment establishes rapid partial reversibility; it does not claim that OPD is the best recovery method or that recovery is free of a learning trade-off. It uses an unrelated instruction dataset and the original model as teacher. Distillation toward pre-SFT behavior can accompany reduced target gains, motivating evaluation of both axes. OPD recovery is excluded from the main rankings because its starting checkpoint and training data differ.

Could the authors compare OPD recovery with replay or model merging at the same number of steps?

As Section 6 explains, our main comparison focuses on general-purpose fine-tuning algorithms under a common target-task data availability assumption, and as such specialized continual learning methods are excluded. Replay can require previous-task data that practitioners lack; a separate retention dataset adds supervision. OPD recovery is included as a side study to demonstrate reversibility, rather than compare specialized recovery methods.

Reviewer_9JdCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Empirical study of the learning-forgetting trade offs incurred by SFT, RL and OPD on two backbones trained for two different target tasks. The conclusion is that learning rates modulate the learning forgetting tradeoff, as recent literature has shown.

Critical strengths
  • Overall, this is an interesting exploration of the learning forgetting tradeoff on two backbones on a wide variety of finetuning algorithms.
Critical weaknesses

1 .The positioning of the paper in the literature omits crucial prior work and overstates its novelty. Similarly for the related work.

L016: "we present the first comprehensive empirical study comparing various fine-tuning methods spanning supervised fine-tuning (SFT) variants, reinforcement learning (RL) variants, and on-policy distillation (OPD) variants.". The phrasing suggests more novelty than there is. The novelty is in extending prior work to RL.

L054: "2) For methodology, prior work (Chen et al., 2025; Lai et al., 2026; Zhang et al., 2026) typically reports single-point results". This sentence uses prior work to establish a gap in the literature. But fails to cite other prior works that already addressed this problem catalan-tatjer, marek, lin.

L065: "(3) Finally, we are the first to systematically examine the relations between hyperparameters and learning-forgetting trade-offs across different fine-tuning algorithms.". False statement, not the first.

L070: "We introduce the first quantifiable Pareto front-based framework to evaluate learning-forgetting trade-offs for LLM fine-tuning." See catalan-tatjer.

L074: "We conduct the first comprehensive learning-forgetting study on various LLM fine-tuning algorithms". As in L016.

See catalan-tatjer, to find Pareto frontier analysis, overall a very similar experimental setting, analysis and conclusions. Moreover lin studies multiple learning rates per dataset to delineate a similar Pareto frontier although sparser, and implicitly. marek similarly studies the Pareto frontier. Please address the phrasing throughout, and properly attribute prior work.

2. What would happen with longer finetuning tasks, e.g. 100x the amount of data to learn?

3. Is there any difference between what is forgotten when the target task is knowledge acquisition versus circuit learning? Have you tried with any task other then MMLU, which the model already knows very well and learning delta is relatively low?

4. You cite both of these papers, but you do not try their proposed solution, is there any reason not to explore regularization techniques?KL with self-replay marek, or elastic weight conslidation [kirkpatriclk].

If the lack of attribution of crucial prior work, and novelty claims is resolved I would increase my score. However, I would like the experimental setting to include a target task that is more interesting, either knowledge QA that is truly novel for the model (MMLU performance for both backbones is already high), or more interesting "frontier" capabilities such as tool use, search or any new ability that the base model does not have. If the authors successfully finetune the model to acquire a truly novel capability and study the learning forgetting dynamics there, the paper would greatly improve. As it stands, it is too similar to prior work, and I recommend rejection.

Questions for authors

See above

9 comments
AuthorsOpenReview ↗

Author Response 1/3

Thank you for your review. We address each point below.

The positioning of the paper in the literature omits crucial prior work and overstates its novelty. Similarly for the related work.

We will revise the introduction and related work to acknowledge prior Pareto-front and hyperparameter studies and narrow our novelty claims. Catalan-Tatjer et al. study SFT through learning-rate sweeps and Pareto fronts; Lin et al. examine domain-specific SFT and mitigation techniques; Marek et al. investigate learning rates, training duration, and self-generated replay.

Our contribution is the breadth and quantitative comparison across fine-tuning paradigms. Our study:

  • Compares ten algorithms spanning three fundamentally different training paradigms: five SFT variants, two RL variants, and three OPD variants, under a unified, quantitative, Pareto-front-based evaluation protocol. Prior work either restricts comparison to SFT and a small number of alternatives or evaluates each algorithm with a single model checkpoint.
  • Uses hypervolume to quantitatively compare the Pareto fronts of different algorithms.
  • Examines learning-rate and training-duration effects across these families, finding that SFT remains competitive, no algorithm consistently dominates, and off-policy RL can outperform on-policy variants.

We will revise the following passages:

  • L016, L074: Replace “first comprehensive” with a systematic comparison of ten SFT, RL, and OPD algorithms.
  • L054–L056: Distinguish single-checkpoint studies from prior learning-rate sweeps and Pareto-front analyses.
  • L065, L070: Remove priority claims for hyperparameter analysis and Pareto fronts; emphasize cross-paradigm comparisons using hypervolume.
  • Related Work: Discuss the three studies above and distinguish their scope from ours.

Catalan-Tatjer et al. "Learning-Forgetting Optimality in Supervised Finetuning: A Cliff Perspective". HiLD 2026 (4th Workshop on High-dimensional Learning Dynamics).

Lin et al. "SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs". ICLR 2026.

Marek et al. "Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay". arXiv preprint arXiv:2605.26097, 2026.

AuthorsOpenReview ↗

Author Response 2/3

What would happen with longer fine-tuning tasks, e.g. 100x the amount of data to learn?

Our paper already investigates the effects of both longer training durations and larger training datasets.

First, in Appendix F.2.6, we explicitly extend training from two to ten epochs for SFT, GRPO, and SDFT on Qwen/Countdown. Figures 14–15 show little additional forgetting, while peak learning within two epochs is already close to that within ten. This supports the two-epoch search and the early-forgetting observation in Section 5.2 for these runs.

Second, Appendix F.2.5 investigates the effect of increasing the amount of training data. We expand Countdown from 4,161 to 10,000 queries and MMLU from 9,645 to 12,000 for GRPO and REINFORCE. Figure 13 shows mixed effects on Qwen; RL still does not outperform SFT in either LLaMA setting.

These checks address sensitivity to training duration and data size. They do not establish behavior with 100× more unique examples, which is outside this comparison's scope.

Is there any difference between what is forgotten when the target task is knowledge acquisition versus circuit learning? Have you tried with any task other than MMLU, which the model already knows very well and learning delta is relatively low?

Countdown is one of our two primary target tasks and provides a complementary, high-headroom learning scenario. It is evaluated throughout Figures 1–6. Its initial accuracy is 16.00% for Qwen and 15.50% for LLaMA, versus 55.04% and 58.57% on MMLU (Table 1). Its compositional arithmetic reasoning provides a complementary setting with more learning headroom.

Appendix I also directly examines what is forgotten. Table 26 reports these shares of aggregate init-to-extreme forgetting:

  • Countdown target: MMLU 36.53%, MATH 20.05%, WildJailbreak 19.47%, IFEval 13.79%, and WildGuardTest 10.16%.
  • MMLU target: WildJailbreak 65.91%, IFEval 18.77%, WildGuardTest 7.42%, Countdown 5.49%, and MATH 2.40%.

Thus, forgetting composition depends on the target: knowledge QA and mathematics contribute most for Countdown, while safety behavior dominates for MMLU. We will make this finding more prominent.

The experimental setting should include a more interesting target task, such as knowledge QA that is truly novel for the model, or frontier capabilities such as tool use or search that the base model does not have.

Countdown is one of our two primary target tasks and provides a complementary, high-headroom learning scenario. It has initial accuracy near 16% for both backbones (Table 1) and is evaluated throughout Figures 1–6. As discussed above, it complements MMLU with a substantially higher-headroom reasoning task.

Our objective is to compare fine-tuning algorithms under controlled conditions. Tool use, search, and novel knowledge acquisition would extend the study, but the existing results already show that learning–forgetting trade-offs vary across algorithms, backbones, and target tasks. We will scope our conclusions to the two tasks and two backbones evaluated.

AuthorsOpenReview ↗

Author Response 3/3

Is there any reason not to explore regularization techniques?

Our experiments already include regularization and constraints on updates:

  • SFT: SFT-OKL with different KL coefficients (Appendix E.1).
  • RL: GRPO with zero and nonzero KL penalties against the initial model.
  • Other update-constraining methods: LoRA, DFT, and TrSFT.

Is there any reason not to explore KL with self-replay (Marek et al., 2026), or elastic weight consolidation (Kirkpatrick et al., 2017).

As Section 6 states, our focus is on general-purpose fine-tuning algorithms commonly used by practitioners, rather than methods specifically designed for continual-learning scenarios. We compare them under a common target-task data assumption. Replay and EWC can require previous-task data or prompts that practitioners lack; granting only some methods a separate retention dataset would not constitute a fair comparison under that assumption.

We include OPD recovery as a feasible recovery experiment, but do not evaluate EWC or Marek et al.'s self-generated replay. The latter is relevant because it avoids requiring original pretraining examples. We view these as complementary extensions, not as missing baselines necessary for evaluating the general-purpose fine-tuning algorithms. We will clarify this scope in Related Work and the limitations.

Marek et al. "Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay". arXiv preprint arXiv:2605.26097, 2026.

Kirkpatrick et al. "Overcoming Catastrophic Forgetting in Neural Networks". Proceedings of the National Academy of Sciences (PNAS), 2017.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.