acceptodds
Under review as a conference paper at ICLR 2027

Learning-Forgetting Optimality in Supervised Finetuning: A Cliff Perspective

Abstract

Supervised finetuning (SFT) of pretrained language models trades off the acquisition of new domain capabilities against retention of prior knowledge. In this work, we extensively characterize this Pareto frontier of learning versus forgetting across nine instruction-tuned models from the Qwen2.5, Llama-3, and OLMo-2 families with full learning-rate sweeps for a number of classical and modern methods for continual learning and domain adaptation. Doing so, we find that across models, this frontier almost universally develops into a sharp cliff: a sharp phase transition where training recipes flip from learning without forgetting into catastrophic forgetting without improvements in learning. Across all methods we test, EWC, MAS, MIGU, NanoAdam, KL regularization, LoRA, and two geometry-motivated baselines we propose, we find that, while many methods can improve retention if the SFT baseline does not yet optimally retain, improvements in learnability remain elusive in practice for modern LLMs. This delineates two regimes. Either baseline SFT performance appears as a gradual trade-off between learning and forgetting, in which case many methods can be applied to ”sharpen” the cliff and reduce forgetting, or, baseline SFT is already not forgetting, in which case no method we test can substantially intervene.

Primary area:
optimization
Keywords:
Continual learning, optimization, loss landscapes

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_b4F5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper invesigates the trade-off between learning a new domain and forgetting prior knowledge during SFT, focusing on instruction-tuned models. They use as datasets ChemL3 (chemistry-related, for learning) and GSM8K (math questions, to monitor forgetting) across models from the Qwen2.5, Llama-3 and OLMo-2 families. They compare classical continual learning method, KL regularization, LoRA, and two proposed geometry-motivated approaches. Their results show that the learning-forgetting frontier tends to form a sharp "cliff": when SFT shows a gradual trade-off, many methods reduce forgetting and sharpen it into a cliff, but when SFT already cliffs, no method helps. In neither case does any method learn more than well-tuned SFT.

Critical strengths
  • Experimental setup: the paper's experiments cover three model families and sizes, a wide set of continual learning methods.
  • Practical framing: I like the fact that the paper studies systematically the learning vs. forgetting frontier, with a concrete and controlled setup. Also, they focus on something useful for practitioners: at what learning rate and settings does the model learn without forgetting? They also introduce the concept of cliffness, a simple and interpretable scalar summary of the frontier that makes results easy to compare across models and methods.
Critical weaknesses
  • Unclear contribution beyond the analysis: the empirical analysis is systematic and interesting, as I mentioned above, but I'm unsure what the main takeaway is beyond it. The central claim ("the cliff is a property of the model") is mostly descriptive. The two methods the authors propose don't clearly beat existing ones. It would help if the authors spelled out what the community concretely gains from this work, beyond the observation itself.
  • No explanation of why cliffs emerge: Section 5.3 hypothesizes that forgetting comes from updates along a small set of retention-sensitive directions, and the authors themselves say this is not tested. It seems testable with their existing setup, for example by tracking how much the SFT update projects onto the top retention-gradient. Relatedly, the paper doesn't explain what separates the two regimes. Table 5 shows that the pretrained Qwen models already cliff (0.92/0.94) while their instruction-tuned versions don't (0.67/0.80). Could the "gradual" regime come from how heavily a model's post-training targeted math, which would make the Llama/Qwen gap a matter of post-training recipe rather than model family? In order to understand what is going on, this is worth discussing.

===

Minor:

  • Figure 1 is hard to read (with the information that the reader has at that stage): consider reviewing the caption or the figure annotations.
Questions for authors
  • The paper studies SFT only, but modern post-training pipelines increasingly combine SFT with reinforcement learning (e.g., RLHF or RL). What is your intuition for those settings? Would the learning-forgetting frontier also form a cliff under RL, where the KL penalty to the reference model already acts as a built-in regularizer? Would the two regimes still appear?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We want to thank Reviewer b4F5 for supporting our submission with an acceptance recommendation. We are grateful to the reviewer for pointing out the coverage and practicality of our submission.


W1: We posit that our paper brings three things to the community. First, we show that a fair comparison between methods requires drawing their Pareto frontiers. Second, we find that no method achieves better learning than plain SFT, and that all the gains are in retention, indicating that the open problem is plasticity [1, 2, 3]. Third, we bring a perspective that unifies retention across methods, which connects forgetting in LLMs [4] to the work on sharpness for post-training [5]. SIB and PTGI act directly on the sharpness of the loss landscape, and we use them to illustrate this perspective, which we expand in Section 5.6.

W2.1, why cliffs emerge: Motivated by the suggestions of Reviewers b4F5 and HnFT, we compute the Fisher information FF of the GSM8K loss at the base model and measure how much of each method's update lands on the sensitive weights. As seen in Section 5.6, Figure 8 - Fisher penalty, SFT updates the few most sensitive weights, whereas the continual learning methods update them less than half as much. FF is a local approximation, thus, we also measure the GSM8K loss along the direction between the base weights and the finetuned weights of each method in Figure 26 - Loss along the update. Although KL keeps a total penalty close to that of SFT, it moves the base model in a direction along which the GSM8K loss barely rises. Finally, we show the distribution of FF for different sizes of Qwen3, Llama-3 and OLMo-2 figure, where model scale shifts the mass of weight sensitivity to the left in all three. In conclusion, cliffs emerge when the update avoids the sharp directions of the retention loss by steering away from sensitive weights (EWC, MAS, PTGI, SIB), by moving where the loss is flat (KL), or by scale, which leaves more weights of low sensitivity.

W2.2, what separates the two regimes: The regime is a property of the method and the model together. A model that does not cliff under SFT can cliff under another method. As shown in Table 1, Qwen2.5-0.5B and Qwen2.5-3B are gradual under SFT (0.69 and 0.80) but cliff under EWC (0.93 and 0.95). Under SFT, we find the gradual regime in the smaller Qwen models, and we discuss the mechanism of forgetting in the new Section 5.5. As can be seen in Figure 7 - Forgetting mechanism, for Qwen2.5-0.5B the teacher-forced likelihood of the GSM8K solution decreases from the lowest learning rate, before ChemL3 is learned, showing the gradual tradeoff. On the role of the post-training recipe, we have re-evaluated Table 5 under a finer learning rate grid. Pretrained and instruction-tuned models are now on par on Qwen2.5-3B (0.98 and 0.98) and close on Qwen2.5-1.5B (1.00 and 0.89), which no longer supports this connection.

Q1, RL: Our intuition was that on-policy RL keeps the weights closer to the base model. Since this is easily falsifiable, we extend the experiments to include Section 5.3 on cliffs in reinforcement learning. In particular, we have performed RL with group-relative advantages on Reasoning Gym for Qwen3-0.6B, Qwen3-4B, Llama-3.2-3B and OLMo-2-1B, with and without a KL penalty to the reference model. As seen in Figure 5 - RL Cliffs, the Pareto frontier also forms a cliff under RL, for which there exists a learning rate that maximally learns the task and retains GSM8K, for the four models. What differs from SFT is the regime past it, where we have not sampled a learning rate that learns the task fully while forgetting GSM8K, which could be a matter of grid resolution. We share learning and forgetting within the RL trajectory Figure 24 - RL Trajectories for clarity. Similarly to SFT, the KL penalty delays the degradation on Qwen3-4B and OLMo-2-1B, but does not remove it.

We simplified the caption of Figure 1.


  • [1] Dohare et al., Loss of plasticity in deep continual learning, Nature 2024. doi.org/10.1038/s41586-024-07711-7
  • [2] Lyle et al., Understanding plasticity in neural networks, 2023. arXiv:2303.01486
  • [3] Hernandez-Garcia et al., Can Scale Save Us From Plasticity Loss in Large Language Models?, 2026. arXiv:2606.24752
  • [4] Marek et al., Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay, 2026. arXiv:2605.26097
  • [5] Watts et al., Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting, 2026. arXiv:2605.02105

I thank the authors for their response. I maintain my positive assessment and support for acceptance.

Reviewer_mxtvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the trade-off between learning new tasks and retaining existing capabilities during supervised finetuning. Through learning-rate and epoch sweeps, it compares several continual learning methods and two geometry-motivated baselines, SIB and PTGI. The authors observe that retention can deteriorate sharply after new-task performance saturates and identify two regimes: continual learning methods improve retention over standard SFT, or standard SFT already approaches a favorable trade-off and additional interventions offer limited benefits.

Critical strengths
  • The evaluation perspective is valuable. Comparing learning–retention trajectories provides a more informative picture of methods’ benefits and limitations than comparisons at a single hyperparameter setting.
  • The study covers a broad range of models and methods. Experiments include multiple model families, sizes, continual learning methods, and training configurations, supplemented by cross-task evaluations, format controls, and selected multi-seed analyses.
  • The empirical findings have practical relevance. Distinguishing different learning–retention regimes helps identify when continual learning interventions may provide benefits.
Critical weaknesses
  • The study primarily examines a single domain-adaptation stage starting from an initial model. Although the appendix includes additional learning tasks, these are finetuned independently rather than learned sequentially by the same model. Consequently, the experiments do not establish whether the cliff phenomenon and the two-regime conclusion persist under repeated updates, nor do they characterize cumulative forgetting, task-order effects, or differences in retention across historical tasks. Sequential learning experiments with evaluation on all previously learned tasks after each stage would strengthen the generalization to broader continual learning settings.

  • The paper presents the difficulty of improving plasticity as a central conclusion, but does not specify whether plasticity means learning new tasks without retention constraints or learning while preserving existing capabilities. It is also unclear whether plasticity is measured by final accuracy, improvement over the initial model, or learning efficiency. These definitions may lead to different conclusions. The authors should provide an explicit operational definition and explain which experimental results support their conclusion under that definition.

  • The interpretation of catastrophic forgetting does not sufficiently distinguish capability loss from changes in capability elicitation or output behavior. The authors provide useful format controls, but Appendix D shows that simply prefilling “To” restores GSM8K accuracy from 10% to 86% in one configuration. This suggests that at least part of the performance decline cannot be directly equated with losing mathematical capability. The failure of the same prefix at a higher learning rate does not, by itself, establish that the capability has been forgotten. The authors should explain this result and its implications for interpreting the observed collapse as catastrophic forgetting.

Questions for authors

I hope the authors could help to address the weakness above.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We want to thank the reviewer for the careful feedback and for their time. We especially appreciate that the reviewer highlights the value of our evaluation perspective, the breadth of coverage and the practical relevance of our work. We have updated the submitted pdf, showing in green the added text, and we respond to the weaknesses and questions below.


Weakness 1, sequential task learning: The main focus of our work is learning-forgetting dynamics in single-task finetuning. Following the suggestion of Reviewer mxtv, we expand the setting of our paper with the new Section 5.4, where we finetune on two tasks back to back, Astro2026 first and then ChemL3 from the checkpoint that is closest to the optimal corner, and we evaluate GSM8K, Astro2026 and ChemL3 after each stage. In Figure 6 we show the 3D representation of the learning-forgetting relationship between the three tasks for Qwen3-4B, Qwen3-8B, OLMo-2-1B and Llama-3.2-1B, which represent a variety of model families and sizes, and in Figure 29 we show each stage separately. Across the models, learning the second task is unaffected by the first, the best ChemL3 accuracy is within 0.02 of the single-task setting. The difference is on retention (Figure 30), where the recently learned task is forgotten at a lower learning rate than the pretrained capability, and the pretrained capability collapses at about the same learning rate as in the single-task setting. The exception is OLMo-2-1B, where the pretrained capability degrades at a lower learning rate than for the single-task setting, and where the prior task is retained for larger learning rates. As in the single-task setting, the cliff persists, where the Pareto curves approach the 3D optimal corner and the projections onto each retained task exhibit a cliff. We have not studied the effect of the task order.

Weakness 2, plasticity conclusion: We define plasticity as learning new tasks without retention constraints, which we measure by final accuracy. We operate under this definition to cleanly separate learning from forgetting. With this definition we see that no method exceeds the best accuracy of SFT by more than 0.04, regardless of the impact on prior capabilities, which is what supports our conclusion that improving plasticity remains difficult. We update the manuscript clarifying this definition.

Weakness 3, catastrophic forgetting: We agree that the mechanism of forgetting is an interesting object of study. From an evaluation perspective, performance degradation is what a practitioner observes regardless of its mechanism, so the unchanged prompt remains our main metric. To clarify and expand this point, we have added Section 5.5, which builds on the format control of Appendix D, and we have included an additional experiment measuring the teacher-forced likelihood for the reference solution at different finetuning learning rates (Figure 7). The conclusion is that forgetting happens through elicitation first, and at higher learning rates the teacher-forced likelihood collapses as well.


We want to thank the reviewer once again, for we believe that in the process of addressing the weaknesses above we have greatly improved the manuscript, and we would be grateful if the reviewer would reconsider their rating. We hope we have addressed all the weaknesses above, if anything remains unclear, we are happy to discuss it further.

Thanks for the clarification. However, I remain unconvinced that this constitutes catastrophic forgetting. If simply pre-filling the response with "To" restores GSM8K accuracy from 10% to 86%, doesn't this suggest that the underlying capability is largely retained, and that the observed collapse is primarily an elicitation issue rather than genuine forgetting?

AuthorsOpenReview ↗

Response by Authors

Thank you for the quick reply.

If simply pre-filling the response with "To" restores GSM8K accuracy from 10% to 86%, doesn't this suggest that the underlying capability is largely retained, and that the observed collapse is primarily an elicitation issue rather than genuine forgetting?

We agree for the first learning rate past the cliff, there the capability is still present and elicitation is what fails. However, we would note that the "To" prefill is an oracle test because it is written in the assistant turn. It is also not universal, since it depends on the task and model, and it has to be known before finetuning. In practice, the finetuned model no longer solves GSM8K.

Moreover, the "To" prefill only restores performance partially and for a short range. At larger learning rates the capability is lost, as shown in orange in Figure 7 - Forgetting mechanism. The teacher-forced likelihood, which measures performance under the best-case prefill, also degrades at larger learning rates (purple), indicating that the capability collapses even under best-case elicitation.

In conclusion, there is loss of accuracy on GSM8K under the canonical evaluation, with loss of the capability itself appearing at larger learning rates.

Reviewer_HnFTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors plot the learning-forgetting tradeoff across models and tasks by sweeping learning rates. The test a variety of continual learning and domain adaptation methods, and find that many methods make progress toward what they call a 'cliff', a sharp transition where training recipes flip from learning with minimal forgetting to catastrophic forgetting with little additional learning. They quantify proximity to an ideal learning-retention corner using 'cliffness', the point on the Pareto frontier which forms a maximally large square with the origin. However, they find that some models naturally have cliff frontiers, and for those models, the methods they train do not substantially improve performance beyond the existing SFT baseline, except for two new methods they propose; these methods show slight improvement of the Pareto frontier and the cliff metric. These two methods aim to reduce updates to parameters that appear more sensitive to the retention loss, using gradients computed on a retain dataset. From there, they hypothesize that the different methods improve the learning-forgetting tradeoff by limiting updates along retention-sensitive directions.

Critical strengths
  • Comprehensive empirical study of learning-forgetting tradeoffs. The paper evaluates a broad range of methods and model families using learning rate sweeps; this is a sound way to compare methods via attainable tradeoffs. Figure 3 clearly illustrates when interventions improve retention and when well-tuned SFT already performs comparably.
  • Compelling evidence that well-tuned SFT can avoid the learning-forgetting tradeoff, for some models on some benchmarks. They show that, in several settings, ordinary SFT achieves near-maximal performance on the new task while largely retaining prior capabilities.
Critical weaknesses
  • The sharpness-based explanation is insufficiently supported by the experiments. The main empirical finding concerns learning–retention tradeoff curves, which does not itself establish a connection to loss-landscape sharpness. Section 5.3 interprets improvements from SIB and PTGI as evidence that forgetting is driven by updates along a small set of retention-sensitive directions. Sure, these methods target those directions, but that does not established a shared mechanism with other methods like KL or EWC.
Questions for authors
  1. What evidence supports the hypothesis that KL regularization, EWC, and MAS improve retention through the same mechanism proposed for SIB/PTGI?
  2. Cliffness measures proximity to an ideal corner, but does not directly measure whether the learning–forgetting curve exhibits a sharp transition. Can you provide full learning–forgetting curves for the additional benchmarks in Table 2?
  3. You claim that the cliff is a property of the underlying model rather than any specific method, but does it depend on the task as well? For example, OLMo2-1B’s SFT cliffness changes from 1.00 on LogiQA to 0.55 on Astro2026.
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We want to thank Reviewer HnFT for the attention and detailed feedback, especially for acknowledging the soundness of evaluating methods via a full learning rate sweep. We have updated the submitted pdf, showing in green the added text, and we respond to the weaknesses and questions below.


Weakness 1 and Question 1: We expand on the shared mechanism among continual learning methods following Reviewer HnFT and b4F5 suggestions. To do so, we define the retention loss from GSM8K samples on the base model. With this, we compute the per-weight Fisher Information F, whose distribution we show here for different sizes of Qwen3, Llama3 and OLMo2 (figure), where model scale shifts the mass of weight sensitivity to the left in all three families.

With this metric we can compute how much each method's weight updates land on the sensitive weights. We report the elastic weight consolidation penalty, ∑iFiΔi2\sum_i F_i\Delta_i^2, each weight's squared movement weighted by its FF (Figure 8). We can see how SFT places its penalty on the few most sensitive weights (F>10−3F>10^{-3}); in contrast, all the continual learning methods place less than half as much there. PTGI moves farther than SFT, but on weights of low sensitivity. We compare all the methods at the learning rate that is closest to the optimal corner.

FF is a local approximation, so we also measure the GSM8K loss along the direction between the base weights and the finetuned weights of each method (Figure 26). Although KL keeps a total penalty close to that of SFT, it moves the base model in a direction along which the GSM8K loss barely rises.

In conclusion, retention follows from avoiding sharp directions of the retention loss: by steering away from sensitive weights (EWC, MAS, PTGI, SIB), by moving where the loss is flat (KL), or by scale, which leaves more weights of low sensitivity.

Question 2: One of our contributions is centering the Pareto frontier as the main object of study. We update the appendix to include Figure 10, which shows the entire learning rate sweeps for Table 2. LogiQA, Astro2026, and ChemL3 against GSM8K, IFEval and MMLU.

Question 3: For the reviewer's example based on Table 2, OLMo-2-1B’s 0.55 on Astro2026 was partly a grid resolution artifact, which moved to 0.79 with a run at 2e-5 learning rate (figure). Across models and tasks, where plain SFT does not cliff (11 of the 24 model–task settings in Tables 1 and 2), a continual learning method reaches a cliffness of at least 0.9 in 9 of them, with the two exceptions being Qwen2.5-1.5B on ChemL3 (0.85) and on Astro2026 (0.89).


We hope this addresses the concern on the shared mechanism, and we would be grateful if the reviewer would reconsider their rating. If anything remains unclear, we are happy to discuss it further.

I thank the authors for substantively addressing question 1, and appreciate the addition of plots which addresses question 2.

I still think that it is a bit of a strong claim to say that the possible frontier is some property of the model. It is more defensible to say that none of the tested methods substantially pushes the corner outward. Nevertheless, I will raise my score from weak reject to weak accept for the additional sharpness experiments and discussion.

Comments on the paper

1 comment

We thank all the reviewers for their thoughtful feedback, and for recognizing the value of our comparison methodology through full learning rate sweeps. Below, we summarize the new revision, which appears highlighted in green in the updated manuscript.

  1. Shared mechanism (HnFT, b4F5). Using the Fisher information of the retention loss, we show that SFT updates the few most sensitive weights, while the continual learning methods update them less than half as much (Section 5.6).

  2. Sequential learning (mxtv). Finetuning on two tasks back to back on four models, the cliff persists (Section 5.4).

  3. Capability versus elicitation (mxtv). Elicitation fails first, and the capability itself is lost at larger learning rates (Section 5.5).

  4. Reinforcement learning (b4F5). The frontier also forms a cliff under RL on four models (Section 5.3).

  5. Further revisions. Full learning rate sweeps behind Table 2 (HnFT), an explicit definition of plasticity (mxtv), and a re-evaluated Table 5 (b4F5).

We address each reviewer's specific concerns in the corresponding response thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.