- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper invesigates the trade-off between learning a new domain and forgetting prior knowledge during SFT, focusing on instruction-tuned models. They use as datasets ChemL3 (chemistry-related, for learning) and GSM8K (math questions, to monitor forgetting) across models from the Qwen2.5, Llama-3 and OLMo-2 families. They compare classical continual learning method, KL regularization, LoRA, and two proposed geometry-motivated approaches. Their results show that the learning-forgetting frontier tends to form a sharp "cliff": when SFT shows a gradual trade-off, many methods reduce forgetting and sharpen it into a cliff, but when SFT already cliffs, no method helps. In neither case does any method learn more than well-tuned SFT.
Critical strengths
- Experimental setup: the paper's experiments cover three model families and sizes, a wide set of continual learning methods.
- Practical framing: I like the fact that the paper studies systematically the learning vs. forgetting frontier, with a concrete and controlled setup. Also, they focus on something useful for practitioners: at what learning rate and settings does the model learn without forgetting? They also introduce the concept of cliffness, a simple and interpretable scalar summary of the frontier that makes results easy to compare across models and methods.
Critical weaknesses
- Unclear contribution beyond the analysis: the empirical analysis is systematic and interesting, as I mentioned above, but I'm unsure what the main takeaway is beyond it. The central claim ("the cliff is a property of the model") is mostly descriptive. The two methods the authors propose don't clearly beat existing ones. It would help if the authors spelled out what the community concretely gains from this work, beyond the observation itself.
- No explanation of why cliffs emerge: Section 5.3 hypothesizes that forgetting comes from updates along a small set of retention-sensitive directions, and the authors themselves say this is not tested. It seems testable with their existing setup, for example by tracking how much the SFT update projects onto the top retention-gradient. Relatedly, the paper doesn't explain what separates the two regimes. Table 5 shows that the pretrained Qwen models already cliff (0.92/0.94) while their instruction-tuned versions don't (0.67/0.80). Could the "gradual" regime come from how heavily a model's post-training targeted math, which would make the Llama/Qwen gap a matter of post-training recipe rather than model family? In order to understand what is going on, this is worth discussing.
===
Minor:
- Figure 1 is hard to read (with the information that the reader has at that stage): consider reviewing the caption or the figure annotations.
Questions for authors
- The paper studies SFT only, but modern post-training pipelines increasingly combine SFT with reinforcement learning (e.g., RLHF or RL). What is your intuition for those settings? Would the learning-forgetting frontier also form a cliff under RL, where the KL penalty to the reference model already acts as a built-in regularizer? Would the two regimes still appear?
2 comments
I thank the authors for their response. I maintain my positive assessment and support for acceptance.
Rebuttal by Authors
We want to thank Reviewer b4F5 for supporting our submission with an acceptance recommendation. We are grateful to the reviewer for pointing out the coverage and practicality of our submission.
W1: We posit that our paper brings three things to the community. First, we show that a fair comparison between methods requires drawing their Pareto frontiers. Second, we find that no method achieves better learning than plain SFT, and that all the gains are in retention, indicating that the open problem is plasticity [1, 2, 3]. Third, we bring a perspective that unifies retention across methods, which connects forgetting in LLMs [4] to the work on sharpness for post-training [5]. SIB and PTGI act directly on the sharpness of the loss landscape, and we use them to illustrate this perspective, which we expand in Section 5.6.
W2.1, why cliffs emerge: Motivated by the suggestions of Reviewers b4F5 and HnFT, we compute the Fisher information F of the GSM8K loss at the base model and measure how much of each method's update lands on the sensitive weights. As seen in Section 5.6, Figure 8 - Fisher penalty, SFT updates the few most sensitive weights, whereas the continual learning methods update them less than half as much. F is a local approximation, thus, we also measure the GSM8K loss along the direction between the base weights and the finetuned weights of each method in Figure 26 - Loss along the update. Although KL keeps a total penalty close to that of SFT, it moves the base model in a direction along which the GSM8K loss barely rises. Finally, we show the distribution of F for different sizes of Qwen3, Llama-3 and OLMo-2 figure, where model scale shifts the mass of weight sensitivity to the left in all three. In conclusion, cliffs emerge when the update avoids the sharp directions of the retention loss by steering away from sensitive weights (EWC, MAS, PTGI, SIB), by moving where the loss is flat (KL), or by scale, which leaves more weights of low sensitivity.
W2.2, what separates the two regimes: The regime is a property of the method and the model together. A model that does not cliff under SFT can cliff under another method. As shown in Table 1, Qwen2.5-0.5B and Qwen2.5-3B are gradual under SFT (0.69 and 0.80) but cliff under EWC (0.93 and 0.95). Under SFT, we find the gradual regime in the smaller Qwen models, and we discuss the mechanism of forgetting in the new Section 5.5. As can be seen in Figure 7 - Forgetting mechanism, for Qwen2.5-0.5B the teacher-forced likelihood of the GSM8K solution decreases from the lowest learning rate, before ChemL3 is learned, showing the gradual tradeoff. On the role of the post-training recipe, we have re-evaluated Table 5 under a finer learning rate grid. Pretrained and instruction-tuned models are now on par on Qwen2.5-3B (0.98 and 0.98) and close on Qwen2.5-1.5B (1.00 and 0.89), which no longer supports this connection.
Q1, RL: Our intuition was that on-policy RL keeps the weights closer to the base model. Since this is easily falsifiable, we extend the experiments to include Section 5.3 on cliffs in reinforcement learning. In particular, we have performed RL with group-relative advantages on Reasoning Gym for Qwen3-0.6B, Qwen3-4B, Llama-3.2-3B and OLMo-2-1B, with and without a KL penalty to the reference model. As seen in Figure 5 - RL Cliffs, the Pareto frontier also forms a cliff under RL, for which there exists a learning rate that maximally learns the task and retains GSM8K, for the four models. What differs from SFT is the regime past it, where we have not sampled a learning rate that learns the task fully while forgetting GSM8K, which could be a matter of grid resolution. We share learning and forgetting within the RL trajectory Figure 24 - RL Trajectories for clarity. Similarly to SFT, the KL penalty delays the degradation on Qwen3-4B and OLMo-2-1B, but does not remove it.
We simplified the caption of Figure 1.