- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper theoretically investigates why Supervised Fine-Tuning (SFT) causes more catastrophic forgetting than Reinforcement Fine-Tuning (RFT), even when all teacher demonstrations are semantically correct. Using a linear-softmax policy on sequential classification tasks where tokens within each class represent different "expression styles" of the same semantic answer, the authors decompose the policy parameters into a semantic channel () and a style channel (). They prove that while SFT and RFT share parallel semantic updates, RFT preserves style symmetry (), whereas SFT develops off-axis style drift along the new task's prompt mean () when fitting a nonuniform teacher. This style drift shifts the decision boundary and induces a strictly positive finite-time forgetting lower bound for SFT (while RFT maintains zero semantic error), which is also supported by finite-sample off-axis energy bounds and synthetic simulations.
Critical strengths
- Clean Mechanistic Insight: This paper offers a concrete, intuitive mechanism showing how fitting a teacher's stylistic preferences couples with prompt mean shifts () to overturn correct semantic predictions.
- Elegant Decomposition & Rigorous Proofs: The orthogonal decomposition into semantic () and style () channels cleanly separates cross-entropy and policy gradient dynamics across both population (Theorems 1–3) and finite-sample regimes (Theorem 4).
- Solid Synthetic Ablations: Control experiments and ablations in Appendix E (zero task means, uniform teachers, Laplace prompts, reward noise, and finite- REINFORCE) align well with the theoretical conditions.
Critical weaknesses
- No Validation on Real Data or LLMs: All experiments are restricted to 24-dimensional synthetic vectors. Even within the paper's exact setup (linear-softmax head over fixed features), the authors could easily validate their theory on a small frozen LLM (e.g., Llama/Qwen-1B) on domain-shifted sentiment/NLI tasks using synonymous label tokens (e.g.,
{"good", "great"}vs.{"bad", "poor"}), directly measuring and trajectories. - Strong Symmetry Assumption at Initialization (): RFT's zero style drift in Theorems 2 and 3 critically relies on starting from a perfectly uniform within-class policy (). However, real pretrained LLMs already possess strong stylistic priors (). Once , RFT updates are no longer uniform within a class, and it is unclear if RFT still provably avoids style drift.
- Stylized Toy Setup & Transient Forgetting: The analysis assumes fixed representations , single-token outputs, and an identical semantic rule across tasks. Moreover, as shown in Proposition D.1, SFT's semantic error eventually vanishes as training time , meaning the forgetting separation in Theorem 3 is only a finite-time transient effect.
Questions for authors
- Non-uniform initialization (): How do RFT's style dynamics behave in Theorems 2–3 if initialized from a realistic checkpoint with existing stylistic preferences ()?
- Small-scale LLM / real-data experiment: Could the authors validate the semantic–style decomposition on a small pretrained LLM (e.g., probing/fine-tuning the LM head or LoRA on sequential classification tasks with multiple synonymous verbalizers per class)?
- Softmax sampling vs. Greedy decoding: Theorem 3 relies on decoding and asymmetric peak style preferences (). Does a similar forgetting lower bound hold under standard softmax sampling when ?
4 comments
Author Response (2/2)
W3 Stylized Toy Setup and Transient Forgetting
The analysis assumes fixed representations , single-token outputs, and an identical semantic rule across tasks. Moreover, as shown in Proposition D.1, SFT's semantic error eventually vanishes as training time , meaning the forgetting separation in Theorem 3 is only a finite-time transient effect.
Our study uses a simplified classification setting as a first theoretical step toward understanding how style fitting affects semantic forgetting under SFT. This specific setup allows us to derive clean theoretical bounds and gain insights into the role of style fitting in forgetting. We explicitly acknowledge these limitations in the paper's Limitations section and leave extensions to more complex settings for future work.
Regarding the transient nature of SFT forgetting, we would like to clarify that, instead of advocating either SFT or RFT for practical use, our study aims to provide a theoretical understanding of their knowledge retention dynamics. We acknowledge that SFT has its own advantages over RFT in terms of computational efficiency and ease of implementation, but it is still important to understand its limitations. Through a case study in classification tasks, our results show that SFT can exhibit transient semantic forgetting when the teacher has style preferences. This behavior reflects a finite-time phase of SFT’s learning dynamics, that semantic errors can arise during style fitting, but eventually vanish under continued training. Our study provides insight into how style fitting can lead to semantic forgetting and motivates further research into whether more uniform teacher preferences or extended training can help mitigate forgetting.
Q3 Softmax sampling vs. Greedy decoding
Theorem 3 relies on argmax decoding and asymmetric peak style preferences (). Does a similar forgetting lower bound hold under standard softmax sampling when ?
Since we study classification tasks, argmax decoding is a natural choice for evaluating deterministic predictions. If the final accuracy is evaluated with softmax sampling, the semantic error should be averaged over both the input distribution and the sampled outputs. Extending our result to this setting requires analyzing changes in class probability mass rather than the highest-scoring token, which is out of scope for this work.
For the condition , it is a sufficient condition for the closed-form bound in Theorem 3, but it is not necessary for forgetting. What matters is that the teacher profiles of the two classes differ. We added some additional experiments in Appendix D.4 to show that even when , SFT can still forget if the teachers are not the same, although the amount of forgetting is small.
Follow-up on our response
Dear reviewer 7jpi,
Thank you very much for your thoughtful comments about our proposal and for your positive recommendation of our paper.
As the discussion period ends on November 18, we would be grateful to know whether our responses have addressed your concerns. We are happy to clarify anything further.
Best regards,
Authors
Thank you for the detailed rebuttal and additional experiments, which have addressed my concerns. I will maintain my positive score.
Author Response (1/2)
We sincerely thank the reviewer for your valuable comments on our paper. We appreciate the opportunity to address your concerns.
W1 and Q2: small-scale LLM experiments
We thank the reviewer for the construtive suggestion. Following your advice, we have conducted sequential-training experiments on six real-world sentiment-classification datasets using features extracted from a frozen Qwen2.5-1.5B model. Even with a randomly initialized head W with nonzero semantic and style components, SFT exhibits transient forgetting as it fits changing teacher preferences across tasks. In contrast, SFT with a uniform teacher and RFT show more stable trajectories. These observations, together with the measured dynamics of the semantic gap and style shifts, are consistent with the mechanism identified by our theoretical analysis. The detailed experimental setup and results are provided in Section 4.2 of the revised manuscript.
W2 and Q1: non-uniform start
We agree that our main separation result (Theorem 3) assumes starting from a perfect style-uniform checkpoint. This condition is mainly for theoretical tractability, and allows us to isolate a limitation of SFT: even when the checkpoint already makes correct semantic predictions, fitting the teacher's stylistic preferences can break within-class symmetry and induce semantic forgetting, whereas RFT preserves this symmetry and retains perfect semantic accuracy in population-level updates.
In our new real-world experiments in Section 4.2, we start from a randomly initialized head with nonzero semantic and style components. RFT also exhibits style drift from this initialization, but the drift is smooth and its style-induced error decreases toward zero with continued training. Its error trajectories remain stable, whereas SFT with changing teacher preferences exhibits transient forgetting as it fits the teacher's style. In contrast, SFT with uniform teachers does not exhibit style drift and the spiked forgetting patterns. These observations support the qualitative mechanism beyond symmetric initialization.