acceptodds
Under review as a conference paper at ICLR 2027

Learning Style, Forgetting Semantics: A case study of SFT and RFT on Classification Tasks

Abstract

Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.

Primary area:
learning theory
Keywords:
supervised fine-tuning, reinforcement learning, continual learning, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7jpiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper theoretically investigates why Supervised Fine-Tuning (SFT) causes more catastrophic forgetting than Reinforcement Fine-Tuning (RFT), even when all teacher demonstrations are semantically correct. Using a linear-softmax policy on sequential classification tasks where tokens within each class represent different "expression styles" of the same semantic answer, the authors decompose the policy parameters into a semantic channel (α\boldsymbol{\alpha}) and a style channel (β\boldsymbol{\beta}). They prove that while SFT and RFT share parallel semantic updates, RFT preserves style symmetry (β≡0\boldsymbol{\beta} \equiv \mathbf{0}), whereas SFT develops off-axis style drift along the new task's prompt mean (μt\boldsymbol{\mu}_t) when fitting a nonuniform teacher. This style drift shifts the decision boundary and induces a strictly positive finite-time forgetting lower bound for SFT (while RFT maintains zero semantic error), which is also supported by finite-sample off-axis energy bounds and synthetic simulations.

Critical strengths
  1. Clean Mechanistic Insight: This paper offers a concrete, intuitive mechanism showing how fitting a teacher's stylistic preferences couples with prompt mean shifts (μt\boldsymbol{\mu}_t) to overturn correct semantic predictions.
  2. Elegant Decomposition & Rigorous Proofs: The orthogonal decomposition into semantic (α\boldsymbol{\alpha}) and style (β\boldsymbol{\beta}) channels cleanly separates cross-entropy and policy gradient dynamics across both population (Theorems 1–3) and finite-sample regimes (Theorem 4).
  3. Solid Synthetic Ablations: Control experiments and ablations in Appendix E (zero task means, uniform teachers, Laplace prompts, reward noise, and finite-KK REINFORCE) align well with the theoretical conditions.
Critical weaknesses
  1. No Validation on Real Data or LLMs: All experiments are restricted to 24-dimensional synthetic vectors. Even within the paper's exact setup (linear-softmax head over fixed features), the authors could easily validate their theory on a small frozen LLM (e.g., Llama/Qwen-1B) on domain-shifted sentiment/NLI tasks using synonymous label tokens (e.g., {"good", "great"} vs. {"bad", "poor"}), directly measuring α\boldsymbol{\alpha} and β\boldsymbol{\beta} trajectories.
  2. Strong Symmetry Assumption at Initialization (β(W1)=0\boldsymbol{\beta}(\mathbf{W}_1) = \mathbf{0}): RFT's zero style drift in Theorems 2 and 3 critically relies on starting from a perfectly uniform within-class policy (β=0\boldsymbol{\beta} = \mathbf{0}). However, real pretrained LLMs already possess strong stylistic priors (β≠0\boldsymbol{\beta} \neq \mathbf{0}). Once β≠0\boldsymbol{\beta} \neq \mathbf{0}, RFT updates are no longer uniform within a class, and it is unclear if RFT still provably avoids style drift.
  3. Stylized Toy Setup & Transient Forgetting: The analysis assumes fixed representations x\mathbf{x}, single-token outputs, and an identical semantic rule u\mathbf{u} across tasks. Moreover, as shown in Proposition D.1, SFT's semantic error eventually vanishes as training time τ2→∞\tau_2 \to \infty, meaning the forgetting separation in Theorem 3 is only a finite-time transient effect.
Questions for authors
  1. Non-uniform initialization (β(W1)≠0\boldsymbol{\beta}(\mathbf{W}_1) \neq \mathbf{0}): How do RFT's style dynamics behave in Theorems 2–3 if initialized from a realistic checkpoint with existing stylistic preferences (β(W1)≠0\boldsymbol{\beta}(\mathbf{W}_1) \neq \mathbf{0})?
  2. Small-scale LLM / real-data experiment: Could the authors validate the semantic–style decomposition on a small pretrained LLM (e.g., probing/fine-tuning the LM head or LoRA on sequential classification tasks with multiple synonymous verbalizers per class)?
  3. Softmax sampling vs. Greedy decoding: Theorem 3 relies on arg⁡max⁡\arg\max decoding and asymmetric peak style preferences (Δρ>0\Delta_\rho > 0). Does a similar forgetting lower bound hold under standard softmax sampling when Δρ=0\Delta_\rho = 0?
4 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank the reviewer for your valuable comments on our paper. We appreciate the opportunity to address your concerns.

W1 and Q2: small-scale LLM experiments

No Validation on Real Data or LLMs: All experiments are restricted to 24-dimensional synthetic vectors. Even within the paper's exact setup (linear-softmax head over fixed features), the authors could easily validate their theory on a small frozen LLM (e.g., Llama/Qwen-1B) on domain-shifted sentiment/NLI tasks using synonymous label tokens (e.g., {"good", "great"} vs. {"bad", "poor"}), directly measuring α\boldsymbol{\alpha} and β\boldsymbol{\beta} trajectories.

Small-scale LLM / real-data experiment: Could the authors validate the semantic–style decomposition on a small pretrained LLM (e.g., probing/fine-tuning the LM head or LoRA on sequential classification tasks with multiple synonymous verbalizers per class)?

We thank the reviewer for the construtive suggestion. Following your advice, we have conducted sequential-training experiments on six real-world sentiment-classification datasets using features extracted from a frozen Qwen2.5-1.5B model. Even with a randomly initialized head W\mathbf{W} with nonzero semantic and style components, SFT exhibits transient forgetting as it fits changing teacher preferences across tasks. In contrast, SFT with a uniform teacher and RFT show more stable trajectories. These observations, together with the measured dynamics of the semantic gap and style shifts, are consistent with the mechanism identified by our theoretical analysis. The detailed experimental setup and results are provided in Section 4.2 of the revised manuscript.

W2 and Q1: non-uniform start

Strong Symmetry Assumption at Initialization: RFT's zero style drift in Theorems 2 and 3 critically relies on starting from a perfectly uniform within-class policy (β=0\boldsymbol{\beta}=0). However, real pretrained LLMs already possess strong stylistic priors (β≠0\boldsymbol{\beta}\neq 0). Once β≠0\boldsymbol{\beta}\neq 0, RFT updates are no longer uniform within a class, and it is unclear if RFT still provably avoids style drift.

Non-uniform initialization: How do RFT's style dynamics behave in Theorems 2-3 if initialized from a realistic checkpoint with existing stylistic preferences?

We agree that our main separation result (Theorem 3) assumes starting from a perfect style-uniform checkpoint. This condition is mainly for theoretical tractability, and allows us to isolate a limitation of SFT: even when the checkpoint already makes correct semantic predictions, fitting the teacher's stylistic preferences can break within-class symmetry and induce semantic forgetting, whereas RFT preserves this symmetry and retains perfect semantic accuracy in population-level updates.

In our new real-world experiments in Section 4.2, we start from a randomly initialized head with nonzero semantic and style components. RFT also exhibits style drift from this initialization, but the drift is smooth and its style-induced error decreases toward zero with continued training. Its error trajectories remain stable, whereas SFT with changing teacher preferences exhibits transient forgetting as it fits the teacher's style. In contrast, SFT with uniform teachers does not exhibit style drift and the spiked forgetting patterns. These observations support the qualitative mechanism beyond symmetric initialization.

AuthorsOpenReview ↗

Author Response (2/2)

W3 Stylized Toy Setup and Transient Forgetting

The analysis assumes fixed representations , single-token outputs, and an identical semantic rule across tasks. Moreover, as shown in Proposition D.1, SFT's semantic error eventually vanishes as training time τ2→∞\tau_2\rightarrow\infty, meaning the forgetting separation in Theorem 3 is only a finite-time transient effect.

Our study uses a simplified classification setting as a first theoretical step toward understanding how style fitting affects semantic forgetting under SFT. This specific setup allows us to derive clean theoretical bounds and gain insights into the role of style fitting in forgetting. We explicitly acknowledge these limitations in the paper's Limitations section and leave extensions to more complex settings for future work.

Regarding the transient nature of SFT forgetting, we would like to clarify that, instead of advocating either SFT or RFT for practical use, our study aims to provide a theoretical understanding of their knowledge retention dynamics. We acknowledge that SFT has its own advantages over RFT in terms of computational efficiency and ease of implementation, but it is still important to understand its limitations. Through a case study in classification tasks, our results show that SFT can exhibit transient semantic forgetting when the teacher has style preferences. This behavior reflects a finite-time phase of SFT’s learning dynamics, that semantic errors can arise during style fitting, but eventually vanish under continued training. Our study provides insight into how style fitting can lead to semantic forgetting and motivates further research into whether more uniform teacher preferences or extended training can help mitigate forgetting.

Q3 Softmax sampling vs. Greedy decoding

Theorem 3 relies on argmax decoding and asymmetric peak style preferences (Δρ>0\Delta_\rho>0). Does a similar forgetting lower bound hold under standard softmax sampling when Δρ=0\Delta_\rho=0?

Since we study classification tasks, argmax decoding is a natural choice for evaluating deterministic predictions. If the final accuracy is evaluated with softmax sampling, the semantic error should be averaged over both the input distribution and the sampled outputs. Extending our result to this setting requires analyzing changes in class probability mass rather than the highest-scoring token, which is out of scope for this work.

For the condition Δρ>0\Delta_\rho>0, it is a sufficient condition for the closed-form bound in Theorem 3, but it is not necessary for forgetting. What matters is that the teacher profiles of the two classes differ. We added some additional experiments in Appendix D.4 to show that even when Δρ=0\Delta_\rho=0, SFT can still forget if the teachers are not the same, although the amount of forgetting is small.

AuthorsOpenReview ↗

Follow-up on our response

Dear reviewer 7jpi,

Thank you very much for your thoughtful comments about our proposal and for your positive recommendation of our paper.

As the discussion period ends on November 18, we would be grateful to know whether our responses have addressed your concerns. We are happy to clarify anything further.

Best regards,

Authors

Thank you for the detailed rebuttal and additional experiments, which have addressed my concerns. I will maintain my positive score.

Reviewer_7jS5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines the differences between SFT and RL in learning-forgetting trade-offs, a topic that has been studied extensively recently. Using a toy model, they show, both theoretically and empirically, that SFT’s fitting of stylistic preferences can cause semantic forgetting despite the annotation being correct.

Critical strengths
  • The toy model that the authors propose allows a clean study of forgetting and learning trade-offs.

  • The paper is well written and easy to understand.

Critical weaknesses

While mathematically correct, I think the main theorems study a simplified version of the problem that does not tell us much about real-world continual learning problems. Mainly, it studies how training affects a task that the model already knows how to solve, rather than learning a new semantic capability. To be concrete, in Theorem 3, the initial checkpoint already achieves perfect accuracy on both tasks. SFT then damages performance while fitting style. This is a valid demonstration that style fitting can cause errors. But it provides limited evidence about why SFT forgets more while acquiring a new capability, especially at matched new-task performance.

More broadly, I’m not sure how much we, as a community, can learn from this analysis, or how much it will actually change the way we understand learning and forgetting. The main thing I’m missing is experiments connecting the toy setting to a phenomenon that is happening in real LLMs, and an empirical analysis of how much the toy-scale analysis can explain the real LLM results.

Questions for authors

In your comparison, you use one sampled teacher token per prompt for SFT, while in RFT, you take the exact expectation over all tokens. This seems like a bit unfair comparison? SFT has sampling noise while RFT does not. For a fair comparison, I recommend an apples-to-apples comparison by sampling from the policy in RFT.

4 comments
AuthorsOpenReview ↗

Author Response

We sincerely thank the reviewer for your valuable comments on our paper. We appreciate the opportunity to address your concerns.

Weakness: relevance to realistic LLMs

The main theorems study a simplified version of the problem that does not tell us much about real-world continual learning problems. Mainly, it studies how training affects a task that the model already knows how to solve, rather than learning a new semantic capability. To be concrete, in Theorem 3, the initial checkpoint already achieves perfect accuracy on both tasks. SFT then damages performance while fitting style. This is a valid demonstration that style fitting can cause errors. But it provides limited evidence about why SFT forgets more while acquiring a new capability, especially at matched new-task performance.

More broadly, I'm not sure how much we, as a community, can learn from this analysis, or how much it will actually change the way we understand learning and forgetting. The main thing I'm missing is experiments connecting the toy setting to a phenomenon that is happening in real LLMs, and an empirical analysis of how much the toy-scale analysis can explain the real LLM results.

It is indeed true that our Theorem 3 starts from a perfect checkpoint. This condition is mainly for theoretical tractability, and allows us to isolate a limitation of SFT: even when the checkpoint already makes correct semantic predictions, fitting the teacher's stylistic preferences can break within-class symmetry and induce semantic forgetting, whereas RFT preserves this symmetry and retains perfect semantic accuracy in population-level updates.

While the separation guarantee in Theorem 3 relies on this condition, our empirical comparisons are not restricted to perfectly fitted checkpoints. Following standard evaluation practice in continual learning, we report both forgetting and generalization error, where the new task error is already included in the latter index. As shown in Figure 2 and the new Table 1, SFT exhibits both higher forgetting and higher generalization error than RFT when fitting changing teacher preferences across tasks. These results provide empirical support for the proposed mechanism beyond the perfectly fitted setting.

We agree that a real-world experiment would be valuable. To this end, we have conducted sequential-training experiments on six real-world sentiment-classification datasets using features extracted from a frozen Qwen2.5-1.5B model. Even with a randomly initialized head W\mathbf{W} with nonzero semantic and style components, SFT exhibits transient forgetting as it fits changing teacher preferences across tasks. In contrast, SFT with a uniform teacher and RFT show more stable trajectories. These observations, together with the measured dynamics of the semantic gap and style shifts, are consistent with the mechanism identified by our theoretical analysis. The detailed experimental setup and results are provided in Section 4.2 of the revised manuscript.

Question: fair comparison

In your comparison, you use one sampled teacher token per prompt for SFT, while in RFT, you take the exact expectation over all tokens. This seems like a bit unfair comparison? SFT has sampling noise while RFT does not. For a fair comparison, I recommend an apples-to-apples comparison by sampling from the policy in RFT.

There are two levels of stochasticity: prompt x\mathbf{x} sampling and action aa / token a⋆a^\star sampling. As clarified in our footnote on page 4, “exact policy gradient” refers to only evaluating the expectation over actions under the current policy (which is computable because policy parameter W\mathbf{W} and rewards are known), while prompt sampling remains stochastic in the finite-sample setting. We acknowledge that this removes action-sampling noise from RFT, whereas SFT still samples teacher tokens (as the teachers' distributions are unknown to the learner). However, our forgetting separation results (Theorem 3) is established under population level, which compares updates that already take expectation over prompts x\mathbf{x} and teacher tokens a⋆∼q(⋅∣x)a^\star\sim q(\cdot|\mathbf{x}) for SFT, and over both prompts x\mathbf{x} and actions a∼πW(⋅∣x)a\sim \pi_\mathbf{W}(\cdot|\mathbf{x}) for RFT. Thus, the population-level comparison is fair, as the separation arises even when sampling noise is absent from both methods and cannot be attributed to a difference in sampling noise.

We also note that in our original submission, the sampled-action REINFORCE discussion was also provided in Appendix D.1, and experiments with different numbers of sampled actions per prompt were reported in Appendix E.3 (Figure E.4(b,d)).

I thank the authors for their reply, and for answering my question.

However, even after reading their answer and the new experiment section, I maintain my opinion that the results in the paper are not representative of the real LLM phenomenon:

  • The authors claim that the assumption in Theorem 3 is "mainly for theoretical tractability, and allows us to isolate a limitation of SFT," but in my eyes it makes the whole theorem irrelevant, since it completely changes the setting being analyzed.
  • The added experiments are still too toy in my eyes, and examine only the linear regime. My original claim was not that I do not trust the original linear model results, but that these don't necessarily say anything about LLM training. The new results do not add much in this regard.
AuthorsOpenReview ↗

Author's reply

Thank you for your prompt reply and for clarifying your concern. The main question we seek to address is whether SFT can lead to semantic forgetting even when every teacher demonstration is semantically correct. We study this question under a linear-softmax classification head, and all of our theorems focus on this setting.

Given the complex nature of real-world LLMs, unfortunately, all theoretical works necessarily rely on simplifying models and assumptions, and we do not claim that our theoretical guarantees directly extend to full LLM fine-tuning (to our knowledge, no theoretical work can actually generalize to explain real-world LLMs). Therefore, the main distinction we aim to draw in our paper is between the mechanisms of “SFT” versus “RFT”, rather than whether the results from a "small model" can be generalized to "LLM". What we show theoreticaly is a provable insight into the mechanism of how style fitting can lead to semantic forgetting under SFT. We believe that our work provides a useful starting point for understanding the role of style fitting in forgetting. Extending our theoretical analysis to more complex models would be an interesting and non-trivial direction for future work, but it is clearly beyond the scope of this paper.

At the same time, we would like to note that every (large language) model has to map its final hidden representation to a probability distribution over the next token or task lebels. As stated in our paper, this linear-softmax form is widely used as the output heads of pretrained language models: Radford et al. [1] use this form for both next-token prediction and supervised classification, while Devlin et al. [2] use this form for downstream classification. To better connect our setup to the LLM context, it can be seen as a frozen pretrained LLM with a trainable linear-softmax head that maps the final-layer output of the LLM to a probability distribution, and all of our analysis holds under this level of generality (including the new experiment in section 4.2).

Regarding experiments on full LLM trainings, as discussed in our introduction, a line of empirical studies, including Chen et al. [3], Lai et al. [4], and Shenfeld et al. [5], have already observed semantic forgetting under SFT in LLMs. These empirical observations directly motivate our work, which seeks to understand the underlying mechanism of this phenomenon.

References

[1] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training. OpenAI technical report, 2018.

[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, pp. 4171–4186, 2019.

[3] Howard Chen, Noam Razin, Karthik R. Narasimhan, and Danqi Chen. Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting. International Conference on Machine Learning (ICML), 2026.

[4] Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Qingfu Zhang, Hongbin Liu, Gaofeng Meng, and Fei Zhu. Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training. International Conference on Machine Learning (ICML), 2026.

[5] Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RL's Razor: Why Online Reinforcement Learning Forgets Less. International Conference on Learning Representations (ICLR), 2026.

After reading the author responses I was convinced that their results do hold in the setting they claim, and will increase my score to 3.

Reviewer_sRDQOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why supervised fine-tuning (SFT) can cause more forgetting than reinforcement fine-tuning (RFT), even with semantically correct demonstrations. Using a linear-softmax classification model, the authors show that fitting teacher style preferences can induce parameter drift and semantic forgetting under SFT, while RFT preserves style symmetry under exact policy gradients.

Critical strengths

The paper addresses the question about the different forgetting behaviors of SFT and RFT. The semantic–style decomposition provides an framework for analyzing how stylistic preferences can affect semantic predictions. The theoretical results establish a finite-time forgetting gap under explicit assumptions, while the experimental results provide supporting evidence.

Critical weaknesses
  1. The theoretical analysis relies on a simplified linear-softmax classification model. It remains unclear whether the proposed mechanism generalizes to realistic language models.
  2. The paper assumes a predefined partition of output tokens into semantic classes and treats within-class variations as style. This separation may not hold in realistic language generation, where semantics and style are often entangled.
  3. The proven forgetting gap is a finite-time phenomenon, and the authors show that SFT errors eventually vanish under continued population training. This raises questions about the practical significance of the proposed mechanism in realistic fine-tuning scenarios.
Questions for authors
  1. Could the authors validate the proposed mechanism on realistic language models and more complex tasks?
  2. How applicable is the proposed semantic–style decomposition to realistic language generation, where stylistic variations may also affect semantic content?
  3. Given that SFT forgetting eventually vanishes under continued population training, does the forgetting gap remain significant within practical fine-tuning budgets?
6 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank the reviewer for your valuable comments on our paper. We appreciate the opportunity to address your concerns.

W1 and Q1: simplified model / realistic esperiments

The theoretical analysis relies on a simplified linear-softmax classification model. It remains unclear whether the proposed mechanism generalizes to realistic language models.

Could the authors validate the proposed mechanism on realistic language models and more complex tasks?

The motivation behind our work is to take a first theoretical step toward understanding how style fitting can lead to different forgetting behavior under SFT and RFT, using a linear-softmax classification head as a controlled case study. This setting allows us to derive an exact decomposition of their updates into semantic and style components and establish conditions under which their different style dynamics lead to a provable separation in semantic forgetting. We believe that our work provides a useful starting point for understanding the role of style fitting in forgetting. Extending our theoretical analysis to more complex models would be an interesting and non-trivial direction for future work, but it is clearly beyond the scope of this paper.

At the same time, we would like to note that every (large language) model has to map its final hidden representation to a probability distribution over the next token or task lebels. As stated in our paper, this linear-softmax form is widely used as the output heads of pretrained language models: Radford et al. [1] use this form for both next-token prediction and supervised classification, while Devlin et al. [2] use this form for downstream classification. To better connect our setup to the LLM context, it can be seen as a frozen pretrained LLM with a trainable linear-softmax head that maps the final-layer output of the LLM to a probability distribution, and all of our analysis holds under this level of generality.

To complement this theoretical connection, we have conducted sequential-training experiments on six real-world sentiment-classification datasets using features extracted from a frozen Qwen2.5-1.5B model. Even with a randomly initialized head W\mathbf{W} with nonzero semantic and style components, SFT exhibits transient forgetting as it fits changing teacher preferences across tasks. In contrast, SFT with a uniform teacher and RFT show more stable trajectories. These observations, together with the measured dynamics of the semantic gap and style shifts, are consistent with the mechanism identified by our theoretical analysis. The detailed experimental setup and results are provided in Section 4.2 of the revised manuscript.

W2 and Q2: semantic-style decomposition

The paper assumes a predefined partition of output tokens into semantic classes and treats within-class variations as style. This separation may not hold in realistic language generation, where semantics and style are often entangled.

How applicable is the proposed semantic–style decomposition to realistic language generation, where stylistic variations may also affect semantic content?

Given the complex nature of real-world LLMs, unfortunately, all theoretical works are simplified to some extent and we are not claiming our theoretical results can generalize to LLMs. In our formulation, "style" refers to variation among outputs within the same semantic class, which share the same reward for any given input. This is mathematically well-defined and provides an operational definition of style for our theoretical analysis. Specifically, since semantics is defined relative to the reward function, we define reward equivalence by

\quad\Longleftrightarrow\quad R(a,\mathbf{x})=R(a',\mathbf{x}) \quad\text{for every }\mathbf{x}\in\mathbb{R}^p.$$ Thus, whether two output tokens belong to the same semantic class is ultimately determined by how the reward function is defined. More generally, if an existing partition contains outputs with different rewards within the same class, a finer partition can always be obtained by splitting each class according to this equivalence relation. However, such a refinement may introduce additional classes and structures and require theoretical analysis beyond our current setting. We leave this extension to future work, as it is beyond the scope of this paper. ## References [1] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. [*Improving Language Understanding by Generative Pre-Training*](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf). OpenAI technical report, 2018. [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [*BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*](https://aclanthology.org/N19-1423/). In *Proceedings of NAACL-HLT*, pp. 4171–4186, 2019.
AuthorsOpenReview ↗

Author Response (2/2)

W3 and Q3: transient forgetting gap

The proven forgetting gap is a finite-time phenomenon, and the authors show that SFT errors eventually vanish under continued population training. This raises questions about the practical significance of the proposed mechanism in realistic fine-tuning scenarios.

Given that SFT forgetting eventually vanishes under continued population training, does the forgetting gap remain significant within practical fine-tuning budgets?

We would like to clarify that, instead of advocating either SFT or RFT for practical use, our study aims to provide a theoretical understanding of their knowledge retention dynamics. We acknowledge that SFT has its own advantages over RFT in terms of computational efficiency and ease of implementation, but it is still important to understand its limitations. Through a case study in classification tasks, our results show that SFT can exhibit transient semantic forgetting when the teacher has style preferences. This behavior reflects a finite-time phase of SFT’s learning dynamics, that semantic errors can arise during style fitting, but eventually vanish under continued training. Our study provides insight into how style fitting can lead to semantic forgetting and motivates further research into whether more uniform teacher preferences or extended training can help mitigate forgetting.

AuthorsOpenReview ↗

Follow-up on our response

Dear reviewer sRDQ,

Thanks again for your thoughtful comments. As the discussion period ends on November 18, we would be grateful to know whether our responses have properly addressed your concerns. We are happy to clarify anything further.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.