- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper mainly focuses on forgetting in LLMs during SFT. Unlike other papers, the authors conduct experiments in a controlled setting. Specifically, the paper uses a random hierarchical grammar as the object for bilingual full-rule pretraining, then removes the French rules for SFT, and finds that generation under the removed rules is rapidly suppressed, while representational degradation occurs on a longer timescale.
Critical strengths
The paper distinguishes between behavioral forgetting and representational forgetting. Unlike other studies, it uses a controllable experimental subject for experiments, and the experimental results also support the conclusions claimed in the paper.
Critical weaknesses
I think the biggest weakness is that although the conclusions presented in the paper are well demonstrated in controlled experiments, they are not necessarily universally generalizable, because the main results come from synthetic grammar rules and relatively small models, and the experiments only cover English and French (English and French themselves share many similarities in grammar and other aspects).
Questions for authors
Could a new experiment be designed in which, after the generative behavior has been suppressed, fine-tuning is performed again with a very small number of full-rule samples to observe whether the generative behavior recovers quickly and how the representations change?
2 comments
Thank you for your reply, I will keep my score.
We thank the reviewer for taking time making valuable comments. We appreciate their praise of our well controlled experiment, clear results and support for the claimed conclusions.
Below we provide further experiments and explanations to supplement the weaknesses and the questions from the reviewer.
1. More experiments on natural language and model at scale
We initially chose EN-FR pair specifically so that we use morphologically similar languages where there is no biased distribution of tokens in a single tokenizer. We did not attempt experiments with pretrained LLMs at scale mostly due to the fact that we do not have a precise control on pretraining data distribution and limited compute.
To address these limitations we performed additional experiments: (i) the same setting with different language pairs, English and Turkish, which share the alphabets but have different grammatical structures (such as order of subject, verb in a sentence), and (ii) finetuning of a pretrained coding language model (Qwen Code 0.5B, 1.5B) to follow a specific target python syntax among other plausible ones. Specifically, we used indentation (tab vs n-spaces) and string formatting (% formatting vs. .format() ). For all cases, we detected behavioral forgetting early on. For targeted finetuning on removal of tab indentation, we found clear signs of representational forgetting after elongated finetuning (x 10^3 of behavior forgetting onset). For other cases, given the available time and compute, we suspect that the separation gap is even larger and probably requires even longer finetuning.
Overall, these experiments support that our observations and claims are more general phenomena than initial controlled setup.
We include our new results and experimental details in C.3 and D, Figure 19-20, Figure 23-24.
2. Recoverability experiment
We ran this experiment in the RHM (new Appendix A.7, Figure 12). Starting from checkpoints at different steps of the restriction fine-tuning, we train the network again on the full pretraining grammar and measure, at each level of the hierarchy, how many steps it takes for the generation of the deleted rules to return to its pretraining rate. Each step uses a fresh batch of sequences, so the number of relearning steps directly measures the number of full-rule samples that are needed. As a reference, we train the same network on the full grammar from random initialization.
(i) After the generative behavior has been suppressed and while the representations of the deleted rules are still intact, the generative behavior recovers quickly. For the rules at the upper levels of the hierarchy, recovery takes one to two orders of magnitude fewer steps than learning them from random initialization.
(ii) After the slow timescale, when the representations of the deleted rules are lost, this advantage disappears. From the latest checkpoint, relearning the rules at the root takes almost as many steps as from random initialization, and relearning the rules at the lower levels takes more.
(iii) Between these two cases, the relearning time increases gradually with the duration of the fine-tuning, at every level.
The speed at which the suppressed behavior can be recovered therefore depends on how long the model has been fine-tuned: quickly between the two timescales, and no faster than learning from scratch after the slow one.
We ran the same experiment on the EN/FR bilingual transformer: we tested how much training is required to recover French knowledge once the model is finetuned with EN-only. We found that after the onset of representation forgetting (∼103), the model requires a similar amount of relearning as to training from the model that was never trained with FR. Meanwhile, before the representation forgetting but after the behavioral forgetting the model almost immediately recovers. We included these results and experimental details in Figure 21 in Appendix C.2, C.3.
We hope our response clarified the concerns and the questions, and we look forward to the reconsideration of the scoring.