acceptodds
Under review as a conference paper at ICLR 2027

Two Timescales of Forgetting in Fine-tuning of Language Models: Fast Behavioral Suppression and Slow Representational Drift

Abstract

Today, a core pipeline of artificial intelligence systems is pretraining on a broad range of text corpora and then fine-tuning on specific tasks or desired behaviors. Yet we still have limited understanding of how fine-tuning on a restricted data distribution modifies representations acquired during pretraining. In particular, it remains unclear whether surface behavioral suppression happens together with the erasure of latent representations that support it. In this work, we study supervised fine-tuning (SFT) by pretraining transformers on a controlled context-free grammar with hierarchical latent structure, the Random Hierarchy Model (RHM), and fine-tuning with a restricted subset of its generative rules. While during pretraining the model's ability to generate with the correct grammar rules coincides with building the internal representations of the corresponding latent, we find that SFT exhibits a very different dynamics: (i) the generation of deleted rules changes on a fast time scale inversely proportional to the magnitude of the token-latent correlation; meanwhile, their corresponding representations stay intact; (ii) the hidden representations of the deleted rules become forgotten on a much longer time scale, associated with a slow drift of the network parameters. Finally, we observe analogous results in a realistic setup with natural language, suggesting that this timescale separation in forgetting is a more general phenomenon of fine-tuning language models.

Primary area:
learning theory
Keywords:
Deep learning theory, continual learning, finetuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_5bCMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly focuses on forgetting in LLMs during SFT. Unlike other papers, the authors conduct experiments in a controlled setting. Specifically, the paper uses a random hierarchical grammar as the object for bilingual full-rule pretraining, then removes the French rules for SFT, and finds that generation under the removed rules is rapidly suppressed, while representational degradation occurs on a longer timescale.

Critical strengths

The paper distinguishes between behavioral forgetting and representational forgetting. Unlike other studies, it uses a controllable experimental subject for experiments, and the experimental results also support the conclusions claimed in the paper.

Critical weaknesses

I think the biggest weakness is that although the conclusions presented in the paper are well demonstrated in controlled experiments, they are not necessarily universally generalizable, because the main results come from synthetic grammar rules and relatively small models, and the experiments only cover English and French (English and French themselves share many similarities in grammar and other aspects).

Questions for authors

Could a new experiment be designed in which, after the generative behavior has been suppressed, fine-tuning is performed again with a very small number of full-rule samples to observe whether the generative behavior recovers quickly and how the representations change?

2 comments

We thank the reviewer for taking time making valuable comments. We appreciate their praise of our well controlled experiment, clear results and support for the claimed conclusions.

Below we provide further experiments and explanations to supplement the weaknesses and the questions from the reviewer.

1. More experiments on natural language and model at scale

the main results come from synthetic grammar rules and relatively small models, and the experiments only cover English and French (English and French themselves share many similarities in grammar and other aspects)

We initially chose EN-FR pair specifically so that we use morphologically similar languages where there is no biased distribution of tokens in a single tokenizer. We did not attempt experiments with pretrained LLMs at scale mostly due to the fact that we do not have a precise control on pretraining data distribution and limited compute.

To address these limitations we performed additional experiments: (i) the same setting with different language pairs, English and Turkish, which share the alphabets but have different grammatical structures (such as order of subject, verb in a sentence), and (ii) finetuning of a pretrained coding language model (Qwen Code 0.5B, 1.5B) to follow a specific target python syntax among other plausible ones. Specifically, we used indentation (tab vs n-spaces) and string formatting (% formatting vs. .format() ). For all cases, we detected behavioral forgetting early on. For targeted finetuning on removal of tab indentation, we found clear signs of representational forgetting after elongated finetuning (x 10^3 of behavior forgetting onset). For other cases, given the available time and compute, we suspect that the separation gap is even larger and probably requires even longer finetuning.

Overall, these experiments support that our observations and claims are more general phenomena than initial controlled setup.

We include our new results and experimental details in C.3 and D, Figure 19-20, Figure 23-24.

2. Recoverability experiment

Could a new experiment be designed in which, after the generative behavior has been suppressed, fine-tuning is performed again with a very small number of full-rule samples to observe whether the generative behavior recovers quickly and how the representations change?

We ran this experiment in the RHM (new Appendix A.7, Figure 12). Starting from checkpoints at different steps of the restriction fine-tuning, we train the network again on the full pretraining grammar and measure, at each level of the hierarchy, how many steps it takes for the generation of the deleted rules to return to its pretraining rate. Each step uses a fresh batch of sequences, so the number of relearning steps directly measures the number of full-rule samples that are needed. As a reference, we train the same network on the full grammar from random initialization.

(i) After the generative behavior has been suppressed and while the representations of the deleted rules are still intact, the generative behavior recovers quickly. For the rules at the upper levels of the hierarchy, recovery takes one to two orders of magnitude fewer steps than learning them from random initialization.

(ii) After the slow timescale, when the representations of the deleted rules are lost, this advantage disappears. From the latest checkpoint, relearning the rules at the root takes almost as many steps as from random initialization, and relearning the rules at the lower levels takes more.

(iii) Between these two cases, the relearning time increases gradually with the duration of the fine-tuning, at every level.

The speed at which the suppressed behavior can be recovered therefore depends on how long the model has been fine-tuned: quickly between the two timescales, and no faster than learning from scratch after the slow one.

We ran the same experiment on the EN/FR bilingual transformer: we tested how much training is required to recover French knowledge once the model is finetuned with EN-only. We found that after the onset of representation forgetting (∼103\sim 10^3), the model requires a similar amount of relearning as to training from the model that was never trained with FR. Meanwhile, before the representation forgetting but after the behavioral forgetting the model almost immediately recovers. We included these results and experimental details in Figure 21 in Appendix C.2, C.3.

We hope our response clarified the concerns and the questions, and we look forward to the reconsideration of the scoring.

Thank you for your reply, I will keep my score.

Reviewer_4ZNWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates forgetting during language model fine-tuning, focusing specifically on the stages of behavioral suppression versus representation-level forgetting. They first investigate this in a synthetic setting, using the Random Hierarchy Model. The authors pretrain a language model on data from the model and then finetune on a restricted subset of the production rules of the model. They identify a fast phase of forgetting where the deleted rules are no longer generated followed by a slow phase of representational forgetting, the latter of which they connect to parameter drift during fine-tuning. They then introduce a simplified theoretical model of this phenomenon and derive an expression for the change in probability. Finally, they extend their analysis to natural language with an experiment using English and French sentences.

Critical strengths

I think the initial synthetic experiment is strong and provides a clean setting to investigate forgetting in a controllable way. The separation between timescales was well motivated and many aspects were convincing (save the points mentioned elsewhere). The authors also go beyond empirical analysis and connect their work to theory, leading them to derive a scaling relationship for suppression. Finally, I found it commendable that they begin to extend their analysis beyond synthetic experiments to a natural language setting.

Critical weaknesses

The initial representation probe offers preliminary, but weak / insufficient evidence for the representation-level forgetting claim. The specific representations used may become less similar even though the relevant information is still present in the model. The experiments involving predicting continuations from deleted-rule contexts make the argument stronger, but still weak. This claim would be made much stronger by with linear probing experiments using representations from different fine-tuning checkpoints.

The claims involving parameter drift are suggestive, but there needs to be additional experiments to understand the mechanism further. In particular, the authors should provide results at different batch sizes and also include experiments where the representations are frozen but the output layer is tuned, to test the hypothesis that the early suppression phase uses pretrained features.

Finally, the results appear limited. The models trained are at small scale and the sequences used during training are fairly short. Additional experiments are needed at larger model scale and more realistic sequence lengths in order to validate the authors' claims. I also find the natural language experiments interesting but limited. Additional work is needed to show that this holds in a more realistic setting.

Questions for authors

In the synthetic setting, are the suppressed symbols / rules still decodable from the representations?

For the natural language experiments, how do you distinguish between suppression of French capabilities versus a simple change in prior towards generating English?

3 comments

We thank the reviewer for taking time making valuable comments.

Below we provide additional experiments and analyses that we conducted to clarify the raised concerns.

1. Additional results on representation forgetting probes

The initial representation probe offers preliminary, but weak / insufficient evidence for the representation-level forgetting claim.

As suggested by the reviewer, we added linear probing experiments on the fine-tuning checkpoints of the RHM (method in Appendix A.4; results in Figure 10 of Appendix A.5 and Figure 11(c) and (f) of Appendix A.6). These experiments also answer the reviewer’s question on whether the suppressed rules are still decodable from the representations.

At each fine-tuning checkpoint, we fit a linear classifier that predicts the symbol of a latent node from the hidden representation. The probe is fitted on sequences in which this node uses a kept rule, and tested on sequences in which it uses a deleted rule. Its accuracy is therefore high only if the deleted rules are still represented and encode the latent symbol in the same way as the kept rules. To account for the possibility that the information is still present elsewhere in the model, we fit one probe at the input embedding and after every layer, and report the largest accuracy.

(i) After the fast timescale, the suppressed rules are still decodable: the probe accuracy stays close to 1 long after the generation of the deleted rules has been suppressed (Figure 10(a) to (c)).

(ii) After the slow timescale, they are no longer decodable: the probe accuracy decreases over the same interval as qRq^R (Figure 2(b)), to a value comparable to the accuracy that can be obtained without the tokens of the deleted rule (dashed lines in Figure 10).

(iii) The probe accuracy at different learning rates η\eta collapses when plotted against tη2t\eta^2, as qRq^R does (Figure 10(d) to (f) and Figure 3(c)).

The linear probe therefore gives the same result as qRq^R: the decrease of qRq^R reflects the loss of the information about the deleted rules, and not only a change in the similarity of the representations.

We also added the following additional control experiments for EN/FR, suggested by another reviewer.

1) Context ablation: we compared how much target sentence tokens average loss increases if the model is given in context uninformative sentence compared to semantically coherent informative sentence in each language. We show that when the model is finetuned with EN only, the target sentence loss is always lower when the English sentence given in context is informative than the context is non-informative. Meanwhile, the loss difference between informative and non-informative French context is similar to that of English early on, but later on there will be less difference, indicating that there is no context benefit in French. We used the baseline as a loss difference from informative/non-informative French context in EN-only trained model.

2) Recoverability: we tested how much training is required to recover French knowledge once the model is finetuned with EN-only. We found that after the onset of representation forgetting (103∼10^3\sim), the model requires a similar amount of relearning as to training from the model that was never trained with FR. Meanwhile, before the representation forgetting but after the behavioral forgetting the model almost immediately recovers.

We included the above results and experimental details in Figure 21 in Appendix C.2, C.3.

2. Further analyses to clarify understanding on the mechanism

The claims involving parameter drift are suggestive, but there needs to be additional experiments to understand the mechanism further

We added three experiments on the RHM to test the mechanism behind the two timescales (new Section 4.4 and Figure 3). Our hypothesis is that the two timescales correspond to two regimes of the parameter updates. In the first regime, the fine-tuning gradient has a persistent direction, which suppresses the generation of the deleted rules. In the second regime, the behavior is already suppressed and the average gradient has no component that acts on the representations of the deleted rules; these representations degrade through the accumulation of the noise of the mini-batch updates.

(i) Learning-rate dependence. The reviewer suggests varying the batch size to probe the role of the noise in the updates. We tested the role of the noise by varying the learning rate η\eta, for which the two regimes give distinct predictions. For Adam, as for SGD, the mean parameter update is proportional to η\eta, while its variance is proportional to η2\eta^2. If behavioral forgetting is driven by a persistent gradient, its timescale is proportional to η−1\eta^{-1}. If representational forgetting is driven by the accumulation of noise, its timescale is proportional to η−2\eta^{-2}. In Figure 3, the curves of the deleted-rule rate at three learning rates collapse when plotted against tηt\eta (Figure 3(a)), while the curves of qRq^R and of the parameter displacement collapse when plotted against tη2t\eta^2 (Figure 3(b) and (c)). In this late stage the displacement also grows as t1/2t^{1/2}, as expected for accumulated noise. The same collapses hold at every level of the hierarchy and for the linear probes (Figures 9 and 10 in Appendix A.5).

(ii) Bounded suppression. To remove the persistent gradient after the behavior is suppressed, we stop the loss from acting on output logits that are already more than 8 nats below the largest logit at their position. With this loss the behavioral forgetting saturates, while the representational forgetting follows the same curve as without it (Figure 3(d)). The slow change in the representations is therefore not caused by the continued suppression of the deleted rules.

(iii) Regularization toward the pretrained parameters. If the representations are lost because the parameters drift away from their pretrained values, limiting this drift should preserve them. We added an L2L_2 regularization toward the pretrained parameters: it leaves the behavioral forgetting essentially unchanged, limits the late parameter displacement, and for sufficiently strong regularization preserves the representations of the deleted rules over the entire training (Figure 11, bottom row, in Appendix A.6).

include experiments where the representations are frozen but the output layer is tuned, to test the hypothesis that the early suppression phase uses pretrained features.

In Appendix B.3, we added experiments comparing finetuning between the full non-linear model and its linearized version around pre-trained weights, which corresponds to a linear readout out of frozen features given by the model jacobian. We show that the behavioural adaptation timescale of the two models coincides only for the lowest level rules, for which an effectively linear model is enough to represent the correct readout. For the higher-level rules, because of the RHM structure, the model needs to build the readout from non-linear combinations of the hidden latents, which the non-linear model is able to achieve faster than the linearized model. In the main section 4.3, we discuss carefully why our theoretical assumptions are expected to work only for the lowest level adaptation. Understanding why our simple prediction seems to capture also higher-level adaptations would require modelling the full non-linear model dynamics, which we leave open for future work.

3. Additional results on more realistic settings and model at scale

Additional work is needed to show that this holds in a more realistic setting.

This limitation was due to the computational costs of performing pre-training and finetuning with practical language models. We also use EN-FR pair specifically so that we use morphologically similar languages where there is no biased distribution of tokens in a single tokenizer.

To address these limitations we performed additional experiments: (i) the same setting with different language pairs, English and Turkish, which share the alphabets but have different grammatical structures (such as order of subject, verb in a sentence), and (ii) finetuning of a pretrained coding language model (Qwen Code 0.5B, 1.5B) to follow a specific target python syntax among other plausible ones. Specifically, we used indentation (tab vs n-spaces) and string formatting (% formatting vs. .format() ). For all cases, we detected behavioral forgetting early on. For targeted finetuning on removal of tab indentation, we found clear signs of representational forgetting after elongated finetuning (x 10^3 of behavior forgetting onset). For other cases, given the available time and compute, we suspect that the separation gap is even larger and probably requires even longer finetuning.

Overall, these experiments support that our observations and claims are more general phenomena than initial controlled setup.

We include our new results and experimental details in C.3 and D, Figure 19-20, Figure 23-24.

We hope our response clarified the concerns and the questions, and we look forward to the reconsideration of the scoring.

Thank you for the additional experiments. I will raise my score to a 3.

Reviewer_cmFYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the issue of model unlearning specifically in the context of supervised fine tuning (SFT). The paper specifically looks at the discrepancy between a model’s tendency to produce data which has been “unlearned” and its ability to recognize/represent that data. The authors use a toy data setting in which they generate strings from a formal grammar. During unlearning, they remove strings that invoke certain production rules internal to the tree. They use a few control conditions that involve generating similar surface forms while using different production rules. They then analyze, over the SFT timesteps, how likely the model is to generate text that uses the deleted rules vs. how well the model encodes the structure e.g., by encoding well-formed sentences that use the deleted rule as similar/dissimilar to ungrammatical ones. The paper then includes a short theory/proof section that tries to relate the time taken to edit representations to the depth of the rule in the hierarchy and its representation at a give layer (this is pretty opaque to me, see weaknesses). Finally, the paper validates that similar patterns hold on a dataset of English and French.

Critical strengths

The topic of the paper is interesting and overall the work makes some nice points. The experimental design with toy grammars and models trained from scratch is a nice choice, and it leads to some clean empirical results. The controls are well designed as I understand them, and the discussion and presentation are mostly clear.

Critical weaknesses

My biggest gripe with the paper is that I struggled with the point of the theoretical results and what it meant. Not only is that section quite dense and hard to read, but it was not interpreted in a way that makes it possible to appreciate the significance even without following the proof. Perhaps this is a me problem, but I would guess that I am representative of the target reader (intelligent, working on related research topics, mathematically capable, but chronically short on time) and so for the paper to be impactful, the section should be written for such an audience. I.e., the section should include a natural language description and interpretation of the point of the proof, its significance, and why we might care, before walking through a series of equations. More pessimistically, I’ve noticed a trend towards “proof maxxing” in papers, in which researchers are feeling the need to add proofs (often heavily AI assisted) to papers just to appear rigorous. I am not sure if this is that…but that is the point! Some nice interpretation and discussion around why this matters alleviates concern that it is math slop.

Beyond discussion, a nice way to make the proof a real part of the paper is to use it to make some concrete empirical predictions – e.g., for a range of difference grammars can you predict (based on your proof) which rules will be more or less easily forgotten? And then can you run the experiments to validate these predictions?

My second concern is the ecological validity of the experimental setup. The authors mention that RLHF is left for future work. I am okay with that in principle, but in reality most “unlearning” work (e.g., for safety reasons) happens via RLHF not SFT, so the dynamics there are more relevant. In addition/alternatively, the paper would be more interesting if it tried to mirror a realistic setting in which models can be tricked into generating unlearned text. So, for example, could the simulations (or better, the theory) tell us something about how/when it is possible to still prompt a model to generate unlearned text? How does this relate or not to the different timescales of forgetting?

Questions for authors

See weaknesses

1 comment

We thank the reviewer for their valuable comments.

We made the following improvements in our presentation and conducted additional experiments to clarify the concerns.

1. Cleaner proof and interpretation

We understand the reviewer’s concern on the “proof maxing” in papers, and we acknowledge that our section reported technical parts of the derivation. In our new rewrite, all the technical part has been transferred to the appendix and only the conceptual aspect of the theoretical argument is discussed in the main section: the difference of the conditional next-token probabilities between fine-tuning and pre-training distributions gives the right scaling of the behavioral forgetting times for rules at different hierarchical levels. We acknowledge that this is not a full dynamical theory for transformer finetuning, but rather a theoretical scaling argument that is in line with similar first-gradient-step arguments made in the literature for the learnability of the RHM.

This theory is already validated in the RHM experiments, where lower level rules get forgotten sooner than higher-level ones, but with a separation between layers that is less dramatic with respect to the one observed in pre-training.

Testing this prediction in real application settings would require defining hierarchical rules for real data and/or measuring correlation differences between pre-training and finetuning distribution. This is definitely a promising direction and, since it requires significant technical work, we leave it open for a future publication.

2. Ecological validity of the experimental setup

I am okay with that in principle, but in reality most “unlearning” work (e.g., for safety reasons) happens via RLHF not SFT, so the dynamics there are more relevant.

We agree that RLHF is a probably more important topic for forgetting/unlearning. However, to our extent of knowledge, SFT is still a very common part of finetuning such as refusal training, instruction tuning and domain adaptation, where a pretrained model is trained further on a narrower set of data than it was pretrained on. This is the setting of our paper. Fine-tuning on the retained data alone is also a standard baseline in the unlearning literature, and it corresponds directly to our restriction fine-tuning, in which the deleted rules are simply absent from the fine-tuning data. In addition, SFT has a fixed data distribution, which allows us to define exactly what is removed and to relate the two forgetting timescales to the statistics of the data (Section 4.3) and to the noise of the parameter updates (Section 4.4). We see this as a natural first step toward the RL case, where the data distribution changes with the policy during training, and we keep RL post-training as a natural future direction in the Discussion.

In addition/alternatively, the paper would be more interesting if it tried to mirror a realistic setting in which models can be tricked into generating unlearned text. So, for example, could the simulations (or better, the theory) tell us something about how/when it is possible to still prompt a model to generate unlearned text? How does this relate or not to the different timescales of forgetting?

To address this question, we added a new experiment in the RHM that mirrors prompting a fine-tuned model with unlearned text (Appendix A.8, Figure 13). The prompt is the first half of a sequence in which each latent node uses a deleted rule with probability 1/2, as in the pretraining grammar. The fine-tuned network generates the second half, and we parse the whole sequence with the pretraining grammar to count, at each level, the deleted rules, the kept rules and the invalid nodes of the continuation.

(i) Deleted rules in the prompt do not bring back deleted rules in the generation. Once generation is suppressed, nearly every rule inside the continuation is a kept rule, and the deleted-rule rate there keeps decreasing for the rest of fine-tuning (Figure 13(a) and (b)).

(ii) The two timescales determine whether the model still understands the prompt. Between the fast and the slow timescale, the model reads the latent symbol of a prompt written with deleted rules and continues it with kept rules: the rule at the root, which joins the prompt to the continuation, is almost always a valid kept rule. After the slow timescale, the fraction of invalid roots increases strongly (Figure 13(c)), over the same interval in which the deleted-grammar-in-context loss increases (Figure 2(c)). The model then no longer reads the symbol of a prompt written with deleted rules.

In our setting, the fast timescale therefore removes the deleted rules from what the model generates, with or without deleted rules in the prompt, while the slow timescale removes its ability to understand a prompt that contains them.

We hope our response clarified the concerns and the questions, and we look forward to the reconsideration of the scoring.

Reviewer_rGoROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies forgetting under restricted-distribution finetuning and identify two timescales: rapid behavioral suppression and slower representational degradation. Experiments with a hierarchical synthetic grammar and an English–French language model support this separation, while parameter dynamics suggest a possible mechanism.

Critical strengths
  1. The paper clearly distinguishes behavioral suppression from representational forgetting and tracks both throughout finetuning.

  2. The controlled grammar enables targeted intervention and tests whether suppressed rule remains useful for interpreting context.

  3. Multiple grammar configurations, weight-decay interventions, and bilingual experiments provide evidence for the observed separation.

Critical weaknesses
  1. RHM’s preserved latent structure and the closely related and deliberately code switching English–French setup limit generalization to settings with less shared structure.

  2. Sentence initial token loss and coherent shuffled representation similarity provide limited evidence of retained understanding or knowledge erasure without context ablation and recoverability controls.

  3. The first step gradient analysis and parameter displacement measurements do not establish the full timescale dynamics or isolate gradient noise as the cause of delayed forgetting.

Questions for authors
  1. What motivated the English–French choice, and does the timescale separation persist for less similar language pairs?

  2. Do LoRA or regularization toward pretrained weights delay representational degradation?

2 comments

We thank the reviewer for their appreciation of the work and their valuable comments.

Below, we provide explanations and additional experiments added to the paper to address the weaknesses pointed out by the reviewer.

1. Limited setup by EN - FR

English–French setup limit generalization to settings with less shared structure

What motivated the English–French choice, and does the timescale separation persist for less similar language pairs?

We agree that our initial experiment on EN-FR is a limited realization. This limitation was due to the computational costs of performing pre-training and fine-tuning with practical language models and the necessity of having two languages with similar enough morphology to use a single tokenizer without a biased distribution of tokens to a specific language.

To address these limitations, we performed additional experiments: (i) the same setting with different language pairs, English and Turkish, which share the alphabet but have different grammatical structures (such as the order of subject and verb in a sentence), and (ii) fine-tuning of a pretrained coding language model (Qwen Code 0.5B, 1.5B) to follow a specific target Python syntax among other plausible ones and check the behavioral/representational persistence of the removed rule. Specifically, we used indentation (tab vs n-spaces) and string formatting (% formatting vs. .format() ). For all cases, we detected behavioral forgetting early on. For targeted fine-tuning on removal of tab indentation, we found a clear sign of representational forgetting after elongated fine-tuning (x 10^3 than behavioral forgetting onset). For other cases, given the available time and compute, we suspect that the separation gap is even larger and probably requires even longer fine-tuning.

Overall, these experiments support that our observations and claims are more general phenomena than the initial controlled setup.

We include our new results and experimental details in C.3 and D, Figure 19-20, Figure 23-24.

2. Limited evidence of knowledge erasure

Sentence initial token loss and coherent shuffled representation similarity provide limited evidence of retained understanding or knowledge erasure without context ablation and recoverability controls.

The representation probe qRq^R measures whether the model gives similar hidden representations to different contexts that correspond to the same latent variable, compared with other non-synonymic contexts. This is a standard metric in the RHM literature, since the similar representation of synonymic contexts has been shown to be key for the learnability of the task, as we reproduce in the appendix for the pretraining phase (see Appendix A.2, Figure 5(b)).

To provide an additional metric to detect the encoding of latent variables, we have added experiments to the paper using linear probes, which are a standard tool in the deep learning literature. The probes are trained to decode the latent variables from the representations of contexts written with kept rules, and then tested on synonymic contexts written with deleted rules. As shown in Figure 10 in Appendix A.5 (method in Appendix A.4), the linear probe accuracy perfectly matches the behavior of qRq^R, showing that the two measurements are consistent and provide the same information.

We also added a recoverability control in the RHM (Appendix A.7, Figure 12), in which fine-tuned checkpoints are trained again on the full pretraining grammar. Between the two timescales, the generation of the deleted rules at the upper levels of the hierarchy recovers in far fewer steps than learning them from random initialization. After the slow timescale, this advantage disappears.

We added the following additional control experiments as suggested for the EN/FR bilingual transformer experiment.

1) Context ablation: We compared how much target sentence tokens average loss increases if the model is given in context uninformative sentence compared to a semantically coherent informative sentence in each language. We show that when the model is finetuned with EN only, the target sentence loss is always lower when the English sentence given in context is informative than the context is non-informative. Meanwhile, the loss difference between informative and non-informative French context is similar to that of English early on, but later on there will be less difference, indicating that there is no context benefit in French. We used the baseline as a loss difference from informative/non-informative French context in EN-only trained model.

2) Recoverability: We tested how much training is required to recover French knowledge once the model is finetuned with EN-only. We found that after the onset of representation forgetting (∼103\sim 10^3), the model requires a similar amount of relearning as to training from the model that was never trained with FR. Meanwhile, before the representation forgetting but after the behavioral forgetting the model almost immediately recovers.

We included the above results and experimental details in Figure 21 in Appendix C.2, C.3.

3. Clearer explanation and evidence on mechanism behind the separation of timescales

The first step gradient analysis and parameter displacement measurements do not establish the full timescale dynamics or isolate gradient noise as the cause of delayed forgetting.

To further test our hypothesis for the origin of the two forgetting timescales, we performed two additional experiments: (i) varying the learning rate η\eta, and (ii) bounded suppression, in which the loss stops acting on tokens that are already suppressed.

(i) Learning-rate dependence. To test our hypothesis, we exploit the different dependence of the mean and variance of stochastic parameter updates on the learning rate η\eta. For Adam, as for SGD, the mean parameter update is proportional to η\eta, while its variance is proportional to η2\eta^2, at a fixed parameter configuration and optimizer state. This suggests two distinct timescales: if behavioral forgetting is primarily driven by a persistent gradient, we expect its characteristic timescale to scale as τbeh∝η−1\tau_{\mathrm{beh}}\propto\eta^{-1}. Conversely, if representational forgetting is driven by the accumulation of noise in the updates, we expect a timescale proportional to η−2\eta^{-2}, provided the noise accumulates diffusively. In the new Figure 3 of Section 4.4, we find that the curves of the deleted-rule rate collapse remarkably well when plotted against tηt\eta (Figure 3(a)), and the curves of qRq^R and of the parameter displacement when plotted against tη2t\eta^2 (Figure 3(b) and (c)). The linear probe accuracy shows the same collapse (Figure 10 in Appendix A.5). This provides strong additional evidence for our interpretation of the two forgetting mechanisms.

(ii) Bounded suppression. In a complementary experiment, we stop the loss from acting on output logits that are already far below the largest logit at their position (by more than 8 nats), with the aim of removing the persistent gradient on already suppressed tokens while leaving the noise of the updates largely unaffected. As shown in Figure 3(d) of Section 4.4, once the output logits reach this bound, behavioral forgetting saturates. In contrast, the subsequent representational forgetting continues on essentially the same timescale as in the baseline, consistent with the persistence of noise-driven parameter motion.

Together, these two experiments provide strong additional evidence for our proposed mechanisms.

4. Do LoRA or regularization toward pretrained weights delay representational degradation?

In the revised manuscript, we added new experiments with an L2L_2 regularization toward the pretrained parameters. We find that increasing the regularization strength leaves behavioral forgetting essentially unaffected, while eventually preventing representational forgetting over the entire observed training period for sufficiently strong regularization (Figure 11, bottom row, in Appendix A.6).

This result contrasts sharply with standard weight decay, which instead accelerates representational forgetting, as shown in the original manuscript (Figure 11, top row, in Appendix A.6). The difference is consistent with our proposed mechanism: while standard weight decay drives the parameters away from their pretrained values, the regularization toward the pretrained parameters constrains their displacement, suppressing the long-term parameter drift associated with representational degradation (Figure 11(d)).

We hope our response clarified the concerns and the questions, and we look forward to the reconsideration of the scoring.

Comments on the paper

1 comment

We thank all reviewers for their careful reading and valuable comments. The reviewers appreciated the controlled setting of the Random Hierarchy Model, the clear distinction between behavioral and representational forgetting, and the extension to natural language. The main concerns were shared by several reviewers, and we addressed them with new experiments and a revised presentation. All changes are marked in blue in the revised manuscript. We summarize them below.

1. Stronger evidence for representational forgetting.

  • Linear probes (RHM). We fit linear probes on the fine-tuning checkpoints, trained on contexts written with kept rules and tested on synonymic contexts written with deleted rules. The linear probe accuracy stays close to 1 long after the generation of the deleted rules is suppressed, and decreases over the same interval as the representation probe qRq^R (method in Appendix A.4; Figure 10 in Appendix A.5).
  • Recoverability (RHM and EN/FR). We train fine-tuned checkpoints again on the full pretraining data. Between the two timescales the suppressed behavior recovers quickly; after the slow timescale, recovery is no faster than learning from scratch (Appendix A.7, Figure 12 for the RHM; Figure 21 for EN/FR).
  • Context ablation (EN/FR). The benefit of an informative French context for prediction persists after the behavioral forgetting and disappears later in the fine-tuning (Figure 21 in Appendix C.2, C.3).
  • Prompting with deleted rules (RHM). Deleted rules in the prompt do not bring back deleted rules in the generation. Between the two timescales the model still understands such a prompt and continues it with kept rules; after the slow timescale it no longer does (Appendix A.8, Figure 13).

2. Mechanism behind the two timescales (new Section 4.4 and Figure 3).

  • Learning-rate dependence. A persistent gradient gives a timescale proportional to η−1\eta^{-1}, and the accumulation of noise in the updates gives a timescale proportional to η−2\eta^{-2}. The curves of the deleted-rule rate at three learning rates collapse when plotted against tηt\eta, while the curves of qRq^R, of the linear probe accuracy and of the parameter displacement collapse when plotted against tη2t\eta^2 (Figure 3(a) to (c); Figures 9 and 10 in Appendix A.5).
  • Bounded suppression. When the loss stops acting on tokens that are already suppressed, behavioral forgetting saturates, while representational forgetting continues on the same timescale as in the baseline (Figure 3(d)).
  • Regularization. An L2L_2 regularization toward the pretrained parameters leaves behavioral forgetting essentially unaffected and, when sufficiently strong, preserves the representations of the deleted rules over the entire training. Standard weight decay, which drives the parameters away from their pretrained values, instead accelerates representational forgetting (Figure 11 in Appendix A.6).
  • Linearized model. In Appendix B.3, we finetuned a linearized model around the pre-trained weights and compared its behavioral forgetting with that of the full non-linear model. We show that the timescale for the lowest-level rule forgetting coincides between the two models, while for deeper hierarchical rules (which require non-linear latent combinations) the linearized model is slower than the non-linear one. We discuss this finding in the theoretical section in the main text.

3. Presentation of the theory (Section 4.3). The technical derivations are now in the appendix, and the main text presents the conceptual argument: the correlation difference between the fine-tuning and pretraining token distributions gives the scaling of the behavioral forgetting times for rules at different hierarchical levels. We also state explicitly the assumption behind this scaling argument and its limitations.

4. More realistic settings. We repeated the bilingual experiment with a less similar language pair, English and Turkish, and fine-tuned pretrained coding language models (Qwen Code 0.5B and 1.5B) to follow one Python formatting convention among other plausible ones (indentation and string formatting). In all cases behavioral forgetting occurs early. For the removal of tab indentation, representational forgetting appears clearly after a much longer fine-tuning (Appendix C.3 and D, Figures 19-20 and 23-24).

We respond to the specific points of each reviewer in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.