acceptodds
Under review as a conference paper at ICLR 2027

Forking: Sudden Overfitting Under Replay

Abstract

This paper studies forking, a generalization failure discovered in NanoGPT autoresearch. Under data replay, models with an over-encoding n-gram memory branch show a sharp separation of training and validation loss at epoch boundaries, resembling the shape of forks. We study this phenomenon in a controlled vanilla NanoGPT setting and reproduce it in a DeepSeek-style model with Engram. Mechanistically, repeated updates sharpen the continuations observed in training while suppressing the probability of unseen continuations, whose loss grows with each pass. The n-gram module creates weakly interacting context-specific subspaces, amplifying this effect. Low-frequency contexts contribute most of the gap, whereas larger training budgets and heavily crowded tables suppress it. We also observe forking in short-budget, heavily repeated SFT and RL-like regimes. The contributions of this paper are twofold: (1) Forking reveals yet another curious phenomenon in deep learning, in addition to grokking and double descent. (2) Forking is an unexpected and unpleasant by-product of tricks proposed by autoresearch agents. While these agents produce an enormous number of results that seem useful, we should always be careful with their results.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model, pretrain, memorization, data replay, generalization gap

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_AmGDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a new term called forking under data replay in over-encoded transformers. It provides analysis on why forking happens under this setting and extends this to fine-tuning as well.

Critical strengths

Observing overfitting under data replay in over-encoding has not been previously studied and analysing this phenomenon is interesting.

Critical weaknesses

I am a big fan of analysis based papers, primarily because they offer more insights than a solution based paper. However I believe there are several critical weaknesses that I want the authors to address:

  1. Forking is essentially "overfitting" (irrespective of whatever the setting is, i.e., over-encoding setting). This creates a lot of confusion because authors are just giving a new term to an already known phenomenon. It would be better if they remove the term "forking" and just say that the model is "overfitting".

  2. Over-encoding has been the main theme of this paper, however, it has never been officially defined throughout the paper. It made it extremely difficult to understand what it really means. The authors should provide proper definition of over-encoding rather than just citing [1] (in line 68). Furthermore, they should provide a clear example of over-encoding by a giving a simple naive example. In this way, any naive reader can understanding what over-encoding even means in the first place.

  3. The authors use a vanilla NanoGPT and DeepSeek model. However, they have not shown whether the trends hold for larger model sizes within these model families. Furthermore, no analysis has been done in other open-source models like Llama, Qwen etc. The trends can differ across model families and especially when sizes vary within a family. An experiment regarding this is required.

  4. In line 140, the authors mentioned that "retrieved embeddings are injected at one of three routes— token representations before attention (input), attention values (v), or attention output (y)". This doesn't tell us whether the n-gram modules are inserted at every position or randomly. If it is randomly, then again we cannot tell inserting at which position has the impact on overfitting. A position-wise impact is required. Also, in Fig 3 there are considering one transformer block. Are these inserted at every transformer block (8 transformer blocks in NanoGPT and 12 in DeepSeek) or only at some of them. For a proper analysis, the authors should provide experiments whether there is a trend when insertion of these modules are compared within a transformer block (attention (input), attention values (v), or attention output (y)) and across transformer blocks (block 1,,...., block 12). Also, why is insertion of these modules not done after FFN/MLP?

  5. In line 195 and 200, authors are talking about context frequency and table capacity, but never defined it - I still do not understand what they really mean by this after repeated reading of their paper. They must also provide concrete examples of this from their training corpus considered. In lines 225-226 the authors say that they count context frequencies and mask the n-gram based on a varying threshold. But again they never explained how they counted the context frequencies and masked the corresponding n-gram contributions. This is critical because as a practitioner I need to know how this was done.

  6. Section 3.1 is introduced as an empirical scaling law, a scaling law is generally an equation which estimates a quantity based on some arguments (like test loss wrt model size, training data size etc). I was expecting something similar but nothing like that has been introduced. Hence, what is the point to even use such a section heading. The authors should change the title of the section to a more neutral line or provide a proper scaling law.

  7. Section 4 of the paper, tries to explain why rare context frequency is the cause of the observed phenomenon and even defines something called as "context subspace". However, all subsections in Section 4 are written so poorly and are presented in an over-complex way for one to even understand. I still cannot understand what the authors even mean by Section 4 as a whole. Also, what is context subspace, the authors should define it properly.

  8. In line 453, the authors say their study is limited due to scarce availability of open source over-encoding models. However, in their vanilla NanoGPT setting, they themselves added the n-gram modules. This means they can do the same for other open-source models and it is not a problem of availability.

Questions for authors
  1. I believe the initial NanoGPT results are provided under pertaining, but what dataset has been used for it? I don't see any information regarding this.

  2. In supervised finetuning setup (Section 5), are the authors using LoRA or full-fine tuning?

  3. In Fig 7b of table freezing, why does the online gap suddenly decrease at the start of the second epoch, i.e., optimizer step 674? This is peculiar because this happens only when the n-gram modules are frozen (purple line) and not when they are trainable (blue line).

[1] Over-Tokenized Transformer: Vocabulary is generally worth scaling

If the authors wish to submit a rebuttal, I would like them to properly clarify all questions raised by me in Critical Weaknesses and Questions for Authors. (PS: I have not used any AI tool to review your paper).

[Update after seeing the rebuttal]

In the current form the paper has a lot of writing-related, title-related, lack of experiments issues as stated above, which I don't believe can be resolved in the current review cycle. Hence, due to all these reasons, I standby my current rating. I hope the authors can carefully read and utilize the above comments to improve their paper for the next conference submission.

Reviewer_xgS9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies ``forking'', a sharp training–validation loss divergence during repeated training on a fixed corpus, amplified by injecting learned, hash-indexed bigram and trigram embeddings into a Transformer. Its proposed explanation is that these embeddings facilitate context-specific memorization: replay reinforces continuations observed in training while suppressing valid alternatives, particularly for rare contexts with incomplete continuation coverage.

The paper's main contributions include:

  • Empirical characterization: empirically showing how the gap varies with context frequency, memory-table capacity, corpus size, and number of training passes.
  • A proposed mechanism: connecting unseen-continuation frequency with the generalization gap, and arguing that n-gram memory enables relatively independent fitting of individual contexts.
  • Intervention evidence: examining memory removal, frequency-based retrieval masking, hash reassignment, parameter freezing, and withholding supervision for selected contexts.

Validation primarily uses controlled NanoGPT experiments with no-memory baselines and several memory-injection routes. The authors additionally reproduce the phenomenon in a compact DeepSeek-style architecture and examine continued training on code, math, and same-distribution data. The evidence supports over-encoding's failure mode in the tested regimes.

Critical strengths
  • Descriptive strength: The paper identifies when over-encoding harms generalization. In the tested replay settings, models with n-gram over-encoding produce substantially worse validation loss than a baseline without it. By analyzing context frequency and varying table capacity, corpus size, and the number of passes, the paper identifies a failure regime involving small corpora repeatedly trained with large n-gram tables. Rare contexts are particularly vulnerable because training captures only some valid continuations, and replay can reinforce those continuations while suppressing unseen alternatives. Masking the prediction loss for selected contexts further suggests that their gap growth depends strongly on their own replay supervision.

  • Prescriptive strength: The paper identifies targeted interventions in how n-gram tables are accessed and reused. Masking retrieval for low-frequency contexts reduces both the gap and validation loss in the minimal model and the DeepSeek-style replication. Changing the mapping between contexts and table rows at replay boundaries also suppresses gap accumulation, supporting the role of stable mappings in repeated memorization. In the minimal model’s input-injection comparison, neither tested intervention fully recovers the validation performance of the baseline without over-encoding. Nevertheless, the results suggest concrete mitigation directions through restricting poorly supported lookups and disrupting repeated reinforcement of context-specific associations.

Critical weaknesses
  • The framing is broader than the explanation supports. The paper emphasizes autoresearch and compares forking with grokking and double descent, while explaining it through ordinary overfitting within individual contexts. Hash reseeding supports the importance of stable table mappings, and Figure 9a provides evidence of reduced beneficial learning across contexts. However, the logit analysis does not fully explain how the injected embeddings produce this reduction. The results support a plausible mechanism, but the claim that the mechanism is largely understood seems stronger than the evidence warrants.

  • Practical significance under training tuned for replay remains unclear. The original search aimed to improve validation performance, but the main controlled settings were chosen to expose a clear replay gap within short runs. The paper does not establish how much worse over-encoding performs when learning-rate schedules, regularization, and checkpoint selection are tuned for repeated training. Its extensions largely reuse the same settings. The authors appropriately leave production-scale testing to future work, but their continued language-model experiments and static RL proxy provide limited support for broader claims about the severity of the problem in practical SFT and RL.

Questions for authors
  • Clarification requests

    • Could the authors clarify how the injected n-gram embeddings reduce beneficial learning across contexts? Hash reseeding and Figure 9a support this interpretation, but Section 4.2’s logit analysis does not explicitly connect this behavior to the embeddings. It seems standard to me as how overfitting influences the token probability distribution.

    • Do existing results show how much worse over-encoding performs when training settings and checkpoints are selected for good validation performance under replay? A comparison with a similarly tuned model without over-encoding would help establish whether the reported severity persists beyond the settings chosen to expose the failure.

Reviewer_EU3oOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that training with bigram and trigram embedding tables shows sharp divergence between training and validation loss at epoch boundaries, a phenomena they term forking. The authors show this phenomenon can also be replicated in SFT settings.

Critical strengths
  1. The problem is important as n-gram is frequently used in training LLMs. Studying their generalization behavior in the data bound setting is increasingly relevant.
  2. Experiment details are well documented.
Critical weaknesses
  1. I find the claim that forking is a new phenomena exaggerating. The sudden increase in train-validation gap is a well-documented phenomena in deep learning without n-gram tables, as shown in several of the cited references in this paper, and comes down to the loss being lower on data the model has already seen due to memorization. The magnitude of that gap reflects the strength of memorization, which naturally becomes large with the massive parameter count of n-gram tables.
  2. The size of this demonstrated gap depends on extreme hyperparameter choices, where the n-gram table LR is 128x the other parameters. It's not surprising that if one primarily learns an n-gram model one would see large generalization gap under multiple epochs.
Questions for authors

RMSProp and Adam is known to be prone to producing over-sized updates for parameters with sparse gradients, which could have contributed to the extent of overfitting here. Do you see the same phenomena if training with signSGD or SGD?

Reviewer_FYVuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents Forking, a phenomenon happening under overencoding and data-replay: after the first epoch, the training and validation losses separate and the gap increases at each epoch. The authors describe how it was discovered and establish a minimal setup to reproduce it, and follow with an empirical study of the phenomenon, including post-training trainings. The authors claim the problem is due to rare contexts.

Critical strengths

The paper correctly presents the authors method. From its discovery to the empirical analysis, the authors search the cause. The experiments are sound and clearly demonstrate the phenomenon.

Critical weaknesses

The phenomenon shown in the paper is predictable. Using a small model (nanoGPT) with large hash table means that most of the parameters are dedicated to the ‘memory’ part of the model. Hence, overfitting / memorization is not surprising. Moreover, there is no baseline in the post-training experiments.

Questions for authors

Is the phenomenon established for larger models? Do you have baselines for the post-training section?

Comments on the paper

9 comments

Hi authors, I'll read your rebuttal and get back to you in the next 1-2 days.

Thanks for your clarifications and new experiments. However, most of my concerns still remain.

W1: In your paper, what you found is that overfitting (train-test gap) increases when we do over-encoding in a language model. Indeed this phenomenon is not observed before. However, naming this observation as Forking is misleading. Instead the title of the paper should be something like - "Over-Encoding Models Overfit More under Data Replay". The word "Forking" should be removed from the entire paper since it only implies the model is overfitting. Just saying that overfitting looks like a fork, doesn't mean that it is not overfitting. Using such an absurd term is just misleading for future researchers who can perceive that forking is a different concept from increase in train-test gap. All you have shown is overfitting increases in over-encoding models.

W2: Fig 1 and Fig 3 never explain how these n-gram embeddings are created. They should be clearly defined in the first place (also by providing an example) on what they are and how they are made. In the current version it is very difficult to understand the term. Both the figures must be revisited to explain the setting well.

W3: DeepSeek-style MoE in the original submission has way too many lookup parameters than the original backbone. It is very likely that just because of adding these overfitting is happening. Instead the authors should test what happens when they keep backbone parameters constant (272.6M) and vary the lookup-parameters (5M,10M,20M and so on). If overfitting increases as lookup-parameters increase, the cause of overfitting is just huge number of lookup-parameters. Regarding the new experiments spanning NanoGPT scale sweep: how many backbone parameters were there?

W4: New results by inserting these over-encoding modules at different transformer blocks vs components are still pending. It is an important that they provide these results in the first place, so that we can see what is the trend.

W6: I have to respectfully disagree here, a scaling law is generally supported by an equation in almost all seminal scaling law papers. Currently, you are just showing how overfitting varies based on the 3 parameters - F, R and M. You must have an equation to study the scaling law among these (like Overfitting = A * F^alphaR^betaM^gamma, where A, alpha, beta, gamma are terms to be estimated). Furthermore almost all scaling law papers provide a goodness of fit (R2 score) and how it performs on a held-out validation set. In the current setting this is not an appropriate to introduce the section as "Empirical Scaling Laws". Moreover, I noticed 2 more issues in Section 3.1: (1) in line 192 it is mentioned analysis of overfitting vs F is done for 3 seeds but no standard-deviation error plots are provided and the other 2 parameters analysis (R and M) are done only for 1 seed - 1 seed tells us barely anything - analysis should be done for atleast 3 seeds for every varying value of F, R and M, (2) in line 202, the authors say that bigram curve is flattening but it seems to still increase. They must increase the M values more to see if they truly saturate quickly or not (again for 3 seeds).

W7: This definition of context subspace does not tell me or any person reading this paper what it truly means. Proper examples must be provided for this along with any prior works which have used this term before. Your general response answer is not what I had asked for.

Q1: The post-training (FT) results are provided only for the NanoGPT model and not for Deepseek. One model is not enough to confirm this for post-training analysis.

Q2: Generally most of the community uses LoRA for finetuning and not full-fine tuning. What happens to over encoding caused overfitting under LoRA? Does it still occur? A sweep across LoRA ranks will clarify it.

Q3: To verify this the authors must show the batch train loss analysis seperately. I cannot take what they said for granted without proper evidence in the updated PDF (As you know you can always update the PDF during the rebuttal).

Global Response for why their research matters: The authors say that 2 frontier labs approached them once they saw their preprint. This is not a fair and valid way to say that your paper is very important. At first I neither know what your preprint is nor who these frontier labs are (especially because knowing such irrelevant things can hamper the double blind review process).This in anyway is not a correct way to justify one's research value.

In the current form the paper has a lot of writing-related, title-related, lack of experiments issues as stated above, which I don't believe can be resolved in the current review cycle. Hence, due to all these reasons, I standby my current rating. I hope the authors can carefully read and utilize the above comments to improve their paper for the next conference submission.

Could the authors please make all the updates that they made throughout the whole rebuttal in the manuscript color-coded (blue). I cannot keep track of the new changes otherwise to see what all has been finally added. This will help me to make a fair decision on my final rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.