acceptodds
Under review as a conference paper at ICLR 2027

Pretraining with Masked Backstories in a Toy World

Abstract

Context-enhanced learning (CEL) involves augmenting context in large language models (LLMs) with special masked context to accelerate learning. It has previously only been explored using LLMs with billions of parameters during finetuning because CEL requires in-context learning (ICL) abilities to work. Here, we leverage a toy world (symbolically labeled randomly interleaved vector time-series from linear deterministic dynamical systems) that admits LLM-style “next token” pretraining and has been shown to exhibit multiple emergences of different ICL/recall abilities in tiny transformer models with mere millions of parameters. In this toy world, we can see a late transition from ICL to in-weights learning that also corresponds to a degradation of ICL performance. We enhance pretraining with additional masked context that allows the model to make near perfect predictions on the original training examples. Masking this additional context disincentivizes the model from memorizing it, and the capability of perfect prediction on the training example disincentivizes the model from memorizing the remaining portion of the training example. Not only does this enhancement suppress in-weights learning of the specific training systems, but we also show that it improves the model's performance in the seemingly unrelated task of associative recall. The study of this enhancement led to a few counterintuitive findings. First, despite such a model during training only seeing losses (and hence gradients) for tokens that are perfectly predictable, it generalizes well at test time when predicting tokens that are not perfectly predictable, nearly matching the performance of the optimal solution for those cases. Second, we found that memorization of 0.025% of the training systems was enough to degrade ICL performance on test data. Finally, when the enhancement is applied only at a specific position in each training example, the model still memorizes the training data, but can only recall these memories at the position where the enhancement was applied and not where the data was exposed to be memorized.

Primary area:
interpretability and explainable AI
Keywords:
content-enhanced learning, toy problems, modulating emergence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_g5WKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether context-enhanced learning (CEL) can suppress the transition from in-context learning (ICL) to in-weights learning (IWL) during pretraining. Using a toy setting of interleaved linear dynamical systems, the authors introduce masked backstories, which prepend informative historical states while masking their training losses. The intervention prevents late-stage memorization, preserves ICL generalization, and improves associative recall. Further experiments reveal several surprising behaviors, including near-optimal predictions at positions without direct supervision, ICL degradation when only 0.025% of training systems remain exposed to memorization, and position-dependent retrieval of memorized information.

Critical strengths
  • Effective suppression of ICL transience. The masked-backstory intervention successfully prevents late-stage memorization while preserving held-out ICL performance and improving associative recall (Figs. 2 and 4). The unmasked and uninformative-backstory controls further support the importance of combining informative context with loss masking.

  • Interesting empirical observations about ICL–IWL interactions. The experiments reveal unexpected behaviors, including near-optimal predictions without direct supervision (Fig. 5), ICL degradation from memorizing only ten training systems (Fig. 6), and position-dependent memorization (Fig. 8). These findings raise potentially valuable questions about how competing learning strategies develop in transformers.

Critical weaknesses
  • Insufficient isolation of the benefits of masked backstories. Although the authors examine several backstory variants, they do not compare against simply masking losses on initially unpredictable positions without adding backstories. Such a baseline is important for determining whether the improvement requires informative additional context or primarily results from removing the training incentive to memorize difficult positions.

  • Limited generality and robustness of the findings. The experiments primarily use a single 10.7M-parameter transformer and deterministic five-dimensional orthogonal systems, leaving generalization to other architectures and tasks unclear. Furthermore, the absence of reported variability across independent training runs makes it difficult to assess the robustness of subtler findings, particularly the degradation caused by exposing only ten systems.

Questions for authors
  • How does masked-backstory training compare against a baseline that simply masks losses on the first unpredictable observations of each system, without prepending additional context? Would this baseline similarly suppress IWL and improve associative recall?

  • Are the observed phenomena consistent across independent training seeds and different model sizes or dynamical systems? In particular, how robust are the ICL degradation caused by the ten exposed systems and the position-dependent memorization results?

3 comments

Thank you for your time and your thoughtful review. Here are our responses to your constructive criticisms:

AuthorsOpenReview ↗

Masking the Loss Without Adding Backstories

We appreciate you pointing out a natural baseline that we missed. We have now run this experiment and the results can be seen in Appendix A. Masking the first seven indices of the first occurrence of a system with a training example does suppress the transition from ICL to IWL just like the masked backstories intervention. Nonetheless, associative recall and performance on later indices that can reach zero error lag behind the performance of the model trained with masked backstories.

AuthorsOpenReview ↗

Robustness of Findings to Multiple Training Runs, Sources of Randomness and Architectures

In Appendix A, we now have a training run of a 63M-parameter model that shows the same dynamics of transitioning from ICL-IWL, except earlier in training than the 10.7M-parameter model. We also have another run of the ten exposed systems from a different random initialization. We see that the effect still appears. We have also run two more training runs of the 10.7M-parameter model on different random initializations and different random draws of training systems and initial states. The masked backstories intervention works just as it does in our original training run.

Reviewer_khWpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work studies ICL transience in a toy model. The data are interleaved trajectories from multiple linear dynamical systems. The task is next-token prediction. Two capabilities are tested: 1. inferring U from the context (ICL), 2. recalling a previously seen system when it is queried again (associative recall). This work proposes a new method called masked backstories to suppress ICL transience in the toy model. This work reveals that: 1. masked backstories added in the middle of training can stop and even reverse the memorization and maintain ICL performance on held-out data. 2. training with masked backstories from the beginning leads to easy-to-hard generalization: the model makes near-optimal predictions at the early, underdetermined positions even though it never received any loss on them. 3. even when only 0.025% of the training systems (10 out of 40,000) have no backstories, these systems are still memorized. This memorization still degrades ICL on held-out data. 4. when masked backstories are added only to the initial segment, the model can still memorizes the training data, but it can also recall these memories only at the initial segment, where the backstory is applied, and not at the later segments where the data is exposed to be memorized. 5. the backstory should carry real predictive information: backstories of length one or two, or backstories made of i.i.d. Gaussian noise, they do not prevent memorization.

Critical strengths

The experimental setup is very clean. The pseudoinverse predictor can be precisely calculated, and memorization can be precisely confirmed whenever the model’s error on the training data falls below this baseline. Memorization and ICL are usually very hard to distinguish in realistic language models. Then, the method is simple and comes with very comprehensive comparisons. The effectiveness of the masked backstory is carefully studied, and the ablation experiments rule out other possible explanations.

Critical weaknesses

I think the most obvious problem is that there is almost no mechanistic explanation, although the discovered phenomena are rich. Then, it is unclear how the masked backstories method proposed in this work could be implemented in real LLMs. The reason why the method works so cleanly in this toy model is also the reason why it is hard to use in reality: here the dynamics are fully known and invertible, so the backstory can be constructed exactly, making the subsequent tokens perfectly inferable in context. For natural language, such context generally cannot be constructed.

Questions for authors
  1. What would happen if you applied only the loss mask, without the backstory?
  2. Why do you consider an average of 0.64 unmasked systems per batch to be rare? Whether a model can memorize a system should be determined by its total number of appearances during training, rather than by its appearance rate per batch.
5 comments

Thank you for your time and your thorough review. We believe that rigorous and controlled experiments are a path towards principles that can hopefully generalize to more realistic or complex settings. Here are our responses to your helpful criticisms:

AuthorsOpenReview ↗

Mechanistic Analysis

We agree that this setting is ripe for a study of the mechanisms behind the behavior. It is a high-priority future direction for us to have a unified picture of how the ICL, IWL, and associative recall mechanisms interact within the model.

AuthorsOpenReview ↗

Relevance to Real-world LLMs

We agree that the masked backstories intervention is specific to this toy setting, but we see ways that concepts touched upon in this simplified setting have parallels in practice. One example is supervised finetuning (SFT), where practitioners have noticed that, in most cases, it is better to mask out the prompt in the training example and only have loss be attributed to the model’s response. They have found that when there is no masking on the prompt, the model tends to learn to imitate the user asking questions instead of learning how to answer them. For instruction finetuning specifically, works [1,2] have tried to understand when you want to mask the prompt and when you do not. An even closer example is goldfish loss [3], where random tokens within a training example are masked from the loss to try and mitigate memorization in LLMs. Our work is studying how masking relevant context affects knowledge acquisition and generalization capabilities which are two things that matter to someone who wants to pretrain or finetune a model.

To address the question of how masked backstories could be implemented at scale, [4] showed that another LLM can be used to generate concise summaries of a training example. If this other LLM provides relevant enough information that can be used to predict subsequent indices, to implement masked backstories, you could run this LLM on your dataset and prepend the relevant summary to each training example and ensure the training loss is masked on the summary.

With regards to the puzzling results, recent work has shown that LLMs exhibit positional fragility [3], where it is easier for models to memorize tokens that appear near the beginning of their context window versus other positions. Our result in Section 6, where our model can only apply the memorized knowledge at early indices even though that knowledge only appeared at late indices adds to the richness of the phenomena within positional fragility. We believe that our toy setting will be a valuable testbed for understanding the principles behind how LLMs’ memorization capabilities are distributed throughout their context window.

We hope to include these references in a future version of our paper in an expanded version of our related work section.

[1] Chatterjee, Anwoy, H. S. V. N. S. Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. 2025. “On the Effect of Instruction Tuning Loss on Generalization.” Transactions of the Association for Computational Linguistics (Cambridge, MA) 13: 1360–80. https://doi.org/10.1162/tacl.a.42.

[2] Huerta-Enochian, Mathew, and Seung Yong Ko. 2024. “Instruction Fine-Tuning: Does Prompt Loss Matter?” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, edited by Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.1267.

[3] Hans, Abhimanyu, Yuxin Wen, Neel Jain, et al. 2024. “Be like a Goldfish, Don’ t Memorize! Mitigating Memorization in Generative LLMs.” Advances in Neural Information Processing Systems 37: 24022–45. https://doi.org/10.52202/079017-0757.

[4] Qi, Siyuan, Bangcheng Yang, Kailin Jiang, et al. 2025. “In-Context Editing: Learning Knowledge from Self-Induced Distributions.” https://arxiv.org/abs/2406.11194.

[5] Xu, Yixuan, Antoine Bosselut, and Imanol Schlag. 2025. “Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks.” Advances in Neural Information Processing Systems 38, Main Conference: 156352–84. https://doi.org/10.52202/085713-5223.

Reviewer_7ogTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how gradient flow interacts with ICL and IWL in a simple synthetic task setting. Authors introduce "backstories" - additional context that provides enough information for ICL to predict upcoming tokens, but with gradient flow blocked so as to discourage memorization - in order to disrupt IWL and promote ICL. They find that this largely works, though with several surprising consquences: (i) backstories promote generalizable ICL abilities, even to tokens that did not receive gradient supervision, (ii) small amounts of permissible IWL degrade ICL, (iii) memorization occurs for data with no gradient flow.

Critical strengths
  1. The backstories intervention is a useful, targeted approach for encouraging the development and maintenance of ICL abilities over IWL abilities. This enables several deeper investigations of the interaction between supervision signal and ICL, which may enable further controlled analysis.

  2. The authors go beyond offering a simple solution to ICL transcience, instead probing several non-obvious effects engendered by backstories. Authors correctly and clearly denote open questions, which can inspire future work. In particular, section 4.2 appears to suggest that models induce a general, rather than index-specific, ICL function from data, which can generalize earlier in time to examples that received no explicit supervision. Studying the particular aspects of the data generating process + model training that encourage such a generalizable solution would be a valuable addition to the literature. This work serves as a bedrock for such future investigation.

Critical weaknesses
  1. The core motivation behind prepending loss-masked "backstories" is not clear from the manuscript. More time should be dedicated to explaining this intervention, perhaps with an enhanced version of figure 3 that clearly denotes that the training loss (rather than e.g., token identity) is masked, and why this only leaves an ICL path toward learning the task.

  2. It seems very plausible that section 5's main effect is the result of a capacity bottleneck. This work frames the ICL degradation as a mystery due to a mix of training signal and task, but one must first rule out this simpler explanation.

Questions for authors

In section 5., did you attempt to train with a larger capacity model? Perhaps the degradation in ICL performance is a due to IWL and ICL fighting over a capacity bottleneck?

3 comments

Thank you for your time and your thorough review. We appreciate your insights and agree that this work is just a first step towards a better understanding of the relationship between supervision and the development of ICL and/or IWL in neural networks. We thank you for your constructive criticisms. Here are our responses to each of your criticisms:

AuthorsOpenReview ↗

Motivation of Masked Backstories Intervention and Clarity of Explanation

Thank you for this suggestion, we’ve added the following paragraph to the beginning of section 4, where we introduce the intervention: “In Section 3, we showed that late in training the model begins to remember the exact values of traces from the training trace library and treat them as singleton facts as opposed to examples to improve its ICL abilities. Since the pseudoinverse predictor error is non-zero on these early indices, the high occurrence of them during training means that the model accumulates training loss on these indices even when its predictions have reached optimal performance. We believe this is what drives the model to resort to an in-weights approach for predicting the early indices of a segment, as opposed to an ICL approach.

To suppress this in-weights learning (in the style of Lampinen et al. (2025); Park et al. (2025b)), we ensure that training loss does not accumulate if the model is predicting optimally… ”

Additionally, in the caption of Figure 3, we’ve added the following sentence for clarity, “Light opacity blocks correspond to indices where training loss is masked, while full opacity blocks are indices that receive training loss.”

AuthorsOpenReview ↗

Is the Degradation of ICL Caused by a Model Capacity Bottleneck?

In the newly added Appendix A we provide plots for a training run of a 63M-parameter model, where the model used in our submission was 10.7M-parameters. What we find is that the 63M-parameter model memorizes the early indices earlier in training than the 10.7M-parameter model, and the degradation of ICL occurs earlier in training than the 10.7M-parameter model as well. We have also found this trend to hold for models smaller than 10.7M-parameters. Smaller models take longer to memorize and ICL capabilities degrade later in training. Under the model capacity bottleneck hypothesis, we would expect that as model size increases, the degradation of ICL will happen later in training if at all. Since we are seeing the opposite behavior, we believe that this degradation, along with the degradation that we see in Section 5, is not strictly due to model capacity constraints. Nonetheless, it remains to be seen whether this trend with model scale holds for models in the billions or trillions of parameters, but running enough pretraining runs for thorough experimentation at this scale is prohibitively resource intensive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.