acceptodds
Under review as a conference paper at ICLR 2027

Can In-Context Learning Survive Recursive Pretraining?

Abstract

Can an in-context learner generate data from which its successors can relearn the same abilities? We study recursive pretraining of attention models on regression tasks, with each generation trained from scratch and evaluated on the original distribution. We distinguish optimization failure caused by small labels from structural degradation that survives energy normalization. For linear–quadratic tasks under population-context gradient flow, arbitrarily accurate single-generation learning coexists with a nonvanishing recursive clean-risk gap at every fixed training time. We derive a cubic threshold for preserving quadratic structure under polynomial training schedules. Alternatively, a calibrated signed combination of two attention-temperature views, followed by normalization, exactly preserves the task law and every generation's clean risk at fixed training time in the same population setting, without real-data refresh or parameter inheritance. We provide fixed-horizon SGD approximations and demonstrate improved retention in dense-attention experiments.

Primary area:
interpretability and explainable AI
Keywords:
In-context learning, transformer, model collapse

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BixGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether in-context learning (ICL) capabilities degrade when models are repeatedly trained on data generated by their predecessors. The authors find that, even when the overall energy of the data is preserved, the task structure gradually erodes, leading to a decline in ICL performance.

Critical strengths

This paper aims to provide a theoretical analysis of how recursive pre-training affects in-context learning, while distinguishing between the concepts of "task energy decay" and "loss of task structure."

Critical weaknesses
  1. This paper should further clarify how it differs from existing theories on "model collapse", including the work of Shumailov et al. (2024), Dohmatob et al. (2024), and Fu et al. (2025). The authors should explicitly state the new theoretical insights offered by this paper and explain the limitations of previous research.
  2. Theorems 3 through 5 rely on simplified linear-quadratic tasks, assumptions regarding the underlying population distribution, and the use of gradient flow. These assumptions differ significantly from the pre-training processes of large language models (LLMs) in real-world scenarios. The paper should further explore the applicability of its theoretical results to more complex Transformer architectures and more realistic training environments.
  3. The experimental results fail to confirm the cubic threshold predicted by Theorem 4. Across all tested values ​​of γ (0, 2, 3, and 4), the quadratic component exhibits a downward trend within five iteration cycles (generations). More extensive experiments are required to provide stronger empirical support for this theoretical result.
Questions for authors

NA

Reviewer_xrSAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors study a model of recursive pretraining, and characterize the conditions under which recursive pretraining degrades vs stabilizes model performance.

Critical strengths

The question of model collapse is interesting and relevant to the current state of machine learning

Critical weaknesses

The writing and presentation were confusing and could be substantially improved, by using clear language and motivating the choices made in the model studied.

Ex; “Alternatively, a calibrated signed combination of two attention-temperature views, followed by normalization, exactly preserves the task law and every generation’s clean risk at fixed training time in the same population setting, without real-data refresh or parameter inheritance.”

Assumptions are too restrictive and there is no effort made to show that the insight derived from results obtained in this setting transfer to more general settings

In particular, there is no way for the models to improve at all in the paradigm studied, missing a key element behind practical uses of recursive training

Further experiments are necessary to validate that the model studied yields real insight in practical cases.

Questions for authors

Can the authors extend their model to include the possibility of recursive improvement?

Can the authors clarify what predictions their model makes that might be testable in more naturalistic settings? (Training a full LLM is not necessary, but it would be great to see more effort made to bridge the gap between theory and practice)

Reviewer_RvGAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies recursive self-training of attention models on regression tasks, where each generation is trained from scratch and evaluated on the original data distribution. The authors distinguish optimization failure due to limited labels from structural degradation that persists even after energy normalization. Under a linear-quadratic population-context gradient-flow setting, they show that near-perfect single-generation learning can coexist with a non-vanishing recursive clean-risk gap under fixed per-generation training time. They derive a cubic threshold that preserves the quadratic structure and propose a calibration-based signed combination of two attention-temperature views followed by renormalization, which exactly preserves the task distribution and per-generation clean risk without requiring real-data refresh or parameter inheritance.

Critical strengths
  1. The paper provides gradient-flow-level exact analysis rather than purely empirical study on model collapse and recursive self-training, with notable theoretical depth.
  2. The paper clearly distinguishes between optimization failure and structural degradation, offering an actionable repair path via a dual-temperature combination and renormalization that exactly preserves task structure.
Critical weaknesses
  1. The theory is highly idealized, relying on linear-quadratic objectives, population gradient flows, and specific temperature settings, limiting extrapolation to real large-model self-training.
  2. The claim of exact preservation depends on calibration of signed combinations and renormalization, but how these calibration quantities are obtained under the no-real-data-refresh setting is unclear, limiting its practical utility.
Questions for authors
  1. What calibration does the dual-temperature combination require, and does it contradict the no-real-data claim if real distribution statistics are needed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.