- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether in-context learning (ICL) capabilities degrade when models are repeatedly trained on data generated by their predecessors. The authors find that, even when the overall energy of the data is preserved, the task structure gradually erodes, leading to a decline in ICL performance.
Critical strengths
This paper aims to provide a theoretical analysis of how recursive pre-training affects in-context learning, while distinguishing between the concepts of "task energy decay" and "loss of task structure."
Critical weaknesses
- This paper should further clarify how it differs from existing theories on "model collapse", including the work of Shumailov et al. (2024), Dohmatob et al. (2024), and Fu et al. (2025). The authors should explicitly state the new theoretical insights offered by this paper and explain the limitations of previous research.
- Theorems 3 through 5 rely on simplified linear-quadratic tasks, assumptions regarding the underlying population distribution, and the use of gradient flow. These assumptions differ significantly from the pre-training processes of large language models (LLMs) in real-world scenarios. The paper should further explore the applicability of its theoretical results to more complex Transformer architectures and more realistic training environments.
- The experimental results fail to confirm the cubic threshold predicted by Theorem 4. Across all tested values of γ (0, 2, 3, and 4), the quadratic component exhibits a downward trend within five iteration cycles (generations). More extensive experiments are required to provide stronger empirical support for this theoretical result.
Questions for authors
NA