acceptodds
Under review as a conference paper at ICLR 2027

Break Step: Recursive Training Resonates with Replayed Sampling Noise

Abstract

How fast does a language model degrade when trained on its own outputs? Theory traces it to gradually accumulating errors, while experiments report repeated phrases within ten generations. Under a fixed sampling seed in vLLM, the fast loss of lexical diversity comes from the sampler. When vLLM serves a batch from one seeded sampling configuration, every request receives the same random draws, and a fixed seed replays them every generation. Fine-tuning raises the tokens that won, and the replayed draws let them win by more. Sharing across requests and replay across generations matter only together. Remove either one, by changing the shared seed every generation or by giving each request its own seed that repeats every generation, and the unique-4-gram fraction of two StableLM checkpoints stays near its starting value of about 0.98 through generation 3. Keep both, and the replayed shared seed takes seven checkpoints from five families to between 0.045 and 0.38 by then. Three generations of replay write the favoured phrases into the weights: decoded with one seed per request, the generation-3 weights of the replayed StableLM-2-1.6B chain recover most of their diversity, yet the phrase that filled every sample under the shared seed still opens 46% of them. Without replay, five checkpoints drift slowly, consistent with the gradual accumulation that theory describes, and three turn incoherent though their diversity scores stay high. One peer-reviewed model-collapse pipeline that fine-tunes Gemma-2-27B samples identical prompts under one seeded configuration, and three quarters of the rows it released for one iteration repeat nearly as often as one such batch copies them. A seed per request restores the fresh sample that stability analyses assume.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model collapse, recursive training, synthetic data, self-consuming loops, sampling seeds, vLLM, reproducibility, lexical diversity, language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wnY7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2(−1)
Presentation
3
Soundness
2(−1)
Read review
Paper summary

This paper investigates the rapid loss of lexical diversity in language models during recursive self-training. While theoretical analyses attribute "model collapse" to the gradual accumulation of statistical errors, empirical studies often reveal severe phrase repetition occurring within just a few generations. The authors discover that this rapid degradation stems primarily from the characteristics of inference samplers (such as those used in vLLM), rather than from flaws within the models themselves. When requests within a batch share a fixed random seed—and this seed is reused across generations—the sampler repeatedly injects the same sampled noise. This creates a feedback loop, or "resonance," which continuously reinforces specific tokens and encodes them into the model's weights. Crucially, decoupling this noise by assigning an independent random seed to each request entirely eliminates this rapid collapse, restores lexical diversity, and brings empirical results into alignment with theoretical predictions.

Critical strengths
  1. This paper reveals an interesting and potentially overlooked interaction between sampling at inference time and recursive training. This finding underscores the importance of implementation details when interpreting experiments on model collapse.
  2. The factorial experimental design—which separates sharing across requests from replay across generations—is well-justified. Cross-decoding experiments also provide useful evidence for distinguishing between repetition induced by sampling and changes encoded within the model weights.
  3. The investigation into token-level sampling behavior, along with the reconstruction of seed-dependent sampling patterns, offers compelling mechanistic evidence for the observed phenomena.
Critical weaknesses
  1. The primary finding concerns implementation-related sampling biases within vLLM. While the interplay between stochasticity and recursive training is of empirical interest, the underlying mechanisms are largely consistent with well-established principles of correlated sampling and feedback amplification. The proposed solution restores standard statistical assumptions rather than introducing novel learning principles or algorithmic contributions.
  2. The paper assesses "model collapse" primarily through metrics such as lexical diversity, self-BLEU scores, and the perplexity of generated samples. While these metrics reflect output distributions, they do not directly demonstrate a degradation in the underlying model’s predictive capability or its performance on downstream tasks. Cross-decoding results further suggest that a significant portion of the observed collapse can be attributed to the sampling process during inference. However, in the absence of held-out language modeling evaluations or downstream task benchmarks, it remains unclear whether this phenomenon truly constitutes "model collapse."
  3. The experiments rely on relatively small synthetic datasets (2,100 examples per round), a single fine-tuning regime, and a setup where synthetic data entirely replaces—rather than supplements—real data. These choices may amplify the impact of repetitive or correlated samples. The paper does not investigate how this phenomenon scales with respect to dataset size, training intensity, or the ratio of real to synthetic data; thus, its relevance to modern, large-scale post-training pipelines remains uncertain.
  4. Most experimental conditions are evaluated based on a single training chain, with limited replication across different random seeds. This is particularly concerning, given that the proposed mechanism explicitly depends on the specific characteristics of individual stochastic realizations. Consequently, more extensive multi-seed experiments are required to characterize the variability, generality, and robustness of the reported "resonance" effects.
  5. The theoretical analysis is based on a phenomenological scalar update that is not derived from the actual neural network optimization process. It overlooks prompt dependency, autoregressive token interactions, and parameter sharing, rendering it incapable of predicting relative sensitivities across different checkpoints. As such, while the analysis offers an intuitive explanation, it provides limited predictive power or theoretical insight into the observed training dynamics.
Questions for authors
  1. Does the same phenomenon occur in other inference engines (e.g., Transformers or SGLang) when comparable sampling dependencies are introduced? How frequently does this type of configuration arise in practical synthetic data generation pipelines?
  2. How does this effect vary with the size of the synthetic dataset, the number of training epochs, and the proportion of real data included in the mix? Specifically, does the rapid performance decay persist when training on significantly larger synthetic datasets?
  3. Can deduplication or frequency-based reweighting mitigate the observed effect without altering the sampling configuration? This would help clarify the relative contributions of repeated training examples versus sampling feedback loops across generations.
  4. Does the reuse of seed data degrade language modeling performance or accuracy on downstream tasks? Can the authors distinguish between a decline in model capability and the repetition-related issues primarily driven by sampling during inference?
Reviewer_Has8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the rapid loss of lexical diversity in language models undergoing recursive training on their own generated data. The authors identify a previously overlooked cause: the interaction between shared sampling noise across requests and repeated random seeds across training generations in vLLM. They demonstrate that these two factors jointly create a resonance effect, progressively amplifying specific token preferences and leading to rapid output repetition. Through controlled experiments across multiple language models and four sampling strategies, the paper shows that assigning independent random seeds to individual requests effectively prevents this rapid diversity collapse. The findings highlight the critical role of sampling configurations in recursive training and provide a simple practical solution to mitigate sampling-induced model collapse.

Critical strengths
  • The paper identifies a previously overlooked factor contributing to rapid model collapse, revealing how shared and replayed sampling noise can amplify repetitive patterns during recursive training.
  • The findings highlight an important implementation issue in synthetic data generation and offer a simple, effective solution through independent per-request random seeds, potentially improving the reliability of recursive training experiments.
Critical weaknesses
  • The paper suffers from significant clarity and organization issues, making it unnecessarily difficult to follow. Although the authors conduct extensive experiments and provide detailed technical analyses, the central research problem and motivation are not clearly articulated. In particular, the abstract and introduction introduce numerous implementation details and experimental observations before adequately explaining why the investigated problem is important and how it relates to existing studies on model collapse. Furthermore, the experimental sections are densely packed with technical details, while the underlying research questions and logical connections between individual experiments are not sufficiently emphasized. As a result, readers must make considerable effort to reconstruct the overall research narrative and understand the significance of the findings. I strongly encourage the authors to substantially improve the presentation, particularly by clarifying the research motivation, highlighting the main contributions, and reorganizing the experimental analysis around clearly defined research questions.

  • The main finding primarily concerns a specific sampling behavior in vLLM rather than a fundamental limitation of recursive training. The proposed solution, assigning independent random seeds to individual requests, is straightforward and does not introduce a new algorithm or training strategy. It remains unclear whether the identified phenomenon generalizes to other widely used inference frameworks.

  • The analysis heavily relies on lexical diversity metrics, particularly the unique 4-gram fraction, which cannot adequately capture semantic degradation, knowledge loss, or deterioration in downstream task performance.

  • The experiments mainly involve relatively small language models (1B–4B parameters), a single synthetic data generation pipeline, and a fixed fine-tuning configuration. It is unclear whether the findings hold for larger, instruction-tuned models or more realistic synthetic data training scenarios involving diverse prompts, data filtering, and mixtures of real and synthetic data.

Questions for authors

Please refer to weaknesses.

Reviewer_cJMzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper identifies a form of rapid lexical diversity collapse in recursive self-training with vLLM, attributing it to sampler-induced resonance. It demonstrates that sharing random draws across requests and replaying them across generations jointly amplify favored tokens and phrases, causing the unique 4-gram fraction of seven checkpoints to fall to 0.045–0.38 within three generations. Removing either factor largely prevents this rapid collapse, although gradual drift and incoherent outputs can still occur.

Critical strengths

The paper offers a novel perspective by characterizing a specific form of model collapse in vLLM as resonance.

Critical weaknesses
  1. Insufficient ablation of seed configurations, model families, and model scales. Although the paper evaluates multiple checkpoints and seed-handling rules, the experiments do not systematically investigate how the observed resonance effect varies across a broader range of random seed configurations, model families, and model scales.
  2. Lack of a non-vLLM inference baseline. All self-training experiments rely on vLLM, without a direct comparison against alternative inference engines or sampling implementations. Consequently, it remains unclear whether the observed rapid diversity collapse is specific to vLLM's sampling behavior or whether similar effects arise under other inference systems.
Questions for authors

Please refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.