acceptodds
Under review as a conference paper at ICLR 2027

Separating Training-Data Source from Update Rate in Verified Self-Training

Abstract

Iterative self-training filters a model's own generated solutions by answer verification and fine-tunes on the survivors. Correct final answers, however, do not guarantee that repeated updates preserve performance, and lowering the learning rate to reduce forgetting also changes the data the model generates in later rounds, so retention differences cannot be attributed to the update or to the data alone. We separate these two factors for Llama-3.1-8B-Instruct over forty rounds on a fixed GSM8K stream. Generator and learner controls first establish that retention depends on the recursive update policy. A crossed design then freezes the generated training data from high-rate and low-rate runs and trains fresh receivers at each rate, with three seeds per cell. At the fixed low receiver rate, data generated under the low-rate policy improves the final-five-round near-domain accuracy change by 7.33 percentage points on average, positive in every seed. Yet every primary run still loses strictly held-out accuracy, and a reference trained on authored rationales also declines in every seed. Answer verification therefore leaves a data-source component of forgetting that update-rate tuning does not remove, and neither condition yields useful self-improvement within this protocol.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM, Fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f2dbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper works as a diagnostic study and explores the root cause of performance degradation of LoRA iterative self-training using answer verified data. The authors designed a 40 rounds LoRA iterative self-training with both high and low learning rate on GSM8K benchmark and found that the model self trained with high learning rate loses 5–8 points on a fixed test set while the model self trained with low learning rate almost lost nothing. To separate the update strength from the data the loop generates, the authors then designed replay training with high/low rate replayed training data on high/low rate model. They found that the low rate helps through the data it produces rather than through gentler updates. However, the authors didn't provide any reason or find out any mechanism under it.

Critical strengths

This paper provide an explorational result on what will cause the performance degradation on LoRA self training on verified answer. It designed detailed and clear experiment to separate the update strength from the data the loop generates. The The authors are honest by disclosing a mid-study change to the protocol and show it doesn’t affect the result, and report the failures.

Critical weaknesses
  1. This paper only work on one incremental experiment without any further explanation or exploration. The main contribution of this work is a single experiment that freeze the training data produced by high and low rate self-training runs and show that the data from the high rate run accounts for most of the forgetting. Its core observation that training on the outputs of a drifted model transmits the drift to a fresh model is already the central result of the model collapse literature [1] [2]. The paper offers no explanation for the effect and present with low novelty

  2. The abstract’s conclusion is inconsistent with the paper’s own results. The abstract concludes that “verification leaves a data-source component of forgetting that rate tuning does not remove,”. Following this logic, in round 1, both donors generate from the same base model with the same seed and problem order, so they train on the same (or near identical) examples and differs only in learning rate. If rate does not matter, the two models after round 1 are the same. The same logic holds by induction from round 1 to the final round 40. This makes the resulting Low and High banks should be identical after these 40 self training rounds. However, they are not in reality. The high bank data degrade more with 7 points. So “rate does not matter” cannot be literally true.

[1]Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv:2305.17493, 2023. [2] Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go MAD. arXiv:2307.01850, 2023.

Questions for authors
  1. An abstract should stand on its own, but this paper's abstract relies on terms and quantities that are only defined later in the paper. "Generator" and "receiver" can be inferred from context. But the term "bank" is not standard jargon and may confuse with other concepts. The reported numbers have the same problem: “final-five-round near-domain accuracy change,” “source contrast,” “fixed-bank receiver-rate contrasts,” and “round-zero spread” are all metrics whose definitions appear

  2. The authors didn't discuss the only successful training recipe in the paper in details. How was the only performance-improving bank D1 subsampled, were steps held fixed, and does D1’s advantage over LL survive the anchor-batch noise band?

  3. The abstract claims training data is the source of performance degradation instead of learning rate. If receiver leraning rate barely moves near accuracy at fixed data, what creates the divergence between the H- and L-banks? When and how the H and L bank begin to diverge.

Reviewer_zkSzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates performance degradation in large language models during iterative self-training with answer verification. Using Llama-3.1-8B-Instruct on GSM8K over 40 training rounds, the authors find that lowering the learning rate mitigates performance degradation but also changes the training data generated in subsequent rounds. To disentangle these effects, they freeze the data generated under high- and low-learning-rate self-training policies and conduct a crossed experiment using freshly initialized models trained at different learning rates. The results show that, at a fixed low receiver learning rate, data generated under the low-rate policy yields a 7.33-percentage-point higher change in final-five-round GSM8K test accuracy relative to the initial baseline than data generated under the high-rate policy. This effect is reproduced with a different data split and receiver model, but not on ARC-Challenge, and no meaningful self-improvement is observed. The findings suggest that training-data source is an important factor in performance retention during iterative self-training, although the underlying mechanisms remain unclear.

Critical strengths
  1. By freezing training data generated under different learning-rate policies and crossing data source with the receiver learning rate, the paper helps disentangle the effects of training-data source and parameter update rate. This provides reasonably convincing empirical evidence for understanding performance degradation during iterative self-training.
  2. The effect of training-data source on performance retention is reproduced across different random seeds, a new data split, and a different receiver model. Although cross-task generalization remains limited, these additional experiments partially strengthen the credibility of the main findings.
Critical weaknesses
  1. Although the paper identifies a substantial effect of training-data source on performance retention, it does not establish the specific factors responsible for the observed performance differences. This limits the explanatory power of the findings and their practical implications for improving self-training methods.

  2. The core experiments are primarily conducted on GSM8K with only three random seeds, and the same effect is not reproduced on ARC-Challenge. Additionally, some experiments exhibit substantial variability in round-zero evaluations, without sufficient uncertainty analysis. These limitations weaken the robustness and cross-task generalizability of the conclusions.

Questions for authors
  1. The crossed experiment simultaneously varies several aspects of the training data, including content, length, and training volume. Could the authors provide further analysis or controlled evidence to identify which factors may be primarily responsible for the observed phenomenon?

  2. The same effect was not reproduced on ARC-Challenge. How do the authors explain this discrepancy? Is there evidence supporting the generalizability of this phenomenon to other tasks?

  3. The core experiments use only three random seeds, and some experiments exhibit substantial variability in round-zero evaluations. Could the authors provide repeated evaluations under identical settings or a more comprehensive uncertainty analysis to strengthen the reliability of the main findings?

Reviewer_4AU4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why self-training can reduce performance even when generated answers are correct. The authors freeze data from runs with different learning rates, then train fresh models on these datasets at different rates. This separates the effects of data source and receiver learning rate. Experiments on GSM8K show that data from lower-rate runs better preserves performance across different data splits and receiver models. However, the main experiments do not demonstrate useful self-improvement, and the source effect is not reproduced on other data.

Critical strengths
  • The crossed design provides a useful way to separate data-source effects from the receiver’s learning rate.

  • Experiments with new data splits and a different receiver model strengthen the evidence for the GSM8K finding.

Critical weaknesses
  • The study provides limited insight into what makes answer-verified training data harmful or how such data could be identified.The crossed replay experiment provides an interesting diagnostic finding, but the paper does not establish which properties of the generated data account for the retention difference. Assigning an entire training bank leaves possible differences in reasoning correctness, solution style, question coverage, training exposure and so on unresolved. The statistics in Section 4.4 do not identify a mechanism, while the token and length controls in Section 4.5 primarily investigate the authored-rationale reference rather than the main contrast between the two generated banks.

  • All primary receivers lose performance on the held-out set, and the authored-rationale reference also declines. The paper does not establish a reference setting with useful held-out learning. It therefore remains unclear how much of the overall degradation reflects problems specific to recursively generated data and how much reflects sensitivity to the chosen SFT recipe. Evidence for generalization is also limited: the source effect is supported mainly by GSM8K with a single donor model and is not consistently reproduced on ARC-Challenge.

Questions for authors
  • Could the authors further examine which properties of the generated data may explain the observed source effect? eg correct final answers do not guarantee reliable intermediate reasoning. A focused analysis of the existing banks would help assess these possible explanations for the retention gap.

  • Authors may provide more training details, including the learning-rate schedule and warmup configuration. These choices may affect fine-tuning stability for instruct models.

  • Authors may clarify the conditions under which the source effect is expected to hold. Task characteristics and the model’s initial performance may influence the outcome. A brief analysis of the existing runs would help assess these factors and clarify the generalizability of the findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.