acceptodds
Under review as a conference paper at ICLR 2027

When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain

Abstract

Self-evolution lets large language models (LLMs) improve iteratively using their own generated data, but often suffers from self-evolution degeneration: performance improves, plateaus, then declines. Existing methods address this issue at the component level, targeting either the Questioner or the Solver, and overlook that self-evolution is a tightly coupled system. We propose a holistic framework based on learnable information gain, which measures how much novel, parameterizable information a round provides relative to the previous round. Theoretically, this gain equals the Kullback–Leibler divergence between the two rounds' data distributions plus their entropy change. Practically, it is estimated by fitting a small language model to the previous round and scoring new data via negative log-likelihood. Based on this diagnostic, we propose ATRI (Adaptive Training Regulation via Information-gain), which reweights samples within a round and halts training across rounds when information gain remains low. Experiments on popular datasets demonstrate the superiority of our proposal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolution, Self-Evolution Degeneration, Learnable Information Gain, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fxkcOpenReview ↗
Rating
4(+1)
Confidence
4(+2)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ATRI to address performance degeneration in LLM self-evolution. Its central diagnostic, learnable information gain, measures the difference in average negative log-likelihood between consecutive rounds of generated data, evaluated using a small proxy model fitted to the previous round. ATRI uses this signal to reweight Questioner and Solver updates and to stop training when the gain remains low. An optional extension selects external samples using an additional directional criterion. Experiments report improvements over the compared self-evolution methods on Qwen3-4B and Qwen3-8B, together with evidence that the diagnostic can anticipate performance peaks without using validation accuracy for the stopping decision.

Critical strengths
  1. Determining when self-generated training data stops being useful is an important operational problem. The paper provides a computable signal that supports both sample-level weighting and round-level stopping, rather than only describing degeneration retrospectively.
  2. The paper separates stopping from sample weighting and reports an additional 1.06-point Math AVG gain from weighting under the same stopping round. It also includes validation-based stopping and oracle checkpoint selection, making the practical value of validation-free stopping easier to assess.
  3. The main results average five training runs. Additional experiments examine proxy size, an alternative proxy family, length and formatting controls, and lifecycle behavior with another base-model family and a code-generation task.
Critical weaknesses
  1. CtC_t measures a change in text predictability under a proxy, not the improvement the main model would obtain from training on that text. The length and formatting controls are useful, but they do not establish that high-gain examples are correct, relevant, or learnable by the main model. Conversely, familiar examples may still provide useful supervision. The paper therefore needs a more direct connection between proxy-based novelty and actual learning benefit.
  2. The paper motivates restarting a small proxy each round and fitting it to Dt−1D_{t-1} for one epoch. However, the reported capacity sweep does not isolate whether this procedure is necessary compared with a frozen pretrained proxy, a continually updated proxy, or scoring with the main model.
Questions for authors

What evidence shows that this coupled objective, together with information-gain weighting, improves reasoning correctness rather than merely making the Solver more consistent with potentially incorrect self-generated answers?

If the authors can provide clarification and improvements, I would be very happy to raise my score to clear acceptance.

8 comments

Response to Reviewer 3 [1/6]

We sincerely thank the reviewer for the careful reading and the constructive comments. We thank the reviewer for recognizing the importance of the problem, the separate test of weighting, the comparison with validation and oracle stopping, and the robustness checks.

We have uploaded a revised paper, with all changes marked in blue. Table numbers below refer to the initial version unless stated otherwise.

[W1] C_t measures a change in text predictability under a proxy, not the improvement the main model would obtain from training on that text. The length and formatting controls are useful, but they do not establish that high-gain examples are correct, relevant, or learnable by the main model. Conversely, familiar examples may still provide useful supervision. The paper therefore needs a more direct connection between proxy-based novelty and actual learning benefit.

Response: We thank the reviewer for this careful and constructive comment. The comment raises four interrelated concerns: (i) whether CtC_t reflects the learning progress of the main model or only that of the proxy model; (ii) whether high-gain examples are correct, relevant, and learnable by the main model; (iii) whether familiar examples may nonetheless provide useful supervision; and (iv) whether proxy-based novelty is directly connected to actual learning benefit. We address each of these points below.

W1-1. C_t measures a change in text predictability under a proxy, not the improvement the main model would obtain from training on that text.

A1-1. We thank the reviewer for this valuable suggestion. We would like to clarify that CtC_t measures the change in text predictability of the main model's own outputs under a proxy model, which we term novelty. Novelty quantifies how much new information is contained in newly generated samples. Usefulness, by contrast, refers to the improvement that the main model would obtain by training on that text. Usefulness quantifies how much of this information contributes to improving model capability. Therefore, CtC_t serves as a proxy for novelty rather than a direct estimate of the learning benefit for the main model. Nevertheless, ATRI explicitly targets both novelty and usefulness in different phases of the self-evolution loop, thereby helping ensure that the generated samples are both novel and useful.

(a) Usefulness is enforced through two complementary signals in (§3.1, §3.2): a reward/filter on the Questioner side and a training selection plus answer reward on the Solver side.

Specifically:

  • The Solver answers each question mm times. The most frequent answer is treated as the pseudo-label, and p^\hat p denotes the fraction of answers that agree with this pseudo-label.

  • The Questioner receives the reward 1−2∣p^−12∣1-2\lvert\hat p-\tfrac{1}{2}\rvert, which is highest when the Solver’s answers are maximally split. Near-duplicate questions are penalized, and malformed questions receive zero reward.

  • The Solver is trained only on questions whose p^\hat p is close to 1/21/2. For these questions, an answer receives reward 11 if it matches the pseudo-label and 00 otherwise.

  • Both rewards are then converted into GRPO advantages: samples with reward above the batch average receive positive advantage, and the remaining samples receive negative advantage.

Consequently, by optimizing these stringent objectives, ATRI encourages the generated samples to focus on challenging but solvable questions, which in turn contributes to improving model capability.

Response to W1 continues in the next comment.

Response to Reviewer 3 [2/6]

[W1] (continued)

(b) Novelty is judged by the learnable information gain of each sample, as defined in Eqs. (8) and (12):

ct=ℓ(x)−ℓˉ(Dt−1)c_t=\ell(x)-\bar\ell(\mathcal{D}_{t-1})

Here, ℓ\ell denotes the loss of a small proxy model trained on the previous round's data, ℓˉ\bar\ell denotes the average of ℓ\ell over the previous round's data, Dt−1\mathcal{D}_{t-1} denotes the question–answer pairs generated in the previous round. Thus, ctc_t measures how much harder a sample xx is to predict than that data on average. The average of ctc_t over a round is denoted by CtC_t, which represents the novelty of the generated samples in this round. ATRI multiplies the GRPO advantage by w=max⁡(ct,0)w=\max(c_t,0). Because ww is non-negative, novelty therefore only modulates how strongly a sample is learned, whereas usefulness still determines whether the model is pushed toward or away from the answer. Consequently, a novel but incorrect answer is penalized more strongly rather than being reinforced.

(c) Moreover, beyond the principled framework design, we conduct a brief controlled experiment to verify that generated samples with higher novelty also tend to be helpful, i.e., training on high-ctc_t samples yields larger gains. Specifically, we select three equal-sized sets of 1,000 samples from a single ATRI round: the highest-ctc_t samples, the lowest-ctc_t samples, and randomly selected samples. Each set is used to train for one round starting from the same checkpoint. The table below reports the performance gain relative to the starting checkpoint. As shown, the highest-ctc_t group provides the largest improvement at both sizes, while the lowest-ctc_t group barely helps. These results indicate that the samples generated by ATRI are not only novel but also helpful. This experiment is added to Appendix K.2 (Table 19) of the revision.

GroupMath AVG gain, Qwen3-4BMath AVG gain, Qwen3-8B
Highest ctc_t+1.9+1.6
Random+1.0+0.9
Lowest ctc_t+0.2+0.1

W1-2. The length and formatting controls are useful, but they do not establish that high-gain examples are correct, relevant, or learnable by the main model.

A1-2. We thank the reviewer for acknowledging our experiments on length and formatting control. To explain why high-gain examples generated by ATRI are expected to be correct, relevant, and learnable by the main model, we provide a rationale from two perspectives.

(a) From a model-design perspective, ATRI incorporates two strategies that help alleviate such risks.

  • First, prior to novelty-score computation, the loop filters out questions that the Solver either never solves or always solves. This filtering restricts the scored samples to the Solver’s current learning range: questions that exceed the Solver’s capacity and those it has already mastered are excluded, so samples with high ctc_{t} are unlikely to merely reflect content that the main model cannot learn or has already learned.

  • Second, incorrect answers are handled by the GRPO reward rather than by the novelty weight. Specifically, ATRI multiplies the GRPO advantage by a nonnegative weight. As a result, novelty modulates only the strength of learning, while the sign of the advantage, which is determined by the reward, still decides whether the model is pushed toward or away from the answer. In this way, the filter determines which samples are eligible for novelty estimation, the reward determines the direction of learning, and the novelty weight controls only the magnitude of the update. This separation consequently helps prevent high ctc_{t} from turning erroneous or unreachable samples into positive supervision.

Response to W1 continues in the next comment.

Response to Reviewer 3 [3/6]

[W1] (continued)

(b) From an experimental standpoint, we conduct a new experiment to examine whether samples with the highest ctc_t contain a larger proportion of incorrect pseudo-labels than the other samples. We sort the training data before stopping by ctc_t into five equal-sized groups. Group 1 contains the lowest values and Group 5 the highest. The experiment uses the same Qwen3-4B-Base ATRI run, and each question is sorted by its own ctQc_t^{Q} (Eq. (8)). A stronger reference model, DeepSeek-V4-Pro, verifies each pseudo-label and flags unclear questions. The pseudo-label is the majority vote across the Solver's 8 answers, i.e., the label used by the reward. The Solver success rate is defined as the share of these 8 answers that match the pseudo-label. Three metrics are calculated:

  • Correct pseudo-labels (%): The proportion of questions in a group whose pseudo-label DeepSeek-V4-Pro judges correct.

  • Solver success rate (%): For each question, the share of the Solver's 8 answers that match the pseudo-label, averaged within the group. A lower rate indicates a harder group.

  • Unclear questions (%): The proportion of questions in a group that DeepSeek-V4-Pro flags as ambiguous or missing a condition.

As shown in the table, label accuracy and the share of unclear questions remain essentially flat across ctc_t bins, while the Solver success rate decreases monotonically, indicating that high-ctc_t samples are somewhat harder. Crucially, this increase in difficulty is not accompanied by a higher error rate, supporting the claim that samples with high ctc_t are no more likely to contain incorrect pseudo-labels than other samples. This experiment is added to Appendix K.3 (Table 20) of the revision.

Group by ctc_t1 (lowest)2345 (highest)
Correct pseudo-labels (%)72.171.672.871.972.4
Solver success rate (%)5553514947
Unclear questions (%)3.12.83.33.03.4

W1-3. Conversely, familiar examples may still provide useful supervision.

A1-3. We thank the reviewer for raising the concern that familiar samples may still provide useful supervision. To address this, we conduct additional experiments that increase the weight assigned to familiar samples and examine their effect on final performance. Specifically, we first describe how familiar samples are identified, and then test whether assigning them a larger weight improves final results.

(a) How familiar samples are identified. We identify familiar samples based on the learnable information gain ctc_t. In each round, ATRI trains a small proxy model on the data from the previous round. The proxy then measures how difficult each new sample is to predict using its loss on that sample. ctc_t compares this loss with the proxy’s average loss on the previous round’s data. A positive ctc_t indicates that the sample is harder to predict than the previous round’s data and thus brings new content. A sample with ct≤0c_t \le 0 is as easy to predict as the previous round’s data, so its content has already appeared in that round; we refer to such samples as familiar samples. In default ATRI, the GRPO advantage of each sample is multiplied by w=max⁡(ct,0)w=\max(c_t,0). Familiar samples therefore receive zero weight and do not contribute to the update.

Response to W1 continues in the next comment.

Reviewer_2ExDOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies self-evolution degeneration in LLMs, where performance initially improves, then plateaus and eventually declines. The authors propose to measure changes in data across consecutive rounds using learnable information gain. They estimate this quantity using the difference in NLL between data from the current and previous rounds, evaluated by a small language model trained on the previous round. Based on this metric, they propose ATRI, which reweights training samples and stops training when information gain remains low. Experiments show that the metric can provide early warnings of performance degradation and that ATRI improves performance across multiple tasks.

Critical strengths
  • This paper addresses performance degradation in self-evolution and attempts to develop a diagnostic metric based on changes in data across rounds, which is a meaningful research direction.
  • ATRI uses proxy-based information gain estimation for within-round sample reweighting and cross-round stopping. The overall method is relatively simple and introduces limited computational overhead.
  • The authors investigate the metric's early-warning capability and the effects of different training control mechanisms through cross-method training trajectory analysis, experiments across multiple models and tasks, and component ablations.
Critical weaknesses
  • The motivation behind the paper's notion of learnable information gain has notable connections to existing work that warrant further clarification. For example, a position paper [1] discusses the role of learnable information gain in sustained self-play and the impact of self-synthetic pipelines on continued evolution. In addition, V-information [2] also theoretically investigates usable information under computational constraints and provides relevant background for understanding the information measure proposed in this paper. Although this paper further introduces a proxy-based estimation method and corresponding training control mechanisms, the connections to these works and the specific conceptual and methodological advances remain insufficiently discussed. A clearer account of these relationships would help establish the paper's distinct contributions.
  • The paper estimates learnable information gain using the NLL difference (Ct)(C_t) obtained from a small proxy LM. However, this metric primarily reflects the relative encoding cost of consecutive rounds of data under a particular proxy. Whether the target model can learn from new data depends on factors such as its capacity, training state, and the structure of the information contained in the data. Distributional changes and novelty relative to the proxy do not necessarily correspond to information that the target model can actually learn. The authors investigate the sensitivity of final task performance to proxy model size, but this does not directly validate the relationship between (Ct)(C_t) and the target model's actual learning gains. Therefore, although the existing experiments support the metric's predictive value in the evaluated settings, more direct evidence is needed to establish its validity as a measure of learnable information gain.
  • The paper proves that, under an exact fit, (Ct)(C_t) decomposes into the KL divergence between consecutive rounds' data distributions plus their entropy change. This identity provides an information-theoretic interpretation of the metric, but it does not establish that (Ct)(C_t) is equivalent to the target model's learnable information gain, nor does it guarantee a monotonic relationship between (Ct)(C_t) and the target model's learning gains. Since changes in entropy can offset part of the KL divergence, a small or even negative (Ct)(C_t) does not necessarily imply that the new data lacks valuable learning signals. Therefore, the paper's interpretation that information gain approaching zero signals the imminent failure of self-evolution still requires more explicit theoretical conditions or more direct empirical support.
  • ATRI combines sample reweighting with early stopping, and the experimental results show that this combination can improve training performance. However, these results do not by themselves establish that the proposed metric accurately captures learnable information gain, nor do they sufficiently establish the independent advantage of the resulting control strategy over simpler alternatives. In particular, early stopping may yield substantial gains simply by avoiding degradation during subsequent training. Although the paper uses ablation experiments to investigate the contribution of the stopping strategy, it lacks direct comparisons with conventional early stopping or validation-based checkpoint selection. Comparisons under consistent training budgets and evaluation conditions would help clarify the practical value of (Ct)(C_t) and the independent contribution of ATRI.

Reference

[1] Liu, W., Qi, S., Du, Y., & He, Y. Position: Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain. In the Forty-third International Conference on Machine Learning Position Paper Track.

[2] Xu, Y., Zhao, S., Song, J., Stewart, R., & Ermon, S. (2020). A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689.

Questions for authors
  • Is there a stable relationship between (Ct)(C_t) and the target model's actual learning gains? Does this relationship remain consistent across different model capacities, training stages, and tasks? Can the authors provide experiments that directly measure this relationship?
  • If the new round's data differs substantially from the previous round in its textual distribution, but the new content exceeds the target model's current learning capacity, would (Ct)(C_t) still indicate high information gain? Conversely, if the data changes little on the surface but contains reasoning structures that the target model has not yet mastered, can the metric identify their learning value?
  • What are the paper's main conceptual and methodological contributions relative to two reference papers? How do the authors characterise the relationship between their metric and these existing works?
  • What advantages does the proposed (Ct)(C_t) -based stopping strategy offer over conventional early stopping or checkpoint selection? Under the same training budget and evaluation conditions, can simpler training monitoring strategies that do not use (Ct)(C_t) achieve comparable gains?
13 comments

Response to Reviewer 2 [1/10]

We sincerely thank the reviewer for the careful reading and the constructive comments. We thank the reviewer for recognizing the meaningful research direction, the simplicity and low cost of ATRI, and the usefulness of our experiments and ablations.

We have uploaded a revised paper, with all changes marked in blue. Table numbers below refer to the initial version unless stated otherwise.

[W1] The motivation behind the paper's notion of learnable information gain has notable connections to existing work that warrant further clarification. For example, a position paper [1] discusses the role of learnable information gain in sustained self-play and the impact of self-synthetic pipelines on continued evolution. In addition, V-information [2] also theoretically investigates usable information under computational constraints and provides relevant background for understanding the information measure proposed in this paper. Although this paper further introduces a proxy-based estimation method and corresponding training control mechanisms, the connections to these works and the specific conceptual and methodological advances remain insufficiently discussed. A clearer account of these relationships would help establish the paper's distinct contributions.

Response: We thank the reviewer for pointing out these two related works. The concerns center on how ATRI relates to the position paper [1], how it relates to V-information [2], and what ATRI contributes beyond both. We try to address each concern below.

W1-1. For example, a position paper [1] discusses the role of learnable information gain in sustained self-play and the impact of self-synthetic pipelines on continued evolution.

A1-1. We agree that [1] is closely related to ATRI. Both works start from the same premise: sustained self-evolution requires that the loop continue to produce learnable information. ATRI operationalizes this premise as a cheap signal that is used to control training. To make the comparison precise, let CtC_t denote the learnable information gain of a round and ctc_t the learnable information gain of a single sample. The two works differ in three aspects, as summarized below.

Aspect[1]ATRI
What is measuredLearnable information in one dataset on its own, using epiplexity [Finzi et al., 2026]How much the current round adds over the last one (CtC_t), plus a score for each sample (ctc_t)
How it is estimatedQwen2.5 observers from 0.5B to 14B, fine-tuned for up to 20 epochs, with the best epoch picked by MDLPythia-160M fine-tuned for one epoch, about 0.35 GPU hours per round
How it is usedDiagnosis across roles, model sizes, and data sources, over nine self-play iterations on a coding taskWeighting samples, deciding when to stop, and selecting external data

The estimation choices differ because CtC_t is used differently in the two settings. ATRI computes CtC_t in every round and scores every sample, so the estimator must be computationally inexpensive. A small proxy trained for one epoch is therefore sufficient for this purpose: ATRI only requires the downward trend of CtC_t and the within-round ranking of samples, not a calibrated estimate of the absolute amount of information. Training for one epoch also mitigates proxy memorization of the previous round (Appendix D). This difference in use also clarifies the connection between the two works. [1] suggests introducing external information to sustain self-play, and ATRI provides a concrete rule for selecting such information (Appendix J).

Response to W1 continues in the next comment.

Response to Reviewer 2 [2/10]

[W1] (continued)

W1-2. In addition, V-information [2] also theoretically investigates usable information under computational constraints and provides relevant background for understanding the information measure proposed in this paper.

A1-2. We agree that [2] provides useful theoretical grounding for ATRI. We relate the two works along three dimensions.

  • Shared premise. [2] asks how much information a model with limited compute can actually use. In ATRI, the small proxy instantiates exactly such a compute-limited model.

  • Distinct question. [2] asks how much one variable reveals about another. In contrast, CtC_t asks how much of the current round remains learnable relative to a small proxy trained only on the preceding round. The two quantities are related but not identical: the former is a general information-theoretic measure, whereas the latter is a round-level signal tailored to the self-evolution loop.

  • Complementary implications. [2] shows that computation can make information usable, but it cannot create new information. Appendix B establishes the analogous limit for the closed loop: self-evolution can convert existing information into a usable form, but it cannot generate new information by itself. Together, these results explain the non-monotonic trend in accuracy. In early rounds, the loop converts information it already possesses into a usable form, so CtC_t remains positive. In later rounds, this reservoir is exhausted, no new information enters the loop, and CtC_t drops toward zero, after which accuracy eventually declines.

W1-3. Although this paper further introduces a proxy-based estimation method and corresponding training control mechanisms, the connections to these works and the specific conceptual and methodological advances remain insufficiently discussed. A clearer account of these relationships would help establish the paper's distinct contributions.

A1-3. We agree that the contributions of ATRI should be stated more clearly. The main advance lies in how the signal is used once it has been measured. Both [1] and [2] primarily study the measure itself, whereas ATRI tests the signal as a rule that controls training. This yields one conceptual and one methodological contribution.

(a) Conceptual contribution. ATRI measures the change between two consecutive rounds rather than the total learnable information contained in a single dataset. The total amount does not indicate whether the loop continues to produce new learnable content, whereas the round-to-round change does.

(b) Methodological contribution. ATRI estimates this round-to-round change using a 160M proxy trained for one epoch. This proxy is cheap enough to be executed in every round and to score every sample, which enables both sample weighting and stopping. We validate the resulting rule in three ways:

  • The alarm round is defined as the first round in which CtC_t falls below the threshold. It precedes the accuracy peak by one to four rounds across six self-evolution methods (§4.3).

  • Stopping based on CtC_t requires no labels and remains within 0.83 points of the best checkpoint selected with hindsight (Table 7; Table 10 in the revision).

  • The rule for selecting external data is compared with random, difficulty-based, and R-Diverse selection under the same budget (§4.4).

The revision positions ATRI relative to [1] and [2] in the introduction and connects CtC_t to V-information in Section 3.1.2. Appendix A adds a paragraph on the three differences from [1] and the three links to [2], including epiplexity. Section 4.7 now also reports the comparison with label-free stopping signals.

Response to Reviewer 2 [3/10]

[W2] The paper estimates learnable information gain using the NLL difference C_t obtained from a small proxy LM. However, this metric primarily reflects the relative encoding cost of consecutive rounds of data under a particular proxy. Whether the target model can learn from new data depends on factors such as its capacity, training state, and the structure of the information contained in the data. Distributional changes and novelty relative to the proxy do not necessarily correspond to information that the target model can actually learn. The authors investigate the sensitivity of final task performance to proxy model size, but this does not directly validate the relationship between C_t and the target model's actual learning gains. Therefore, although the existing experiments support the metric's predictive value in the evaluated settings, more direct evidence is needed to establish its validity as a measure of learnable information gain.

Response: We thank the reviewer for this insightful comment. The concerns center on whose behavior CtC_t reflects, how the target model's capacity and training state enter CtC_t, and whether CtC_t is directly linked to the target model's learning gains. We try to address each concern below.

W2-1. The paper estimates learnable information gain using the NLL difference C_t obtained from a small proxy LM. However, this metric primarily reflects the relative encoding cost of consecutive rounds of data under a particular proxy.

A2-1. We agree that CtC_t is computed with a proxy scorer. This does not imply that CtC_t reflects the proxy rather than the target model. The data scored by CtC_t are generated by the target model: in each round, the Solver produces responses after learning the data from the preceding round. Accordingly, CtC_t quantifies whether the target model, after learning the preceding round, produces content beyond what it has just learned. The proxy is used only as a shared scorer across the two rounds, so it provides a common measurement scale rather than the object of measurement. Under this shared scorer, a positive CtC_t indicates that the target model generates new content relative to the preceding round, whereas a CtC_t near zero indicates that the target model largely repeats what it has just learned. Thus, CtC_t should be interpreted as a measure of the target model's behavioral change across rounds computed with a proxy, not as a property of the proxy itself.

W2-2. Whether the target model can learn from new data depends on factors such as its capacity, training state, and the structure of the information contained in the data. Distributional changes and novelty relative to the proxy do not necessarily correspond to information that the target model can actually learn.

A2-2. We thank the reviewer for this comment. We agree that capacity and training state are relevant to learning. ATRI incorporates both considerations into CtC_t through two design choices: filtering questions before scoring, and scoring the question and the solution separately. These choices are examined in two cases. In the first case, the new content exceeds the model's current capacity. In the second case, the data exhibit only minor surface-level changes but contain reasoning that the model has not yet mastered. Together, these cases test whether CtC_t remains sensitive to capacity constraints and to latent reasoning demands, rather than only to surface-level changes in the data.

Response to W2 continues in the next comment.

Reviewer_XAaTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ATRI (Adaptive Training Regulation via Information-gain) to mitigate the common "rise–plateau–degradation" pattern in the performance of large language models during self-evolution training. The method fits a small proxy language model to the data generated in the previous round, and then estimates "learnable information gain" using the negative log-likelihood difference of the current samples relative to the previous round's data. ATRI applies this signal at two levels: increasing the weight of novel samples within each training round, and early-stopping self-evolution when the information gain stays below a threshold for multiple consecutive rounds. The authors evaluate the method on Qwen3-4B and Qwen3-8B using seven mathematical reasoning benchmarks and three general reasoning benchmarks, and report that ATRI's average performance outperforms several self-evolution baselines; in addition, the information-gain signal fires before the accuracy peak in several settings.

Critical strengths
  • It starts from the entire self-evolution pipeline rather than a single component. ATRI regulates the training of both the Questioner and the Solver, and adds a cross-round stopping mechanism, targeting the interplay between data generation, filtering, and learning in self-evolution systems.
  • It attempts to establish a quantifiable cross-round diagnostic signal. The method uses the proxy model's NLL to compare the data generated in the current and previous rounds, and provides a theoretical connection to KL divergence and entropy change. This offers a unified metric for observing changes in the training data distribution.
  • It combines intra-training weighting with inter-training stopping. ATRI does not only stop after performance degradation, but attempts to detect potential saturation via information gain, and allocates more updates to novel samples within a single round; the design logic is fairly complete.
  • The experiments cover a variety of reasoning tasks and self-evolution baselines. The paper tests multiple mathematical and general reasoning benchmarks on Qwen3-4B and Qwen3-8B, and reports averaged results over multiple runs. The authors also analyze the relationship between accuracy peaks, declines in information gain, and changes in answer diversity.
Critical weaknesses
  • Informational novelty does not necessarily equal useful information. Current samples that are hard to predict under the previous round's proxy model may stem from novel and valuable reasoning, but may also be just wording variation, noise, or erroneous content. Using high NLL directly as the basis for training weighting may not stably improve task capability.
  • There is a gap between the theoretical explanation and the actual estimation. The decomposition in terms of KL divergence and entropy change relies on an exactly fitted data distribution; in practice, the method uses a Pythia-160M proxy model, a limited number of training rounds, and NLL approximation. The information-gain threshold also depends on the initial value setting, and its stability across models, data scales, and training settings is not sufficiently clear.
  • The sources of the performance gains are not yet fully disentangled. ATRI introduces both sample reweighting and early stopping at the same time. If overall performance improves, it still needs to be determined how much of the gain comes from each part, and whether the two mechanisms are complementary or counteract each other.
  • The experimental scope remains limited. The main results are concentrated on Qwen3-4B/8B, and the life-cycle analysis is mainly based on MATH training with Qwen3-4B. Although the authors mention supplementary experiments on other tasks, the current evidence is still insufficient to confirm that the signal applies to different architectures, training paradigms, and a broader range of tasks.
Questions for authors
  • Regarding novelty and task value: Have the authors verified that samples with higher information gain are indeed more likely to improve the Solver's or Questioner's task performance? Can the relationship between NLL-based novelty and metrics such as correctness, difficulty, and noise rate be compared, and can the authors evaluate whether quality or correctness constraints should be added?
  • Regarding estimation reliability and the threshold: When the proxy model capacity, number of training rounds, or training data scale changes, are the information-gain rankings and the stopping round stable? Is setting the threshold as a certain proportion of the initial information gain sensitive to hyperparameters or random seeds?
  • Regarding component contributions: Can the authors separately evaluate "sample reweighting only," "cross-round stopping only," and the full ATRI, and compare them under the same number of training rounds or computational budget? This would help clarify where the performance gains come from, and also test whether the two mechanisms must be used together.
  • Regarding method generalization: Can ATRI be validated on more model families, scales, and different types of self-evolution tasks? In particular, when the system introduces external data or the task distribution changes actively, does a low information gain still imply that training should be stopped?
14 comments

Response to Reviewer 1 [1/10]

We sincerely thank the reviewer for the careful reading and the constructive comments. We thank the reviewer for recognizing ATRI’s holistic self-evolution design, its quantifiable cross-round diagnostic signal, its combined intra-training weighting and inter-training stopping mechanism, and the breadth of our reasoning experiments and baselines.

We have uploaded a revised paper, with all changes marked in blue. Table numbers below refer to the initial version unless stated otherwise.

[W1] Informational novelty does not necessarily equal useful information. Current samples that are hard to predict under the previous round's proxy model may stem from novel and valuable reasoning, but may also be just wording variation, noise, or erroneous content. Using high NLL directly as the basis for training weighting may not stably improve task capability.

Response: We thank the reviewer for this comment. The concerns center on the relationship between novelty and usefulness, the effect of trivial modifications on generated samples, and the potential risks of directly using high NLL. We try to address each concern below.

W1-1. Informational novelty does not necessarily equal useful information.

A1-1. We agree that novelty and usefulness are two important and complementary metrics. Novelty measures how much new information is contained in newly generated samples, whereas usefulness measures how much of that information contributes to improving model capability. ATRI explicitly targets these two metrics in different phases of the self-evolution loop, thereby helping ensure that the generated samples are both novel and useful.

(a) Usefulness is enforced through two complementary signals in (§3.1, §3.2): a reward/filter on the Questioner side and a training selection plus answer reward on the Solver side.

Specifically:

  • The Solver answers each question mm times. The most frequent answer is treated as the pseudo-label, and p^\hat p denotes the fraction of answers that agree with this pseudo-label.

  • The Questioner receives the reward 1−2∣p^−12∣1-2\lvert\hat p-\tfrac{1}{2}\rvert, which is highest when the Solver’s answers are maximally split. Near-duplicate questions are penalized, and malformed questions receive zero reward.

  • The Solver is trained only on questions whose p^\hat p is close to 1/21/2. For these questions, an answer receives reward 11 if it matches the pseudo-label and 00 otherwise.

  • Both rewards are then converted into GRPO advantages: samples with reward above the batch average receive positive advantage, and the remaining samples receive negative advantage.

Consequently, by optimizing these stringent objectives, ATRI encourages the generated samples to focus on challenging but solvable questions, which in turn contributes to improving model capability.

Response to W1 continues in the next comment.

Response to Reviewer 1 [2/10]

[W1] (continued)

(b) Novelty is judged by the learnable information gain of each sample, as defined in Eqs. (8) and (12):

ct=ℓ(x)−ℓˉ(Dt−1)c_t=\ell(x)-\bar\ell(\mathcal{D}_{t-1})

Here, ℓ\ell denotes the loss of a small proxy model trained on the previous round's data, ℓˉ\bar\ell denotes the average of ℓ\ell over the previous round's data, Dt−1\mathcal{D}_{t-1} denotes the question–answer pairs generated in the previous round. Thus, ctc_t measures how much harder a sample xx is to predict than that data on average. The average of ctc_t over a round is denoted by CtC_t, which represents the novelty of the generated samples in this round. ATRI multiplies the GRPO advantage by w=max⁡(ct,0)w=\max(c_t,0). Because ww is non-negative, novelty therefore only modulates how strongly a sample is learned, whereas usefulness still determines whether the model is pushed toward or away from the answer. Consequently, a novel but incorrect answer is penalized more strongly rather than being reinforced.

(c) Moreover, beyond the principled framework design, we conduct a brief controlled experiment to verify that generated samples with higher novelty also tend to be helpful, i.e., training on high-ctc_t samples yields larger gains. Specifically, we select three equal-sized sets of 1,000 samples from a single ATRI round: the highest-ctc_t samples, the lowest-ctc_t samples, and randomly selected samples. Each set is used to train for one round starting from the same checkpoint. The table below reports the performance gain relative to the starting checkpoint. As shown, the highest-ctc_t group provides the largest improvement at both sizes, while the lowest-ctc_t group barely helps. These results indicate that the samples generated by ATRI are not only novel but also helpful. This experiment is added to Appendix K.2 (Table 19) of the revision.

GroupMath AVG gain, Qwen3-4BMath AVG gain, Qwen3-8B
Highest ctc_t+1.9+1.6
Random+1.0+0.9
Lowest ctc_t+0.2+0.1

W1-2. Current samples that are hard to predict under the previous round's proxy model may stem from novel and valuable reasoning, but may also be just wording variation, noise, or erroneous content. Using high NLL directly as the basis for training weighting may not stably improve task capability.

A1-2. We agree that wording variation, noise, or erroneous content may also make samples harder to predict under the previous round's proxy model, thereby increasing ctc_t. However, we respectfully note that such cases have limited influence in practice due to both design and practical considerations.

(a) From the perspective of model design, novelty depends on what the proxy model actually learns in each round. The proxy learns patterns that repeat across the previous round's data rather than individual sentences (§3.1.2). A reworded solution preserves the same formulas, numbers, and reasoning steps, so its score should barely change. As clarified in A1-1, incorrect answers are handled by the reward: a wrong answer may still be novel, but its advantage is negative, and a large weight then only pushes it down more strongly. Therefore, rewording and noise can hardly inflate ctc_t, and thus have limited influence on the novelty estimate.

Response to W1 continues in the next comment.

Response to Reviewer 1 [3/10]

[W1] (continued)

(b) From the perspective of practical experiments, we already conducted experiments to investigate the influence of length and format, as reported in Appendix H, Table 9 of the initial version (Table 15 in the revision). The experiment uses the full ATRI run of Qwen3-4B-Base on mathematical reasoning. The proxy is Pythia-160M, trained on the previous round's data as in the main experiments. Two constraints are applied to the data before CtC_t is computed to ensure that the length or format of the generated samples is aligned with the previous rounds. Length matching aligns the length distributions of two consecutive rounds, while format standardization unifies templates, spacing, and equivalent LaTeX forms. Every sample in both rounds is rewritten accordingly before the proxy is trained and used to score it. Results under these constraints are compared with the vanilla ATRI model. If the CtC_t curve of the ATRI model is close to that of these rigidly constrained variants, this verifies that the influence of length and format is minimal. As shown in Appendix H, Table 9 of the initial version, we design several metrics to quantify the influence of length and format constraints:

  • Correlation with the original CtC_t curve. The Spearman correlation between the CtC_t values of all rounds under a specific constraint and those under the vanilla setting. It measures how closely the constrained CtC_t curve follows the original one; a value of 11 indicates that the two curves exhibit the same trend.

  • Alarm shift. The number of rounds by which the alarm round shifts under the constraint, where the alarm round is the round at which CtC_t first falls below the stopping threshold. For constraints with repetitions, the most common alarm round across the repetitions is used.

  • Stop shift. The number of rounds by which the stop round shifts under the constraint. The most common stop round across the repetitions is used in the same way.

  • Same stop round. The fraction of repetitions that stop at exactly the same round as in the original setting.

These metrics collectively quantify whether the length and format constraints materially alter the novelty dynamics and the stopping behavior of ATRI. As shown by the results, the behavior of ATRI is rarely influenced by the rigid length and format constraints, demonstrating that such superficial changes have limited impact on novelty discovery.

SettingCorrelation with the original CtC_t curveAlarm shiftStop shiftSame stop round
Original1.00000100%
Length matched0.982 ± 0.0070095.6%
Format standardized0.96900100%
Both controls0.958 ± 0.0100093.1%

Second, rewording alone barely raises the novelty score. Rewording changes the wording of a solution while preserving its original semantic meaning. Owing to the strong semantic understanding capability of the LLM and the GRPO optimization, such trivial surface-level changes have minimal effect on the novelty estimation. We further conduct a new experiment to verify this phenomenon. The test uses the same Qwen3-4B-Base ATRI run on mathematical reasoning, and the proxy is trained on the last round's data. Given each generated question, we generate two types of solutions:

  • Vanilla solution: the original solution generated by the Solver.

  • Reworded solution: a reworded version of the vanilla solution with the same semantic meaning, modified by DeepSeek-V4-Pro.

Each type of solution is scored by ctSc_t^{S} (Eq. (12)), and the table averages this score over all questions. We use the average difference of ctSc_t^{S} compared to the vanilla solution as the metric. As shown, the reworded solution has limited influence on the calculation of the novelty score compared to the vanilla solution, demonstrating that such surface-level modification does not affect model performance. A fuller version with two more solution types is added to Appendix K.4 (Table 22) of the revision.

Version of the solutionAverage difference of ctSc_t^S compared to vanilla solution
Vanilla0.00
Reworded solution0.03

Response to W1 continues in the next comment.

Comments on the paper

2 comments

Dear Reviewers, please take a moment to read the authors' rebuttal and discuss any unresolved concerns with them before the discussion period concludes on Nov 18 AoE. Thanks!

Dear reviewers, a friendly reminder that the deadline for author–reviewer discussion is November 18 (AoE).

XAaT, you asked whether high-NLL samples provide useful training signal rather than reflecting rewording, noise, or errors. You also raised the gap between the exact-fit theory and the proxy estimate, the stability of the stopping threshold across settings, and how sample reweighting and early stopping contribute separately and interact. Finally, you questioned whether the signal generalizes across model families, training paradigms, and tasks.

Do the authors’ responses adequately address these concerns? If any remain, please let us know before the discussion deadline.

2ExD, thank you for following up. Please feel free to share any additional comments before the deadline.

fxkc, thank you for following up. Please feel free to share any additional comments before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.