- Rating
- 4(+1)
- Confidence
- 4(+2)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes ATRI to address performance degeneration in LLM self-evolution. Its central diagnostic, learnable information gain, measures the difference in average negative log-likelihood between consecutive rounds of generated data, evaluated using a small proxy model fitted to the previous round. ATRI uses this signal to reweight Questioner and Solver updates and to stop training when the gain remains low. An optional extension selects external samples using an additional directional criterion. Experiments report improvements over the compared self-evolution methods on Qwen3-4B and Qwen3-8B, together with evidence that the diagnostic can anticipate performance peaks without using validation accuracy for the stopping decision.
Critical strengths
- Determining when self-generated training data stops being useful is an important operational problem. The paper provides a computable signal that supports both sample-level weighting and round-level stopping, rather than only describing degeneration retrospectively.
- The paper separates stopping from sample weighting and reports an additional 1.06-point Math AVG gain from weighting under the same stopping round. It also includes validation-based stopping and oracle checkpoint selection, making the practical value of validation-free stopping easier to assess.
- The main results average five training runs. Additional experiments examine proxy size, an alternative proxy family, length and formatting controls, and lifecycle behavior with another base-model family and a code-generation task.
Critical weaknesses
- measures a change in text predictability under a proxy, not the improvement the main model would obtain from training on that text. The length and formatting controls are useful, but they do not establish that high-gain examples are correct, relevant, or learnable by the main model. Conversely, familiar examples may still provide useful supervision. The paper therefore needs a more direct connection between proxy-based novelty and actual learning benefit.
- The paper motivates restarting a small proxy each round and fitting it to for one epoch. However, the reported capacity sweep does not isolate whether this procedure is necessary compared with a frozen pretrained proxy, a continually updated proxy, or scoring with the main model.
Questions for authors
What evidence shows that this coupled objective, together with information-gain weighting, improves reasoning correctness rather than merely making the Solver more consistent with potentially incorrect self-generated answers?
If the authors can provide clarification and improvements, I would be very happy to raise my score to clear acceptance.
8 comments
Response to Reviewer 3 [2/6]
[W1] (continued)
(b) Novelty is judged by the learnable information gain of each sample, as defined in Eqs. (8) and (12):
Here, denotes the loss of a small proxy model trained on the previous round's data, denotes the average of over the previous round's data, denotes the question–answer pairs generated in the previous round. Thus, measures how much harder a sample is to predict than that data on average. The average of over a round is denoted by , which represents the novelty of the generated samples in this round. ATRI multiplies the GRPO advantage by . Because is non-negative, novelty therefore only modulates how strongly a sample is learned, whereas usefulness still determines whether the model is pushed toward or away from the answer. Consequently, a novel but incorrect answer is penalized more strongly rather than being reinforced.
(c) Moreover, beyond the principled framework design, we conduct a brief controlled experiment to verify that generated samples with higher novelty also tend to be helpful, i.e., training on high- samples yields larger gains. Specifically, we select three equal-sized sets of 1,000 samples from a single ATRI round: the highest- samples, the lowest- samples, and randomly selected samples. Each set is used to train for one round starting from the same checkpoint. The table below reports the performance gain relative to the starting checkpoint. As shown, the highest- group provides the largest improvement at both sizes, while the lowest- group barely helps. These results indicate that the samples generated by ATRI are not only novel but also helpful. This experiment is added to Appendix K.2 (Table 19) of the revision.
| Group | Math AVG gain, Qwen3-4B | Math AVG gain, Qwen3-8B |
|---|---|---|
| Highest | +1.9 | +1.6 |
| Random | +1.0 | +0.9 |
| Lowest | +0.2 | +0.1 |
W1-2. The length and formatting controls are useful, but they do not establish that high-gain examples are correct, relevant, or learnable by the main model.
A1-2. We thank the reviewer for acknowledging our experiments on length and formatting control. To explain why high-gain examples generated by ATRI are expected to be correct, relevant, and learnable by the main model, we provide a rationale from two perspectives.
(a) From a model-design perspective, ATRI incorporates two strategies that help alleviate such risks.
-
First, prior to novelty-score computation, the loop filters out questions that the Solver either never solves or always solves. This filtering restricts the scored samples to the Solver’s current learning range: questions that exceed the Solver’s capacity and those it has already mastered are excluded, so samples with high are unlikely to merely reflect content that the main model cannot learn or has already learned.
-
Second, incorrect answers are handled by the GRPO reward rather than by the novelty weight. Specifically, ATRI multiplies the GRPO advantage by a nonnegative weight. As a result, novelty modulates only the strength of learning, while the sign of the advantage, which is determined by the reward, still decides whether the model is pushed toward or away from the answer. In this way, the filter determines which samples are eligible for novelty estimation, the reward determines the direction of learning, and the novelty weight controls only the magnitude of the update. This separation consequently helps prevent high from turning erroneous or unreachable samples into positive supervision.
Response to W1 continues in the next comment.
Response to Reviewer 3 [3/6]
[W1] (continued)
(b) From an experimental standpoint, we conduct a new experiment to examine whether samples with the highest contain a larger proportion of incorrect pseudo-labels than the other samples. We sort the training data before stopping by into five equal-sized groups. Group 1 contains the lowest values and Group 5 the highest. The experiment uses the same Qwen3-4B-Base ATRI run, and each question is sorted by its own (Eq. (8)). A stronger reference model, DeepSeek-V4-Pro, verifies each pseudo-label and flags unclear questions. The pseudo-label is the majority vote across the Solver's 8 answers, i.e., the label used by the reward. The Solver success rate is defined as the share of these 8 answers that match the pseudo-label. Three metrics are calculated:
-
Correct pseudo-labels (%): The proportion of questions in a group whose pseudo-label DeepSeek-V4-Pro judges correct.
-
Solver success rate (%): For each question, the share of the Solver's 8 answers that match the pseudo-label, averaged within the group. A lower rate indicates a harder group.
-
Unclear questions (%): The proportion of questions in a group that DeepSeek-V4-Pro flags as ambiguous or missing a condition.
As shown in the table, label accuracy and the share of unclear questions remain essentially flat across bins, while the Solver success rate decreases monotonically, indicating that high- samples are somewhat harder. Crucially, this increase in difficulty is not accompanied by a higher error rate, supporting the claim that samples with high are no more likely to contain incorrect pseudo-labels than other samples. This experiment is added to Appendix K.3 (Table 20) of the revision.
| Group by | 1 (lowest) | 2 | 3 | 4 | 5 (highest) |
|---|---|---|---|---|---|
| Correct pseudo-labels (%) | 72.1 | 71.6 | 72.8 | 71.9 | 72.4 |
| Solver success rate (%) | 55 | 53 | 51 | 49 | 47 |
| Unclear questions (%) | 3.1 | 2.8 | 3.3 | 3.0 | 3.4 |
W1-3. Conversely, familiar examples may still provide useful supervision.
A1-3. We thank the reviewer for raising the concern that familiar samples may still provide useful supervision. To address this, we conduct additional experiments that increase the weight assigned to familiar samples and examine their effect on final performance. Specifically, we first describe how familiar samples are identified, and then test whether assigning them a larger weight improves final results.
(a) How familiar samples are identified. We identify familiar samples based on the learnable information gain . In each round, ATRI trains a small proxy model on the data from the previous round. The proxy then measures how difficult each new sample is to predict using its loss on that sample. compares this loss with the proxy’s average loss on the previous round’s data. A positive indicates that the sample is harder to predict than the previous round’s data and thus brings new content. A sample with is as easy to predict as the previous round’s data, so its content has already appeared in that round; we refer to such samples as familiar samples. In default ATRI, the GRPO advantage of each sample is multiplied by . Familiar samples therefore receive zero weight and do not contribute to the update.
Response to W1 continues in the next comment.
Response to Reviewer 3 [1/6]
We sincerely thank the reviewer for the careful reading and the constructive comments. We thank the reviewer for recognizing the importance of the problem, the separate test of weighting, the comparison with validation and oracle stopping, and the robustness checks.
We have uploaded a revised paper, with all changes marked in blue. Table numbers below refer to the initial version unless stated otherwise.
Response: We thank the reviewer for this careful and constructive comment. The comment raises four interrelated concerns: (i) whether Ct reflects the learning progress of the main model or only that of the proxy model; (ii) whether high-gain examples are correct, relevant, and learnable by the main model; (iii) whether familiar examples may nonetheless provide useful supervision; and (iv) whether proxy-based novelty is directly connected to actual learning benefit. We address each of these points below.
W1-1. C_t measures a change in text predictability under a proxy, not the improvement the main model would obtain from training on that text.
A1-1. We thank the reviewer for this valuable suggestion. We would like to clarify that Ct measures the change in text predictability of the main model's own outputs under a proxy model, which we term novelty. Novelty quantifies how much new information is contained in newly generated samples. Usefulness, by contrast, refers to the improvement that the main model would obtain by training on that text. Usefulness quantifies how much of this information contributes to improving model capability. Therefore, Ct serves as a proxy for novelty rather than a direct estimate of the learning benefit for the main model. Nevertheless, ATRI explicitly targets both novelty and usefulness in different phases of the self-evolution loop, thereby helping ensure that the generated samples are both novel and useful.
(a) Usefulness is enforced through two complementary signals in (§3.1, §3.2): a reward/filter on the Questioner side and a training selection plus answer reward on the Solver side.
Specifically:
The Solver answers each question m times. The most frequent answer is treated as the pseudo-label, and p^ denotes the fraction of answers that agree with this pseudo-label.
The Questioner receives the reward 1−2∣p^−21∣, which is highest when the Solver’s answers are maximally split. Near-duplicate questions are penalized, and malformed questions receive zero reward.
The Solver is trained only on questions whose p^ is close to 1/2. For these questions, an answer receives reward 1 if it matches the pseudo-label and 0 otherwise.
Both rewards are then converted into GRPO advantages: samples with reward above the batch average receive positive advantage, and the remaining samples receive negative advantage.
Consequently, by optimizing these stringent objectives, ATRI encourages the generated samples to focus on challenging but solvable questions, which in turn contributes to improving model capability.
Response to W1 continues in the next comment.