- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates what improves during self-evolving reinforcement learning, separating solver improvement from proposer utility. It evaluates six methods across six base models on seven mathematical reasoning benchmarks. The authors find that most pass@1 gains occur on questions whose initial plurality answer is already correct, while later proposers do not consistently provide more useful training tasks. Generated-task audits identify deteriorating question validity and label quality in some methods. A simplified single-update analysis explains how label sources determine answer reinforcement and how proposers reweight training questions.
Critical strengths
An interesting evaluation perspective. The paper distinguishes improved answer consistency from error correction and separately examines proposer utility. These diagnostics help assess claims of recursive self-improvement beyond aggregate accuracy.
Substantial empirical coverage. The study covers multiple methods and model families, tracks training checkpoints, and supplements rule-based grading with semantic regrading and a smaller reasoning-validity analysis.
Critical weaknesses
Limited novelty and insight. The main observations overlap substantially with prior analyses of sharpening and collapse in unsupervised RLVR [1, 2] and degradation in R-Zero. The broader evaluation is valuable, but the paper does not clearly establish a new mechanism or explain when proposer evolution improves learning.
Insufficient support for the broader conclusions. Plurality-based grouping and net probability gains do not directly establish limited capability acquisition or fewer corrected errors. The proposer results also require reconciliation with R-Zero’s trained-versus-untrained comparison. The simplified theory does not resolve these gaps or establish an early–later proposer ordering.
References
[1] Shumailov et al. AI models collapse when trained on recursively generated data. Nature, 2024.
[2] He et al. How Far Can Unsupervised RLVR Scale LLM Training? ICLR 2026.
Questions for authors
- For Finding 1 (Most solver gains come from questions the base model already answers correctly) and Finding 2 (Self-evolving RL corrects far fewer errors than human-annotated RL.). Since an initially incorrect plurality answer does not imply that the model cannot solve the question, could you report gains by initial correct-answer probability and separate question-level improvements from regressions? Could you clarify how the additional analyses support these interpretations, and what they establish beyond the recursive-data degradation studied in AI Models Collapse [1] and the sharpening mechanism analyzed in Unsupervised RLVR [2] ?
- For Finding 3 (Proposers do not teach more after further training.). Section 4.1 finds no benefit from updating the proposer, whereas R-Zero reports that a trained Challenger outperforms an untrained one [3, Table 1]. How do you reconcile these results? Does this discrepancy persist under matched training and evaluation settings, and how does it affect the scope of Finding 3?
- Theoretical explanation of the findings. What theoretical insight does Section 5 add beyond the sharpening analysis in [2]? Please clarify which findings follow from the analysis, particularly since Proposition 2 does not establish whether later proposers teach more.
References
[1] Shumailov et al. AI models collapse when trained on recursively generated data. Nature, 2024.
[2] He et al. How Far Can Unsupervised RLVR Scale LLM Training? ICLR 2026.
5 comments
Reply to Q2
We thank the reviewer for this question. It led us to a direct, matched test.
R-Zero's Table 1 (arXiv v1) gives "Iter 1 minus Base Challenger" = +1.52 (Qwen3-8B) and +3.70 (Qwen3-4B) on a seven-benchmark average. Each row is one training run; the paper reports no seeds or intervals, and it does not give the training budget of the Base Challenger row. We tested whether the gap persists under matched settings. Our full-loop and frozen-proposer runs use the same code, solver budget and data seed; only proposer training differs. We evaluated them with R-Zero's released evaluation code and with our protocol.
| Evaluation | Model | Runs | Round 1, trained minus untrained proposer (pp) |
|---|---|---|---|
| R-Zero Table 1 (v1) | 8B / 4B | 1 per row | +1.52 / +3.70 |
| R-Zero's released evaluation code, our runs | 8B | 3 paired seeds | +0.91, −0.60, +0.41 (mean +0.24) |
| same | 4B | 1 pair | −1.50 |
| Our protocol, 128 samples, 2,866 questions | 8B | 3 paired seeds | +0.06, −0.45, −0.28 (mean −0.22; t interval −0.87 to +0.43) |
| Our protocol, 128 samples, 1,847 questions | 4B | 1 pair | −0.08 |
- Under matched training and evaluation, the gap does not persist. With R-Zero's evaluation code, every paired 95% interval includes zero, and round 2 gives the same picture (8B mean +0.07; 4B −0.80). With 128 samples, the 8B interval excludes +1.52.
- The published 8B gap is within the noise of that metric. In the seven-benchmark average, AMC and the two AIME sets hold 100 questions but 3/7 of the weight. The question-sampling standard error of a difference of two such averages is about 2.5 points, so +1.52 is 0.6 standard errors (+3.70 for 4B is 1.6). Between arXiv v1 and v4, a new measurement of the AIME columns alone moved R-Zero's final 8B average by −0.97.
- R-Zero's own table agrees in part. The untrained Challenger already gives 64% (8B) and 32% (4B) of the first-iteration gain over the base model.
- Limits. Our proposer gets the update budget stated in R-Zero's paper (20 optimizer updates per round); the released script uses 192, and we did not test it. Our 4B pair has one seed. Our run of R-Zero's evaluation code omits the GPT-4o re-judging step, so we compare only within one harness.
Scope of Finding 3. Finding 3 rests mainly on the fixed-solver comparison of early and later proposers in five methods (Section 4.2). R-Zero's Table 1 does not test this. We also repeated that comparison at the last stage of the R-Zero run: the round-9 solver, trained on tasks from the round-1 proposer or from the round-10 proposer (which was trained against that solver), with three paired seeds. Later minus early is −0.66, −1.15 and −0.13 pp (mean −0.64), against a mean of −0.67 at round 5 in the paper.
Dear Authors,
Thank you for the detailed rebuttal and the additional experiments. Most of my concerns have been addressed, and I will raise my rating to 2.
Best regards,
Reviewer AszS
We sincerely thank the reviewer for the careful reading and the valuable questions. To address these concerns, we ran the additional analyses that were requested. All numbers use the paper's protocol (2,866 questions, 128 samples each).
What the paper adds. Here we summarize three of our main contributions that are not in previous papers.
Reply to Q1
We agree: a wrong plurality answer does not show that the model cannot solve the question. To better understand what actually happens, we regrouped all 42 settings by p0, the fraction of the 128 base-model responses that are correct. The table gives medians over the 36 self-evolving (SE) and the 6 human-annotated RL (HA) settings. "Up / down" counts the questions whose correct-response rate changes by at least 10 pp.
We hope that these results address the concern about the plurality-based grouping and the net gains.
Our comparison with [1], [2] and other related work:
[1] studies likelihood retraining without RL. Our result is of a different kind: pass@1 rises in all 42 settings, and we ask what the rise consists of.
[2] is the closest work. Its main experiments use intrinsic rewards on fixed question sets without a proposer, so it does not test results (2) and (3). For external rewards, [2] gives preliminary in-domain evidence (self-verification on Countdown) that they "may escape the confidence-correctness ceiling". Result (1) is complementary: external verification on code or games (AZR, SPIRAL) and document answers (SPICE) give no extra error correction on math.
We will add the p0 table, its figure and this comparison with prior work to the revised manuscript.