- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how large language models can continue to improve using their own generated responses without reference answers or external verifiers. The authors argue that repeatedly rewarding majority answers can cause alternative solutions to disappear, and propose EVOL-RL: majority voting determines the priority of responses, while reasoning-trace novelty adjusts rewards within each group, supported by entropy regularization and asymmetric clipping to maintain exploration. Experiments across multiple models, training datasets, and evaluation tasks show improved multi-attempt success rates and generally higher single-response accuracy. Component ablations, training-dynamics analyses, and validation of differences between solutions further investigate the sources of these gains.
Critical strengths
-
The method performs well across a relatively broad range of settings. Across the two main model scales, three training datasets, and multiple evaluation benchmarks, EVOL-RL consistently improves pass@16 over TTRL and generally increases pass@1. Comparisons with other label-free methods and transfer results on tasks not used for training further support its effectiveness.
-
The paper provides detailed, targeted analyses that help explain the method's effectiveness. Beyond final performance, the authors examine the roles of individual components, changes in generation behavior during training, whether correct responses remain reachable when the majority answer is wrong, and whether the preserved correct responses use different solution strategies. These analyses make the role of diversity more concrete and provide valuable evidence for understanding the performance gains.
Critical weaknesses
-
The current theory does not sufficiently explain why diversity improves actual accuracy under potentially incorrect majority-vote supervision. The paper provides substantial empirical support, but its theoretical analysis assumes that response correctness is known and mainly establishes that encouraging differences can broaden coverage of correct solutions. In actual training, the majority answer may be wrong, and novelty may preserve both minority correct solutions and novel incorrect ones. The analysis therefore does not establish the conditions under which preserving these responses improves the supervision provided by subsequent majority votes. The experiments also do not track a fixed set of problems to show how the generation probabilities of preserved minority correct solutions change and whether they eventually become the correct majority. This limits our understanding of the proposed mechanism.
-
The performance comparisons do not sufficiently control for the actual computational budget. EVOL-RL generates substantially longer responses later in training, so matching the number of sampled responses and training steps does not imply equal computation. The appendix reports the fraction of its own training time spent computing novelty rewards, but does not provide a comparison of total training costs against the baselines or results under the same total generated-token budget. This makes it difficult to assess the method's advantage under equal computational resources and the extent to which longer generations contribute to the gains.
Questions for authors
-
When the majority answer is wrong, under what conditions does the novelty reward increase the probability of generating the correct answer, rather than merely preserve a wider range of responses? Can the current theory provide an analysis of this? Tracking a fixed set of problems with an incorrect majority and a correct minority would also help test the proposed explanation.
-
What are the total training-token counts and GPU times for EVOL-RL and TTRL? Do the main performance advantages persist when the total generated-token budget is matched?
-
Are the overall improvements in pass@1 and pass@k accompanied by reduced reliability on some problems that were previously solved consistently? Could the existing samples be used to report pass^k for small k—the probability that all k independent attempts on the same problem are correct—or changes in per-problem success rates, to distinguish broader problem coverage from consistent correctness?
4 comments
Follow-up on our response
Dear Reviewer n6K6,
Thank you again for your time and feedback. We wanted to gently follow up on our response and the revised manuscript. We would appreciate hearing whether the additional analyses address your concerns, and whether there are any remaining questions we could clarify before the discussion closes.
Thank you for considering our response.
Thank you for the honest and thoughtful response, and for taking these concerns so seriously. I really appreciate the care you put into this work. Your replies helped me understand the paper much better, and I believe the added analyses will also be useful to others working on similar problems. Still, I think some limitations in the method’s design remain, so I will keep my current rating.
Response to Reviewer n6K6's Follow-up Comment
Thank you very much for reading our response so carefully, and for your kind words. The fixed-problem tracking, the cost comparison, and the all-pass@k results all came from your suggestions, and they have made the paper more complete.
On the limitations in the method's design, we understand one of them to be fundamental: without labels, novelty cannot distinguish a correct minority answer from an incorrect one. We now state this explicitly in the Limitations of the revised paper. This limit is shared by methods that learn from their own outputs; our design targets what can be done within it, which is to avoid discarding the correct answer before voting has completed its selection, and the fixed-problem tracking tests this directly (§4.6).
If you have another specific design issue in mind, we would be glad to address it with further explanation or analysis during the discussion period.
Response to Reviewer n6K6
Thank you for the review. New analyses are in the revised paper (§4.6, App. B.14).
P1 (W1, Q1). When does preserving diversity improve accuracy if the vote can be wrong? Two conditions: the correct answer must be repeatable enough to win over repeated votes, and training must not lock in whichever answer happens to lead. Voting supplies the first; the variation terms (novelty, entropy regularization, asymmetric clipping) supply the second. Novelty alone does not raise the correct answer's probability: it shifts the advantage of correct and incorrect minorities almost equally (+0.08 vs. +0.07, §4.6). It separates responses the vote treats alike; without it, MATH pass@16 falls from 93.8 to 88.7 (MATH-500 training, Table 3).
What the theory covers. §3.4 assumes correctness is known: selection alone does not prevent collapse to one correct solution, and with a small enough weight (Theorem 1) variation prevents it without overriding selection. Noisy selection is not covered; we address it empirically, as you suggested.
P2 (W1, Q1). Fixed-problem tracking. We trained EVOL-RL and TTRL from the same base model on the 30 AIME24 problems for 60 steps, recording 64 responses at each of 16 observations per problem.
Under EVOL-RL the correct share rises on 11 of the 15 problems. The 3 that become majority-correct start with the correct answer at 5.2% and the leading wrong answer at 7.0%. Later supervision improves: maj@16 on the 30 training problems rises from 0.175 to 0.272 under EVOL-RL and does not improve under TTRL (0.185 to 0.166).
When a wrong answer is itself repeatable, step 2 applies to it too and it is reinforced, as in any method that learns from its own outputs.
P3 (W2, Q2). Cost and matched budget. Up to the step where both methods stop improving, EVOL-RL uses 1.5–2.7× the generated tokens and 2.0–2.9× the training time of TTRL. The advantage holds at a matched budget: truncated at the token total of TTRL's full run (B), EVOL-RL is ahead. AIME25 pass@1/pass@16, 5-checkpoint averages; "best" is TTRL's best window for each metric:
Across AIME25, AIME24 and MATH, EVOL-RL at B is higher than both TTRL's final and its best in 17 of 18 comparisons, so the gain is not bought with more generation. Both methods have the same generation limit; the token gap arises because TTRL's training responses shrink from about 1,500 to 650 tokens (the collapse itself) while EVOL-RL's lengthen. Over our full runs (far past the plateau, to test stability) the token ratio is 2.7–6.1×, with no further gain.
P4 (Q3). Reliability. Consistency improves relative to the base model, together with coverage. all-pass@k (your pass^k), 32 samples per problem, MATH-500-trained models, re-evaluated from released weights:
EVOL-RL is more consistent than the base model at every k, and its pass@16 also rises (Table 1). all-pass@k reflects both accuracy and how deterministic the output is: for a fully deterministic policy it does not decrease with k. TTRL is close to this (75.3 to 68.0 on MATH from k = 2 to 8), which is why its all-pass@k on MATH is higher while its pass@16 falls below the base model (Table 1); on held-out AIME25, EVOL-RL is higher at every k. EVOL-RL can also be made deterministic by greedy decoding at comparable accuracy (Table 18, MATH-TRAIN model).