acceptodds
Under review as a conference paper at ICLR 2027

On the Evolution of Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

Abstract

Large language models (LLMs) can improve through reinforcement learning with verifiable rewards (RLVR), but many settings lack reference answers or external verifiers. Label-free self-improvement instead constructs training signals from the model's own outputs. Because the same policy generates both the candidate responses and the signals derived from them, each update also changes the candidates available for future training, creating a self-reinforcing feedback loop. We identify a failure mode of this loop, which we call selection-induced diversity collapse: repeated self-selection can progressively narrow the candidate pool and make alternative correct solutions harder to sample. To address this problem, we propose Evol-RL, which balances selection and variation. Majority voting provides a stable selection signal, while embedding-based novelty refines credit among responses with the same selection status; entropy regularization and asymmetric clipping further support this process by maintaining diverse rollouts and preserving rare, high-advantage updates. Across multiple training distributions, model scales, backbones, and matched label-free baselines, Evol-RL consistently improves pass@n while generally maintaining or improving pass@1. For example, when trained without labels on AIME24, Qwen3-4B-Base improves over TTRL on held-out AIME25 from 4.6% to 17.1% in pass@1 and from 18.5% to 42.0% in pass@16. Training analyses further show that when the current majority is wrong, Evol-RL better preserves alternative correct solutions. Code is available at https://anonymous.4open.science/r/EVOL-RL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, RL, Math Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_n6K6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how large language models can continue to improve using their own generated responses without reference answers or external verifiers. The authors argue that repeatedly rewarding majority answers can cause alternative solutions to disappear, and propose EVOL-RL: majority voting determines the priority of responses, while reasoning-trace novelty adjusts rewards within each group, supported by entropy regularization and asymmetric clipping to maintain exploration. Experiments across multiple models, training datasets, and evaluation tasks show improved multi-attempt success rates and generally higher single-response accuracy. Component ablations, training-dynamics analyses, and validation of differences between solutions further investigate the sources of these gains.

Critical strengths
  1. The method performs well across a relatively broad range of settings. Across the two main model scales, three training datasets, and multiple evaluation benchmarks, EVOL-RL consistently improves pass@16 over TTRL and generally increases pass@1. Comparisons with other label-free methods and transfer results on tasks not used for training further support its effectiveness.

  2. The paper provides detailed, targeted analyses that help explain the method's effectiveness. Beyond final performance, the authors examine the roles of individual components, changes in generation behavior during training, whether correct responses remain reachable when the majority answer is wrong, and whether the preserved correct responses use different solution strategies. These analyses make the role of diversity more concrete and provide valuable evidence for understanding the performance gains.

Critical weaknesses
  1. The current theory does not sufficiently explain why diversity improves actual accuracy under potentially incorrect majority-vote supervision. The paper provides substantial empirical support, but its theoretical analysis assumes that response correctness is known and mainly establishes that encouraging differences can broaden coverage of correct solutions. In actual training, the majority answer may be wrong, and novelty may preserve both minority correct solutions and novel incorrect ones. The analysis therefore does not establish the conditions under which preserving these responses improves the supervision provided by subsequent majority votes. The experiments also do not track a fixed set of problems to show how the generation probabilities of preserved minority correct solutions change and whether they eventually become the correct majority. This limits our understanding of the proposed mechanism.

  2. The performance comparisons do not sufficiently control for the actual computational budget. EVOL-RL generates substantially longer responses later in training, so matching the number of sampled responses and training steps does not imply equal computation. The appendix reports the fraction of its own training time spent computing novelty rewards, but does not provide a comparison of total training costs against the baselines or results under the same total generated-token budget. This makes it difficult to assess the method's advantage under equal computational resources and the extent to which longer generations contribute to the gains.

Questions for authors
  1. When the majority answer is wrong, under what conditions does the novelty reward increase the probability of generating the correct answer, rather than merely preserve a wider range of responses? Can the current theory provide an analysis of this? Tracking a fixed set of problems with an incorrect majority and a correct minority would also help test the proposed explanation.

  2. What are the total training-token counts and GPU times for EVOL-RL and TTRL? Do the main performance advantages persist when the total generated-token budget is matched?

  3. Are the overall improvements in pass@1 and pass@k accompanied by reduced reliability on some problems that were previously solved consistently? Could the existing samples be used to report pass^k for small k—the probability that all k independent attempts on the same problem are correct—or changes in per-problem success rates, to distinguish broader problem coverage from consistent correctness?

4 comments
AuthorsOpenReview ↗

Response to Reviewer n6K6

Thank you for the review. New analyses are in the revised paper (§4.6, App. B.14).

P1 (W1, Q1). When does preserving diversity improve accuracy if the vote can be wrong? Two conditions: the correct answer must be repeatable enough to win over repeated votes, and training must not lock in whichever answer happens to lead. Voting supplies the first; the variation terms (novelty, entropy regularization, asymmetric clipping) supply the second. Novelty alone does not raise the correct answer's probability: it shifts the advantage of correct and incorrect minorities almost equally (+0.08 vs. +0.07, §4.6). It separates responses the vote treats alike; without it, MATH pass@16 falls from 93.8 to 88.7 (MATH-500 training, Table 3).

What the theory covers. §3.4 assumes correctness is known: selection alone does not prevent collapse to one correct solution, and with a small enough weight (Theorem 1) variation prevents it without overriding selection. Noisy selection is not covered; we address it empirically, as you suggested.

P2 (W1, Q1). Fixed-problem tracking. We trained EVOL-RL and TTRL from the same base model on the 30 AIME24 problems for 60 steps, recording 64 responses at each of 16 observations per problem.

  1. Losing one vote does not eliminate the correct answer. Wrong answers spread over ~40 values, so the leader often leads by one or two votes. With 4–6 of 64 votes the correct answer wins 43% of the time; with 7–12 votes, 81%.
  2. The correct answer is the one still present at the next vote. Early per-token entropy is 0.27 for correct vs. 0.71 for incorrect responses. An answer seen once among 64 samples reappears next time 58% of the time if correct and 19% if incorrect. Voting rewards what many samples reproduce, so repeated votes favor correct answers.
  3. This holds only if the leader is not amplified for its own sake. On the 15 problems with an initially wrong majority and a sampled correct answer (start → step 60):
TTRLEVOL-RL
Share of the leading wrong answer13.6% → 42.2%11.5% → 17.5%
Share of the correct answer2.0% → 2.0%1.8% → 7.8%
Majority-correct problems0 → 00 → 3

Under EVOL-RL the correct share rises on 11 of the 15 problems. The 3 that become majority-correct start with the correct answer at 5.2% and the leading wrong answer at 7.0%. Later supervision improves: maj@16 on the 30 training problems rises from 0.175 to 0.272 under EVOL-RL and does not improve under TTRL (0.185 to 0.166).

When a wrong answer is itself repeatable, step 2 applies to it too and it is reinforced, as in any method that learns from its own outputs.

P3 (W2, Q2). Cost and matched budget. Up to the step where both methods stop improving, EVOL-RL uses 1.5–2.7× the generated tokens and 2.0–2.9× the training time of TTRL. The advantage holds at a matched budget: truncated at the token total of TTRL's full run (B), EVOL-RL is ahead. AIME25 pass@1/pass@16, 5-checkpoint averages; "best" is TTRL's best window for each metric:

Train setTokens to plateau, TTRL / EVOL-RLTime to plateau (8 GPUs)BTTRL finalTTRL bestEVOL-RL at B
AIME2453M / 145M10.3h / 22.7h143M5.5/20.610.0/30.218.5/40.5
MATH-50037M / 57M11.1h / 22.6h106M7.2/26.19.8/32.217.8/35.4
MATH-TRAIN69M / 164M19.8h / 57.2h160M7.9/29.410.1/33.013.6/34.5

Across AIME25, AIME24 and MATH, EVOL-RL at B is higher than both TTRL's final and its best in 17 of 18 comparisons, so the gain is not bought with more generation. Both methods have the same generation limit; the token gap arises because TTRL's training responses shrink from about 1,500 to 650 tokens (the collapse itself) while EVOL-RL's lengthen. Over our full runs (far past the plateau, to test stability) the token ratio is 2.7–6.1×, with no further gain.

P4 (Q3). Reliability. Consistency improves relative to the base model, together with coverage. all-pass@k (your pass^k), 32 samples per problem, MATH-500-trained models, re-evaluated from released weights:

Test setModelall-pass@2all-pass@4all-pass@8
MATHBase57.146.736.3
MATHTTRL75.371.768.0
MATHEVOL-RL73.567.060.4
AIME25Base1.70.20.0
AIME25TTRL5.72.91.6
AIME25EVOL-RL9.85.73.2

EVOL-RL is more consistent than the base model at every k, and its pass@16 also rises (Table 1). all-pass@k reflects both accuracy and how deterministic the output is: for a fully deterministic policy it does not decrease with k. TTRL is close to this (75.3 to 68.0 on MATH from k = 2 to 8), which is why its all-pass@k on MATH is higher while its pass@16 falls below the base model (Table 1); on held-out AIME25, EVOL-RL is higher at every k. EVOL-RL can also be made deterministic by greedy decoding at comparable accuracy (Table 18, MATH-TRAIN model).

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer n6K6,

Thank you again for your time and feedback. We wanted to gently follow up on our response and the revised manuscript. We would appreciate hearing whether the additional analyses address your concerns, and whether there are any remaining questions we could clarify before the discussion closes.

Thank you for considering our response.

Thank you for the honest and thoughtful response, and for taking these concerns so seriously. I really appreciate the care you put into this work. Your replies helped me understand the paper much better, and I believe the added analyses will also be useful to others working on similar problems. Still, I think some limitations in the method’s design remain, so I will keep my current rating.

AuthorsOpenReview ↗

Response to Reviewer n6K6's Follow-up Comment

Thank you very much for reading our response so carefully, and for your kind words. The fixed-problem tracking, the cost comparison, and the all-pass@k results all came from your suggestions, and they have made the paper more complete.

On the limitations in the method's design, we understand one of them to be fundamental: without labels, novelty cannot distinguish a correct minority answer from an incorrect one. We now state this explicitly in the Limitations of the revised paper. This limit is shared by methods that learn from their own outputs; our design targets what can be done within it, which is to avoid discarding the correct answer before voting has completed its selection, and the fixed-problem tracking tests this directly (§4.6).

If you have another specific design issue in mind, we would be glad to address it with further explanation or analysis during the discussion period.

Reviewer_dG5QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EVOL-RL for training without answer labels. It combines majority-based rewards with response novelty to reduce the loss of useful answer diversity during training. Experiments across several models include held-out tasks and tests of novelty, entropy, clipping, and embedding choices.

Critical strengths
  • Novelty rewards reduce pressure to copy the same reasoning trace during self-training.
  • Component tests show gains from novelty beyond entropy and clipping changes.
  • Gains on new questions across several models support benefits beyond the training set.
Critical weaknesses
  • A wrong majority answer still receives more reward than a correct minority answer. Novelty preserves alternatives but cannot directly favor the correct one, as Appendix B.3 also notes.
  • Training groups responses by their numeric final answers. This rule does not directly apply to writing or dialogue, where different answers may all be valid.
  • Embedding distance can reward different wording or different mistakes. Appendix B.9 gives useful examples, but does not establish that this reward consistently measures better reasoning diversity.
Questions for authors

Please respond to the concerns raised in Critical Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dG5Q

Thank you for the review. New analyses are in the revised paper (§4.6, App. B.14, Table 15).

W1. A wrong majority outranks a correct minority; novelty cannot favor the correct one. Both are true, and the first is by design. What favors correct answers is not novelty but repeated majority voting.

Why not rank the correct minority higher. Without labels this means raising every novel minority, right or wrong. In Table 16, once the two reward bands touch or overlap (β = 1.5, 2.0), MATH pass@1 drops from 79.8 to 59.6 and 36.4. The vote is the only signal related to correctness, so it sets the direction; novelty only reorders responses within a group.

How correct answers win nonetheless. A systematic error, which the model reproduces consistently, cannot be told apart from a correct answer without labels, and any method learning from its own outputs reinforces it. Our gains come from a different case, where no wrong answer truly leads: on the problems that become majority-correct, the leading wrong answer starts with 3–5 of 64 votes among ~40 distinct wrong answers, and the risk is losing the correct answer before it can win. We tracked the 30 AIME24 training problems for 60 steps under TTRL and EVOL-RL.

  1. Voting rewards repeatability, and correct answers are more repeatable. Early per-token entropy is 0.27 for correct vs. 0.71 for incorrect responses. An answer seen once among 64 samples reappears at the next observation 58% of the time if correct and 19% if incorrect. A single vote under-rewards a correct minority, but repeated votes favor it.
  2. Selection alone destroys this signal: the leader takes all the positive reward and leads again, right or wrong. On the 15 problems that start with a wrong majority and a sampled correct answer (start → step 60):
TTRLEVOL-RL
Leading wrong answer's share13.6% → 42.2%11.5% → 17.5%
Correct answer's share2.0% → 2.0%1.8% → 7.8%
Majority-correct problems0 → 00 → 3
  1. With the variation terms (novelty, entropy regularization, asymmetric clipping) the lead is not amplified, so the correct answer stays available until it wins. maj@16 on the 30 problems rises from 0.175 to 0.272 under EVOL-RL and does not improve under TTRL (0.185 to 0.166).

The claim is not that EVOL-RL identifies the correct minority, but that it does not discard it before selection completes.

W2. Grouping by numeric answers does not apply to writing or dialogue. Agreed; this is the current scope, stated in the revised Limitations. The restriction comes only from the selection signal (answer extraction and voting, as in TTRL); novelty, entropy regularization, and asymmetric clipping do not depend on the answer format. Where several answers can all be valid, agreeing with the majority no longer indicates quality, so open-ended tasks need a different selection signal. Beyond mathematics we show transfer: models trained without labels on math improve on GPQA (Table 1), MMLU-Pro, SuperGPQA, and BBEH (§4.8).

W3. Embedding distance can reward different wording or different mistakes. B.9 is only an example. Embedding distance is an imperfect proxy, and we test it in four ways.

Does it track reasoning? In B.10, an independent judge from another model family labels response pairs on 60 problems, sampled from the most and least similar thirds under the embedding, as using the same or a different approach. It agrees with the embedding ranking on 83.9% of these pairs (90.6% for pairs the embedding places far apart).

Wording or reasoning? In B.11, another model rewrites the language and notation of a solution while keeping every reasoning step. A rewrite's similarity to its original is 0.834; two different solutions to the same problem score 0.681. Rewording thus earns clearly less novelty than a different solution.

Do different mistakes benefit? Any minority response, however novel, receives less reward than any majority response. Within the minority, a more novel response is penalized less, but this does not keep wrong answers in the pool: they recur 19% of the time, against 58% for correct ones (W1, step 1). Which answers stay is decided by the vote, not by novelty.

Do gains come from the embedding's information? We keep the reward structure fixed and change only the vectors (MATH-500 training, pass@1/pass@16):

Vectors used for noveltyMATHAIME24AIME25
None79.3/88.712.1/27.011.1/34.8
Random vectors (new)77.6/90.311.9/30.510.4/29.8
OpenAI embedding79.3/93.816.2/39.715.6/40.8
Qwen3 embedding (default)79.8/93.819.0/43.216.1/41.9

Random vectors perform like no novelty; gains appear only when the vectors reflect response content, and persist across embedding families.

These results support embedding distance as a training signal, not a claim that every rewarded difference is better reasoning.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer dG5Q,

Thank you again for your time and feedback. We wanted to gently follow up on our response and the revised manuscript. We would appreciate hearing whether the additional analyses address your concerns, and whether there are any remaining questions we could clarify before the discussion closes.

Thank you for considering our response.

Reviewer_8GRQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EVOL-RL, a reinforcement learning method for label-free self-improvement. The method uses majority voting to provide a selection signal, employs a reasoning-trace Novelty Reward to refine rewards within each selection group, and combines Entropy Regularization and Asymmetric Clipping to mitigate Selection-Induced Diversity Collapse. The authors evaluate the method across multiple models and reasoning benchmarks, demonstrating improvements over several existing label-free methods.

Critical strengths

The authors address Selection-Induced Diversity Collapse in label-free self-improvement by combining majority voting with a novelty-based reward. The full method achieves consistent improvements across multiple models and reasoning benchmarks. Component ablations, correct-minority availability analysis, reasoning-strategy diversity analysis, and majority-vote accuracy analysis also provide supporting evidence for the method's effectiveness.

Critical weaknesses

EVOL-RL still relies on majority voting to determine reward priority. Even when the majority answer is incorrect, correct minority responses receive lower rewards than incorrect majority responses. Although the Novelty Reward can reduce the suppression of minority reasoning paths, it cannot distinguish correct alternative solutions from novel incorrect ones. Consequently, when an incorrect consensus persists, the method may continue to reinforce incorrect answers.

Questions for authors
  • When the majority answer is incorrect, correct minority responses still receive lower rewards than incorrect majority responses. Although the Novelty Reward can reduce the penalties imposed on some minority paths, how much does it change the advantages assigned to correct minority responses during training? Is the observed preservation mainly attributable to the Novelty Reward or to exploration mechanisms such as Entropy Regularization?
  • The Novelty Reward cannot distinguish correct alternative solutions from novel incorrect ones. Appendix B.3 shows improved correct-minority availability, but are novel incorrect minority solutions preserved to a similar extent? How does the availability of these two types of reasoning paths change over the course of training?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8GRQ

Thank you for the careful review. Your points share one premise: novelty cannot tell correct from incorrect. We agree, and this is by design. Novelty separates repeated reasoning from different reasoning; what favors correct answers is repeated majority voting. New results below are in the revised paper (§4.6, App. B.14, Table 15).

P1 (W1, Q1). A correct minority receives less reward than a wrong majority. This is intended: without labels, raising the minority means raising every novel minority, right or wrong. Table 16 tests this: once the two reward bands touch or overlap (β = 1.5, 2.0), MATH pass@1 falls from 79.8 to 59.6 and 36.4. The vote is the only signal related to correctness, so it sets the direction and novelty only reorders responses within a group.

P2 (W1). Is a persistent wrong consensus reinforced? Yes, for systematic errors the model reproduces consistently: from its own outputs alone, these cannot be told apart from correct answers (App. F). Our gains come from a different case, where no wrong answer truly leads: on the problems that become majority-correct, the leading wrong answer starts with 3–5 of 64 votes among ~40 distinct wrong answers, and the risk is losing the correct answer before it can win. We tracked the 30 AIME24 training problems for 60 steps under both methods (same base model, 64 samples per observation).

  1. Voting rewards repeatability, and correct answers are more repeatable. In the first 10 steps, mean per-token entropy is 0.27 for correct versus 0.71 for incorrect responses. Among minority answers seen only once in 64 samples, 58% of correct answers recur at the next observation, versus 19% of incorrect answers.
  2. Selection alone destroys this signal: whichever answer leads takes all the positive reward and leads again, right or wrong. On the 15 problems that start with a wrong majority and a sampled correct answer (start → step 60):
TTRLEVOL-RL
Share of the leading wrong answer13.6% → 42.2%11.5% → 17.5%
Problems with a wrong answer above 50%0 → 60 → 0
Share of the correct answer2.0% → 2.0%1.8% → 7.8%
Majority-correct problems0 → 00 → 3
  1. Variation limits runaway reinforcement. Novelty, entropy regularization, and asymmetric clipping jointly constrain wrong-answer concentration, keeping correct alternatives available for later votes. Across all 30 problems, maj@16 rises from 0.175 to 0.272 under EVOL-RL, but falls from 0.185 to 0.166 under TTRL.

The claim is therefore not that EVOL-RL identifies the correct minority, but that it does not discard it before selection completes.

P3 (W1, Q2). Are novel incorrect minority solutions preserved to a similar extent? In reward, yes: novelty treats both alike (P4). In outcome, no: incorrect answers recur at 12%–21% at every novelty level, vs. 58% for correct ones, so even the most novel wrong answers are mostly replaced. From steps 1–10 to 51–60, correct-answer share rises from 4.1% to 14.2%, incorrect-minority share falls from 85.9% to 71.9%, and distinct wrong answers per problem fall from 41.4 to 29.6. (Availability of incorrect minorities is always 100%, so we report shares.) What is preserved is candidate-pool diversity, not specific wrong answers.

P4 (Q1). How much does novelty change the advantage of correct minority responses? It reduces their penalty by roughly 1/6, with a similar effect on incorrect minorities. On the same samples, advantages without → with novelty are −0.444 → −0.367 for correct minorities (n = 123) and −0.397 → −0.326 for incorrect ones (n = 3158). Its main effect is within each voting group: the most novel third receives an advantage of 2.02 vs. 1.61 for the least novel third in the majority group, and +0.01 vs. −0.52 in the minority group. Novelty favors non-redundant responses within each group, not correct minorities as a class; which answers persist is determined by repeated voting (P3).

P5 (Q1). Novelty or entropy regularization? Both contribute, but differently. Entropy regularization prevents premature policy concentration: removing it under AIME24 training eliminates the gain (MATH pass@1: 79.6 → 66.7; base model: 67.4). Novelty distinguishes responses with the same voting status, mainly benefiting multi-sample coverage. Removing it under MATH-500 training barely changes MATH pass@1 (79.8 → 79.3) but reduces pass@16 (93.8 → 88.7; AIME24: 43.2 → 27.0). The benefit depends on content-sensitive novelty (MATH-500 training, pass@1/pass@16):

Novelty signalMATHAIME24AIME25
None79.3/88.712.1/27.011.1/34.8
Random vectors (new)77.6/90.311.9/30.510.4/29.8
Qwen3 embedding (default)79.8/93.819.0/43.216.1/41.9

Without novelty, responses to the same problem become more alike: mean pairwise embedding distance falls from 0.058 to 0.045 (21 of 23 problems), and an embedding-independent trigram distance from 0.800 to 0.750 (19 of 23).

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 8GRQ,

Thank you again for your time and feedback. We wanted to gently follow up on our response and the revised manuscript. We would appreciate hearing whether the additional analyses address your concerns, and whether there are any remaining questions we could clarify before the discussion closes.

Thank you for considering our response.

Thank you for the clarification. I now better understand how novelty helps preserve correct alternatives for subsequent voting. However, recovering correct minority answers under a persistent incorrect consensus would be more compelling to me in the label-free setting. I will keep my rating.

AuthorsOpenReview ↗

Response to Reviewer 8GRQ's Follow-up Comment

Thank you very much for your follow-up, and we are glad the mechanism is now clearer.

We agree that recovering a correct answer under a persistent incorrect consensus would be a stronger result. We do not think it can be achieved from the model's own outputs alone. When the model consistently produces the same incorrect answer, its outputs look the same as when it consistently produces a correct one: the answer is repeated, stable, and confident. Any label-free rule that overturns a persistent incorrect consensus would also overturn a persistent correct consensus, which is exactly what label-free training is meant to reinforce. This is why we state it as a limitation (App. F), and why we see label-free methods as complementary to external verification: such cases need a signal from outside the model, such as a verifier or a small number of labels.

What EVOL-RL addresses is the stage before a consensus becomes persistent. Majority-only training amplifies a weak early wrong lead into a persistent one: in our tracking, the leading wrong answer's share grows from 13.6% to 42.2% under TTRL, but only from 11.5% to 17.5% under EVOL-RL (Table 4). Preventing a persistent incorrect consensus from forming is the part of the problem a label-free method can address.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

General response: summary of the revision

We thank all reviewers for their careful and constructive comments. We have uploaded a revised PDF in which new text is marked in blue. Responses to individual points follow under each review; here we summarize what changed in the paper.

1. How a correct minority answer comes to win (§4.6, Table 4, new App. B.14). Reviewers 8GRQ (W1, Q1, Q2), n6K6 (W1, Q1), dG5Q (W1)

All three reviewers asked what happens when the majority answer is wrong, given that novelty cannot tell correct from incorrect. Following Reviewer n6K6's suggestion, we tracked a fixed set of problems: EVOL-RL and TTRL were trained from the same base model on the 30 AIME24 problems, and all 64 responses to each problem were recorded at every observation.

  • §4.6 now reports this analysis in the main text, with Table 4 comparing the two methods on the 15 problems that start with a wrong majority and a sampled correct answer.
  • App. B.14 gives the setup and full results in five new tables: how often the correct answer wins a vote, by the number of votes it receives (Table 19); outcomes by problem category for all 30 problems (Table 20); how the shares of correct and incorrect answers change during training (Table 21); how novelty changes the advantages of correct and incorrect minority responses and redistributes credit within each group (Table 22); and a direct measurement that responses to the same problem become more alike when novelty is removed (Table 23).

The resulting account is the following. Voting rewards answers that the model reproduces, and correct answers are reproduced more often than incorrect ones. Majority-only training amplifies whichever answer currently leads, right or wrong. The variation terms limit this amplification, so a correct answer remains in the sampled pool until it wins. EVOL-RL does not identify the correct minority; it avoids discarding it.

2. Validation of the embedding-based novelty signal (Table 15, App. B.9, §4.6). Reviewers dG5Q (W3), 8GRQ (Q1)

  • Table 15 has a new control: replacing the embedding with content-independent random vectors, with the reward structure unchanged, performs close to removing novelty altogether. The benefit of novelty therefore depends on the information carried by the embedding.
  • App. B.9 is now presented as an illustration of how the proxy behaves rather than as evidence of its reliability, and §4.6 points to the systematic checks in App. B.10–B.12.

3. What the reward design does and does not do (App. C.1). Reviewers 8GRQ (W1), dG5Q (W1)

We state explicitly that novelty is computed without access to correctness, that it does not identify which minority responses are correct, and that any minority response receives a lower reward than every majority response.

4. Limitations (App. F). Reviewers 8GRQ (W1), dG5Q (W1, W2)

  • Systematic errors: when the model consistently produces the same incorrect answer, it may be indistinguishable from a correct one without external verification, and methods that learn from self-generated signals may reinforce it.
  • Scope: the current implementation requires final answers that can be extracted and compared across responses, as in TTRL. This applies only to the selection signal; extending the framework to open-ended generation requires a different selection signal and remains unexplored.

Provided in the individual responses. The training-cost and matched-budget comparison (Reviewer n6K6, W2/Q2) and the all-pass@k reliability results (Reviewer n6K6, Q3) are given in our response to Reviewer n6K6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.