acceptodds
Under review as a conference paper at ICLR 2027

-Evolve: LLMs Evolve Their Own Generator Programs for Self-Improvement

Abstract

In group-relative reinforcement learning with binary verifiable rewards, all-correct or all-incorrect response groups contribute no reward-driven policy gradient. Sustained self-improvement therefore requires problem instances that continue to yield both correct and incorrect responses as the Solver changes. We introduce , which retains generator programs in an archive, each representing a task family of problem instances sharing the same structure. Each iteration, the Solver reassesses retained programs on newly sampled instances and, as the Evolver, mutates them into child programs. Reasoning Quality () estimates the current usefulness of retained programs and their children by multiplying learnability, measured as verifier reward variance, by uncertainty, measured as mean token entropy. We motivate this product by deriving an upper bound on the squared norm of a centered-reward surrogate gradient in output-logit space. Fitness ranks eligible retained programs for training, while retention confines competition to cells defined by mathematical domain and output form, leaving programs in other cells available for later reassessment and mutation. With Qwen3-4B-Base and Qwen3-8B-Base, achieves final mathematical benchmark gains of and percentage points over the base models. Both scales reach their highest averages late in training, and the final 8B checkpoint exceeds R-Zero's peak by points. Classified generated instances remain broadly distributed across domains and output forms late in the 8B run.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning with verifiable rewards, self-evolving reasoning, automatic curriculum learning, quality-diversity, MAP-Elites, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9QU2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper propose a R-zero style self-play framework that includes two key differences: 1) the proposer is replaced by a store of generator programs that generate problems and correct answers directly; 2) the solver evolves these programs with several heuristics centered on a metric called RQ (reasoning quality), which is a slight change to existing learnability signals. Results show that the propose pipeline improves over R-zero on several mathematical benchmarks, on average, slightly.

Critical strengths

The proposed pipeline is relatively complete engineering-wise, and similar ideas could be more broadly applied to other domains that can benefit from generator programs with sufficient human heuristics are priors.

Critical weaknesses

The major weakness is with the scientific contributions of this paper, as I do not clearly see how this advances the communities' understandings. Individual components, namely using program generators for math questions to produce synthetic training signals, the RQ design, and the evolution algorithm, are mostly methods broadly applied in previous works with minor changes. I do like the overall formulation of evolving the program stores as an replacement of the proposer in the original R-zero framework, yet there is no convincing evidences or ablation studies on whether the mutation itself actually contributes to the gains. As a result, the gains could largely be from models getting more consistent (via the program-generated questions) and moving slightly to regions that they were good at in the first place (as shown by the RQ selection beating random). I do not see any clear evidence that any real and sustainable evolution from the proposed framework is possible.

Questions for authors

Please see the weaknesses.

Reviewer_2AAAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents RQ-Evolve, an evolutionary framework for improving mathematical reasoning without relying on a separately trained problem generator. The main idea is to let the Solver generate and refine executable problem-generation programs, which are maintained in a MAP-Elites archive. These programs are ranked using an RQ fitness metric based on reward variance and token entropy. The generated problems are then used to further train the Solver. The authors evaluate the method on Qwen3-4B and Qwen3-8B across several mathematical reasoning benchmarks. They report improvements over the original models, as well as better average performance and reduced training cost compared to R-Zero. Several ablation experiments are included to examine the framework.

Critical strengths

I think the use of reusable generator programs is a reasonable design choice. Compared with generating each problem independently, storing programs in an archive makes it possible to reuse and update different types of problems as the Solver improves. This could be helpful for building an adaptive training curriculum. The evaluation includes two model sizes and multiple mathematical reasoning benchmarks. The lower computational cost compared to R-Zero is another positive aspect of the work.

Critical weaknesses
  1. The methodological novelty is limited, and the contribution of individual components is not sufficiently demonstrated.

The proposed framework mainly combines techniques that have already been explored in previous work, such as MAP-Elites, program evolution, and curriculum learning based on problem difficulty. While their combination is reasonable, I do not see a particularly new technical idea that clearly distinguishes this method from existing approaches.

The ablation experiments also leave several important questions unanswered. For instance, the authors do not compare their approach against a fixed collection of generator programs or a simpler evolutionary approach without MAP-Elites. In addition, the Random baseline modifies both the replacement strategy and the selection of training problems. Therefore, it is unclear whether the improvements actually come from the archive mechanism.

I am also not fully convinced by the justification for the RQ fitness score. Proposition 1 only establishes an upper bound on a surrogate gradient norm. However, this does not necessarily imply that selecting programs with higher RQ values will produce better learning outcomes. The bound may not be tight, and the paper does not directly examine whether RQ is correlated with actual training progress. I believe stronger evidence is needed to support this design choice.

  1. The experimental results need more careful validation.

The advantage over R-Zero is fairly modest in some of the main results. On the 8B model, the mathematical average increases from 54.80 at R-Zero's peak to 55.86 with RQ-Evolve, which is only a 1.06-point improvement. Moreover, RQ-Evolve is noticeably worse on AIME25 for the same model, scoring 14.69 compared to 19.38 for R-Zero.

The paper reports results from single training runs without providing standard deviations or confidence intervals. Since the method involves stochastic evolutionary search and reinforcement learning, the observed differences could be affected by randomness. This is especially relevant to the ablation results, where some improvements are quite small. Without multiple runs, it is hard to determine how stable these findings are.

Another issue is the way generator programs are evaluated. Each program is reassessed using just one sampled problem, even though it is intended to generate a distribution of problems. I am not sure that a single sample can provide a reliable estimate of the quality of the entire program. Since these estimates are used for both archive updates and training selection, noisy evaluations could have a considerable impact on the method. The authors should investigate how the results change when more samples are used.

The efficiency comparison also needs some clarification. RQ-Evolve avoids training an additional Challenger model, which appears to explain a large part of the computational savings relative to R-Zero. While this is useful, it does not necessarily mean that the proposed curriculum is more effective. A comparison under similar compute budgets would make the efficiency claim more convincing.

Questions for authors

1.Can the authors separate the effect of MAP-Elites from that of evolutionary program generation? Results using a fixed archive or a simpler evolution strategy without MAP-Elites would be helpful.

2.What is the reason for evaluating each generator program with only one sampled problem? Have the authors tested whether using more samples changes the archive contents or final performance?

3.How stable are the results across independent training runs? Specifically, does the 1.06-point gain over R-Zero on the 8B model remain consistent across different seeds?

4.Have the authors measured the relationship between RQ scores and subsequent learning progress? It would be useful to see empirical support for the fitness metric beyond the theoretical bound.

Reviewer_onmkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RQ-Evolve, a self-improvement framework in which LLMs evolve reusable problem generator programs. The method uses the Solver itself to generate and modify programs, maintains diversity through MAP-Elites, and evaluates training utility using reward variance and token entropy. Experiments on mathematical and general reasoning tasks report performance improvements and faster training compared with R-Zero.

Critical strengths
  1. The paper addresses an interesting problem in RLVR: continuously generating useful training tasks as the Solver improves.
  2. The authors evaluate the method on multiple reasoning benchmarks and provide wall-clock measurements, analyses of program evolution, and component ablations.
Critical weaknesses
  1. The novelty of the method is unclear. DéjàQ [1] already combines MAP-Elites, LLM-based problem evolution, and learnability-based selection, while EvoEnv [2] uses LLM-generated executable programs to produce reusable training tasks. The paper does not clearly distinguish its contributions from these methods or provide direct comparisons.
  2. The comparison with existing diversity-preserving methods is insufficient. R-Diverse [3], OCNR [4], and Prism [5] already address repetition and curriculum collapse through memory-based penalties, novelty rewards, or semantic coverage. The advantages of MAP-Elites over these approaches are not adequately demonstrated.
  3. The Reasoning Quality (RQ) metric lacks sufficient justification. Eq. (8) multiplies reward variance by mean token entropy, but the upper bound in Eq. (7) does not establish that higher RQ leads to larger actual gradients or better learning gains. Token entropy may also reflect uncertainty unrelated to useful reasoning.
  4. Program fitness estimation may be unreliable. Each task family is evaluated using only one newly sampled instance and a limited number of rollouts. Since instances generated by the same program can vary considerably in difficulty, it is unclear whether this estimate reliably reflects the value of the entire task family.
  5. The ablations do not sufficiently isolate the contributions of the main components. Although No Refresh, No U, and Random variants are provided, comparisons with evolution without MAP-Elites, instance-level evolution, and simpler learnability-based selection are missing. The individual benefits of program reuse, archive retention, and RQ remain unclear.
  6. The experimental results provide limited evidence for sustained self-improvement. The 8B model improves the average mathematical score over R-Zero's best checkpoint by only 1.06 percentage points and performs worse on AIME 2025. Multiple-seed results are also missing. Moreover, mixing published baseline scores with reproduced results makes fair comparison more difficult.
  7. The reliability of generated programs is not sufficiently evaluated. Validation on a limited number of seeds cannot guarantee correct answers across an entire task family. A more systematic analysis of program errors, verification failures, and their effects on Solver training is needed.

[1] DéjàQ: Open-Ended Evolution of Diverse, Learnable and Verifiable Problems.

[2] Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis.

[3] R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training.

[4] OCNR: Stabilizing Self-Play by Mitigating Iteration-Collapse With One-Class Novelty Rewards.

[5] Preventing Curriculum Collapse in Self-Evolving Reasoning Systems.

Questions for authors

See Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.