- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Direct Self-Evolving Optimization (DEO), a method that removes challenger training from self-evolving LLM training. Instead of updating a challenger model, DEO uses a frozen LLM to generate and mutate math problems. The current solver answers each problem several times, and the answer agreement is used as a difficulty score. A Metropolis-style rule then decides whether to keep each mutated problem. The selected problems are labeled by the solver’s majority answer and used to train the solver with GRPO. The paper also gives a theoretical analysis of an idealized version of DEO. Experiments on OctoThinker-3B and Qwen3-4B/8B show that DEO is competitive with R-Zero while using less wall-clock time. The method can also use an API-only model such as Claude as the frozen task generator.
Critical strengths
-
Clear practical motivation. Removing challenger training makes the self-evolution pipeline simpler and reduces the need to maintain and update a second model.
-
Lower training time. DEO reports 52%–61% less wall-clock time than R-Zero while achieving similar or slightly better average math performance at the third iteration.
-
Flexible task generator. Since the generator only needs inference access, DEO can use an API-only model. The Claude generator experiment shows a useful setting that is difficult to support with methods that train the challenger.
-
Useful implementation details. The paper clearly reports the mutation operations, acceptance rule, filtering process, retained dataset sizes, prompts, and training settings. The no-walk ablation also helps measure the effect of solver-guided task mutation.
Critical weaknesses
-
Limited novelty. The main method combines LLM-based problem rewriting, solver uncertainty, and a Metropolis-style selection rule. Each part is closely related to existing work on instruction evolution, self-training, and adaptive curriculum construction. The exponential-tilt form of the challenger distribution follows a standard KL-regularized objective. Therefore, the main contribution is an incremental change to the challenger-solver framework rather than a new self-evolution method.
-
Large gap between theory and practice. The theory assumes exact independent sampling from the optimal task distribution, the reward , fresh labels, and an L2 proximal solver update. The actual method uses a triangular agreement score, a BLEU-based batch penalty, noisy finite-step mutation, reused responses, and GRPO. It also drops the base-distribution and proposal-probability ratio required by Metropolis-Hastings. Since the paper does not estimate the resulting sampling error, the theory does not directly support the practical algorithm.
-
Pseudo-label and task quality are not verified. Training labels are based only on the solver’s majority answer. Answers are grouped by extracted strings rather than full mathematical equivalence, and the format checker does not verify correctness. High answer disagreement may also come from unclear or invalid questions rather than useful difficulty. The reported in-band statistics reuse the same responses used for selection, so they are not an independent check of task quality.
-
Limited evaluation. The training experiments focus on math and use only three relatively small base models. The paper reports three iterations and appears to use one training run per setting, without variance or confidence intervals. It is unclear whether the method remains stable over more rounds or transfers to other self-evolving settings.
Questions for authors
-
Can the authors provide a comparison with R-Zero using the same candidate pool, retained training-set size, solver updates, hardware allocation, and total inference budget?
-
What fraction of the generated pseudo-labels are correct when checked by an independent verifier or against problems with known answers? How many generated questions are ambiguous or mathematically invalid?
-
How close is the practical LLM walk to the target distribution used in the theory? Is there any empirical estimate of the sampler error, mixing behavior, or effect of the missing proposal ratio?
-
Are the reported gains stable across random seeds? Standard deviations or confidence intervals could materially change the interpretation of the small gains over no walk.
-
How does DEO compare with other recent self-evolving methods, such as Absolute Zero and Agent0, under a similar training and inference budget?
8 comments
Thank you for the response. It addresses some of my concerns. However, I would still like to see a more controlled comparison with R-Zero under the same budget, especially with matched inference cost and wall-clock time. Since efficiency is one of the main contributions of this paper, it is difficult to assess the advantage of DEO without such a comparison.
I also have a remaining concern about the theoretical analysis. The error bound assumes a symmetric transition kernel, but it is unclear whether this assumption can hold when the proposal is produced by an LLM-based rewriting process. It would be helpful to provide some approximate empirical analysis of the proposal asymmetry, sampling error, or mixing behavior. I will keep my current score.
Thanks for your reply. Here we provide same-budget comparison with R-Zero. We shorten the training steps for challenger in R-Zero, and decrease the number of generated questions from 8000 to 2000 every iteration. It ensures two methods consume similar tokens every iteration (DEO 53M vs R-Zero 58M). Here are the results:
| Method | AVG7 (%) | Runtime (min) |
|---|---|---|
| Base | 43.5 | NA |
| R-Zero | 47.17 ± 1.39 | 329.0 ±3.8 |
| DEO | 48.66 ± 1.66 | 340.4 ±11.6 |
We can see that DEO performs better under the similar budget. We hope this can address your concern.
Thank you for your constructive comments. We address them below.
W1: Limited novelty
Our central contribution is to replace explicit challenger training with task sampling in self-evolving LLM training. This removes a separate optimization stage and reduces training time and memory costs relative to two-stage approaches such as R-Zero. Technically, we develop an LLM-based mutation walk with Metropolis-style selection to approximate the desired task distribution, which is not a trivial combination. Theoretically, we characterize an idealized version of self-evolving training through a distributionally robust uncertainty bound, with an accuracy implication under additional assumptions. This provides a theoretical perspective on why adversarial task generation can support self-improvement.
Q1: Multi-seed comparison with R-Zero under a matched budget
For the no-walk comparison, we increase the candidate pool to 6,000 questions per round while retaining 2,000 for DEO, with the aim of obtaining comparable token budgets. Across three paired random seeds, we obtain the following results on math AVG7:
DEO can achieve better averaged math reasoning performance. We will include more seeds comparison in the updated version. Thanks for raising this point!
For the model size, we follow the setup of R-Zero and Agent0 where they also use Qwen-4B and Qwen-8B models.
Q2: Independent verification of pseudo-labels and question validity
We use Claude Sonnet as an independent judge to assess question validity and pseudo-label accuracy. Separately, we draw 9 fresh rollouts from the corresponding solver to re-estimate the modal-answer frequency p.The label quality and uncertainty are inspected on the VALID questions only.
DEO will lose some question validity and label quality during mutation, while it achieves higher solver uncertainty for the remaining valid questions than no-walk, which verifies that the mutation makes the questions more challenging.
Q3: The practical sampler and the theoretical target distribution
The error between the approximate distribution and the actual task distribution can be bounded, if we assume the transition kernel to be symmetric. Fix an outer training round t, and let μT,m denote the task distribution after T MCMC steps using m rollouts per reward evaluation. Then the total deviation distance bound is
\le (1-a)^T +\tanh\left( \frac{1}{4\beta\sqrt m} +\frac{1}{32\beta^2m} \right), \qquad a=ce^{-1/\beta}.$$ The first term is the finite-step mixing error. The remaining term bounds the finite-rollout target bias. The mixing term decreases geometrically in $T$, while the finite-rollout term decreases as $m$ grows. ## Q5: Comparison with Absolute Zero and Agent0 Thank you for suggesting these comparisons. We chose R-Zero as our primary baseline because it provides the closest controlled reference for our central question: can explicit challenger training be replaced by task sampling while retaining the solver-training pipeline? Absolute Zero jointly trains a single policy for task proposal and solving, using code-execution feedback. The R-Zero paper reports higher mathematical-reasoning averages than Absolute Zero. These findings motivate our choice of R-Zero as the primary baseline. Our results suggest that training only the solver can achieve performance comparable to R-Zero's two-model co-evolutio. Agent0 shares the curriculum–executor co-evolution structure, while additionally incorporating tool-integrated reasoning, multi-turn interactions, and an ambiguity-aware executor update (ADPO). DEO could, in principle, be adapted to its framework by replacing curriculum-agent training with task sampling while retaining the tool environment and executor update. We agree that a matched-budget Agent0 comparison would strengthen the evaluation. Completing its multi-round training and evaluation is beyond the available rebuttal time, and we will include this comparison in a subsequent version.