acceptodds
Under review as a conference paper at ICLR 2027

Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training

Abstract

Self-evolving training improves large language models by generating tasks and learning from self-produced supervision. Existing approaches can alternate between training a challenger and training a solver, requiring an additional reinforcement-learning optimization for task generation which is costly in computation and memory. Hence we introduce and study a new self-evolving framework, Direct Self-Evolving Optimization (DEO), which directly samples from optimal challenger distribution and only trains the solver. Under stated regularity and gradient-dominance conditions, we prove that DEO can learn distributionally robust model. The error bound holds over the KL ball induced by the final optimal challenger. Motivated by this conceptual algorithm, we use a finite-step, Metropolis-style mutation walk to simulate sampling task from the optimal challenger distribution and update only the solver. Our implementation uses finite-rollout uncertainty estimates and an approximate acceptance rule. We evaluate the basic method on Qwen3-4B-Base and Qwen3-8B-Base with 2,000 candidate questions per round and a fixed temperature. At iteration three, DEO attains seven-benchmark average accuracies of 47.57% and 53.60%, respectively, compared with 45.67% and 51.77% for the corresponding no-walk ablation and 45.93% and 52.88% for R-Zero. Meanwhile, our method saves memory and compute time compared to R-Zero. These comparisons use the same three-round horizon and nominal solver-update budget; the methods differ in question-pool size and generation cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_49utOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Self-Evolving Optimization (DEO), a method that removes challenger training from self-evolving LLM training. Instead of updating a challenger model, DEO uses a frozen LLM to generate and mutate math problems. The current solver answers each problem several times, and the answer agreement is used as a difficulty score. A Metropolis-style rule then decides whether to keep each mutated problem. The selected problems are labeled by the solver’s majority answer and used to train the solver with GRPO. The paper also gives a theoretical analysis of an idealized version of DEO. Experiments on OctoThinker-3B and Qwen3-4B/8B show that DEO is competitive with R-Zero while using less wall-clock time. The method can also use an API-only model such as Claude as the frozen task generator.

Critical strengths
  • Clear practical motivation. Removing challenger training makes the self-evolution pipeline simpler and reduces the need to maintain and update a second model.

  • Lower training time. DEO reports 52%–61% less wall-clock time than R-Zero while achieving similar or slightly better average math performance at the third iteration.

  • Flexible task generator. Since the generator only needs inference access, DEO can use an API-only model. The Claude generator experiment shows a useful setting that is difficult to support with methods that train the challenger.

  • Useful implementation details. The paper clearly reports the mutation operations, acceptance rule, filtering process, retained dataset sizes, prompts, and training settings. The no-walk ablation also helps measure the effect of solver-guided task mutation.

Critical weaknesses
  1. Limited novelty. The main method combines LLM-based problem rewriting, solver uncertainty, and a Metropolis-style selection rule. Each part is closely related to existing work on instruction evolution, self-training, and adaptive curriculum construction. The exponential-tilt form of the challenger distribution follows a standard KL-regularized objective. Therefore, the main contribution is an incremental change to the challenger-solver framework rather than a new self-evolution method.

  2. Large gap between theory and practice. The theory assumes exact independent sampling from the optimal task distribution, the reward (1−pθ(x))(1-p_\theta(x)), fresh labels, and an L2 proximal solver update. The actual method uses a triangular agreement score, a BLEU-based batch penalty, noisy finite-step mutation, reused responses, and GRPO. It also drops the base-distribution and proposal-probability ratio required by Metropolis-Hastings. Since the paper does not estimate the resulting sampling error, the theory does not directly support the practical algorithm.

  3. Pseudo-label and task quality are not verified. Training labels are based only on the solver’s majority answer. Answers are grouped by extracted strings rather than full mathematical equivalence, and the format checker does not verify correctness. High answer disagreement may also come from unclear or invalid questions rather than useful difficulty. The reported in-band statistics reuse the same responses used for selection, so they are not an independent check of task quality.

  4. Limited evaluation. The training experiments focus on math and use only three relatively small base models. The paper reports three iterations and appears to use one training run per setting, without variance or confidence intervals. It is unclear whether the method remains stable over more rounds or transfers to other self-evolving settings.

Questions for authors
  1. Can the authors provide a comparison with R-Zero using the same candidate pool, retained training-set size, solver updates, hardware allocation, and total inference budget?

  2. What fraction of the generated pseudo-labels are correct when checked by an independent verifier or against problems with known answers? How many generated questions are ambiguous or mathematically invalid?

  3. How close is the practical LLM walk to the target distribution used in the theory? Is there any empirical estimate of the sampler error, mixing behavior, or effect of the missing proposal ratio?

  4. Are the reported gains stable across random seeds? Standard deviations or confidence intervals could materially change the interpretation of the small gains over no walk.

  5. How does DEO compare with other recent self-evolving methods, such as Absolute Zero and Agent0, under a similar training and inference budget?

8 comments

Thank you for your constructive comments. We address them below.

W1: Limited novelty

Our central contribution is to replace explicit challenger training with task sampling in self-evolving LLM training. This removes a separate optimization stage and reduces training time and memory costs relative to two-stage approaches such as R-Zero. Technically, we develop an LLM-based mutation walk with Metropolis-style selection to approximate the desired task distribution, which is not a trivial combination. Theoretically, we characterize an idealized version of self-evolving training through a distributionally robust uncertainty bound, with an accuracy implication under additional assumptions. This provides a theoretical perspective on why adversarial task generation can support self-improvement.

Q1: Multi-seed comparison with R-Zero under a matched budget

For the no-walk comparison, we increase the candidate pool to 6,000 questions per round while retaining 2,000 for DEO, with the aim of obtaining comparable token budgets. Across three paired random seeds, we obtain the following results on math AVG7:

MethodAVG7 (%)Runtime
Base43.38N/A
No-walk46.87 ± 0.1310.36 ± 0.04
DEO48.66 ± 1.6617.71 ± 0.13

DEO can achieve better averaged math reasoning performance. We will include more seeds comparison in the updated version. Thanks for raising this point!

For the model size, we follow the setup of R-Zero and Agent0 where they also use Qwen-4B and Qwen-8B models.

Q2: Independent verification of pseudo-labels and question validity

We use Claude Sonnet as an independent judge to assess question validity and pseudo-label accuracy. Separately, we draw 9 fresh rollouts from the corresponding solver to re-estimate the modal-answer frequency p^\widehat p.The label quality and uncertainty are inspected on the VALID questions only.

MethodSeedValid questions (%)Pseudo-label accuracy (%)p^\widehat p: 25th / 50th / 75th percentilep^<0.3\widehat p<0.3 (%)0.3≤p^≤0.80.3\le\widehat p\le0.8 (%)p^>0.8\widehat p>0.8 (%)
No-walk177.085.70.38 / 0.56 / 0.6914.367.018.7
No-walk266.765.00.31 / 0.50 / 0.7515.765.319.0
No-walk363.073.00.38 / 0.56 / 0.7515.062.023.0
DEO160.774.60.25 / 0.44 / 0.5625.367.77.0
DEO241.065.60.31 / 0.50 / 0.6221.070.09.0
DEO342.367.70.31 / 0.44 / 0.6216.072.711.3

DEO will lose some question validity and label quality during mutation, while it achieves higher solver uncertainty for the remaining valid questions than no-walk, which verifies that the mutation makes the questions more challenging.

Q3: The practical sampler and the theoretical target distribution

The error between the approximate distribution and the actual task distribution can be bounded, if we assume the transition kernel to be symmetric. Fix an outer training round tt, and let μT,m\mu_{T,m} denote the task distribution after TT MCMC steps using mm rollouts per reward evaluation. Then the total deviation distance bound is

\le (1-a)^T +\tanh\left( \frac{1}{4\beta\sqrt m} +\frac{1}{32\beta^2m} \right), \qquad a=ce^{-1/\beta}.$$ The first term is the finite-step mixing error. The remaining term bounds the finite-rollout target bias. The mixing term decreases geometrically in $T$, while the finite-rollout term decreases as $m$ grows. ## Q5: Comparison with Absolute Zero and Agent0 Thank you for suggesting these comparisons. We chose R-Zero as our primary baseline because it provides the closest controlled reference for our central question: can explicit challenger training be replaced by task sampling while retaining the solver-training pipeline? Absolute Zero jointly trains a single policy for task proposal and solving, using code-execution feedback. The R-Zero paper reports higher mathematical-reasoning averages than Absolute Zero. These findings motivate our choice of R-Zero as the primary baseline. Our results suggest that training only the solver can achieve performance comparable to R-Zero's two-model co-evolutio. Agent0 shares the curriculum–executor co-evolution structure, while additionally incorporating tool-integrated reasoning, multi-turn interactions, and an ambiguity-aware executor update (ADPO). DEO could, in principle, be adapted to its framework by replacing curriculum-agent training with task sampling while retaining the tool environment and executor update. We agree that a matched-budget Agent0 comparison would strengthen the evaluation. Completing its multi-round training and evaluation is beyond the available rebuttal time, and we will include this comparison in a subsequent version.

Thank you for the response. It addresses some of my concerns. However, I would still like to see a more controlled comparison with R-Zero under the same budget, especially with matched inference cost and wall-clock time. Since efficiency is one of the main contributions of this paper, it is difficult to assess the advantage of DEO without such a comparison.

I also have a remaining concern about the theoretical analysis. The error bound assumes a symmetric transition kernel, but it is unclear whether this assumption can hold when the proposal is produced by an LLM-based rewriting process. It would be helpful to provide some approximate empirical analysis of the proposal asymmetry, sampling error, or mixing behavior. I will keep my current score.

Thanks for your reply. Here we provide same-budget comparison with R-Zero. We shorten the training steps for challenger in R-Zero, and decrease the number of generated questions from 8000 to 2000 every iteration. It ensures two methods consume similar tokens every iteration (DEO 53M vs R-Zero 58M). Here are the results:

MethodAVG7 (%)Runtime (min)
Base43.5NA
R-Zero47.17 ± 1.39329.0 ±3.8
DEO48.66 ± 1.66340.4 ±11.6

We can see that DEO performs better under the similar budget. We hope this can address your concern.

Reviewer_WRn8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Self-Evolving Optimization (DEO), an algorithm that uses frozen LLM under the guidance of current solver feedback to replace challenge parameter updates with task generation and variance. KL-regularized challenger objective targets incentivize exponentially skewed task distribution, while the actual algorithm uses approximate Metropolis-style selection, a triangular answer-agreement reward, and a diversity penalty. The GRPO updates are then monitored by pseudo-labels generated by the solver. The analysis of ideal variables establishes a boundary of conditional uncertainty and accuracy. Experiments at three basic checkpoints report competitive mathematical reasoning performance, with wall clocks running for less time than archived R-Zero, and demonstrate an API-only task generator.

Critical strengths
  1. A useful self-evolving training simplification. Separating curriculum adaptation from generator parameter updates is actually very attractive. Experiments using only the API have practical benefits: replacing the local generator with Claude improves Qwen3-4B AVG7 from 47.57 to 49.60, while retaining the solver-generated target answer (see Section 5.3 and Table 3).
  2. Consistent positive aggregation results were obtained at all three checkpoints. In the third phase of the iteration, the DEO improved AVG7 by 0.79, 1.89, and 1.82 percentage points in non-walking ablation of OctoThinker-3B, Qwen3-4B, and Qwen3-8B, respectively. he matched candidate-pool and nominal solver-update configuration is a useful control, and the reported wall clock reduction of 52-61% relative to R-Zero is significant (Table 1 and Figure 2)
  3. A cautious statement of theoretical scope. This paper clearly distinguishes between accurate sampling and approximate optimization in analysis from the implemented walk and GRPO. It also explains that this assurance under the correctness assumption strengthens predictions of initial modal answers, rather than establishing the acquisition of correct answers previously unavailable (Sections 3.2-4 and Appendix A-E).
Critical weaknesses
  1. This comparison has not yet isolated the value of the proposed selection mechanism. No walk removed the mutation and additional solver score, while the DEO performed five sweeps, with nine responses per evaluation proposal. For 4B and 8B, the DEO required 14.6 and 20.6 hours, while no steps required 4.9 and 6.2 hours, obtaining 1.89 and 1.82 AVG7 scores. Similar inference budgets were required on more independent candidates or no-leading mutations to distinguish the benefits of the acceptance mechanism from additional searches. The paper acknowledges this limitation, but it remains at the core of the algorithmic contribution. Similarly, the larger native resource pool in Qwen R-Zero training makes timing a useful methodological level comparison, rather than using all the savings to eliminate the controlled allocation of challenger training. Timing decomposition at the stage level will illuminate this interpretation (Sections 5.1–5.2; Appendix F.1).
  2. Selection tasks and pseudo-label quality are not independently established. Protocols are measured by extracted answer strings, selection responses are reused for labels, and format judgments are obviously not validators of correctness. Therefore, improved in-band output establishes operational selection output, but cannot distinguish between truly useful difficulty and fuzzy questions, mathematically equivalent response strings, or incorrect majority answers. A small, independently examined seed sample, accepted mutations, and retained labels will help determine whether refinement improves useful supervision. This is particularly important because the accuracy interpretation of theorems depends on the initial mode-answer correctness (Table 2; Appendix A; Appendix E.2 and E.5).
  3. Evidence does not quantify uncertainty in relatively mild accuracy differences. The 8B AVG7 has a margin of 0.72 points to R-Zero; rankings vary in different rounds, and Omni-MATH is not always favorable to DEO. Appendix F correctly points out that repeated AMC/AIME rows are not independent problems, but it provides neither the variability of training runs nor the matching uncertainty of the original benchmark problem. These omissions limit confidence in how reliably small reported differences will be reproduced. The idealized theoretical results do not resolve this empirical uncertainty, nor do they demonstrate the robustness of the implemented samplers (Tables 1, 5, and 6; Appendix F.1).
Questions for authors
  1. On a representative scale, how does the DEO use similar proposal/scoring budgets, as well as the same filtering and solver training methods, to compare with the freezing generator baseline? a targeted mutation-without-score-guided-selection control elucidate the contribution of the mechanism? Please distinguish between the size of the retained dataset and the actual sample used for training.
  2. In an independently examined sample, what was the ratio of improving before and after asking questions and correcting false labels? Would new solution-seeking answers or mathematically equivalent grouping significantly change the reported in-band advantages?
  3. Which stages and accelerator allocations are included in Figure 2, and how much of each method's time is spent on generation, scoring, challenger updates, and solver updates? What evidence is available for variability across training runs or uncertainty over original evaluation questions?
5 comments

Thank you for your constructive comments. We address them below.

Q1: Fair-budget, multi-seed comparison with the baseline

We increase the no-walk baseline's candidate pool to 6,000 questions per round while retaining 2,000 for DEO, with the aim of obtaining comparable token budgets. We run both methods using three paired random seeds and report the math AVG7 results on Qwen-4B:

MethodAVG7 (%)Runtime
Base43.38N/A
No-walk46.87 ± 0.1310.36 ± 0.04
DEO48.66 ± 1.6617.71 ± 0.13

DEO can achieve better averaged math reasoning performance. We will include more seeds comparison in the updated version. Thanks for raising this point!

Q2: Selected-task and pseudo-label quality

We use Claude Sonnet as an independent judge to assess question validity and pseudo-label accuracy. Separately, we draw 9 fresh rollouts from the corresponding solver to re-estimate the modal-answer frequency p^\widehat p. The label quality and uncertainty are inspected on the VALID questions only.

MethodSeedValid questions (%)Pseudo-label accuracy (%)p^\widehat p: 25th / 50th / 75th percentilep^<0.3\widehat p<0.3 (%)0.3≤p^≤0.80.3\le\widehat p\le0.8 (%) (In-band)p^>0.8\widehat p>0.8 (%)
No-walk177.085.70.38 / 0.56 / 0.6914.367.018.7
No-walk266.765.00.31 / 0.50 / 0.7515.765.319.0
No-walk363.073.00.38 / 0.56 / 0.7515.062.023.0
DEO160.774.60.25 / 0.44 / 0.5625.367.77.0
DEO241.065.60.31 / 0.50 / 0.6221.070.09.0
DEO342.367.70.31 / 0.44 / 0.6216.072.711.3

DEO will lose some question validity and label quality during mutation, while it achieves higher solver uncertainty for the remaining valid questions than no-walk, which verifies that the mutation makes the questions more challenging.

Q3: Time allocation

We report the Wall-clock breakdown of the runs in Figure 2 (Qwen3-4B-Base, iterations 1–3, one 8×H100 node, hours).

StageGPUs usedR-ZeroDEONo-walk ablation
Challenger update (questioner rollouts + GRPO update + checkpointing)R-Zero: 0–311.51——
Challenger reward scoring (solver vLLM servers)R-Zero: 4–73.69——
Question generation + solver scoringR-Zero: 0–7. DEO: generator on GPU 0, solver on 1, 4, 51.2811.481.90
of which: initial pool (generation + scoring)—1.531.54
of which: MCMC walk (mutation + scoring + MH)—9.73—
of which: filtering / deduplication—0.220.36
Solver update (verl GRPO)R-Zero: 0–7. DEO: 2, 313.692.792.69
Checkpoint merge + per-iteration MATH-500 monitoringR-Zero: 0–7. DEO: 60.260.270.27
Total (as in Figure 2)30.4314.534.87

Thank you for the additional experiments and runtime breakdown. They address some of my concerns. However, the contribution of score-guided selection remains unclear without a mutation-only control under matched budgets. The quality assessment also shows substantially lower question validity for DEO, so higher uncertainty does not necessarily imply more useful supervision. Finally, the large difference in solver-update time needs clarification to support the efficiency claim. These concerns remain central to my assessment, so I will maintain my current rating.

Thanks for your reply. We address your remaining concerns here.

Fair runtime comparison

The slow runtime of solver-update in R-Zero is due to GPU communication issues and idle time. We fix that and re-run the comparison with fair GPU allocation. DEO is still ~2x faster than R-Zero.

MethodStageGPUsIter 1Iter 2Iter 3Mean
R-Zero (default config)Challenger GRPO training (5 steps, including reward-server launch and merge)0–3: training; 4–7: reward137.9136.2145.0139.7
Question generation (8,000 questions)0–71.91.81.81.8
Question scoring (10 solver samples per question)0–721.021.225.422.5
Solver GRPO training (20 steps, including merge) + MATH-500 evaluation0–731.529.030.730.4
Iteration total192.3188.1202.8194.4
DEO (ours)Pool initialization (2,000 questions)0: base; 1–7: solver13.817.617.516.3
MCMC mutation walk (5 steps)0: base; 1–7: solver45.356.955.752.6
Filtering / LLM judge / upload / miscellaneous—3.13.56.24.3
Solver GRPO training (20 steps, including merge) + MATH-500 evaluation0–734.433.633.533.8
Inference-server restore—1.61.5—1.0
Iteration total98.3113.2113.0108.2

Question quality While DEO has low validity rate, it improves the uncertainty (the p^\hat{p}). The fair comparison of testing acc also demonstrated the effect of mcmc walk.

Mutation-only without selection Mutation-only without selection is equivalent to DEO with β=∞\beta = \infty. It will lower the average uncertainty p^\hat{p} due to the absence of selection. We will provide such ablation.

Reviewer_gHUPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Self-Evolving Optimization (DEO), a method that improves LLM reasoning without training a separate challenger model. Instead, it uses a frozen LLM to generate and modify tasks, and selects challenging tasks based on the current solver's feedback. Experiments show that DEO achieves competitive performance with R-Zero while reducing training time by over 50%.

Critical strengths

The main idea is simple and effective. Replacing challenger training with task sampling significantly reduces training costs.

The method is flexible and can use closed-source LLMs as task generators without accessing their parameters.

The experiments cover multiple model sizes and show consistent improvements over the no-walk baseline.

Critical weaknesses

The theoretical analysis assumes exact sampling from the optimal task distribution, while the actual algorithm uses an approximate sampling procedure. It is unclear how well the theory explains the practical results.

The paper does not compare its Metropolis-based selection with simpler alternatives, such as best-of-K sampling.

Questions for authors

Does the practical sampler actually approximate the claimed optimal distribution? If the challenge reward is constant, the optimal distribution is simply the base distribution. However, the implemented algorithm accepts every mutation, which generally does not preserve the base distribution. What justifies the connection between the practical algorithm and the theoretical sampling target?

Is the improvement really driven by solver-guided selection? The paper combines LLM mutation with Metropolis selection, but does not isolate their contributions. Would simply accepting all mutations, or selecting the best candidates without an MH walk, achieve similar results under the same inference budget?

3 comments

Thank you for your positive and insightful comments. We address them below.

Q1: The gap between exact and approximate sampling

The gap can be bounded under explicit assumptions on the proposal kernel, if we assume the transition kernel to be symmetric. Fix an outer training round tt, and let μT,m\mu_{T,m} denote the task distribution after TT MCMC steps using mm rollouts per reward evaluation. Then the total deviation distance bound is

\le (1-a)^T +\tanh \left( \frac{1}{4\beta\sqrt m} +\frac{1}{32\beta^2m} \right), \qquad a=c e^{-1/\beta}.$$ The first term is the finite-step mixing error. The remaining term bounds the finite-rollout target bias. The mixing term decreases geometrically in $T$, while the finite-rollout term decreases as $m$ grows. ## Q2: Comparison with Best-of-K sampling DEO aims to approximately sample from the reward-tilted distribution $q_t(x)$. Best-of-K instead selects the highest-scoring candidate among $K$ proposals, generally inducing a different output distribution. We agree that it is a useful heuristic baseline and will include a comparison in an updated version.

Thank you for the clarification and additional analysis. I still have some reservations about the gap between the theoretical analysis and the practical sampling because the given bound is under very strong assumptions and is loose. Nevertheless, I find the core idea interesting, and the empirical results are promising. I will maintain my current score.

Thank you for your positive feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.