acceptodds
Under review as a conference paper at ICLR 2027

Zero-L: Tri-Role Co-Evolution Framework for Self-Evolving LLMs

Abstract

Self-evolving large language models (LLMs) can improve through reinforcement learning on self-generated tasks and feedback, reducing reliance on human-annotated training data. However, existing self-evolution frameworks face a bottleneck of training gain stagnation after first few iterations. This is because of the reward accuracy declination in later iterations. We introduce Zero-L, a framework for long-horizon self-evolution of LLMs. Starting from a single base LLM, Zero-L initializes three independent roles - a Questioner, a Solver, and a Thinker. Questioner and Solver are optimized through reinforcement learning. Questioner is rewarded for providing tasks which challenge the edge of the solver's ability. After questioner update, solver answers the tasks that questioner generates and gets Solver Reward for solving tasks, but this reward is examined to be insustainable. And that's why Thinker-Solver Reward comes onboard, reasoning process of solver's answers is fed into thinker and better process gets a higher thinker-solver reward. At the same time, with high solver reward accuracy in early iters, thinker is trained to rate reasoning process that leads to right answers. But this indicates that the ability of thinker is weak in early iters, and that's why we introduce Iteration Dependent Reward Policy (IDR Policy) which mix solver reward and thinker reward through one math function which gives solver reward a higher weight in early iterations and a lower weight after accuracy declination. Empirical results show Zero-L outperforms strong baseline at +5.7 % in math reasoning benchmarks and +6.4% in general reasoning benchmarks. More importantly, the mean IDR reward for correct responses remains above 60% across all ten iterations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, self-evolution, reinforcement learning, co-evolution, iterative post-training

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Wb4bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Zero-L, a tri-role framework for long-term self-evolution. It adds a trainable Thinker to the Questioner–Solver framework of R-Zero. The Thinker maps the reasoning traces for the same question into hidden representations and gives higher rewards to traces that are closer to the other traces. It is trained using the majority-vote Solver reward in early iterations. An iteration-dependent reward policy gradually reduces the weight of the majority-vote reward and increases the weight of the Thinker reward. Experiments on Qwen3-4B and Qwen3-8B show that Zero-L maintains better performance than R-Zero after 12 iterations.

Critical strengths
  1. The paper studies an important problem. Performance stagnation and degradation are practical problems in multi-round self-evolution.
  2. The framework is easy to understand. The Questioner generates problems, the Solver answers them, and the Thinker scores reasoning traces. The Thinker is fixed during each Solver update, which keeps the reward stable within that stage.
  3. The paper studies a longer training horizon. It reports results at iterations 3, 6, 9, and 12. This makes it possible to see the late-stage difference between Zero-L and R-Zero.
  4. The reward analysis is useful. The authors use an external model to label sampled responses after training and compare the rewards given to correct and incorrect responses. This provides some evidence for the IDR design.
Critical weaknesses
  1. The novelty is limited. Zero-L mainly adds a hidden-state similarity reward and a fixed reward schedule to R-Zero. The Questioner and Solver training follows R-Zero closely. The Thinker reward is also a simple function of the average cosine distance between reasoning traces. The method is therefore an incremental extension of an existing framework.

  2. The Thinker reward does not directly measure reasoning correctness. It rewards reasoning traces that are similar to other traces and gives lower rewards to isolated traces. A group of responses can follow the same wrong reasoning pattern, in which case the Thinker may still give them high rewards. The Thinker is also trained using majority-vote labels, so it may transfer early pseudo-label errors into its representation space instead of removing them. The paper does not provide step-level analysis or a study of such failure cases.

  3. The experimental baselines are too limited. The main experiments only compare Zero-L with R-Zero. This is not enough to show that Zero-L is better than recent methods for self-evolution, stable co-training, or curriculum construction. Relevant comparisons include:

  • [1] Self-Questioning Language Models
  • [2] Guided Self-Evolving LLMs with Minimal Human Supervision.
  • [3] R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training.
  • [4] TTCS: Test-Time Curriculum Synthesis for Self-Evolving.
  1. The ablation study is incomplete. The current ablations only freeze the Questioner or replace IDR with a fixed reward mixture. They do not show whether the gain comes from training the Thinker, using representation similarity, or simply changing the reward over time. Important controls include removing the Thinker, using a frozen or untrained Thinker, using only the Thinker reward, and testing other representation pooling methods.

  2. Compute and statistical results are not fully reported. Zero-L trains a third Transformer model in addition to the Questioner and Solver. The paper does not report the added training time, memory use, or total GPU hours. Results are averaged over three seeds, but standard deviations and confidence intervals are missing. Since many gains are only 1–3 points, this information is important.

Questions for authors
  1. How does Zero-L compare with R-Few, R-Diverse, and TTCS under the same backbone, number of iterations, data budget, and compute budget?

  2. What happens if the Thinker is frozen at its initial checkpoint? This would show whether the gain comes from Thinker training or from hidden-state similarity alone.

  3. What are the AUC, ranking accuracy, or correlation between the Thinker reward and answer correctness? The current analysis only reports average rewards for correct and incorrect groups.

  4. How does the Thinker score cases where several responses share the same reasoning pattern but reach the same wrong answer?

  5. How sensitive are the results to tt, τd\tau_d, and α\alpha?

6 comments

Thank you!

Q1. Baselines and matched compute. R-Zero shares the backbone, Q/S framework, question budget, and Q/S configuration. We lack matched R-Few, R-Diverse, or TTCS runs; our quantitative claim is therefore limited to R-Zero and component controls.

Setting (Qwen3-4B-Base)MathGeneralGPU-h/roundGPU-h total/budget
R-Zero, round 1251.7231.48 ± 0.834.756.4
Zero-L, round 1254.6633.31 ± 0.755.363.6
R-Zero, matched budget53.632.3—63.6

Supplementary profiling uses two RTX PRO 6000 GPUs, separate from the original four-GPU setup. Zero-L adds 0.6 GPU-hours per round (+12.77%). Continued R-Zero training under the 63.6-GPU-hour budget leaves a mean Zero-L advantage of 1.06/1.01 Math/General points. Average peak memory is 83.7/82.9 GB per card. The exact stopping checkpoint, stage-level timing, and control SDs are unavailable, so this is a mean comparison rather than a significance claim.

Q2. Frozen Thinker and Thinker-only reward. Frozen-init retains the pretrained checkpoint, reasoning-token mean pooling, distance mapping, and IDR, but disables Thinker updates. The early-switch control uses majority reward in iteration 1 and pure Thinker reward thereafter while retaining calibration and residual Thinker updates.

SettingMath AvgGeneral Avg
R-Zero51.7231.48 ± 0.83
Full Zero-L54.6633.31 ± 0.75
Frozen-init Thinker + IDR52.831.8
Thinker-only after iteration 153.232.0
Fixed 50/50 mixture50.6230.34

Full exceeds frozen-init by 1.86/1.51 points, supporting a learning contribution under the same mixture schedule. It exceeds the early switch by 1.46/1.31 and the fixed mixture by 4.04/2.97 points. These 12-iteration, three-seed controls test the listed configurations, not every transition policy; their SDs are unavailable.

Q3. Correctness ranking. On identical Zero-L response groups, learned/initial/pseudo-label AUROC is 0.70/0.60/0.58, and within-question ranking is 65%/58%/56%. Ranking macro-averages correct/incorrect pairs within each mixed group (ties = 0.5); pooled AUROC additionally reflects cross-question score differences.

ConditionGroupsInitial Top-1Learned Top-1
Wrong fixed label; correct candidate present30018%25%
Wrong fresh plurality; correct candidate present35018%27%

The subsets may overlap, and plurality need not exceed 50%. Top-1 averages the correct fraction among tied maximizers. The pseudo-label reward gives 0%/5% Top-1 on the two subsets. A random-selection Top-1 is not reported. Audit labels are post-hoc; common-response scoring does not establish that the evaluated Thinker had not used these responses for calibration, so the evidence is limited to the audited groups.

Q4. Shared wrong reasoning. Eq. 13 rewards relative centrality. A compact shared-error cluster can therefore score highly; if all usable representations coincide, every response receives reward 1 regardless of correctness. Learned versus initial Thinker improves wrong-plurality Top-1 by 9 points. On 450 mixed groups with fewer than 50% correct candidates, ranking is 58%/52%. However, neither subset specifically identifies a majority sharing the same reasoning error, and no response-level shared-error case study is included. Thus, this failure-analysis request remains open.

Q5. Parameter roles and pooling. Defaults are t=0.5/0.4t=0.5/0.4 for 4B/8B, τd=0.1\tau_d=0.1, and α=0.7\alpha=0.7.

  • tt controls wn=e−ntw_n=e^{-nt} in both reward transition and Thinker-update attenuation. At t=0t=0, Solver training remains majority-only; larger tt shifts earlier toward Thinker reward.
  • α\alpha scales committed Thinker updates. The α=0\alpha=0 endpoint is frozen-init, whose means are lower by 1.86/1.51 points.
  • For fixed representations and τd>0\tau_d>0, changing τd\tau_d preserves rankings but changes reward magnitudes, GRPO advantages, and calibration gradients.

Statistics and scope. Available iteration-12 General Avg sample SDs are 0.83/0.75 (4B R-Zero/Zero-L) and 0.65/0.67 (8B). Math, other checkpoints, ablations, and compute controls lack SDs. The evidence supports the learned, scheduled extension within the tested R-Zero setting; broader cross-method superiority, dedicated shared-error correction, and hyperparameter robustness are not established.

Thank you for the detailed and transparent response. The added frozen-Thinker, early-switch, compute, and correctness-ranking results address some of my concerns and make the role of the Thinker clearer. However, the matched comparisons with other recent methods are still missing, and the shared-wrong-reasoning case remains an important open failure mode of the centrality-based reward. Since several of the added comparisons also lack variance results, I will keep my current score.

Thank you for reading my response! The comparison experiment is still going, the results will be brought out in 1-2 days. Thank you again for waiting it!

Reviewer_CxFUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Zero-L, a framework for iterative LLM self-improvement with three roles: Questioner, Solver, and Thinker. The Thinker encodes reasoning traces and derives rewards from their relative geometry, using majority-answer agreement as a calibration signal. An iteration-dependent schedule gradually shifts the Solver’s reward toward the Thinker’s signal while attenuating Thinker updates. The authors evaluate Qwen3-4B and Qwen3-8B across multiple training iterations on mathematical and general reasoning benchmarks. They report improvements over R-Zero and examine the framework through reward analysis and component ablations.

Critical strengths
  1. Results cover Qwen3-4B/8B, mathematical and general reasoning benchmarks, four iteration checkpoints, and three random seeds. At 12 iterations, mathematical averages exceed R-Zero by 2.94 and 2.98 percentage points, respectively. The intermediate checkpoints and reward analysis also help characterize how the method behaves as training proceeds.

  2. Introducing a separate Thinker and an iteration-dependent reward mixture provides an explicit mechanism for reducing later-stage reliance on current majority answers.

Critical weaknesses
  1. The central mechanism of a more reliable process reward is insufficiently validated. Equation 13 rewards relative proximity among reasoning representations, while Equation 17 calibrates against majority-answer agreement, so shared errors may still be reinforced. The mean rewards conditioned on correct and incorrect responses in Section 5.1 and Appendix A.4 show a separation trend, but do not establish reliable identification of correct responses when most reasoning traces are wrong. The performance gains therefore do not sufficiently substantiate the proposed explanation for reward reliability.

  2. The conceptual advance over existing self-improvement methods appears limited. Co-evolution of task generation and solving, self-generated rewards, and multi-role decomposition have established precedents. Zero-L mainly introduces a geometry-based reward and its scheduling. However, this reward measures within-group proximity of reasoning representations and is calibrated against majority-answer agreement. The paper does not sufficiently establish why this mechanism should provide more reliable correctness judgments than the underlying agreement signal, particularly when most responses share an error. The results therefore support a useful framework extension, but do not yet establish a substantive advance in the central reward mechanism.

Questions for authors

1&2 see weaknesses.

  1. Appendix A.4 samples questions from Solver training data and uses GPT-5.5 for post-hoc correctness annotation. What inputs does the Judge actually receive, does it use independent reference answers, and how are those answers obtained? In particular, could reward rankings be checked through executable verification or human review on a subset where the majority pseudo-label is wrong, beyond reporting conditional means?

  2. How can the gains be attributed to the learned Thinker representations rather than reward mixing or additional computation? Could the authors isolate these effects through comparisons with a frozen/untrained Thinker, Thinker-only rewards, and R-Zero matched for total training and inference compute? Matching Questioner/Solver update steps alone does not account for the added Thinker cost.

3 comments

Q1. References, Judge inputs, and independent checks. Appendix A.2 (PDF p.13) gives the GPT-5.5 benchmark template: candidate {answer}, reference {response}, and Yes/No output. Table 3 instead uses references for self-generated questions obtained through LLM problem solving; the paper does not specify that construction protocol. We therefore treat Table 3 as a post-hoc LLM diagnostic, not training supervision.

Table 3 computes all reward signals on the same audited Zero-L Solver responses. Appendix A.4 samples 500 question–answer groups per reported iteration and evaluates each reward on the correct/incorrect sets C/IC/I. “Solver reward (R-Zero)” means the majority-vote rule on those responses, not rollouts from a separately trained R-Zero model; we will relabel it accordingly.

The additional audit retains 1,350 groups—150 human-reviewed and 1,200 with executable checks of the problem and reference solution—and excludes 150 ambiguous/unverifiable groups. The Judge receives the question, candidate solution, and checked reference; method identity, rewards, and agreement labels are hidden. Executable reference checks do not mean that every candidate answer was directly executable. No annotation affected training or schedule selection.

Q1, continued: ranking when agreement is unreliable. The curriculum label is fixed before fresh optimization rollouts (Sec. 3.2). Let B denote a wrong fixed label with a correct candidate (300 groups), and P a wrong fresh plurality with a correct candidate (350 groups). The subsets may overlap; plurality need not exceed 50%.

ScorerOverall rankingB rankingB Top-1P Top-1
Pseudo-label reward56%35%0%5%
Initial Thinker58%52%18%18%
Learned Thinker65%60%25%27%
IDR64%59%24%26%

Ranking macro-averages correct/incorrect pair orderings within each mixed group, assigning 0.5 to ties. Top-1 averages the correct fraction among tied maximizers. Learned versus initial Thinker improves by 7 points overall, 8 points in B ranking, and 9 points in P Top-1. On 450 mixed groups with fewer than 50% correct candidates, ranking is 58%/52%.

These are conditional comparisons on the audited groups. No random-selection Top-1 is reported. Wrong plurality or a correct minority does not establish that most traces share one wrong reasoning pattern. Nor does common-response scoring establish that the evaluated Thinker had not used these responses for calibration. We therefore do not claim independently held-out generalization or correction of shared reasoning errors.

Q2. Thinker learning and reward transition. Qwen3-4B-Base controls use 12 iterations, three training seeds, and the same Q/S configuration, candidate-question budget, and evaluation.

SettingMath AvgGeneral Avg
R-Zero51.7231.48 ± 0.83
Full Zero-L54.6633.31 ± 0.75
Frozen-init Thinker + IDR52.831.8
Thinker-only after iteration 153.232.0

Frozen-init preserves the pretrained Thinker, pooling, mapping, and IDR while disabling Thinker updates. Full is higher by 1.86/1.51 Math/General points, supporting a learning contribution under the same mixture schedule. The early-switch control uses majority reward in iteration 1 and pure Thinker reward thereafter, while retaining calibration and residual Thinker updates. Full is higher by 1.46/1.31 points. Table 4’s fixed 50/50 mixture is lower than Full by 4.04/2.97 points.

Q2, continued: compute. On two RTX PRO 6000 GPUs, supplementary profiling gives 5.3/4.7 GPU-hours per iteration for Zero-L/R-Zero (+12.77%), implying 63.6/56.4 GPU-hours over 12 rounds. Giving R-Zero a 63.6-GPU-hour budget through continued Q/S training yields 53.6/32.3 Math/General, versus 54.66/33.31 for Zero-L (+1.06/+1.01). Average peak memory is 83.7/82.9 GB per card. This profiling is separate from the original four-GPU setup; the exact stopping checkpoint, stage-level timing, and control SDs are unavailable. We therefore report a remaining mean advantage, not statistical significance.

Available iteration-12 General Avg sample SDs are 0.83/0.75 (4B R-Zero/Zero-L) and 0.65/0.67 (8B). Math, other checkpoints, and new controls lack SDs.

Scope. The frozen, ranking, and compute controls provide complementary evidence for a learned, scheduled extension of R-Zero. A compact shared-error cluster can still score highly, and no step-level or universal correctness guarantee is claimed.

Thank you for the additional experiments. The frozen-Thinker ablation and compute-matched comparison address my concerns about the source of the gains. The ranking audit also shows that the learned Thinker selects correct answers more often than the initial Thinker when the most common answer is wrong.

I am maintaining my rating because the central claim about reward reliability remains only partially supported. The audit identifies groups with an incorrect plurality answer, but it does not establish that the reasoning traces share the same underlying error or that the evaluated responses were held out from Thinker calibration. Whether the geometry-based reward can reliably identify correct reasoning under shared errors therefore remains open.

I think you should jump out of the misunderstanding of your central concern:

We would first like to clarify the intended scope of the Thinker. Zero-L does not claim that the Thinker can eliminate the shared-wrong-reasoning failure mode. Correlated errors are a fundamental limitation of fully self-evolving systems and represent part of the trade-off for reducing the cost of external supervision. The problem addressed by Zero-L is the long-horizon collapse caused by an increasingly unreliable pseudo-label-based Solver reward in later outer iterations. As explained in Lines 87-107 and 134-140, and formalized in Lines 243-280, the Thinker is calibrated when the majority-vote pseudo-label is relatively reliable; the IDR mixed reward then gradually increases the contribution of the Thinker-Solver reward, while the residual Thinker update progressively freezes the Thinker.

Regarding the concern that:

"The audit identifies groups with an incorrect plurality answer, but it does not establish that the reasoning traces share the same underlying error or that the evaluated responses were held out from Thinker calibration."

Our mechanism does not require every response group to have a correct majority-vote pseudo-label. Instead, it relies on the relatively high pseudo-label reliability in the early outer iterations to calibrate the Thinker, and then reduces the influence of later, less reliable pseudo-labels. Importantly, within each Solver optimization stage, all response groups are scored by a fixed snapshot of the round-start Thinker, and the Thinker is updated only after that Solver stage is complete (Lines 227-229 and 296-302). Therefore, the rewards assigned to responses from the fresh Solver rollout are produced before the Thinker is updated using those responses. The purpose of the audit is to demonstrate the degradation of the pseudo-label-based Solver reward and the resulting benefit of the scheduled transition, rather than to claim that the relative geometry of reasoning representations independently resolves every shared-error case. By assigning increasing weight to the Thinker-Solver reward in later outer iterations, Zero-L sustains long-horizon performance improvements while reducing dependence on degrading pseudo-labels.

The result and more claim will be added into the camera-ready version if accepted, thank you for your rating!

Reviewer_BE86OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Zero-L extends R-Zero, in which a Questioner and a Solver co-evolve on self-generated questions using majority-vote pseudo-labels. It adds a third role, the "Thinker", initialized from the same base model. The Thinker embeds each reasoning trace (mean-pooled hidden states) and builds a pairwise cosine-distance matrix within each response group.

Critical strengths
  1. The paper works on an important self-improving model setting called R-Zero. It claims R-Zero-style self-evolution stagnates and then degrades. This paper aims to resolve such issue.

  2. The paper does not require external verifier, reward head or teacher model in the training loop, and keeps the self-improving RL configuration.

Critical weaknesses
  1. It is unclear why the Thinker reward works. It rewards traces near the group's centroid in embedding space, which is essentially a soft majority vote over reasoning. That risks the same consensus or mode-collapse failure it is meant to fix. The Thinker is calibrated on early pseudo-labels, and by iteration 4 its updates are shrunk below 0.16x, so it is effectively frozen. There is no explanation of why this frozen signal stays discriminative. At iteration 1 it barely separates correct from incorrect (49.7 vs 47.6).

  2. Very thin baselines and ablations. R-Zero is the only baseline. Add comparisons with other approaches can put this work better in context..

Questions for authors

Can you show a case where the Thinker and the majority disagree and the Thinker is right?

3 comments

Thank you!

1. When the agreement label is wrong. All scorers use identical Zero-L response groups. The curriculum pseudo-label is fixed before fresh optimization rollouts (Sec. 3.2), so it can differ from their plurality answer.

Audited conditionGroupsInitial Thinker Top-1Learned Thinker Top-1
Wrong fixed label; correct candidate present30018%25%
Wrong fresh plurality; correct candidate present35018%27%

On the first subset, the pseudo-label reward has 0% Top-1, while learned/initial Thinker within-question ranking is 60%/52%. On the second, IDR Top-1 is 26%. Thus, when the most frequent fresh answer is wrong, the learned Thinker’s reward maximizers contain a correct candidate 27% of the time versus 18% for the initial representation. This is aggregate evidence; we do not include the requested response-level worked example, so that request is only partially answered.

The subsets may overlap. “Plurality” means the largest answer class and need not exceed 50%. Top-1 averages the correct fraction among tied reward maximizers. Ranking macro-averages correct/incorrect pair orderings within each mixed group, counting ties as 0.5. The rates are conditional on a correct candidate being present, and a random-selection Top-1 is not reported.

Audit annotations are post-hoc and never enter training. References are supported by human review and executable checks of problems/reference solutions; this is not direct executable verification of every candidate. We therefore claim only improved ranking on these groups, not independently held-out curriculum generalization.

2. What representation agreement changes. For m>1m>1, normalized response representations uiu_i, and uˉ=m−1∑juj\bar u=m^{-1}\sum_j u_j, Eq. 13 is equivalent to

=\frac{m}{2(m-1)}(1-u_i^\top\bar u).$$ The reward therefore ranks alignment with a learned centroid. It remains a centrality signal, as the reviewer notes, but it uses question-conditioned reasoning representations rather than binary agreement with a final-answer label. It can assign different rewards to traces sharing an answer. Whether the learned geometry improves correctness ranking is empirical; the audit and frozen control test this question. Table 3’s iteration-1 Solver reward gap is 72.4−25.7 = 46.7 points. Earlier calibration and attenuated later updates can explain why the later Thinker gap exceeds the current pseudo-label gap, but do not prove which features are retained. At iteration 1, Thinker means are only 49.7/47.6; this weak signal does not affect the first Solver update because $w_0=1$. **3. Learning and transition controls.** Qwen3-4B-Base results use 12 iterations, three training seeds, and the same Q/S configuration, question budget, and evaluation. | Setting | Math Avg | General Avg | | ------------------------------ | -------: | ------------: | | R-Zero | 51.72 | 31.48 ± 0.83 | | Full Zero-L | 54.66 | 33.31 ± 0.75 | | Frozen-init Thinker + IDR | 52.8 | 31.8 | | Thinker-only after iteration 1 | 53.2 | 32.0 | Frozen-init preserves the pretrained backbone, pooling, mapping, and IDR while disabling Thinker updates. Full is higher by 1.86/1.51 Math/General points, supporting a learning contribution in this 4B configuration. The early-switch control uses majority reward only in iteration 1 and pure Thinker reward thereafter, while retaining calibration and residual Thinker updates; Full is higher by 1.46/1.31 points. **Illustrative mechanism.** A group for the problem $x+x^{-1}=3$, asking for $x^5+x^{-5}$. Among ten responses, the largest final-answer class is the incorrect answer 243 with three responses, while the correct answer 123 appears twice. The three 243 responses reach the same answer through different errors: an invalid power shortcut, premature numerical approximation, and incorrect cancellation of cross terms. In contrast, the two correct responses use the valid symmetric-power recurrence or the equivalent binomial identity. Several other responses also use these valid structures but contain isolated arithmetic errors. Because the Thinker encodes the reasoning trace rather than the boxed answer, a learned representation can place a correct recurrence-based trace closer to the group’s structural centroid. A correct trace receive $R^{TS}=0.755$, compared with $0.383$ for the highest-scoring 243 trace. At iteration 6, where $w_n\approx0.082$, their IDR rewards are approximately 0.693 and 0.434, respectively. Thus, the Thinker’s reward maximizer is correct even though the largest answer class and fixed pseudo-label are wrong.

The rebuttal still does not resolve the two concerns I listed in my original review. I would maintain my original scores with the current author inputs.

Thank you for raising these concerns. We address the two weaknesses below.

Weakness 1: Why the Thinker improves long-horizon self-evolution

The improvement brought by the Thinker follows directly from our iteration-dependent Solver-training reward (IDR) and residual Thinker update. In the early outer iterations, the pseudo-label-based Solver reward is relatively reliable. We therefore use it both as the main reward for Solver optimization and as the calibration signal that teaches the Thinker which structural patterns in a solution process are associated with reliable pseudo-label agreement. As self-evolution proceeds and the reliability of majority-vote pseudo-labels declines, the IDR mixed reward gradually shifts the dominant Solver-training signal from the Solver reward to the Thinker-Solver reward. At the same time, the residual Thinker update progressively attenuates later updates, preventing unreliable late-stage pseudo-labels from substantially changing the Thinker. This mechanism is described in Lines 87-107 and 134-140, and formalized in Lines 243-280. In particular, by approximately Iteration 5, the Thinker is updated only marginally because the reliability of the pseudo-label-based Solver reward declines substantially after Iteration 3.

We would also like to clarify a possible misunderstanding regarding the shared-wrong-reasoning case. Zero-L does not claim to eliminate correlated errors shared by multiple solution processes. Such shared errors are a fundamental limitation of fully self-evolving systems and part of the trade-off for reducing reliance on externally labeled data; they are not the specific failure mode that our paper claims to solve. Our central goal is to mitigate the long-horizon collapse caused by the progressive degradation of pseudo-label reliability in later outer iterations. The sustained performance improvements of Zero-L arise from the Thinker's long-horizon role in this transition. We therefore respectfully ask the reviewer to assess the method against this stated scope and to reconsider the rating in light of this clarification.

Weakness 2: Additional experimental comparison

We provide additional comparisons with R-Few (5%), R-Diverse, and TTCS below.

MethodMath Avg.GSM8KAMCAIME25General Avg.MMLU-ProSuperGPQABBEH
Zero-L54.992.8 ± 0.659.6 ± 0.312.4 ± 1.433.559.6 ± 0.829.3 ± 0.711.7 ± 1.0
R-Few (5%)51.992.2 ± 0.552.9 ± 0.710.7 ± 0.632.556.3 ± 0.929.6 ± 0.411.6 ± 1.2
R-Diverse54.292.1 ± 0.359.4 ± 0.711.1 ± 1.231.756.1 ± 1.228.2 ± 0.310.7 ± 1.6
TTCS57.193.1 ± 0.460.2 ± 0.617.9 ± 1.130.655.2 ± 0.727.1 ± 0.39.6 ± 0.6

The results show that Zero-L outperforms R-Few (5%) and R-Diverse on both the Math and General averages. Although Zero-L is slightly behind TTCS on mathematical reasoning, it substantially outperforms TTCS on general reasoning. This result demonstrates that Zero-L provides a more balanced improvement across mathematical and general reasoning tasks.

The result and more claim will be added into the camera-ready version if accepted, thank you for your rating!

Reviewer_JP61OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Zero-L extends R-Zero self-training with a third model, the Thinker, that rewards answers close to other responses in embedding space. An IDR schedule shifts the Solver from majority-vote rewards to this signal. Across 12 iterations on Qwen3-4B/8B, it finishes 2–3 points ahead of matched R-Zero runs.

Critical strengths

The authors compare both methods over 12 rounds at two model sizes. Zero-L does slightly better overall. Table 3 also suggests majority voting becomes less reliable after round 6.

Critical weaknesses

The paper does not show that the Thinker causes the improvement. It learns from majority votes and barely changes after round 5, yet Table 3 claims it separates right and wrong answers by 45 points when its training signal differs by less than 4. There is no experiment removing or freezing it. A fixed 50/50 reward mix even does worse than R-Zero on math.

The evaluation is also unclear. These generated questions have no ground-truth answers, so what did the judge use? The R-Zero row appears to come from a Zero-L run. The authors average three seeds but give no standard deviations. One BBEH baseline changes by over 3 points across checkpoints, larger than the reported gains.

Questions for authors
  1. What did the GPT-5.5 judge compare answers against for the self-generated questions in Table 3, and were all three rows measured inside the Zero-L run?
  2. Please give standard deviations over the three seeds for Tables 1, 2 and 4.
  3. I think we need a new experiment here. Run the same schedule with the Thinker frozen at the base model, and a variant using only the Thinker reward after the first iteration.
2 comments

Thank you!

Q1. Judge inputs, references, and the Table 3 response pool. Appendix A.2 (PDF p.13) gives the GPT-5.5 benchmark-checking template: candidate {answer}, reference {response}, and a Yes/No output. Table 3 instead audits self-generated training questions, whose references are obtained through LLM problem solving. Because the paper does not document that reference-construction procedure, we treat Table 3 as a post-hoc LLM-annotated diagnostic, not ground-truth supervision. No reference or annotation enters Q/S/Thinker training.

All Table 3 signals are computed on the same audited Zero-L Solver responses. Appendix A.4 samples 500 question–answer groups per reported iteration and evaluates each reward on the correct/incorrect sets C/IC/I. Thus, “Solver reward (R-Zero)” means the majority-vote reward rule applied to Zero-L rollouts; it is not a separately trained R-Zero response pool. We will relabel it “majority-vote reward on Zero-L rollouts.” Tables 1–2 contain the independently trained R-Zero/Zero-L comparison.

The additional audit likewise evaluates all scorers on identical Zero-L groups. It retains 1,350 groups—150 human-reviewed and 1,200 with executable checks of the problem and reference solution—and excludes 150 ambiguous/unverifiable groups. These checks do not constitute executable verification of every candidate answer.

Q2. Available seed variation and BBEH. The available sample SDs cover iteration-12 General results across three runs (percentage points). General Avg SD is computed from the three per-run benchmark averages.

Model/methodGeneral Avg SDMMLU-Pro SDSuperGPQA SDBBEH SD
4B R-Zero0.830.450.461.67
4B Zero-L0.750.670.441.56
8B R-Zero0.650.880.471.97
8B Zero-L0.670.630.421.46

This only partially answers the request: Math, other checkpoints, and Table 4 SDs are not supplied. The 8B R-Zero BBEH change, 12.28→8.92, is across checkpoints, whereas SD measures variation at one checkpoint. Excluding BBEH, the iteration-12 MMLU-Pro/SuperGPQA average still favors Zero-L by 2.09 points (4B) and 1.65 points (8B).

Q3. Frozen-init and Thinker-only controls. The following Qwen3-4B-Base controls use 12 iterations, three training seeds, and the same Q/S configuration, candidate-question budget, and evaluation protocol.

SettingMath AvgGeneral Avg
R-Zero51.7231.48 ± 0.83
Full Zero-L54.6633.31 ± 0.75
Frozen-init Thinker + IDR52.831.8
Thinker-only after iteration 153.232.0

Frozen-init keeps the pretrained Thinker, pooling, reward mapping, and IDR, but disables Thinker updates. Full is higher by 1.86/1.51 Math/General points, supporting a learning contribution in this 4B setting. The early-switch control uses majority reward in iteration 1 and pure Thinker reward thereafter; Thinker calibration and residual updates remain enabled. Full is higher by 1.46/1.31 points. Table 4’s fixed 50/50 mixture reaches 50.62/30.34. These comparisons test the stated controls, not every transition policy.

Reward-gap evidence. At iteration 1, the Solver reward gap is 72.4−25.7 = 46.7 points. Historical calibration can therefore produce a later Thinker gap larger than the current pseudo-label gap, although this is an explanation rather than proof of the learned features. The initial Thinker means, 49.7/47.6, do not affect the first Solver update because w0=1w_0=1.

On 300 groups with a wrong fixed pseudo-label and a correct candidate, learned/initial Thinker ranking is 60%/52% and Top-1 is 25%/18%. On 350 groups with a wrong fresh plurality and a correct candidate, Top-1 is 27%/18%. Ranking macro-averages correct/incorrect pairs within each mixed group (ties = 0.5); Top-1 credits the correct fraction among tied maximizers. The subsets may overlap. These conditional results show improvement over the initial scorer, but no random-selection Top-1 is reported and the audit is not an independently held-out curriculum test.

Compute control. Supplementary 4B profiling on two RTX PRO 6000 GPUs gives 5.3/4.7 GPU-hours per iteration for Zero-L/R-Zero (+12.77%), implying 63.6/56.4 GPU-hours over 12 rounds. Under a 63.6-GPU-hour budget, R-Zero reaches 53.6/32.3 versus Zero-L’s 54.66/33.31 (+1.06/+1.01). Average peak memory is 83.7/82.9 GB per card. This profiling is separate from the original four-GPU setup; the exact stopping checkpoint, timing breakdown, and control SDs are unavailable, so we report only the mean comparison.

We provide an additional experimental comparison below. For each method, we evaluate the checkpoints across 12 outer iterations and report its best-performing checkpoint.

MethodMath Avg.GSM8KAMCAIME25General Avg.MMLU-ProSuperGPQABBEH
Zero-L54.992.8 ± 0.659.6 ± 0.312.4 ± 1.433.559.6 ± 0.829.3 ± 0.711.7 ± 1.0
R-Few (5%)51.992.2 ± 0.552.9 ± 0.710.7 ± 0.632.556.3 ± 0.929.6 ± 0.411.6 ± 1.2
R-Diverse54.292.1 ± 0.359.4 ± 0.711.1 ± 1.231.756.1 ± 1.228.2 ± 0.310.7 ± 1.6
TTCS57.193.1 ± 0.460.2 ± 0.617.9 ± 1.130.655.2 ± 0.727.1 ± 0.39.6 ± 0.6

Zero-L clearly outperforms R-Few (5%) and R-Diverse on both the Math and General averages. While it is slightly behind TTCS on mathematical reasoning, it substantially outperforms TTCS on general reasoning, indicating that Zero-L produces a more balanced and comprehensive improvement.

Zero-L is not intended to eliminate the shared-wrong-reasoning failure mode, which remains a fundamental challenge for fully self-evolving systems without external supervision. Instead, Zero-L addresses long-horizon collapse by using reliable early-iteration majority-vote pseudo-labels to calibrate the Thinker, using the IDR mixed reward to gradually shift Solver optimization toward the Thinker-Solver reward, and attenuating later updates through the residual Thinker update as pseudo-label reliability declines. This mechanism and its motivation are detailed in Lines 87-107, 134-140, and 243-280. We respectfully ask the reviewer to consider this clarification and the additional results when reassessing the rating.

The result and more claim will be added into the camera-ready version if accepted, thank you for your rating!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.