acceptodds
Under review as a conference paper at ICLR 2027

J-Zero: Unified Challenger–Solver–Judge Self-Evolution from Zero Data

Abstract

Self-evolving language models have recently emerged as a promising path toward superintelligence, with the advantage of reducing the cost of human supervision. While considerable progress has been made in verifiable domains, self-evolution in unverifiable domains remains less explored. We propose udge co-adaptation from data (), a unified Challenger–Solver–Judge self-evolution framework that supports self-improvement across both domains. The Challenger and Solver co-evolve through an adversarial interaction: the Challenger generates increasingly difficult tasks, while the Solver learns to produce higher-quality responses to them. We augment this Challenger–Solver setup with a co-adapting Judge, trained using preference pairs whose ordering is known in advance from how each response was produced, , the Solver's answer over the Challenger's, and the Solver's decomposed-and-recombined answer over its one-shot answer, rather than from the Judge's own scores. J-Zero outperforms the baselines by an average of 4.2 points on verifiable and 8.0 points on unverifiable domains, and continues to improve through at least ten iterations, whereas the baselines degrade after two. Further analysis identifies Judge co-adaptation as the key driver of this sustained improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Self-evolving, Self-improvement, Self-play, Continual Learning, Superintelligence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LkBMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes co-adapting the judge model alongside the challenger and solver in LLM self-play. In particular, the judge is trained together with the challenger and solver, using preference pairs whose order is determined by the source of the responses. The solver's responses are preferred over the challenger's. Responses from a more sophisticated pipeline are preferred over the Solver's vanilla responses. Finally, the challenger and solver are trained via adversarial GRPO, and the judge is trained via preference learning. All roles are trained in an interleaved manner, and each role is trained for several steps before switching to other roles. Experiment results on both verifiable and non-verifiable domains show the effectiveness of their method

Critical strengths
  • The paper identifies a meaningful bottleneck in self-play: the reliability of evaluation in the absence of strong supervision. This arises naturally: in self-play, the generation-verification gap is small, and with a frozen judge model, training easily saturates to the judge model's internal reward model. This paper addresses it by constructing preference training data, using different prompting conditions and trained weights.
  • The evaluation is extensive, covering both verifiable and non-verifiable domains, while prior work mostly focuses only on one of them.
Critical weaknesses
  • Training seems to be validated directly against the test benchmarks. In Section 4.1, the author states that they stop training once the average score on the verifiable or non-verifiable tasks begins to drop. This raises concerns about the generalizability of the performance of their selected checkpoints.
  • The preference learning framework for the judge is flawed both theoretically and empirically. The role asymmetry assumes that the Solver's responses are of higher quality than those from the Challenger, because it is trained to answer the questions correctly. However, this is not true at the start of training. This can, in turn, result in a judge model that is cold-started with wrong training data and misleads Challenger and Solver, causing ribbling effects. This is also reflected empirically: According to Section 5.1, the accuracy of the preference directions from role asymmetry is capped at 87.9% and continues to decrease as training progresses. The observed accuracy is quite low and thus the preference learning for the judge is quite noisy.
Questions for authors
  • Are the best checkpoints selected using the scores from the test benchmarks?
  • How many repeated runs are there for your method and the baselines, and what is the statistical significance of the reported results?
4 comments
AuthorsOpenReview ↗

Official Comment to Reviewer LkBM (1/2)

We thank the reviewer for the careful review and constructive comments. We appreciate the recognition that our work identifies a meaningful bottleneck in self-play: training easily plateaus at the performance level of the frozen Judge model. We are also grateful for the acknowledgment of our extensive evaluation across both verifiable and unverifiable domains.


Weakness 1. Training seems to be validated directly against the test benchmarks. In Section 4.1, the author states that they stop training once the average score on the verifiable or non-verifiable tasks begins to drop. This raises concerns about the generalizability of the performance of their selected checkpoints.

Question 1. Are the best checkpoints selected using the scores from the test benchmarks?

Thank you for raising this concern. We would like to clarify that we did not select the best-performing checkpoints based on test benchmark scores. In principle, we followed the evaluation protocol of the baselines. In practice, it does not affect J-Zero as it improves monotonically through iteration 10 (our iteration budget), so its reported results are from the final iteration and involve no checkpoint selection.


Weakness 2. The preference learning framework for the Judge is flawed both theoretically and empirically. The role asymmetry assumes that the Solver’s responses are of higher quality than those from the Challenger, because it is trained to answer the questions correctly. However, this is not true at the start of training. This can, in turn, result in a Judge model that is cold-started with wrong training data and misleads Challenger and Solver, causing ribbling effects. This is also reflected empirically: According to Section 5.1, the accuracy of the preference directions from role asymmetry is capped at 87.9% and continues to decrease as training progresses. The observed accuracy is quite low and thus the preference learning for the Judge is quite noisy.

We agree that the preference data construction for Judge co-evolving does not guarantee a correct preference for every pair. However, their primary purpose is to prevent the Judge's learning from being limited by the Solver's capabilities. While role-asymmetry pairs provide stable supervision across iterations, their preference signals are inherently bounded by the Solver's performance, as the chosen and rejected responses originate from the Solver and Challenger, respectively. To sustain self-evolution, the Judge must develop evaluation capabilities beyond those of the current Solver. Subtask-amplification pairs become particularly important in later stages, when the benefits of role-asymmetry supervision begin to saturate. Thus, the two sources play complementary roles: role-asymmetry provides reliable early supervision, while subtask-amplification supports continued Judge improvement.

Unlike conventional datasets, preference datasets inherently involve subjective annotations, making perfectly accurate labels difficult to obtain. For instance, HH-RLHF reports only 63% agreement among human annotators [1], yet remains one of the most widely used preference datasets. Prior studies have also shown that reward models can be effectively trained with weak supervision [2, 3]. From this perspective, although role-asymmetry and subtask-amplification pairs do not provide perfectly accurate preference labels, they can still offer useful supervision for Judge training. The strong results in Tables 1–2 are consistent with the effectiveness of these complementary preference sources. We will add this discussion in the revised manuscript.

[1] Bai, Yuntao, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv 2022.

[2] Stiennon et al., Learning to summarize from human feedback, NeurIPS 2020.

[3] Ouyang et al., Training language models to follow instructions with human feedback, NeurIPS 2022.

AuthorsOpenReview ↗

Official Comment to Reviewer LkBM (2/2)

Question 2. How many repeated runs are there for your method and the baselines, and what is the statistical significance of the reported results?

All of our experiments were run once with a fixed seed. To further investigate statistical significance, we additionally conducted training runs and evaluations for each method on Qwen3-4B-Base, repeating the entire process three times with different seeds.

MethodMath Reasoning Avg.General Reasoning Avg.IFEval Avg.Verifiable Avg.
R-Zero47.73±0.7649.71±0.1852.00±1.0649.81±0.15
G-Zero45.73±0.1047.04±0.8150.88±1.3047.88±0.69
J-Zero51.52±0.3153.30±0.3657.29±1.4854.04±0.29
J-Zero-frozen50.73±1.7250.80±0.3654.07±0.9551.87±0.91
MethodAlpacaEvalArena-Hard (H.P.)Arena-Hard (C.W.)EQ-Bench C.W.Unverifiable Avg.
R-Zero11.20±0.493.27±0.711.23±0.3124.00±0.7712.48±0.38
G-Zero11.27±2.243.20±0.531.30±0.1021.84±0.5811.79±0.89
J-Zero28.27±0.724.47±0.582.30±0.1729.45±0.7920.37±0.46
J-Zero-frozen23.31±5.613.97±0.291.47±0.2126.61±0.2917.54±1.92

As shown in the tables, J-Zero consistently achieves the highest mean performance across both domains, with improvements substantially exceeding the observed standard deviation. We will include this result and discussion in the revised manuscript.

Dear Reviewer LkBM,

We sincerely appreciate your time and effort in reviewing our work and providing valuable feedback.

We have carefully addressed your concerns through detailed responses and additional experiments, and have updated our manuscript accordingly. We would be grateful if you could take another look at our rebuttal and share any further feedback or remaining concerns. We would be happy to provide additional clarification if needed.

Thank you again for your time and consideration.

Best, The Authors

Dear Reviewer LkBM,

We have made substantial efforts to address your concerns, including conducting additional experiments and providing detailed responses.

With only 24 hours remaining in the author–reviewer discussion period, we would greatly appreciate it if you could review our responses and reconsider your rating if you find that your concerns have been adequately addressed.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_7BebOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes J-ZERO, a zero-data self-evolution framework in which a Challenger, Solver, and Judge are jointly adapted through iterative self-play. The Challenger is trained to generate tasks that are difficult for the current Solver, while the Solver is trained with GRPO to improve on the selected tasks. The main addition over Challenger-Solver self-play is that the Judge is also updated rather than kept fixed.

To train the Judge without using the Judge's own scores as preference labels, the paper constructs two types of in-loop preference pairs. Role-asymmetry pairs prefer the Solver's response over the Challenger's response to the same task, while subtask-amplification pairs prefer a decomposed-and-recombined response over the Solver's one-shot response. The Judge is then trained using a Bradley-Terry preference loss.

The method is evaluated on Qwen3-4B-Base and Qwen3-8B-Base across verifiable and unverifiable benchmarks. J-ZERO improves the reported aggregate performance over R-Zero and G-Zero in both settings. The paper further studies the reliability of the self-generated preference pairs , ablates Judge adaptation and the two preference sources, and reports that the full method continues improving for more iterations than the frozen-Judge variants and baselines. Additional experiments test a Llama-3.2-3B-Instruct policy and a smaller Judge.

Critical strengths
  1. The paper addresses an important bottleneck in zero-data self-evolution with a concrete and well-motivated mechanism. Existing Challenger-Solver self-play can become limited when the evaluation signal is fixed. J-ZERO turns the Judge into an adaptive component and integrates Challenger, Solver, and Judge updates in a unified loop. This is particularly relevant for unverifiable/open-ended tasks, where an external deterministic verifier is unavailable.

  2. The paper provides reasonably strong empirical evidence that Judge adaptation contributes to the observed improvement. The frozen-Judge variant underperforms the full method, and removing either preference source also reduces performance . In addition, Fig. 3 shows that the full method continues improving after the frozen-Judge variant begins to plateau. I find this combination of ablation and training-trajectory evidence more convincing than an end-point comparison alone.

Critical weaknesses
  1. The reliability assumption behind the Judge-training preferences is not fully supported by the paper’s own analysis. The method treats role-asymmetry and subtask-amplification pairs as reliable preference supervision (Sec. 3.2), but Fig. 2 shows that the subtask-amplification label is wrong in the majority of cases during the first three iterations (21.1% win rate at iteration 1). Nevertheless, the two preference sources are mixed equally throughout training (Appendix A.1). The paper does not provide an ablation showing whether delaying or filtering the unreliable amplification pairs would change the results.
  2. The methodological novelty is not sufficiently isolated from a combination of existing self-evolution components. The Challenger–Solver loop is closely related to prior zero-data self-play, while GRPO, Bradley–Terry reward-model training, and iterative improvement of judging capability are established techniques. The main new elements appear to be the specific Judge coupling and the two constructed preference sources. However, the paper only provides ablations within J-ZERO in Table 3, and does not compare against a controlled baseline that augments an existing Challenger–Solver framework with a simpler adaptive Judge under comparable training conditions. Consequently, it is unclear how much of the gain comes from the proposed J-ZERO mechanism itself rather than from combining already established components.
Questions for authors
  1. How should the large gains on the Qwen Base models be interpreted relative to the much smaller gains on the instruction-tuned model? In Table 5, J-ZERO improves the Llama-3.2-3B-Instruct unverifiable average only from 16.97 to 17.32, and only by 0.27 over R-Zero. Do the authors have results on a comparably sized instruction-tuned model that support the stronger open-ended improvements reported in Tables 1–2?
6 comments
AuthorsOpenReview ↗

Official Comment to Reviewer 7Beb (1/3)

We thank the reviewer for the thoughtful review and constructive feedback. We appreciate the recognition that J-Zero provides a concrete and well-motivated approach to an important bottleneck in zero-data self-evolution, particularly for open-ended tasks. We are also grateful for the acknowledgment that our ablations and training trajectories together support the contribution of Judge adaptation to the observed improvements.


Weakness 1. The reliability assumption behind the Judge-training preferences is not fully supported by the paper’s own analysis. The method treats role-asymmetry and subtask-amplification pairs as reliable preference supervision, but Fig. 2 shows that the subtask-amplification label is wrong in the majority of cases during the first three iterations. Nevertheless, the two preference sources are mixed equally throughout training. The paper does not provide an ablation showing whether delaying or filtering the unreliable amplification pairs would change the results.

We thank the reviewer for this comment. Following the reviewer's suggestion, we conducted two additional experiments to investigate the impact of unreliable preference labels: (1) reliability-aware scheduling of preference pairs and (2) excluding subtask-amplification pairs during early training.

(1) Reliability-aware scheduling. We adjusted the proportion of role-asymmetry and subtask-amplification pairs based on their relative reliabilities (87.9% vs. 21.1%), starting with an 80:20 ratio and linearly transitioning to 50:50 by iteration 7. As shown below, the comparable performance suggests that J-Zero is relatively robust to noisy preference labels during early training.

MethodVerifiableUnverifiableOverall
J-Zero54.3820.8137.59
J-Zero-Scheduling54.2920.5037.39

(2) Excluding early subtask-amplification pairs. To further examine their impact, we trained a variant using only role-asymmetry pairs for the first four Judge updates, restoring the original mixture thereafter.

MethodVerifiableUnverifiableOverall
J-Zero54.3820.8137.59
J-Zero-role-only-until-iter452.9421.5337.24

Interestingly, at iteration 5, this variant achieves higher Solver performance on unverifiable tasks than the original J-Zero (19.92 vs. 18.41). However, upon reintroducing subtask-amplification pairs, the Judge's mean training loss sharply increases from 0.14 at iteration 4 to 8.16 at iteration 5, indicating substantial optimization instability. This instability appears to hinder sustainable Judge co-evolution, with the overall self-evolution process saturating by iteration 8. We will include this result and discussion in the revised manuscript.

AuthorsOpenReview ↗

Official Comment to Reviewer 7Beb (2/3)

Weakness 2. The methodological novelty is not sufficiently isolated from a combination of existing self-evolution components. The Challenger–Solver loop is closely related to prior zero-data self-play, while GRPO, BT RM training, and iterative improvement of judging capability are established techniques. The main new elements appear to be the specific Judge coupling and the two constructed preference sources. However, the paper only provides ablations within J-Zero in Table 3, and does not compare against a controlled baseline that augments an existing Challenger-Solver framework with a simpler adaptive Judge under comparable training conditions. Consequently, it is unclear how much of the gain comes from the proposed J-Zero mechanism itself rather than from combining already established components.

To demonstrate the effectiveness of our two proposed preference sources, role-asymmetry and subtask-amplification, we compared J-Zero against R-Zero and G-Zero augmented with an adaptive Judge using their respective preference construction strategies. For R-Zero, we treated the majority-voted response as chosen and the remaining responses as rejected. For G-Zero, we used responses generated with and without hints as chosen and rejected, respectively. As shown in the table below, J-Zero achieves the best performance, highlighting the effectiveness of our proposed preference pair construction beyond simply introducing an adaptive Judge.

MethodMath Reasoning Avg.General Reasoning Avg.IFEval Avg.Verifiable Avg.
J-Zero51.2052.9358.9954.38
J-Zero - (R-Zero style pair)47.8349.7851.3049.64
MethodVerifiableUnverifiableOverall
J-Zero54.3820.8137.59
J-Zero - (R-Zero style pair)49.64--
J-Zero - (G-Zero style pair)51.0518.7734.91

First, to the best of our knowledge, J-Zero is the first framework to introduce Judge co-evolution into adversarial Challenger–Solver self-training. We emphasize that achieving sustainable Challenger–Solver–Judge co-evolution in a fully closed-loop setting without external data or supervision is a non-trivial challenge. Our results further demonstrate that the proposed preference pair construction, relying solely on Challenger and Solver outputs, plays a crucial role in sustaining self-improvement, rather than merely combining existing self-evolution components. We will include this result and discussion in the revised manuscript.

AuthorsOpenReview ↗

Official Comment to Reviewer 7Beb (3/3)

Question 1. How should the large gains on the Qwen Base models be interpreted relative to the much smaller gains on the instruction-tuned model? In Table 5, J-Zero improves the Llama-3.2-3B-Instruct unverifiable average only from 16.97 to 17.32, and only by 0.27 over R-Zero. Do the authors have results on a comparably sized instruction-tuned model that support the stronger open-ended improvements reported in Tables 1~2?

We thank the reviewer for this question. To further investigate the effectiveness of J-Zero on instruction-tuned models, we conducted additional experiments using Llama-3.1-8B-Instruct and OLMo-3-7B-Instruct-SFT. As shown below, J-Zero outperforms the strongest baseline by 0.56 and 0.97 points on unverifiable tasks, respectively. As the reviewer pointed out, the gains on instruction-tuned models are smaller than those on Qwen Base models reported in Tables 1–2 of the manuscript. This trend is also observed in R-Zero and G-Zero, suggesting a broader challenge for zero-data self-play rather than a limitation specific to J-Zero.

MethodVerifiableUnverifiableOverall
Llama-3.1-8B-Instruct48.3518.8533.60
R-Zero48.9919.4034.19
G-Zero48.1919.2833.73
J-Zero49.0419.9634.50
MethodVerifiableUnverifiableOverall
Olmo-3-7B-Instruct-SFT55.3321.8338.58
R-Zero55.5221.4738.49
G-Zero55.3121.2038.26
J-Zero56.8022.4439.62

One potential explanation is that instruction tuning substantially narrows the Challenger's question distribution, limiting the diversity of supervision available to the Solver. To investigate this, we sampled 2,000 questions from Qwen3-4B-Base and Qwen3-4B-Instruct-2507 using the same instruction prompt (Appendix D.1). Using Qwen3-Embedding-4B, the instruction-tuned model achieves a substantially lower Vendi score of 27.6 [24.2, 30.7], compared to 61.2 [57.4, 65.1] for the Base model, evaluated over groups of 128 questions. Notably, all 2,000 questions generated by the instruction-tuned model begin with the same phrase, and increasing the sampling temperature to 1.6 does not fully close the diversity gap (Vendi score: 48.3). These findings suggest that reduced generation diversity may constrain the effectiveness of self-evolution from instruction-tuned models.

Although the improvements on instruction-tuned models are relatively modest, the substantial gains on base models remain valuable, highlighting the potential of zero-data self-evolution to unlock pretrained capabilities without external supervision. Looking forward, as pretraining recipes and datasets continue to advance, increasingly capable pretrained models will require effective post-training supervision to fully leverage their knowledge on complex tasks. However, obtaining reliable human annotations for such tasks may become increasingly challenging. In this regard, we believe that zero-data self-play offers a promising direction for scalable post-training without reliance on human annotations, potentially contributing to the development of increasingly capable AI systems. We will include this result and discussion in the revised manuscript.

Reviewer_RNocOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

J-Zero jointly evolves a Challenger, Solver, and Judge without additional external training data. The Challenger and Solver are updated through GRPO, while the Judge learns from two assumed preference relations, Solver responses over Challenger responses and decomposed responses over direct responses. Experiments on Qwen3-4B-Base and Qwen3-8B-Base report improvements across verifiable and open-ended tasks and continued gains through ten iterations.

Critical strengths
  1. The evaluation covers 11 verifiable and three unverifiable benchmarks at two model scales, with additional experiments on Llama-3.2-3B-Instruct and a smaller Judge. This breadth provides useful evidence beyond a single task or configuration.

  2. The frozen Judge comparison and preference source ablations support the value of Judge adaptation within J-Zero. The reported ten-round trajectory is encouraging, and the analysis of preference reliability exposes meaningful differences between early and late training.

Critical weaknesses
  1. Preference reliability remains a concern. Role preference agreement falls from 87.9% to about 66%, while amplification agreement is only 21.1% in the first round. The effects of these noisy labels and safeguards against preference reversals are unclear.

  2. J-Zero shares G-Zero's core idea of learning from automatically constructed preference pairs. Its preference orderings lack objective guarantees, while the separate Judge introduces additional training and inference costs. The benefit over direct preference learning remains unclear.

Questions for authors

1.Beyond replacing G-Zero's hint preferences with role and subtask assumptions, what is the substantive novelty? How much extra training and inference compute does the Judge require, and are the gains worth this cost?

  1. Does preference validation with a strong Qwen evaluator support the Claude-based findings? Does J-Zero also improve strong, instruction-tuned models, or does it mainly help small Base models?

  2. How much do early unreliable amplification labels harm training? How does the method handle declining role preference reliability or a reversal of the assumed ordering?

  3. Figure 2 excludes ties, but Appendix D.3 requires a forced choice. How are ties defined?

7 comments
AuthorsOpenReview ↗

Official Comment to Reviewer RNoc (1/3)

We thank the reviewer for the careful review and constructive feedback. We appreciate the recognition of our broad evaluation across benchmarks and model scales and of the ablation evidence supporting the value of Judge adaptation. We are also grateful that the reviewer viewed the training trajectory as encouraging and recognized the insights from our analysis of how preference reliability changes during training.


Weakness 1. Preference reliability remains a concern. Role preference agreement falls from 87.9% to about 66%, while amplification agreement is only 21.1% in the first round. The effects of these noisy labels and safeguards against preference reversals are unclear.

We agree that the preference data construction for Judge co-evolving does not guarantee a correct preference for every pair. However, their primary purpose is to prevent the Judge's learning from being limited by the Solver's capabilities. While role-asymmetry pairs provide stable supervision across iterations, their preference signals are inherently bounded by the Solver's performance, as the chosen and rejected responses originate from the Solver and Challenger, respectively. To sustain self-evolution, the Judge must develop evaluation capabilities beyond those of the current Solver. Subtask-amplification pairs become particularly important in later stages, when the benefits of role-asymmetry supervision begin to saturate. Thus, the two sources play complementary roles: role-asymmetry provides reliable early supervision, while subtask-amplification supports continued Judge improvement.

Unlike conventional datasets, preference datasets inherently involve subjective annotations, making perfectly accurate labels difficult to obtain. For instance, HH-RLHF reports only 63% agreement among human annotators [1], yet remains one of the most widely used preference datasets. Prior studies have also shown that reward models can be effectively trained with weak supervision [2, 3]. From this perspective, although role-asymmetry and subtask-amplification pairs do not provide perfectly accurate preference labels, they can still offer useful supervision for Judge training. The strong results in Tables 1–2 are consistent with the effectiveness of these complementary preference sources. We will add this discussion in the revised manuscript.

[1] Bai, Yuntao, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv 2022.

[2] Stiennon et al., Learning to summarize from human feedback, NeurIPS 2020.

[3] Ouyang et al., Training language models to follow instructions with human feedback, NeurIPS 2022.


Weakness 2. J-Zero shares G-Zero’s core idea of learning from automatically constructed preference pairs. Its preference orderings lack objective guarantees, while the separate Judge introduces additional training and inference costs. The benefit over direct preference learning remains unclear.

By incorporating the Judge, we establish a unified evaluation criterion for adversarial training between the Challenger and Solver. Specifically, the Challenger is optimized to generate tasks on which the Solver performs poorly (i.e., receives low rewards). As the reviewer suggested, one could technically adopt a G-Zero-style approach, where the Challenger is trained with a separately designed objective to generate harder tasks, while the Solver is optimized via DPO on generated preference pairs. However, we believe that directly optimizing the Challenger to minimize the Solver's reward provides a more coherent adversarial objective under a shared evaluation criterion, thereby facilitating more stable training throughout the self-evolution process.

To empirically validate the benefit of this design, we conducted additional experiments comparing J-Zero with three variants: (1) freezing the Judge, (2) replacing GRPO with DPO for Solver training while freezing the Judge, and (3) removing the Judge and training the Solver with DPO.

MethodVerifiableUnverifiableOverall
J-Zero54.3820.8137.59
J-Zero- Frozen-Judge52.7216.3734.54
J-Zero- Frozen-Judge-Solver-DPO50.1418.2034.17
J-Zero- without-Judge-Solver-DPO42.989.6826.34

As shown above, J-Zero consistently achieves the best performance, demonstrating the empirical benefit of jointly optimizing the Challenger, Solver, and adaptive Judge over the evaluated alternatives. We acknowledge that maintaining an adaptive Judge introduces additional training and inference costs (13.8% more GPU-hours per iteration than G-Zero). Nevertheless, we believe this computational trade-off is justified by the stronger performance gains, particularly given that sustained capability improvement is a central objective of self-evolving language models. We will include this result and discussion in the revised manuscript.

AuthorsOpenReview ↗

Official Comment to Reviewer RNoc (2/3)

Question 1. Beyond replacing G-Zero’s hint preferences with role and subtask assumptions, what is the substantive novelty? How much extra training and inference compute does the Judge require, and are the gains worth this cost?

According to the conference guidelines, papers published within two months of the full-paper deadline are considered contemporaneous, and comparisons with arXiv-only papers are not required (https://iclr.cc/Conferences/2027/ReviewerGuidelines). We therefore regard G-Zero as concurrent work. Nevertheless, we provide a detailed comparison to clarify the similarities and key differences between G-Zero and J-Zero.

G-Zero and J-Zero share the same overarching goal: extending zero-data self-evolution beyond verifiable domains to unverifiable domains, which have been largely overlooked by prior self-evolution frameworks. As illustrated in Figure 1 of G-Zero, unverifiable tasks can be evaluated using a reward model (Judge). However, recognizing that the Judge's limited evaluation capability may become a bottleneck for self-evolution, G-Zero adopts a Judge-free approach. In contrast, J-Zero addresses the same fundamental limitation by continuously improving the Judge rather than eliminating it.

Both approaches have distinct advantages and trade-offs. G-Zero is computationally lighter, as it does not require a Judge. However, it relies on an indirect training objective that encourages the Challenger to generate tasks and hints that alter the Solver's response distribution. In contrast, J-Zero incurs additional computational overhead (13.8% longer average iteration time than G-Zero, as shown in the table below) but enables more direct adversarial optimization. By employing a shared reward model, J-Zero trains both the Challenger and Solver using a unified GRPO-based framework, where the Challenger minimizes the Solver’s reward while the Solver maximizes it. We believe this shared evaluation criterion provides a more coherent optimization objective, thereby facilitating more stable self-evolution.

GPU-hR-ZeroG-ZeroJ-Zero
avg. over iter3.684.324.92

We report the average GPU-hours per training iteration, measured on 4×B200 GPUs with Qwen3-4B-Base, excluding idle and model-loading time. Each iteration includes Challenger and Solver training (including rollouts), while J-Zero additionally includes Judge-related computation: reward scoring, preference-data generation (additional response generation, decomposition, subtask solving, and recomposition), and Judge optimization. We will include this result and discussion in the revised manuscript.


Question 2-a. Does preference validation with a strong Qwen evaluator support the Claude-based findings?

We thank the reviewer for this suggestion. To assess whether our preference-reliability findings are consistent across evaluators, we evaluated the same preference pairs using Qwen-3.6 27B, following the same evaluation protocol as in our Claude-based analysis.

IterationClaude Opus 4.8Qwen 3.6 27B
Role AsymSubtask AmpRole AsymSubtask Amp
Iteration 187.921.192.821.8
Iteration 285.436.986.644.2
Iteration 385.040.688.442.6
Iteration 477.350.284.454.4
Iteration 582.263.588.259.4
Iteration 676.069.988.374.7
Iteration 774.070.582.678.0
Iteration 865.877.076.377.7
Iteration 965.578.978.485.0
Iteration 1066.771.374.680.0

The strong Qwen-based results are consistent with the Claude-based findings: role asymmetry decreases from 92.8% to 74.6%, while subtask amplification increases from 21.8% to 80.0% as self-evolving progresses. These consistent trends suggest that the observed patterns are not specific to the Claude evaluator. We will include this result and discussion in the revised manuscript.

AuthorsOpenReview ↗

Official Comment to Reviewer RNoc (3/3)

Question 2-b. Does J-Zero also improve strong, instruction-tuned models, or does it mainly help small Base models?

We thank the reviewer for this question. To further investigate the effectiveness of J-Zero on instruction-tuned models, we conducted additional experiments using Llama-3.1-8B-Instruct and OLMo-3-7B-Instruct-SFT. As shown below, J-Zero outperforms the strongest baseline by 0.56 and 0.97 points on unverifiable tasks, respectively. The gains on instruction-tuned models are smaller than those on Qwen Base models reported in Tables 1–2 of the manuscript. This trend is also observed in R-Zero and G-Zero, suggesting a broader challenge for zero-data self-play rather than a limitation specific to J-Zero.

MethodVerifiableUnverifiableOverall
Llama-3.1-8B-Instruct48.3518.8533.60
R-Zero48.9919.4034.19
G-Zero48.1919.2833.73
J-Zero49.0419.9634.50
MethodVerifiableUnverifiableOverall
Olmo-3-7B-Instruct-SFT55.3321.8338.58
R-Zero55.5221.4738.49
G-Zero55.3121.2038.26
J-Zero56.8022.4439.62

One potential explanation is that instruction tuning substantially narrows the Challenger's question distribution, limiting the diversity of supervision available to the Solver. To investigate this, we sampled 2,000 questions from Qwen3-4B-Base and Qwen3-4B-Instruct-2507 using the same instruction prompt (Appendix D.1). Using Qwen3-Embedding-4B, the instruction-tuned model achieves a substantially lower Vendi score of 27.6 [24.2, 30.7], compared to 61.2 [57.4, 65.1] for the Base model, evaluated over groups of 128 questions. Notably, all 2,000 questions generated by the instruction-tuned model begin with the same phrase, and increasing the sampling temperature to 1.6 does not fully close the diversity gap (Vendi score: 48.3). These findings suggest that reduced generation diversity may constrain the effectiveness of self-evolution from instruction-tuned models.

Although the improvements on instruction-tuned models are relatively modest, the substantial gains on base models remain valuable, highlighting the potential of zero-data self-evolution to unlock pretrained capabilities without external supervision. Looking forward, as pretraining recipes and datasets continue to advance, increasingly capable pretrained models will require effective post-training supervision to fully leverage their knowledge on complex tasks. However, obtaining reliable human annotations for such tasks may become increasingly challenging. In this regard, we believe that zero-data self-play offers a promising direction for scalable post-training without reliance on human annotations, potentially contributing to the development of increasingly capable AI systems. We will include this result and discussion in the revised manuscript.


Question 3. How much do early unreliable amplification labels harm training? How does the method handle declining role preference reliability or a reversal of the assumed ordering?

To investigate whether unreliable subtask-amplification labels in early iterations adversely affect training, we conducted an additional experiment. We adjusted the proportion of preference pairs based on their reliability, starting with a higher proportion of role-asymmetry pairs (80:20), reflecting their relative reliabilities (87.9% vs. 21.1%), and linearly transitioning to a balanced ratio (50:50) by iteration 7.

MethodVerifiableUnverifiableOverall
J-Zero54.3820.8137.59
J-Zero-Scheduling54.2920.5037.39

While J-Zero does not explicitly detect or correct preference reversals, the comparable performance under reliability-aware scheduling suggests that J-Zero is not highly sensitive to variations in preference reliability, demonstrating its robustness to potentially noisy or unreliable preference labels. We will include this result and discussion in the revised manuscript.


Question 4. Figure 2 excludes ties, but Appendix D.3 requires a forced choice. How are ties defined?

Thank you for pointing this out. To mitigate position bias in LLM-as-a-judge evaluations, we evaluate each response pair twice, once in each presentation order (A vs. B and B vs. A), and treat inconsistent judgments as ties. We will clarify this in the revised manuscript.

Comments on the paper

2 comments

We thank all reviewers for their constructive feedback. We have revised the manuscript to address their concerns, with all additions highlighted in blue. The key updates are summarized below.

  1. Section 4.1 & Appendix B.1–B.3: Expanded experiments across model families (Llama-3.1-8B-Instruct and OLMo-3-7B-Instruct-SFT), Judge scales, and random seeds, including an analysis of limited improvements on instruction-tuned models.
  2. Appendix C.1: Validated preference-label reliability using an additional evaluator, Qwen3.6-27B, confirming consistent trends.
  3. Appendix C.2: Investigated robustness to noisy preference labels through reliability-aware scheduling and delayed subtask-amplification supervision.
  4. Appendix E.1: Added comparisons with DPO-based alternatives and analyzed the computational overhead of Judge co-evolution.
  5. Appendix E.2: Compared alternative preference constructions adapted from R-Zero and G-Zero, further validating our proposed preference pair design.
  6. Appendix F.3: Clarified the LLM-as-a-judge evaluation protocol, including position-bias mitigation.

We also updated Sections 5.1–5.2 and the Limitations to reflect these findings.

We hope these revisions address the reviewers' concerns and further clarify the contributions and effectiveness of J-Zero.

AuthorsOpenReview ↗

Final Remarks by Authors

We thank all reviewers for their thoughtful reviews and constructive feedback that helped strengthen our work. We are grateful for their recognition that J-Zero addresses an important bottleneck in zero-data self-evolution, particularly in open-ended and unverifiable domains (7Beb, LkBM), through a concrete and well-motivated approach (7Beb). We appreciate the reviewers' acknowledgement of our extensive evaluation across diverse benchmarks and model scales (RNoc, LkBM), as well as the strong empirical evidence supporting Judge co-adaptation through ablation studies and training trajectories (RNoc, 7Beb). Additionally, we value the recognition of our insightful analysis of preference reliability throughout self-evolution (RNoc) and the encouraging evidence of sustained self-improvement across training iterations (RNoc, 7Beb).

During the rebuttal, we conducted additional empirical evaluations and analyses in response to the reviewers' valuable feedback.

  • Generalizability across model families. To address concerns about generalization beyond Qwen Base models, we conducted additional experiments using Llama-3.1-8B-Instruct and OLMo-3-7B-Instruct-SFT. J-Zero consistently outperforms the baselines, although the gains are smaller than those on Base models. We further analyzed the reduced question diversity of instruction-tuned Challengers as a potential explanation.

  • Robustness to noisy preference labels. To investigate the reliability of our preference construction, we conducted additional experiments using reliability-aware scheduling and excluding unreliable subtask-amplification pairs during early training. The results demonstrate J-Zero's relative robustness to noisy supervision. We also validated our preference-reliability analysis using an additional evaluator, Qwen3.6-27B, confirming consistent trends.

  • Methodological novelty and controlled comparisons. To further isolate the contributions of J-Zero, we compared it against variants using frozen or removed Judges with DPO-based Solver training. We additionally introduced adaptive Judges into R-Zero- and G-Zero-style frameworks using their respective preference construction strategies. J-Zero consistently achieves the best performance, highlighting the effectiveness of our unified Challenger–Solver–Judge framework and proposed preference pairs.

  • Computational overhead. We measured the actual GPU-hours required by R-Zero, G-Zero, and J-Zero, showing that J-Zero incurs only 13.8% additional computational cost per iteration over G-Zero, while achieving substantially stronger performance.

  • Statistical significance. We repeated the full training and evaluation process with three different random seeds for J-Zero and the baselines. The results consistently demonstrate J-Zero's superior performance across both verifiable and unverifiable domains.

Finally, we addressed the reviewers' clarification questions, including checkpoint selection, LLM-as-a-judge evaluation protocols, and the distinction between G-Zero and J-Zero. We also revised the manuscript to incorporate the additional experiments and discussions.

We sincerely thank the reviewers and area chairs for their time, efforts, and constructive feedback, which have helped strengthen our work. We hope that our detailed rebuttal, additional experiments, and revised manuscript adequately address the reviewers' concerns and further demonstrate the effectiveness and significance of J-Zero for sustainable zero-data self-evolution across both verifiable and unverifiable domains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.