- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes co-adapting the judge model alongside the challenger and solver in LLM self-play. In particular, the judge is trained together with the challenger and solver, using preference pairs whose order is determined by the source of the responses. The solver's responses are preferred over the challenger's. Responses from a more sophisticated pipeline are preferred over the Solver's vanilla responses. Finally, the challenger and solver are trained via adversarial GRPO, and the judge is trained via preference learning. All roles are trained in an interleaved manner, and each role is trained for several steps before switching to other roles. Experiment results on both verifiable and non-verifiable domains show the effectiveness of their method
Critical strengths
- The paper identifies a meaningful bottleneck in self-play: the reliability of evaluation in the absence of strong supervision. This arises naturally: in self-play, the generation-verification gap is small, and with a frozen judge model, training easily saturates to the judge model's internal reward model. This paper addresses it by constructing preference training data, using different prompting conditions and trained weights.
- The evaluation is extensive, covering both verifiable and non-verifiable domains, while prior work mostly focuses only on one of them.
Critical weaknesses
- Training seems to be validated directly against the test benchmarks. In Section 4.1, the author states that they stop training once the average score on the verifiable or non-verifiable tasks begins to drop. This raises concerns about the generalizability of the performance of their selected checkpoints.
- The preference learning framework for the judge is flawed both theoretically and empirically. The role asymmetry assumes that the Solver's responses are of higher quality than those from the Challenger, because it is trained to answer the questions correctly. However, this is not true at the start of training. This can, in turn, result in a judge model that is cold-started with wrong training data and misleads Challenger and Solver, causing ribbling effects. This is also reflected empirically: According to Section 5.1, the accuracy of the preference directions from role asymmetry is capped at 87.9% and continues to decrease as training progresses. The observed accuracy is quite low and thus the preference learning for the judge is quite noisy.
Questions for authors
- Are the best checkpoints selected using the scores from the test benchmarks?
- How many repeated runs are there for your method and the baselines, and what is the statistical significance of the reported results?
4 comments
Official Comment to Reviewer LkBM (2/2)
Question 2. How many repeated runs are there for your method and the baselines, and what is the statistical significance of the reported results?
All of our experiments were run once with a fixed seed. To further investigate statistical significance, we additionally conducted training runs and evaluations for each method on Qwen3-4B-Base, repeating the entire process three times with different seeds.
| Method | Math Reasoning Avg. | General Reasoning Avg. | IFEval Avg. | Verifiable Avg. |
|---|---|---|---|---|
| R-Zero | 47.73±0.76 | 49.71±0.18 | 52.00±1.06 | 49.81±0.15 |
| G-Zero | 45.73±0.10 | 47.04±0.81 | 50.88±1.30 | 47.88±0.69 |
| J-Zero | 51.52±0.31 | 53.30±0.36 | 57.29±1.48 | 54.04±0.29 |
| J-Zero-frozen | 50.73±1.72 | 50.80±0.36 | 54.07±0.95 | 51.87±0.91 |
| Method | AlpacaEval | Arena-Hard (H.P.) | Arena-Hard (C.W.) | EQ-Bench C.W. | Unverifiable Avg. |
|---|---|---|---|---|---|
| R-Zero | 11.20±0.49 | 3.27±0.71 | 1.23±0.31 | 24.00±0.77 | 12.48±0.38 |
| G-Zero | 11.27±2.24 | 3.20±0.53 | 1.30±0.10 | 21.84±0.58 | 11.79±0.89 |
| J-Zero | 28.27±0.72 | 4.47±0.58 | 2.30±0.17 | 29.45±0.79 | 20.37±0.46 |
| J-Zero-frozen | 23.31±5.61 | 3.97±0.29 | 1.47±0.21 | 26.61±0.29 | 17.54±1.92 |
As shown in the tables, J-Zero consistently achieves the highest mean performance across both domains, with improvements substantially exceeding the observed standard deviation. We will include this result and discussion in the revised manuscript.
Dear Reviewer LkBM,
We sincerely appreciate your time and effort in reviewing our work and providing valuable feedback.
We have carefully addressed your concerns through detailed responses and additional experiments, and have updated our manuscript accordingly. We would be grateful if you could take another look at our rebuttal and share any further feedback or remaining concerns. We would be happy to provide additional clarification if needed.
Thank you again for your time and consideration.
Best, The Authors
Dear Reviewer LkBM,
We have made substantial efforts to address your concerns, including conducting additional experiments and providing detailed responses.
With only 24 hours remaining in the author–reviewer discussion period, we would greatly appreciate it if you could review our responses and reconsider your rating if you find that your concerns have been adequately addressed.
Thank you for your time and consideration.
Best regards,
The Authors
Official Comment to Reviewer LkBM (1/2)
We thank the reviewer for the careful review and constructive comments. We appreciate the recognition that our work identifies a meaningful bottleneck in self-play: training easily plateaus at the performance level of the frozen Judge model. We are also grateful for the acknowledgment of our extensive evaluation across both verifiable and unverifiable domains.
Weakness 1. Training seems to be validated directly against the test benchmarks. In Section 4.1, the author states that they stop training once the average score on the verifiable or non-verifiable tasks begins to drop. This raises concerns about the generalizability of the performance of their selected checkpoints.
Question 1. Are the best checkpoints selected using the scores from the test benchmarks?
Thank you for raising this concern. We would like to clarify that we did not select the best-performing checkpoints based on test benchmark scores. In principle, we followed the evaluation protocol of the baselines. In practice, it does not affect J-Zero as it improves monotonically through iteration 10 (our iteration budget), so its reported results are from the final iteration and involve no checkpoint selection.
Weakness 2. The preference learning framework for the Judge is flawed both theoretically and empirically. The role asymmetry assumes that the Solver’s responses are of higher quality than those from the Challenger, because it is trained to answer the questions correctly. However, this is not true at the start of training. This can, in turn, result in a Judge model that is cold-started with wrong training data and misleads Challenger and Solver, causing ribbling effects. This is also reflected empirically: According to Section 5.1, the accuracy of the preference directions from role asymmetry is capped at 87.9% and continues to decrease as training progresses. The observed accuracy is quite low and thus the preference learning for the Judge is quite noisy.
We agree that the preference data construction for Judge co-evolving does not guarantee a correct preference for every pair. However, their primary purpose is to prevent the Judge's learning from being limited by the Solver's capabilities. While role-asymmetry pairs provide stable supervision across iterations, their preference signals are inherently bounded by the Solver's performance, as the chosen and rejected responses originate from the Solver and Challenger, respectively. To sustain self-evolution, the Judge must develop evaluation capabilities beyond those of the current Solver. Subtask-amplification pairs become particularly important in later stages, when the benefits of role-asymmetry supervision begin to saturate. Thus, the two sources play complementary roles: role-asymmetry provides reliable early supervision, while subtask-amplification supports continued Judge improvement.
Unlike conventional datasets, preference datasets inherently involve subjective annotations, making perfectly accurate labels difficult to obtain. For instance, HH-RLHF reports only 63% agreement among human annotators [1], yet remains one of the most widely used preference datasets. Prior studies have also shown that reward models can be effectively trained with weak supervision [2, 3]. From this perspective, although role-asymmetry and subtask-amplification pairs do not provide perfectly accurate preference labels, they can still offer useful supervision for Judge training. The strong results in Tables 1–2 are consistent with the effectiveness of these complementary preference sources. We will add this discussion in the revised manuscript.