- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes ARISE-RL, a self-evolution framework that couples a task/rubric generator and a reasoning solver. The generator is responsible for generating the task and rubrics, and needs to call tools for grounding before generation. The solver is responsible for reasoning and answering. The generator's reward is the task's difficulty relative to the current solver, encouraging the generation of moderately difficult tasks. The solver's reward is based on the rubrics. The authors also propose Reward-Gated Self-Evolution Distillation, a type of OPSD that uses memory as privileged information. It is applied only when the teacher model has higher accuracy. The authors also propose ECR-Bench, an expert-calibrated rubric benchmark, including ECR-DeepResearch and ECR-Travel. The authors demonstrate the framework's effectiveness through extensive experiments.
Critical strengths
- This paper has a clear structure and is quite easy to understand.
- This paper is quite comprehensive and involves a substantial amount of work.
- Reward-Gated Self-Evolution Distillation, compared to standard OPSD, introduces an additional gating based on the accuracy of the teacher model, offering some insight. The authors' empirical analysis also demonstrates the importance of this gating mechanism.
- ARISE-RL shows significant performance improvements across various benchmarks.
Critical weaknesses
- Obvious issues with innovation and overclaim. The authors claim it's a novel, full-cycle self-evolution framework. However, since a year or two ago, numerous similar self-play paradigms have emerged, such as [1-3]. These paradigms include, but are not limited to, problem generators and solvers in RL scenarios, and code generators and unit testers in code scenarios. In my view, the framework proposed by the authors is no different from the paradigms represented by these works. The authors' claim that this is the first such framework is clearly an exaggeration. The paper also lacks specific comparisons with these works. This work seems more like a combination of multiple existing components, including the self-play paradigm and OPSD.
- This comparison may be unfair. In ARISE-RL, the coach and judge are external frontier models (gpt-5.2), providing evaluation and summary signals for the entire framework. However, OPCD and GKD baselines use a Qwen3.5-397B teacher, which provides a relatively weaker signal. The authors also do not report the baseline settings. For example, they do not explicitly state whether the number of training steps, number of rollouts, total training data, and number of evolution rounds for the Dr. Zero and Absolute Zero methods are consistent with those of ARISE-RL.
- There is a lack of necessary baselines, such as standard reinforcement learning baselines that can isolate “co-evolution”. For example, there are methods based on frozen generators or static LLM synthetic query pools, or GRPO methods that use manually written rubrics to reward scores.
[1] Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning
[2] Absolute Zero: Reinforced Self-play Reasoning with Zero Data
[3] Self-Rewarding Language Models
Questions for authors
See Weaknesses.
5 comments
Response to Reviewer ryxS (2/3): Is the comparison fair? (W2 + W3, part 1)
W2: ARISE-RL uses gpt-5.2 as coach and judge, while OPCD/GKD use a weaker Qwen3.5-397B teacher; Dr. Zero and Absolute Zero settings are not reported.
W3: Baselines isolating "co-evolution" are missing: a frozen generator, a static LLM-synthesized pool, or GRPO with manually written rubrics.
A: The judge is shared, the coach advantage is removable by construction, all settings are matched and reported, and all three missing baselines have been run. W2 asks whether our comparison is fair; W3 asks whether it is complete. Both reduce to controlled comparisons on the same backbone, judge, reward and budget.
(1) The judge is shared. gpt-5.2 is the rubric judge for every trained method, including Dr. Zero, Absolute Zero, OPCD and GKD, and all are trained on the same gpt-5.2 rubric reward. The only gpt-5.2 use specific to ARISE-RL is the coach.
(2) Matched and stronger coaches. We ran two comparisons designed to give OPCD and GKD the strongest feasible configuration rather than holding them at a weaker one:
- ARISE-RL with 397B as coach, i.e. the model that serves as the OPCD/GKD teacher.
- OPCD and GKD given gpt-5.2's coach memory in their teacher's context, so they receive the same information as ARISE-RL.
A gpt-5.2 teacher is not feasible: its API exposes no full token distributions and its tokenizer differs from the student's, so a logit-level distillation objective is not well defined. Giving them gpt-5.2's memory, the transferable part of ARISE-RL's advantage, is the strongest version we could construct.
| Method (Qwen3.5-9B) | Guidance | RR | ECR-DR | VitaBench | ECR-Travel | Avg. |
|---|---|---|---|---|---|---|
| OPCD | 397B teacher | 0.455 | 0.748 | 0.377 | 0.553 | 0.533 |
| GKD | 397B teacher | 0.460 | 0.752 | 0.368 | 0.547 | 0.532 |
| OPCD | 397B teacher + gpt-5.2 memory | 0.459 | 0.753 | 0.384 | 0.561 | 0.539 |
| GKD | 397B teacher + gpt-5.2 memory | 0.463 | 0.757 | 0.375 | 0.556 | 0.538 |
| ARISE-RL | self-coach (no external coach) | 0.468 | 0.764 | 0.395 | 0.577 | 0.551 |
| ARISE-RL | 397B coach | 0.473 | 0.773 | 0.405 | 0.587 | 0.560 |
| ARISE-RL | gpt-5.2 coach (paper) | 0.479 | 0.781 | 0.417 | 0.599 | 0.569 |
The row that answers the fairness concern most directly is the no-external-coach one: with no coach at all (ARISE-RL coaching itself with its own 9B model), it scores 0.551, above OPCD (0.533) and GKD (0.532) and above the same two baselines after they were handed gpt-5.2's memory (0.539 and 0.538). That run has strictly less external information than either baseline variant, so the advantage cannot be attributed to the coach. A matched 397B coach gives 0.560 and the paper setting 0.569: the coach contributes +1.8 over self-coaching, but the bulk of the gain is already present without it. The advantage comes from the same-policy, reward-gated design, not from a stronger guidance model.
Response to Reviewer ryxS (3/3): Matched settings and the missing baselines (W2 + W3, part 2)
(3) Baseline settings and compute. All self-evolution methods follow one protocol (App. B):
| Dr. Zero | Absolute Zero | ARISE-RL | |
|---|---|---|---|
| Backbone / tool stack / judge | same | same | same |
| Co-evolution cycles Solver steps | 3 120 | 3 120 | 3 120 |
| Queries per refresh | 500 | 500 | 500 |
| Rollouts per query (with PG) | 16 (16) | 16 (16) | 16 (8) |
| Max tool rounds / temperature | 40 / 0.8 | 40 / 0.8 | 40 / 0.8 |
| Learning rate / batch / ref-KL | same | same | same |
ARISE-RL samples the same number of rollouts but gives policy gradients to only 8; its extra wall-clock cost is $\times$1.17 (coach call plus teacher forward pass). With 1.25$\times$ steps the baselines reach 0.534 for Dr. Zero and 0.512 for Absolute Zero, still below 0.569. This is a lower bound rather than a converged comparison: Absolute Zero is flat there, but Dr. Zero was still rising at 3 cycles, so extra steps would close part of that gap.
(4) Baselines that isolate co-evolution. We ran all three. Every run shares the backbone, judge, Solver reward, Solver steps and query budget (1,500 queries = 3 500); only the source of training queries and rubrics changes:
- Static LLM-synthesized pool. gpt-5.2, with the same tool access and Generator prompts, generates 1,500 queries and rubrics once before training.
- Frozen Generator. The untrained 9B Generator refreshes the pool every cycle: the loop without Generator training.
- Human-written rubrics. 400 expert-written training queries with per-query rubrics (100 per domain), following the ECR handbook and not overlapping any test set. Covering the same 1,500-query budget gives each about 3.75$\times$ more epochs, which favours the baseline, so the comparison is conservative for us.
Each data source is run with plain GRPO (isolating the data source) and with RG-SED (isolating co-evolution in the full system).
| Training data | Informative groups | GRPO Avg. | + RG-SED Avg. |
|---|---|---|---|
| Static pool (gpt-5.2) | 43.5% | 0.491 | 0.533 |
| Frozen Generator | 39.2% | 0.479 | 0.524 |
| Human-written rubrics | 47.8% | 0.498 | 0.539 |
| Co-evolving Generator (ARISE-RL) | 76.4% | 0.502 | 0.569 |
"Informative groups" is the share of training groups with , where GRPO gets a non-zero advantage. Per-benchmark numbers are in App. G.4, largest gaps on the two multi-tool benchmarks. Three findings:
- (i) Co-evolution matches human supervision under plain GRPO. 0.502 versus 0.498 (a 0.4-point difference at the level of run-to-run variation, which we read as a tie), at no annotation cost and no data-volume limit, despite the human baseline's 3.75$\times$ more epochs per query.
- (ii) Co-evolution is clearly better in the full system. With RG-SED it leads human rubrics by +3.0 and the static pool by +3.6. The reason is the curriculum: a static or frozen pool cannot keep up with the Solver, so its share of informative groups falls as the Solver improves (static pool: 58.1% 40.7% 31.7% over three cycles), while the co-evolving Generator holds it near 76%.
- (iii) Generator training is what matters, not pool refresh. The frozen Generator, refreshed every cycle but never trained, is the weakest variant with RG-SED (0.524), within noise of the fully static pool (0.533). Refresh alone buys nothing; what separates 0.524 from 0.569 is training on the difficulty-shaped reward, so the curriculum tracks the Solver's frontier instead of drifting from it.
This supports the paper's claim that the two work together: RG-SED adds most when the curriculum keeps tasks at the Solver's frontier.
Summary. (W1) We accept the overclaim: the self-play template predates our work, and the "first" / "novel full-cycle" wording has been removed. We now compare directly with adapted [1] and [3] and with a naive combination of existing components, which recovers only 16% of our gain over Absolute Zero, and we narrowed the contribution to the coupling of tool-grounded rubrics, difficulty-shaped generation and the reward gate; RG-SED's specific novelty is that it assumes the privileged context can be wrong. (W2) The judge is shared by all methods; the decisive comparison is the no-external-coach run (0.551), which beats OPCD and GKD even when those are given gpt-5.2's memory (0.539/0.538), with the coach worth +1.8 on top. All baseline settings are reported and matched, and compute-matched baselines confirm the gap as a lower bound. (W3) Static-pool, frozen-Generator and human-rubric baselines isolate the contribution of co-evolution; the human baseline was given a training advantage, so that comparison is conservative. We hope these results address the soundness concerns, and we would be grateful if the reviewer would consider raising the score.
Response to Reviewer ryxS (1/3): Novelty and positioning (W1)
Thank you for the thorough review, and for finding the paper clearly structured and comprehensive and seeing value in the reward gate and its empirical analysis.
A: We accept the criticism of our wording, and have narrowed the claim to the coupling. The Generator-Solver self-play template predates our work, including [1-3], R-Zero and Dr. Zero, so "novel full-cycle" and "first self-evolution framework" overstate our contribution. We have removed these claims from the abstract, introduction and related work, and the introduction now discusses [1-3] explicitly.
The novelty we do claim lies in the coupling, not in the self-play template, and has two parts:
These differences matter in practice. We adapted [1] and [3] to our setting, keeping the same backbone, tools, judge and budget (details in App. B), and compare them with Absolute Zero from Table 1 and with a naive combination of existing components:
Self-Rewarding exhibits the failure that motivates a frozen external judge: over three iterations its self-assigned training reward climbs from 0.61 to 0.83 while its test Avg. stays flat (0.483 → 0.491 → 0.490).
The naive combination is the sharpest evidence, because it has every ingredient and none of the couplings: Absolute Zero at 0.508 plus ungated memory self-distillation reaches 0.518, a gain of +1.0, which is only 16% of ARISE-RL's +6.1 over Absolute Zero. The rest comes from how the components are coupled, and Table 2 shows no coupling is decorative: removing any one of the four design choices costs 3.5 to 6.8 points of the four-benchmark average (0.035 without difficulty-shaped reward, 0.041 without tool grounding, 0.055 without the gate, 0.068 without RG-SED).
The revised contributions are: