acceptodds
Under review as a conference paper at ICLR 2027

ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning

12 views

Abstract

Training open-ended agents via reinforcement learning (RL) is hindered by the lack of verifiable gold answers and scalable rubrics. Moreover, even near the model’s capability boundary, long-horizon open-ended agentic tasks often yield brittle and unstable rewards, resulting in weak or noisy rollout contrast that obscures fine-grained optimization signals for group-based policy learning. To address these challenges, we propose ARISE-RL, a novel full-cycle self-evolution framework that couples a task/rubric Generator and a reasoning Solver through rubric-mediated co-evolution. The Generator grounds tool-related rubric criteria in real tool observations and is rewarded for producing valid, intermediate-difficulty tasks aligned with the Solver’s evolving capability boundary. The Solver, in turn, learns from fine-grained rubric satisfaction signals through multi-step reasoning and tool use. We further introduce Reward-Gated Self-Evolution Distillation (RG-SED), which selectively distills a memory-augmented variant of the same policy back into itself only when the memory yields empirical reward improvement, thereby reducing distribution mismatch and avoiding blind imitation of noisy guidance. Finally, to support rigorous evaluation, we present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning. Extensive experiments demonstrate that ARISE-RL consistently achieves robust and stable overall state-of-the-art performance across all evaluated benchmarks.

Primary area:
reinforcement learning
Keywords:
Open-ended Agents, Self-Evolving Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ryxSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ARISE-RL, a self-evolution framework that couples a task/rubric generator and a reasoning solver. The generator is responsible for generating the task and rubrics, and needs to call tools for grounding before generation. The solver is responsible for reasoning and answering. The generator's reward is the task's difficulty relative to the current solver, encouraging the generation of moderately difficult tasks. The solver's reward is based on the rubrics. The authors also propose Reward-Gated Self-Evolution Distillation, a type of OPSD that uses memory as privileged information. It is applied only when the teacher model has higher accuracy. The authors also propose ECR-Bench, an expert-calibrated rubric benchmark, including ECR-DeepResearch and ECR-Travel. The authors demonstrate the framework's effectiveness through extensive experiments.

Critical strengths
  1. This paper has a clear structure and is quite easy to understand.
  2. This paper is quite comprehensive and involves a substantial amount of work.
  3. Reward-Gated Self-Evolution Distillation, compared to standard OPSD, introduces an additional gating based on the accuracy of the teacher model, offering some insight. The authors' empirical analysis also demonstrates the importance of this gating mechanism.
  4. ARISE-RL shows significant performance improvements across various benchmarks.
Critical weaknesses
  1. Obvious issues with innovation and overclaim. The authors claim it's a novel, full-cycle self-evolution framework. However, since a year or two ago, numerous similar self-play paradigms have emerged, such as [1-3]. These paradigms include, but are not limited to, problem generators and solvers in RL scenarios, and code generators and unit testers in code scenarios. In my view, the framework proposed by the authors is no different from the paradigms represented by these works. The authors' claim that this is the first such framework is clearly an exaggeration. The paper also lacks specific comparisons with these works. This work seems more like a combination of multiple existing components, including the self-play paradigm and OPSD.
  2. This comparison may be unfair. In ARISE-RL, the coach and judge are external frontier models (gpt-5.2), providing evaluation and summary signals for the entire framework. However, OPCD and GKD baselines use a Qwen3.5-397B teacher, which provides a relatively weaker signal. The authors also do not report the baseline settings. For example, they do not explicitly state whether the number of training steps, number of rollouts, total training data, and number of evolution rounds for the Dr. Zero and Absolute Zero methods are consistent with those of ARISE-RL.
  3. There is a lack of necessary baselines, such as standard reinforcement learning baselines that can isolate “co-evolution”. For example, there are methods based on frozen generators or static LLM synthetic query pools, or GRPO methods that use manually written rubrics to reward scores.

[1] Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning

[2] Absolute Zero: Reinforced Self-play Reasoning with Zero Data

[3] Self-Rewarding Language Models

Questions for authors

See Weaknesses.

5 comments

Response to Reviewer ryxS (1/3): Novelty and positioning (W1)

Thank you for the thorough review, and for finding the paper clearly structured and comprehensive and seeing value in the reward gate and its empirical analysis.

W1: Self-play paradigms such as [1] CURE, [2] Absolute Zero and [3] Self-Rewarding LMs already exist, so ARISE-RL is not the first such framework and looks like a combination of self-play and OPSD. Specific comparisons with these works are missing.

A: We accept the criticism of our wording, and have narrowed the claim to the coupling. The Generator-Solver self-play template predates our work, including [1-3], R-Zero and Dr. Zero, so "novel full-cycle" and "first self-evolution framework" overstate our contribution. We have removed these claims from the abstract, introduction and related work, and the introduction now discusses [1-3] explicitly.

The novelty we do claim lies in the coupling, not in the self-play template, and has two parts:

  • We identify which assumptions of existing self-play fail in open-ended multi-tool agentic tasks: there is no verifier, so "correct" must be constructed from tool-grounded rubrics, and progress must be shaped by difficulty rather than by solving rate.
  • We propose RG-SED, which, unlike OPCD, GKD or the memory distillation in Dr. Zero, assumes the privileged context itself may be wrong and therefore distils only where the reward measurably improves.
Source of correctness signalProposer rewarded forProposer grounded in environmentDistillation
Absolute Zero [2]code executorlearnabilitycode executionnone
CURE [1]code execution with unit testsdiscriminating correct from incorrect codecode executionnone
Self-Rewarding [3]the model's own LLM-as-judge scorenoneNonone
ARISE-RLtool-grounded generated rubrics, scored by a frozen judgeintermediate solvability (RdiffR_{\mathrm{diff}})tool calls required (RtoolR_{\mathrm{tool}})same-policy, reward-gated

These differences matter in practice. We adapted [1] and [3] to our setting, keeping the same backbone, tools, judge and budget (details in App. B), and compare them with Absolute Zero from Table 1 and with a naive combination of existing components:

  • Self-Rewarding (adapted). The 9B model writes rubrics and then scores its own rollouts against them. For parity, we use GRPO instead of iterative DPO.
  • CURE-style (adapted). The rubric writer plays the unit-tester role, rewarded for how well its rubrics separate the Solver's high-scoring rollouts from its low-scoring ones, without tool grounding or difficulty shaping.
  • Naive combination. Absolute Zero plus ungated memory self-distillation, in the style of OPSD.
Method (Qwen3.5-9B)RRECR-DRVitaBenchECR-TravelAvg.
Self-Rewarding (adapted)0.4510.7390.2810.4890.490
Absolute Zero0.4520.7480.3250.5080.508
CURE-style (adapted)0.4580.7500.3380.5290.519
Absolute Zero + ungated self-distillation0.4550.7440.3520.5210.518
ARISE-RL0.4790.7810.4170.5990.569

Self-Rewarding exhibits the failure that motivates a frozen external judge: over three iterations its self-assigned training reward climbs from 0.61 to 0.83 while its test Avg. stays flat (0.483 →\rightarrow 0.491 →\rightarrow 0.490).

The naive combination is the sharpest evidence, because it has every ingredient and none of the couplings: Absolute Zero at 0.508 plus ungated memory self-distillation reaches 0.518, a gain of +1.0, which is only 16% of ARISE-RL's +6.1 over Absolute Zero. The rest comes from how the components are coupled, and Table 2 shows no coupling is decorative: removing any one of the four design choices costs 3.5 to 6.8 points of the four-benchmark average (0.035 without difficulty-shaped reward, 0.041 without tool grounding, 0.055 without the gate, 0.068 without RG-SED).

The revised contributions are:

  1. A Generator-Solver self-play method for open-ended multi-tool agents, in which tool-grounded rubrics and difficulty-calibrated generation replace the missing verifier.
  2. RG-SED, an on-policy self-distillation method whose privileged context can be wrong, and which therefore distils only when the reward measurably improves.
  3. Benchmark and analysis. ECR-Bench (four domains, expert-written test rubrics), offered as a shared evaluation resource rather than as a methodological claim, together with the analysis isolating why open-ended self-play needs the couplings above. We no longer present the benchmark as a primary contribution, and report the external-benchmark results (RR, VitaBench) alongside it so the method can be judged without relying on our data.

Response to Reviewer ryxS (2/3): Is the comparison fair? (W2 + W3, part 1)

W2: ARISE-RL uses gpt-5.2 as coach and judge, while OPCD/GKD use a weaker Qwen3.5-397B teacher; Dr. Zero and Absolute Zero settings are not reported.

W3: Baselines isolating "co-evolution" are missing: a frozen generator, a static LLM-synthesized pool, or GRPO with manually written rubrics.

A: The judge is shared, the coach advantage is removable by construction, all settings are matched and reported, and all three missing baselines have been run. W2 asks whether our comparison is fair; W3 asks whether it is complete. Both reduce to controlled comparisons on the same backbone, judge, reward and budget.

(1) The judge is shared. gpt-5.2 is the rubric judge for every trained method, including Dr. Zero, Absolute Zero, OPCD and GKD, and all are trained on the same gpt-5.2 rubric reward. The only gpt-5.2 use specific to ARISE-RL is the coach.

(2) Matched and stronger coaches. We ran two comparisons designed to give OPCD and GKD the strongest feasible configuration rather than holding them at a weaker one:

  • ARISE-RL with 397B as coach, i.e. the model that serves as the OPCD/GKD teacher.
  • OPCD and GKD given gpt-5.2's coach memory in their teacher's context, so they receive the same information as ARISE-RL.

A gpt-5.2 teacher is not feasible: its API exposes no full token distributions and its tokenizer differs from the student's, so a logit-level distillation objective is not well defined. Giving them gpt-5.2's memory, the transferable part of ARISE-RL's advantage, is the strongest version we could construct.

Method (Qwen3.5-9B)GuidanceRRECR-DRVitaBenchECR-TravelAvg.
OPCD397B teacher0.4550.7480.3770.5530.533
GKD397B teacher0.4600.7520.3680.5470.532
OPCD397B teacher + gpt-5.2 memory0.4590.7530.3840.5610.539
GKD397B teacher + gpt-5.2 memory0.4630.7570.3750.5560.538
ARISE-RLself-coach (no external coach)0.4680.7640.3950.5770.551
ARISE-RL397B coach0.4730.7730.4050.5870.560
ARISE-RLgpt-5.2 coach (paper)0.4790.7810.4170.5990.569

The row that answers the fairness concern most directly is the no-external-coach one: with no coach at all (ARISE-RL coaching itself with its own 9B model), it scores 0.551, above OPCD (0.533) and GKD (0.532) and above the same two baselines after they were handed gpt-5.2's memory (0.539 and 0.538). That run has strictly less external information than either baseline variant, so the advantage cannot be attributed to the coach. A matched 397B coach gives 0.560 and the paper setting 0.569: the coach contributes +1.8 over self-coaching, but the bulk of the gain is already present without it. The advantage comes from the same-policy, reward-gated design, not from a stronger guidance model.

Response to Reviewer ryxS (3/3): Matched settings and the missing baselines (W2 + W3, part 2)

(3) Baseline settings and compute. All self-evolution methods follow one protocol (App. B):

Dr. ZeroAbsolute ZeroARISE-RL
Backbone / tool stack / judgesamesamesame
Co-evolution cycles ×\times Solver steps3 ×\times 1203 ×\times 1203 ×\times 120
Queries per refresh500500500
Rollouts per query (with PG)16 (16)16 (16)16 (8)
Max tool rounds / temperature40 / 0.840 / 0.840 / 0.8
Learning rate / batch / ref-KLsamesamesame

ARISE-RL samples the same number of rollouts but gives policy gradients to only 8; its extra wall-clock cost is $\times$1.17 (coach call plus teacher forward pass). With 1.25$\times$ steps the baselines reach 0.534 for Dr. Zero and 0.512 for Absolute Zero, still below 0.569. This is a lower bound rather than a converged comparison: Absolute Zero is flat there, but Dr. Zero was still rising at 3 cycles, so extra steps would close part of that gap.

(4) Baselines that isolate co-evolution. We ran all three. Every run shares the backbone, judge, Solver reward, Solver steps and query budget (1,500 queries = 3 ×\times 500); only the source of training queries and rubrics changes:

  • Static LLM-synthesized pool. gpt-5.2, with the same tool access and Generator prompts, generates 1,500 queries and rubrics once before training.
  • Frozen Generator. The untrained 9B Generator refreshes the pool every cycle: the loop without Generator training.
  • Human-written rubrics. 400 expert-written training queries with per-query rubrics (100 per domain), following the ECR handbook and not overlapping any test set. Covering the same 1,500-query budget gives each about 3.75$\times$ more epochs, which favours the baseline, so the comparison is conservative for us.

Each data source is run with plain GRPO (isolating the data source) and with RG-SED (isolating co-evolution in the full system).

Training dataInformative groupsGRPO Avg.+ RG-SED Avg.
Static pool (gpt-5.2)43.5%0.4910.533
Frozen Generator39.2%0.4790.524
Human-written rubrics47.8%0.4980.539
Co-evolving Generator (ARISE-RL)76.4%0.5020.569

"Informative groups" is the share of training groups with 0<c<K0<c<K, where GRPO gets a non-zero advantage. Per-benchmark numbers are in App. G.4, largest gaps on the two multi-tool benchmarks. Three findings:

  • (i) Co-evolution matches human supervision under plain GRPO. 0.502 versus 0.498 (a 0.4-point difference at the level of run-to-run variation, which we read as a tie), at no annotation cost and no data-volume limit, despite the human baseline's 3.75$\times$ more epochs per query.
  • (ii) Co-evolution is clearly better in the full system. With RG-SED it leads human rubrics by +3.0 and the static pool by +3.6. The reason is the curriculum: a static or frozen pool cannot keep up with the Solver, so its share of informative groups falls as the Solver improves (static pool: 58.1% →\rightarrow 40.7% →\rightarrow 31.7% over three cycles), while the co-evolving Generator holds it near 76%.
  • (iii) Generator training is what matters, not pool refresh. The frozen Generator, refreshed every cycle but never trained, is the weakest variant with RG-SED (0.524), within noise of the fully static pool (0.533). Refresh alone buys nothing; what separates 0.524 from 0.569 is training on the difficulty-shaped reward, so the curriculum tracks the Solver's frontier instead of drifting from it.

This supports the paper's claim that the two work together: RG-SED adds most when the curriculum keeps tasks at the Solver's frontier.


Summary. (W1) We accept the overclaim: the self-play template predates our work, and the "first" / "novel full-cycle" wording has been removed. We now compare directly with adapted [1] and [3] and with a naive combination of existing components, which recovers only 16% of our gain over Absolute Zero, and we narrowed the contribution to the coupling of tool-grounded rubrics, difficulty-shaped generation and the reward gate; RG-SED's specific novelty is that it assumes the privileged context can be wrong. (W2) The judge is shared by all methods; the decisive comparison is the no-external-coach run (0.551), which beats OPCD and GKD even when those are given gpt-5.2's memory (0.539/0.538), with the coach worth +1.8 on top. All baseline settings are reported and matched, and compute-matched baselines confirm the gap as a lower bound. (W3) Static-pool, frozen-Generator and human-rubric baselines isolate the contribution of co-evolution; the human baseline was given a training advantage, so that comparison is conservative. We hope these results address the soundness concerns, and we would be grateful if the reviewer would consider raising the score.

Reviewer_vWapOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARISE-RL, a self-evolving reinforcement learning framework for open-ended agentic tasks. The framework consists of a Generator that produces tasks and evaluation rubrics, and a Solver that learns to complete these tasks through multi-turn tool interactions. The two components co-evolve through rubric-based rewards and difficulty-aware task generation. The authors further introduce Reward-Gated Self-Evolution Distillation (RG-SED), which selectively distills behaviors from a memory-augmented version of the same policy when the generated guidance leads to higher rewards. Additionally, the paper introduces ECR-Bench, a rubric-based benchmark covering deep research and multi-tool travel planning. Experiments are conducted on Qwen3-8B and Qwen3.5-9B across four benchmarks, including two newly constructed benchmarks and two existing ones. The reported results demonstrate improvements over several self-evolution and distillation baselines.

Critical strengths

S1. Promising empirical results.

The proposed method demonstrates consistent performance improvements over the evaluated baselines across multiple benchmarks. The experiments cover two different model backbones and both single-tool and multi-tool agentic tasks.

S2. Comprehensive experimental comparisons.

The paper includes comparisons with existing self-evolution frameworks and on-policy distillation methods. The ablation studies also examine the contributions of RG-SED, reward gating, tool-grounded rubrics, and difficulty-shaped rewards, providing empirical evidence for the effectiveness of the proposed components.

Critical weaknesses

W1. Insufficient description of ECR-Bench construction.

Section 4 provides an overly simplified description of the ECR-Bench construction process. Although additional details are provided in Appendix A, the main text does not explicitly refer readers to the relevant appendix sections. Given that ECR-Bench is presented as one of the main contributions, I believe its construction process should be explained more clearly in the main paper. In particular, the descriptions of benchmark data construction, expert calibration, and quality control are insufficient for readers to fully understand how the benchmark was established.

W2. Insufficient justification for constructing a new benchmark.

I am not convinced of the necessity of introducing ECR-Bench instead of relying on existing, established agentic benchmarks. Although the authors evaluate their method on ResearchRubrics and VitaBench, the paper does not sufficiently explain what important limitations of existing benchmarks motivate the construction of ECR-Bench, or what unique evaluation capabilities the new benchmark provides. A clearer justification of the necessity and distinctiveness of ECR-Bench would strengthen this contribution.

W3. Limited evidence for the long-term stability and scalability of self-evolution.

The proposed framework relies on a closed-loop self-evolution process, where the Generator produces both training tasks and evaluation rubrics, while the Solver is optimized using the resulting reward signals. Such a setting raises concerns about potential reward hacking and self-reinforcing biases during prolonged training. However, the paper does not provide sufficient evidence to establish whether these problems can be avoided over longer training horizons. Furthermore, the experiments only cover three co-evolution cycles. As shown in Table 3, performance continues to improve from Iteration 2 to Iteration 3, suggesting that training has not yet converged. Therefore, the current results are insufficient to support strong claims about sustained self-evolution and long-term scalability.

W4. Overstated motivation regarding reward instability in agentic RL.

In the Introduction, the authors claim:

Open-ended agentic tasks typically involve long-horizon, multi-turn tool use and interaction with external environments, where reward signals are often brittle and unstable.

However, this claim lacks sufficient empirical evidence or literature support. The fact that agentic tasks involve long-horizon interactions and open-ended outputs does not necessarily imply that their reward signals are inherently brittle or unstable. Similarly, the subsequent claim that vanilla RL suffers from weak or noisy rollout contrast is not adequately substantiated. These statements appear to generalize potential difficulties in agentic RL without sufficiently establishing their prevalence or severity. Since these claims form an important part of the paper's motivation, I believe they require stronger evidence and more careful qualification.

W5. Insufficient evidence for the claimed limitations of teacher-guided training.

The Introduction argues that conventional teacher-guided training can be unreliable in open-ended agentic tasks, potentially transferring incorrect tool choices, missed constraints, or low-quality reasoning patterns to the student. However, these arguments are largely presented as potential failure modes rather than empirically established limitations. The cited work (Ye et al., 2026) primarily investigates specific distillation settings and does not sufficiently support a general claim that conventional teacher-trajectory SFT is ineffective or unstable for open-ended agentic tasks. In practice, distilling trajectories from stronger models is a common approach to improving agent capabilities. The paper does not provide a direct teacher-trajectory SFT baseline to demonstrate that this approach suffers from the claimed limitations in the considered tasks. As a result, the motivation for replacing conventional teacher guidance with the proposed RG-SED mechanism is not sufficiently established.

W6. Insufficient positioning with respect to existing work.

The Introduction does not adequately discuss or cite the relevant developments in on-policy distillation and self-evolving agents. Neither on-policy distillation nor the general concept of Generator–Solver co-evolution is new. Although some related studies are discussed in Section 2, the Introduction does not clearly explain how the proposed framework builds upon these existing ideas or what fundamental limitations of prior approaches it addresses. This makes it difficult to identify the precise novelty of the proposed framework.

W7. Incomplete specification of the RL optimization objective.

Equation (14) defines the overall optimization objective as

La=LRLa+λtaLSEDa\mathcal{L}^{a}=\mathcal{L}_{\mathrm{RL}}^{a}+\lambda_t^{a}\mathcal{L}_{\mathrm{SED}}^{a}

However, the paper does not clearly specify how LRLa\mathcal{L}_{\mathrm{RL}}^{a} is formulated or which RL algorithm is used to optimize the Generator and Solver. Although the reward functions are described, the underlying RL optimization procedure remains unclear. This is an important implementation detail for understanding and reproducing the proposed method.

W8. Clarity and organization of the presentation.

The overall presentation could be substantially improved. The paper introduces multiple components, including rubric-grounded rewards, difficulty-shaped rewards, Generator–Solver co-evolution, and reward-gated self-distillation. While the individual components are described, the motivation for introducing and combining these mechanisms is not always clearly established. The narrative also moves between several different claimed limitations of existing approaches without sufficiently connecting them to the proposed solutions. Overall, I found the paper difficult to follow, particularly regarding why the different reward mechanisms are necessary and how they collectively address the central research problem. A clearer and more focused presentation would significantly improve readability.

Questions for authors

Q1. Why is a new benchmark necessary?

Could the authors clarify the specific limitations of existing agentic benchmarks, such as ResearchRubrics and VitaBench, that motivated the construction of ECR-Bench?

What evaluation capabilities does ECR-Bench provide that cannot be adequately assessed using existing benchmarks?

Q2. Which RL algorithm is used in Equation (14)?

Could the authors explicitly define LRLa\mathcal{L}_{\mathrm{RL}}^{a} in Equation (14)?

In particular, which RL algorithm is used for the Generator and Solver, and are their optimization procedures identical?

Q3. Why does training stop after three co-evolution cycles?

Table 3 shows that performance continues to improve from Iteration 2 to Iteration 3, particularly on VitaBench and ECR-Travel. Why do the authors stop training at Iteration 3 rather than continuing until convergence?

Given the paper's emphasis on continual self-evolution, could the authors provide evidence that the proposed method remains stable and effective over longer training horizons?

Could the authors clarify the total number of RL optimization steps? Section 5.1 states that training involves three co-evolution cycles, with the query pool refreshed every 120 steps. However, Figure 12 only reports student–teacher KL divergence over 120 training rollout steps. Does Figure 12 cover the entire training process or only one of the three cycles? Please clarify the relationship between training rollout steps, optimization steps, and co-evolution cycles.

7 comments

Response to Reviewer vWap (1/5): How ECR-Bench was built (W1 + W2/Q1, part 1)

Thank you for the detailed review, and for reading the equations and appendix closely. W1 and W2/Q1 both concern ECR-Bench (how it was built and why it is needed), so we answer them together. New results use Qwen3.5-9B and appear in the revised PDF (App. G; changes in blue).

W1: The construction of ECR-Bench (data construction, expert calibration, quality control) is too briefly described, and the main text does not refer to Appendix A.

W2 / Q1: Why is a new benchmark necessary? What can ECR-Bench evaluate that ResearchRubrics and VitaBench cannot?

A: Section 4 now describes the full pipeline with explicit pointers to Appendix A. Six domain experts carried it out (three travel planning, three deep research; App. A.3), in four stages.

  1. Query design. ECR-Travel fixes five sub-task types in advance (Direction, Compare, Search, 1-Day, M-Day), 100 queries each over real cities and POIs. ECR-DR asks open-domain questions requiring synthesis of several sources (69 Chinese, 31 English).
  2. Tool-verified grounding. For each ECR-Travel query the author runs the needed calls live and records them as expected_tools with exact arguments (coordinates, waypoints, mode, date): 1,060 calls, 2.1 per query; unretrievable queries are discarded. For ECR-DR, authors confirm each rubric's evidence is findable by web search.
  3. Rubric writing and calibration. Each item is one independently checkable statement with an explicit pass criterion; ECR-DR has 693 items (6.9 per query), ECR-Travel 2,056 (4.1). A pilot of 30 queries per domain was annotated independently by that domain's three experts; resolving disagreements and revising the handbook lifted item-level Fleiss' κ\kappa from 0.55/0.61 to 0.74/0.79, after which a second expert cross-reviewed every remaining query-rubric pair.
  4. Quality control. Reference trajectories pass a rule-augmented LLM checker for tool-call validity, dialogue correctness and answer consistency; flagged items are rewritten or removed, with a final manual pass. Of 158 and 712 candidates, 100 and 500 were kept.

Honestly stated: 30 pilot queries per domain make the item-level κ\kappa a calibration diagnostic, not a population estimate; final quality rests more on the exhaustive cross-review, the checker, and the 200-response human study in our H1aE response (LLM-human agreement within 2.7-3.3 points of human-human).

Reproducibility. Because the live tool environment moves, the release pins a dated snapshot of the map/route/POI, weather and web-API responses behind every expected_tools annotation, plus the underlying data version identifiers, and a cached-return mode that replays the snapshot offline. Reported scores come from cached mode; where facts legitimately change (opening hours, prices, schedules) the rubric accepts any snapshot-consistent answer.


Response to Reviewer vWap (2/5): Why it is needed (W1 + W2/Q1, part 2)

ResearchRubrics and VitaBench remain our main external evidence, and our gains there are at least as large: +7.2%/+62.9% over base on RR/VitaBench, versus +6.3%/+27.4% on ECR-DR/ECR-Travel. ECR-Bench adds capabilities the two lack:

ResearchRubricsVitaBenchECR-Bench
Tool environmentnone specified (report only)simulated, synthetic databaseslive map/route/POI, weather and web APIs
Argument-level process supervisionNoNoYes (1,060 expected_tools)
Real geographic and temporal feasibilityNolimitedYes (coordinates, waypoints, ETAs, weather)
Balanced skill-level breakdownNoby scenario5 planning skills ×\times 100
Languages (research)Englishn/aChinese + English

The strongest evidence that ECR-Bench measures something distinct is the failure mode it exposes, not the rank correlation: 17.8% of foundation-model responses on ECR-Travel satisfy every outcome rubric yet fail an expected_tools item, typically by stating routes or ETAs without calling direction. An outcome-only benchmark scores them fully correct; ECR-Bench scores them as failures, the behaviour that matters in deployment. Now in Section 4 and the introduction.

Rank evidence is supporting only: over the 8 untrained foundation models, Spearman ρ\rho = 0.64 (VitaBench vs ECR-Travel) and 0.52 (RR vs ECR-DR), neither significant at n = 8 (p ≈\approx 0.09, 0.19). The concrete instance is clearer: Claude-4.6-Sonnet ranks first on VitaBench (0.442) but fourth on ECR-Travel (0.449), concentrated in multi-waypoint Direction tasks (0.223 versus 0.512 for Qwen3.5-397B), a skill VitaBench cannot pose.

Response to Reviewer vWap (3/5): Three cycles, convergence and step accounting (W3/Q3)

W3 / Q3: Only three cycles, with performance still rising, so long-term stability, reward hacking and self-reinforcing bias are not established. Please clarify how rollout steps, optimization steps and cycles relate, and what Figure 12 covers.

A: (1) Longer training. Extending training to six cycles (720 Solver steps) improves Avg. from 0.569 to 0.592, with benchmark scores RR/ECR-DR/VitaBench/ECR-Travel of 0.486/0.794/0.457/0.630. Gains progressively halve (+2.7, +1.4, +0.7, +0.2), suggesting convergence toward 0.596 rather than divergence. We do not claim convergence given the single seed. Multi-tool benchmarks continue improving longest.

(2) No hacking or self-reinforcing bias. Across six cycles, the judge gap remains stable (0.023–0.025), while both scores improve. Rubric validity (93.2–94.4%), trivial rubrics (1.6–2.1%), and query diversity (0.601–0.612) also remain stable. The frozen Solver's pass rate on new queries declines then plateaus, indicating no task degeneration. Structurally, RdiffR_{\mathrm{diff}} gives zero reward to always-pass tasks, while independent test rubrics prevent generated-task exploitation from inflating evaluation scores.

(3) Step accounting. One training step comprises B queries (4; 2 for VitaBench), 16 rollouts (8 standard + 8 memory-augmented), and one policy update. Each cycle contains 120 Solver steps, followed by Generator updates and a 500-query pool refresh; three cycles total 360 steps. Figure 12 shows cycle 1; full curves appear in App. C.3 (mean KL: 0.27 RG-SED vs. 0.48 GKD).


Response to Reviewer vWap (4/5): Are the motivating claims substantiated? (W4 + W5)

W4: The claims that agentic rewards are "brittle and unstable" and that vanilla RL suffers weak or noisy rollout contrast are not substantiated.

W5: The limitations of teacher-guided training are not established empirically, and there is no direct teacher-trajectory SFT baseline.

A: Both were under-supported; we qualified the wording and measured both.

(1) Reward brittleness (W4). Prior work reports collapsing within-group reward variance in multi-turn agent RL (the "Echo Trap" in RAGEN, Wang et al., 2025), while zero-advantage filtering motivates dynamic sampling in DAPO (Yu et al., 2025). Vanilla GRPO, Qwen3.5-9B, w/o RG-SED, cycle 1:

RRECR-DRVitaBenchECR-Travel
Avg. tool calls per trajectory7.48.118.615.3
Groups with reward std < 0.0514.8%16.9%31.6%34.2%
Judge re-scoring std (same trajectory, 5$\times$)0.0120.0150.0210.024

Low-contrast groups grow with horizon: pooled over multi-tool groups the share rises from 11.7% (≤\le 5 tool calls) to 25.3% (6-15) and 42.8% (> 15). Long-task rewards are bimodal (one missed constraint removes the full-completion bonus), so groups collapse to all-fail or all-pass, while judge noise is an order of magnitude smaller: brittleness comes from long horizons, not the evaluator. On the 34.2% low-contrast ECR-Travel groups, coach memory yields Δr>τ\Delta_r>\tau in 61.5% of cases. The introduction now says "can be brittle" and cites these measurements.

(2) Teacher-guided training (W5). Baseline: teacher trajectories on the same ARISE-generated pool, kept where r≥γ=0.9r\ge\gamma=0.9 (rejection sampling), used to fine-tune the 9B model.

Method (Qwen3.5-9B)RRECR-DRVitaBenchECR-TravelAvg.
Base0.4470.7350.2560.4700.477
SFT on Qwen3.5-397B trajectories0.4380.7220.3710.5510.521
SFT on Claude-4.6-Sonnet trajectories0.4510.7160.3890.5480.526
SFT (397B) →\rightarrow GRPO0.4520.7410.3920.5670.538
ARISE-RL0.4790.7810.4170.5990.569
SFT (397B) →\rightarrow ARISE-RL0.4780.7760.4250.6060.571
  • (i) Teachers are not uniformly reliable. Both score below the 9B base on ECR-DR (0.628/0.623 versus 0.735), and teacher SFT lowers it. Even among accepted trajectories 12.4% state ETAs, prices or opening hours no tool call returned; the SFT student copies this in 10.9% of responses, versus 4.2% for ARISE-RL.
  • (ii) Distribution gap. Per-token NLL is 1.37 on teacher trajectories versus 0.52 on the student's own rollouts.
  • (iii) Complementary, not sufficient. SFT →\rightarrow GRPO trails ARISE-RL by 3.1 points, and SFT alone (0.521) by 4.8. Starting ARISE-RL from the SFT model gives 0.571 versus 0.569, reported but not interpreted: at one seed run-to-run variation cannot be ruled out. What carries weight is 0.571 (SFT plus our pipeline) against 0.521 (SFT alone): the pipeline produces the gain.

These are measured limitations in our setting, not a general claim about teacher SFT.

Response to Reviewer vWap (5/5): Positioning, presentation and the RL objective (W6 + W8 + W7/Q2)

W6: The introduction does not position the work against existing on-policy distillation and self-evolving agents, so the novelty is unclear.

W8: The presentation is hard to follow; it is unclear why each reward mechanism is needed and how they jointly address the central problem.

W7 / Q2: How is LRLa\mathcal{L}_{\mathrm{RL}}^{a} in Eq. (14) defined, which RL algorithm is used, and is it the same for both roles?

A: (1) Positioning and presentation. Neither co-evolution nor on-policy distillation is new, and the revised introduction positions against both, dropping the "novel full-cycle" / "first" wording. Self-play methods (Self-Rewarding, Absolute Zero, R-Zero, CURE, Dr. Zero) all need a verifier (code execution, majority vote or short verifiable answers), which open-ended agentic tasks lack; ARISE-RL replaces it with generated rubrics made reliable by tool grounding (RtoolR_{\mathrm{tool}}) and difficulty calibration (RdiffR_{\mathrm{diff}}). On-policy distillation uses either a fixed external teacher (GKD, OPCD), causing the mismatch of Fig. 12, or a same-model teacher given privileged gold answers (OPSD-style), which do not exist here; RG-SED uses group-level coach memory instead, and since that memory can be wrong (27.9% neutral or harmful, Fig. 5) it distils only where a measured reward gap shows it helps.

Sections 1 and 3 follow one problem-to-mechanism chain, each component introduced with the failure it addresses: no data or rubrics →\rightarrow Generator-written queries and rubrics (Table 1; App. G.4); rubrics may hallucinate tool facts →\rightarrow RtoolR_{\mathrm{tool}} ($-$6.1 ECR-DR, $-$5.0 ECR-Travel); tasks far from the frontier give zero GRPO advantage →\rightarrow RdiffR_{\mathrm{diff}} peaking at c=K/2c=K/2 (Fig. 4); frontier tasks still give weak contrast →\rightarrow RG-SED ($-$6.7 / $-$8.1); coach guidance may be wrong →\rightarrow the gate Δr>τ\Delta_r>\tau (Fig. 5). Unsupported claims are qualified (W4-W6), and Table 4 in Appendix G pairs each component with its evidence.

(2) The objective. Both roles use GRPO with PPO-style clipping, on slime. Each query yields 8 standard rollouts yi∼πθ(⋅∣p)y_i\sim\pi_\theta(\cdot\mid p) and 8 memory-augmented ones from πθ(⋅∣p⊕m)\pi_\theta(\cdot\mid p\oplus m); only the former get policy gradients, the latter are loss-masked and only compute Δr\Delta_r. With group-normalized advantage A^i=(ri−mean(r1:8))/std(r1:8)\hat A_i=(r_i-\mathrm{mean}(r_{1:8}))/\mathrm{std}(r_{1:8}), RG-SED enters as advantage shaping:

A~i,t=A^i−λtamin⁡(log⁡πθ(yi,t∣ht,p)−log⁡πθ(yi,t∣ht,p⊕m),c),\tilde A_{i,t}=\hat A_i-\lambda_t^a\min\big(\log\pi_\theta(y_{i,t}\mid h_t,p)-\log\pi_\theta(y_{i,t}\mid h_t,p\oplus m),c\big), La=−E[1∣yi∣∑tmin⁡(ρi,tA~i,t,clip(ρi,t,1−ϵ,1+ϵ)A~i,t)]+βDKL(πθ∥πref),\mathcal L^a=-\mathbb E\Big[\tfrac{1}{|y_i|}\textstyle\sum_t\min\big(\rho_{i,t}\tilde A_{i,t},\mathrm{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\tilde A_{i,t}\big)\Big]+\beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),

with ρ\rho the importance ratio, ϵ=0.2\epsilon=0.2, β=0.001\beta=0.001 (low-variance KL), c=10c=10, no entropy bonus (Adam, lr 1e-6 →\rightarrow 1e-7). Since ∇θDKL(πθ∥sg[q])=Ey∼πθ[∇log⁡πθ(y)(log⁡πθ(y)−log⁡q(y))]\nabla_\theta D_{\mathrm{KL}}(\pi_\theta\Vert\mathrm{sg}[q])=\mathbb E_{y\sim\pi_\theta}[\nabla\log\pi_\theta(y)(\log\pi_\theta(y)-\log q(y))], the shaping term is the score-function estimator of λtaLSEDa\lambda_t^a\mathcal L_{\mathrm{SED}}^a, so Eq. (14) is LRLa\mathcal L_{\mathrm{RL}}^a plus the gated reverse KL. The Eq. (13) indicator is smoothed as σ((Δr−τ)/T)\sigma((\Delta_r-\tau)/T), T=0.0125T=0.0125 (App. C.2); waw_a is a 20-step warm-up, dad_a a cosine decay. This is Section 3.3, with the complete loop as Algorithm 1 in Appendix G.6.


Summary. (W1, W2/Q1) ECR-Bench's construction, including its limited pilot calibration, is described in Section 4, and the release pins a dated tool snapshot for offline reproduction; its justification rests on capability gaps and on the 17.8% of responses that pass every outcome rubric while failing a tool item, rank correlation only supporting. (W3/Q3) Six cycles show gains halving each cycle (+2.7 →\rightarrow +0.2) with no degradation or hacking signature; we no longer call it convergence, and step accounting is explicit. (W4, W5) Brittleness and teacher reliability are measured and the claims qualified; teacher SFT trails ARISE-RL by 4.8 and 4.3 points, and SFT$\rightarrow$ARISE-RL's +0.2 is reported but not interpreted. (W6, W8) Positioning is explicit, overclaims removed, and the presentation follows one problem-to-mechanism chain. (W7/Q2) The full objective is specified. We would be grateful for a reconsideration of the score.

Reviewer_H1aEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ARISE-RL, a reinforcement learning self-evolution framework for open-ended agents that couples a tool-grounded task/rubric Generator and a Solver via rubric-mediated co-evolution. It introduces Reward-Gated Self-Evolution Distillation (RG-SED) to selectively distill memory-augmented rollouts only when empirical rewards improve, and evaluates this on existing benchmarks and a newly curated ECR-Bench.

Critical strengths

The co-evolution setup with tool-grounded generation and difficulty-shaped rewards effectively prevents hallucinated evaluation criteria while dynamically matching the solver's learning frontier. In addition, RG-SED provides a well-motivated, reward-gated mechanism to mitigate distribution mismatch and filter noisy teacher guidance.

Critical weaknesses

The evaluation relies heavily on an LLM-as-a-judge setup and the authors' newly introduced benchmark, which introduces risks of circular evaluation bias and judge hallucination despite the reported human agreement. Furthermore, the coach memory extraction and verification pipeline introduces considerable engineering overhead and prompting sensitivity that are not sufficiently analyzed across architectures.

Questions for authors

How sensitive is the RG-SED gating threshold to different LLM backbones, and how does the framework prevent potential judge reward hacking when the Generator and Solver iteratively optimize against the same evaluation model?

2 comments

Response to Reviewer H1aE (1/4): Is the judge a sound signal? (W1 + Q2, part 1)

Thank you for the review and positive feedback. W1 and Q2 share the concern of relying on a single LLM judge for both reward and evaluation. We address them jointly. Unless stated otherwise, new results use Qwen3.5-9B and are included in the revised PDF (Appendix G, changes in blue).

W1: Evaluation relies on an LLM judge and on the authors' own benchmark, risking circular bias and judge hallucination.

Q2: ...how is judge reward hacking prevented?

A: The evaluation is not circular, the gains do not need our benchmark, and the diagnostics show that no hacking develops.

(1) No train-test circularity. Generator-written rubrics are used only for training. Test rubrics are expert-written (ECR-Bench) or come from other groups (ResearchRubrics, VitaBench), and the Generator never sees a test query. Screening the 1,500 generated training queries against all test queries (embedding similarity ≥\ge 0.9 plus identical entity set) flagged 0 ECR-DR and 3 ECR-Travel queries; removing them moves ECR-Travel by < 0.002. The gains do not need our benchmark either: over base, the external RR/VitaBench improve by +7.2%/+62.9%, versus +6.3%/+27.4% on ECR-DR/ECR-Travel (Table 1). Nor does the judge favour its own family: gpt-5.2 scores 0.442 as a benchmarked model, below Claude-4.6 (0.495) and Qwen3.5-397B (0.508).

(2) Other judges, including a weaker one. We re-scored every system with three more judges. Four-benchmark average:

Eval judgeBase+Dr. Zero+ARISE-RLMargin
gpt-5.2 (paper)0.4770.5310.569+3.8
Claude-4.6-Sonnet0.4570.5060.546+4.0
Qwen3.5-397B0.4680.5190.559+4.0
Qwen3-32B (weaker)0.4370.4840.519+3.5

Absolute scores vary by judge strictness, but rankings remain stable (Kendall τ=0.93\tau=0.93; only one adjacent pair differs).

(3) Judge-swap training. If the gains were fitted to gpt-5.2's preferences they should not survive a change of judge. Training with Qwen3.5-397B in every training-time LLM role (rubric scoring, coach, format check) and evaluating with gpt-5.2 gives 0.556; the paper model evaluated with Claude-4.6 gives 0.546. Measured within each evaluation judge, so the denominators match: +0.092 with gpt-5.2 throughout; +0.079 (86% retained) for the 397B-trained run scored by gpt-5.2; +0.089 (97% of its own gpt-5.2 gain) for the paper model under Claude-4.6. Both also beat Dr. Zero under the same judge (0.531, 0.506).


Response to Reviewer H1aE (2/4): Human validation and reward hacking (W1 + Q2, part 2)

(4) Human validation. 200 responses per ECR benchmark (50 queries ×\times 4 systems), three annotators per rubric item. The item counts are large, but the number of independent decisions is 50 queries per benchmark.

#ItemsLLM-humanCohen's κ\kappaHuman-human
ECR-DR1,38682.9%0.6685.6%
ECR-Travel82279.1%0.6182.4%

Agreement is 2.7 (ECR-DR) and 3.3 (ECR-Travel) points below human-human agreement, close to the ceiling that human disagreement itself imposes. The judge does not favour our model: judge-minus-human is +1.1 (base), +0.9 (Dr. Zero), +1.0 (Absolute Zero), +0.8 (ARISE-RL). The four span 0.3 points, which reads as uniformly mild leniency rather than a system trend; with 50 queries we do not claim 0.3 points is itself significant.

(5) Reward hacking: why it cannot happen, and that it does not. Four structural safeguards:

  1. The Generator is never rewarded for high judge scores. RdiffR_{\mathrm{diff}} peaks when about half the Solver's attempts succeed and is zero at c=0c=0 or c=Kc=K, so vacuous rubrics (always satisfied) and impossible ones earn nothing; Generator-Solver collusion is not profitable.
  2. RtoolR_{\mathrm{tool}} forbids tool-dependent criteria not backed by tool evidence.
  3. The judge is frozen, runs at temperature 0, and is never trained.
  4. Test rubrics are expert-written or external, so exploiting generated rubrics cannot raise reported scores.

Empirically, we extended training from 3 to 6 cycles and tracked the two signatures hacking would leave, namely a widening gap to a held-out judge or degenerate generated rubrics:

Cycle123456
Avg., gpt-5.2 judge0.5080.5420.5690.5830.5900.592
Avg., held-out Claude-4.6 judge0.4850.5180.5460.5590.5660.567
Valid generated rubrics (human audit)94.1%93.6%94.4%93.8%93.2%93.5%
Trivially satisfiable rubrics2.1%1.8%1.6%1.9%2.0%1.7%

The stable 0.023–0.025 judge gap and rubric validity (100 random pairs per cycle), alongside judge-swap results, indicate genuine Solver improvement rather than gpt-5.2 exploitation.

Response to Reviewer H1aE (3/4): Cost and sensitivity (W2 + Q1, part 1)

W2: Coach-memory extraction and verification add engineering overhead and prompt sensitivity not analysed across architectures.

Q1: How sensitive is the gating threshold τ\tau?

A: The pipeline is light, and RG-SED is stable to every knob that feeds it: cost, prompt, coach model and threshold.

(1) What it involves. The coach is one LLM call per group, one fixed template per role (App. D). Its input is only the pass/fail results and tool-call lists of the 8 standard rollouts; it returns ≤\le 300 characters and never sees a reference answer or writes trajectories. There is no separate verifier; the reward gate reuses judge scores the pipeline already computes. RG-SED is a 282-line patch to the slime trainer, in our release.

(2) Cost. Each group has 8 standard + 8 memory-augmented rollouts, the same total as GRPO with G=16G=16; the overhead is the sequential phases, the coach call and one teacher forward pass. Wall-clock per 100 Solver steps (ECR-Travel, 32$\times$H20):

RolloutJudgeCoachTeacher fwdUpdateTotal
GRPO (GG=16)6.8h1.9hn/an/a1.6h10.3h
+ RG-SED7.6h1.9h0.4h0.5h1.6h12.0h ($\times$1.17)

Inference cost is unchanged (the deployed policy runs without memory). Spending the same budget on more GRPO instead (G=20G=20, $\times$1.21) gives 0.507 versus 0.569 for RG-SED.

(3) Prompt and coach model. Three alternative coach prompts (P1 independently paraphrased, P2 structured JSON, P3 a degenerate one-liner) and three alternative coaches:

VariantVitaBenchECR-TravelGate rateCoachAvg.Gate rate
P0 (paper)0.4170.59972.1%gpt-5.2 (paper)0.56972.1%
P1 paraphrase0.4140.59671.4%Claude-4.60.56673.0%
P2 structured0.4190.59472.6%Qwen3.5-397B0.56066.9%
P3 minimal0.4060.58363.5%Self (9B)0.55157.8%
w/o RG-SED0.3500.518n/aw/o RG-SED0.502n/a

The two independently written prompts (P1, P2) move the four-benchmark average by at most 0.5 points. The deliberately degenerate P3 costs more (1.4 points on the multi-tool average), and we report it rather than hiding it. Even self-coaching keeps 73% of the gain. The gate explains why: worse guidance yields fewer memories with Δr>τ\Delta_r>\tau, so the gate fires less often (63.5% for P3, 57.8% self-coach). Low-quality guidance therefore leads to less distillation, not to distilled noise.


Response to Reviewer H1aE (4/4): The threshold across backbones (W2 + Q1, part 2)

(4) Threshold. Multi-tool average (VitaBench + ECR-Travel)/2, gate rate in parentheses:

τ\tauQwen3.5-9BQwen3-8BLlama-3.1-8B-Instruct
00.471 (83.4%)0.262 (85.1%)0.226 (79.2%)
0.020.490 (78.6%)0.279 (80.3%)0.238 (73.5%)
0.05 (paper)0.508 (72.1%)0.288 (74.6%)0.245 (66.8%)
0.100.499 (55.3%)0.291 (58.2%)0.243 (50.1%)
0.200.477 (31.8%)0.271 (34.7%)0.229 (27.4%)
w/o RG-SED0.4340.2510.208

On all three backbones the curve is a flat inverted U with its optimum in [0.05,0.10][0.05,0.10]; the default stays within 0.3 points of each optimum, and every τ>0\tau>0 beats both τ=0\tau=0 and no RG-SED.

Two rows are easy to confuse with Table 2, so we flag them. The τ=0\tau=0 row still applies the gate's sign test (distil only where Δr>0\Delta_r>0, no margin) and scores 0.471; Table 2's w/o reward gating removes the indicator entirely, also distilling the 8.3% of groups with Δr<−τ\Delta_r<-\tau, and scores 0.447 on the same multi-tool average. The sign test alone is thus worth +2.4 and the margin a further +3.7 (0.508): of RG-SED's +7.4 over no RG-SED (0.434), the sign test recovers half.

τ\tau transfers because Δr\Delta_r measures normalized reward gaps, independent of backbone logits. Empty-memory controls show near-zero means and s.d. 0.039–0.044 across backbones, placing τ=0.05\tau=0.05 at 1.1–1.3$\sigma$. False activations add variance, not bias, as memory rollouts provide no advantage to imitate. Real coach memory yields Δr=0.093\Delta_r=0.093 (2.3$\sigma$), clearly above the noise floor. Threshold sweeps further confirm robustness.


Summary. (1) Evaluation is non-circular: test rubrics are independent, overlap is negligible, and external benchmarks improve comparably. Results are robust across four judges (τ=0.93\tau=0.93), judge-swap training (86–97% retention), and a larger human study, with no evidence of reward hacking. (2) RG-SED adds only 1.17$\times$ training cost and no inference overhead. It is robust to prompts, coaches, and backbones; τ=0.05\tau=0.05 requires no backbone-specific tuning, while the sign test alone recovers half the gain. We hope these results address the concerns and merit reconsideration of the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.