acceptodds
Under review as a conference paper at ICLR 2027

Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents

Abstract

Experience-driven self-evolving agents aim to overcome the static nature of large language models by distilling reusable experience from past interactions, thus enabling adaptation to novel tasks at deployment time. This process places substantial demands on the foundation model's capacities for abstraction, generalization, and in-context learning. However, existing methods either rely on external system-level designs or optimize only isolated components, without jointly improving the underlying model's capabilities to extract and utilize experience. To this end, we propose **Evolving-RL**, an algorithmic framework that jointly improves experience extraction and utilization. Specifically, we center the learning process on experience extraction and evaluation, using the two supervisory signals derived from evaluation to optimize the extractor and solver separately and thus enable their coordinated co-evolution. Experiments on ALFWorld, ScienceWorld, and Mind2Web demonstrate that Evolving-RL substantially improves experience extraction and reuse, enabling more effective experience-assisted out-of-distribution (OOD) adaptation. Under the same skill-conditioned protocol, it achieves 98.7% relative improvement over GRPO on ALFWorld unseen tasks and 35.8% in overall action accuracy on Mind2Web, together with a 6.3-point OOD score improvement on ScienceWorld. Furthermore, Evolving-RL also serves as an experience-augmented RL algorithm: the Evolving-RL-trained Qwen2.5-7B-Instruct model substantially outperforms its GRPO counterpart on ALFWorld and Mind2Web even without test-time skill injection, demonstrating benefits for both explicit skill reuse and the underlying policy. The code will be released after the review process.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, self-evolving agents, reinforcement learning, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8SKjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Evolving-RL trains one policy to do two jobs: extract reusable skills from experience and use them. For each source task, the policy solves the task and then writes N=8N=8 candidate skills from its trajectory. Each candidate is scored by the mean reward the same policy gets when solving K=4K=4 retrieved related tasks with that skill (Eq. 1–3), and the score trains the extractor. The same N×KN\times K rollouts also train the solver (Eq. 4–5). Appendix A adds stability fixes and a bound on how reliably KK rollouts can rank skills. Experiments with Qwen2.5-7B and 3B cover ALFWorld with two held-out task types, Mind2Web and ScienceWorld. The headline result is 88.6% on unseen ALFWorld. Ablations, a token-matched control and five seeds back this up.

Critical strengths

S1. The training signal for extraction is simple and well posed. Every candidate skill is scored on the same task set, so the extractor's GRPO group is a fair comparison. The same rollouts are then reused for the solver at the same budget as GRPO (Table 5). This idea is easy to reuse, and it is the main reason I lean positive.

S2. The main comparisons are better controlled than is usual in this area. Co-evolution beats solver-only training on unseen types (Table 3), and the gap holds over five seeds (Table 7). Fig. 1(a) fixes the solver and the source trajectories and still finds the trained skills more useful. The token-matched GRPO run (App. B.1) rules out the simplest compute explanation.

Critical weaknesses

W1. The headline gain is mostly a better policy, not test-time self-evolution. On unseen ALFWorld, 36.5 of the 44-point gap over GRPO is already there with no skills (81.1 vs. 33.7, Table 1). Injecting extracted skills adds about 7.5 points, measured on 35 tasks. The result is interesting, but it is not the "extract and reuse experience at test time" story the abstract leads with. On ScienceWorld OOD the no-skill policy is actually worse than GRPO (9.39 vs. 11.34). Table 4 (3B) has no GRPO (w/ skills) row, so L465–466 compares a with-skills method against a no-skills baseline. On 7B, skill injection alone gave GRPO +10.9 points. I would like the headline split into policy gain and test-time skill gain, plus the missing 3B row.

W2. The OOD and Mind2Web evidence is thinner than the text suggests. ALFWorld "unseen" is two types and 35 tasks, ScienceWorld OOD is a single random split, and the baselines are single runs. On Mind2Web, test-time skills come from other test tasks. Since rollouts stop at the first wrong action, a source trace reveals the gold prefix of a sibling task. This is transductive use of the test split. The metric is also non-standard, and success rates are around 2%. In addition, Fig. 1(a) (7B: 44.0 / 48.8) does not match Table 1 (45.5 / 44.5). A second held-out split, seeds for the baselines, and a Mind2Web run with training-split skill sources would address most of this.

Questions for authors
  1. [Clarification] Does the ALFWorld training retrieval pool exclude all Look and Pick2 instances? Look reaches 100% with no skills, against 53.3% for GRPO, which surprised me.
  2. [Clarification] Can you add GRPO (w/ skills) for 3B in Table 4? If Evolving-RL still clearly wins, W1 matters much less to me.
  3. [Clarification] What differs between Fig. 1(a) and Table 1 for the 7B base model?
  4. [Clarification] How is the GRPO baseline grouped (tasks × rollouts per update)? Evolving-RL's solver sees 64 distinct tasks per update, and that diversity alone could explain part of the unseen gap.
  5. [Suggestion] A Mind2Web run where skills are extracted only from training tasks, as on the other two benchmarks, would remove the transductive concern.
3 comments

We sincerely thank you for the careful reading and for recognizing our controlled skill evaluation, trajectory reuse, and budget matching. We address the distinction between policy improvement and test-time experience use first, then clarify the experimental controls. We also provide the requested GRPO (w/ skills) results for the 3B model and a Mind2Web experiment using only training-task experience.


W1: How can self-evolution capability be evaluated separately?

Your distinction between a stronger policy and better use of experience is central to evaluating self-evolution. Training experience extraction and utilization also strengthens the shared policy, so final task success alone cannot fully separate these contributions. We also recognized this issue and therefore included the controlled experiments in Figure 1(a) and Figure 3(b) in the original submission to examine skill extraction and test-time refinement more directly.

Figure 1(a) changes only the skill extractor, holding the solver, source trajectories, feedback, and target tasks fixed. The larger gains from Evolving-RL's skills thus demonstrate a better ability to turn the same experience into useful guidance. Figure 3(b) examines the next step: refining that guidance through interaction. Both models start with the same trajectories and feedback and iteratively refine their skills with model parameters fixed. Evolving-RL improves task performance through this process, demonstrating self-evolution through experience refinement at test time.


Q1: Does the training retrieval pool exclude Look and Pick2?

Yes. All Look and Pick2 instances are excluded from the training retrieval pool. Source tasks, downstream retrieval, and skill evaluation during training use only Pick, Heat, Cool, and Clean, so neither held-out type contributes to parameter updates.

The strong no-skill result on Look can be understood through the basic capabilities it shares with training task types: search, navigation, object pickup, and state assessment. These capabilities can transfer even though Look itself is held out. We examined this explanation further by reproducing the concurrent work Skill1 under the same task-type split:

Unseen task typeSkill1 (w/o skills)Skill1 (w/ skills)
Look77.78%83.33%
Pick220.00%37.50%
Unseen overall49.7%61.1%

Skill1 also achieves relatively strong performance on Look despite not being trained on that type, consistent with this explanation.


Q2: Additional GRPO (w/ skills) results for the 3B model

We added the missing GRPO (w/ skills) results for Qwen2.5-3B-Instruct so that both methods can be compared with skill guidance. The tables place these new results alongside GRPO (w/o skills) and Evolving-RL (w/ skills) from the original submission.

ALFWorld task success rate (%):

MethodSeenUnseenOverall
GRPO (w/o skills)88.150.978.4
GRPO (w/ skills)89.253.379.8
Evolving-RL (w/ skills)92.960.084.3

Mind2Web action accuracy (%):

MethodCTCWCDOverall
GRPO (w/o skills)19.0313.6613.0214.23
GRPO (w/ skills)19.4414.0512.8914.27
Evolving-RL (w/ skills)25.5316.1917.7318.99

ScienceWorld environment score:

MethodIDOOD
GRPO (w/o skills)7.667.25
GRPO (w/ skills)7.327.83
Evolving-RL (w/ skills)10.1211.44

Evolving-RL retains its advantage over GRPO across all three environments when both methods receive skills, and also outperforms GRPO without skills. We have added the new results to Table 4 (p. 9) and updated the corresponding comparisons in Section 4.3 (p. 9, lines 464–471).

Q3: Why do the results in Figure 1(a) and Table 1 differ?

The reported values come from independent experiments with different purposes; we did not reuse the same measurements. Although we evaluated the base model five times in each experiment, small variations during inference can still accumulate across multi-step trajectories, resulting in differences in the reported success rates.


Q4: How are GRPO tasks and rollouts grouped?

GRPO uses 64 distinct tasks per update. Evolving-RL retrieves 4 downstream tasks for each of 16 source tasks, yielding at most 64 distinct downstream tasks because retrievals may overlap. Its solver therefore sees no more distinct tasks per update than GRPO, so greater task diversity cannot explain the gains.


W2/Q5: Extracting Mind2Web skills only from training tasks

Following your suggestion, we evaluated Evolving-RL using only training-task trajectories for skill extraction. Below, we compare this new setting with the original results from Table 2 under the same evaluation protocol.

Action accuracy (%):

Skill sourceCTCWCDOverall
No skills35.3235.1524.6728.05
test tasks41.9935.1426.9730.87
Training tasks38.2034.7026.3329.67

Task success rate (%):

Skill sourceCTCWCDOverall
No skills1.593.391.211.57
test tasks2.381.811.911.99
Training tasks1.982.821.641.86

Training-task skills still improve both overall metrics over no skills. Their slightly lower gains than test-task skills are consistent with weaker source–target matching across splits. We have added the training-source experiment’s overall results to the existing Mind2Web setting in Appendix E.3 and clarified the distinction from the official action-accuracy metric (p. 21, lines 1129–1134).

Dear Reviewer 8SKj

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve the quality of our work.

We would like to kindly follow up on our previous rebuttal and ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications.

Thank you again for your time, consideration, and valuable feedback.

Best regards, The Authors

Reviewer_4ZYVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Evolving-RL, a framework designed to optimize the self-evolution capabilities of experience-driven agents in an end-to-end manner. The method represents experiences as textual skills and jointly optimizes the extraction and utilization of these skills through a shared policy. The model generates candidate skills from source task trajectories, trains the extractor based on their average performance on related downstream tasks, and reuses the trajectories to train the solver. Ultimately, the extractor and solver losses are jointly optimized, creating a process of co-evolution between experience extraction and utilization. The authors conducted experiments on ALFWorld, Mind2Web, and ScienceWorld; the results demonstrate that Evolving-RL outperforms GRPO, SkillRL, and several prompt-based experience methods in skill-conditioned evaluations.

Critical strengths
  1. This paper points out that experience-driven self-evolution depends not only on a model's ability to utilize experiences but also on its capacity to extract experiences with transfer value. Compared to optimizing the solver alone, the joint study of experience extraction and utilization more closely reflects the complete self-evolution process.

  2. This paper defines the reward based on the average performance of candidate skills across relevant downstream tasks. This directly aligns the skill extraction objective with actual cross-task transfer capability. Furthermore, downstream evaluation provides rewards for both the extractor and the solver. This design enables the efficient reuse of costly environment interactions and establishes a relatively concise joint training framework.

Critical weaknesses
  1. During both training and inference, inference trajectories on the source task receive environmental feedback; in my view, this introduces additional test-time information, environmental interaction, and inference computation. The so-called OOD evaluation effectively allows access to feedback derived from OOD-type results. While this protocol might be suitable for experiments on experience sharing during inference, it should not serve as a standard evaluation against an independent test set.

  2. The paper acknowledges that Skill1 and Complementary RL also investigate the co-evolution of experience and policy, yet no experimental comparisons are provided. Given the high relevance of these methods to the paper's core contributions, comparing solely against baselines that primarily optimize experience utilization is insufficient to fully establish the method's novelty and relative advantages.

Questions for authors
  1. Regarding Weakness 1, the authors are advised to report evaluation results based on scenarios where only source trajectories are provided (without result feedbacks) or where the model itself determines the result labels. Alternatively, other experiments could be conducted to reasonably rule out the introduction of extraneous information during testing.

  2. When the extractor loss and the solver loss act on the same policy, how consistent are their gradient directions? Is there significant conflict between the two optimization objectives, and does joint optimization perform worse than single-objective optimization?

  3. Could control experiments be added to better analyze whether the joint optimization model outperforms single-objective optimization in both skill extraction and skill solving? Specifically, please include the following experiments:

    • Skill extractor uses an Extractor-only model; solver uses Co-evolution model.
    • Skill extractor uses a Co-evolution model; solver uses Solver-only model.
    • Skill extractor uses an Extractor-only model; solver uses Solver-only model. This approach would minimize confounding factors and allow for a direct comparison of improvements in individual capabilities.
  4. Is it possible to conduct a direct comparison with Skill1 or Complementary RL under the same experimental protocol? If comparison is not feasible due to concurrent work or unavailable code, the authors should more clearly articulate the significant contributions of Evolving-RL compared to these methods.

  5. Could the authors separately report the inference costs for "w/o skills" and "w/ skills"?

  6. Figure 1(b) mentions "Solver-only (Static Skills)"; please provide details on how the Solver-only model is trained.

3 comments

We sincerely thank you for the careful reading and for recognizing the value of joint optimization, cross-task skill evaluation, and trajectory reuse. Your questions help distinguish two aspects of the method: the information used to extract skills at test time and the benefits of training extraction and utilization jointly. We examine these through self-assessed trajectory outcomes and controlled extractor–solver combinations, then present a direct Skill1 comparison and clarify inference costs and training details. The appendix now includes the self-assessed-outcome experiment, Heat-task behavioral analysis, crossed extractor–solver results, inference costs, and Static Skills training details.


W1/Q1: The effect of ground-truth outcome labels at test time

To test whether experience transfer depends on externally supplied outcome labels, we removed these labels from the skill-extraction inputs on ALFWorld, as you suggested. In this experiment, the model assesses whether the source trajectory succeeds before extracting a skill. The table compares task success under the two sources of outcome labels and reports the accuracy of the model's own judgments.

Task typesTask success with ground-truth labelsTask success with self-assessed outcomesOutcome-judgment accuracy
Seen98.60%98.32%99.32%
Unseen88.60%88.60%81.63%
Overall96.00%95.77%94.92%

Unseen-task success is unchanged, and overall performance is essentially unaffected. The model can therefore extract useful experience from trajectories using its own outcome judgments, preserving the transfer benefits without externally supplied ground-truth labels. We have added this experiment and its analysis to Appendix E.1.1 and Table 8 (p. 18, lines 937–950).


Q2: Alignment between the extractor and solver objectives

We first examined how the objectives affect the shared parameters by comparing each trained model's full-parameter update relative to the same base model:

Model pairCosine similarity
Extractor-only vs. Solver-only0.0135
Extractor-only vs. Co-evolution−0.0029
Solver-only vs. Co-evolution0.0300

All three pairs are nearly orthogonal, including the joint model compared with either single-objective model. Near-orthogonality is also typical of independently and uniformly sampled directions in high-dimensional spaces, so nearly orthogonal updates under different training objectives are unsurprising. These similarities alone do not reveal whether the objectives promote shared capabilities. We therefore examine their alignment through their effects on agent behavior.

Table 3 already shows that Extractor-only achieves substantially higher overall task success than the base model without skill injection, despite receiving no direct training on task execution. To understand how this improvement manifests in agent behavior, we further compared the base model, Extractor-only, and Solver-only on a single ALFWorld task type, Heat:

MetricBaseExtractor-onlySolver-only
Repeated actions ↓40.82%30.56%9.69%
Ineffective actions ↓31.20%22.81%2.40%
Successful target heating ↑18.84%42.03%100.00%

Repeated and ineffective actions are measured as proportions of actions, with repetitions counted within trajectories and ineffective actions identified by “Nothing happens” feedback. Successful target heating measures the proportion of trajectories that successfully heat the target object.

The reductions in repeated and ineffective actions and the improvement in target heating show that extraction training also promotes more effective execution. Q3 examines how these capabilities combine when skills are used. We have added the Heat-task behavioral analysis to Appendix E.2 and Table 11 (p. 20, lines 1066–1079).

Q3: Controlled combinations of extractors and solvers

The combinations you propose allow us to examine each capability while holding the other component fixed. We evaluated all three alongside the complete Co-evolution model.

Skill extractorSolverSeen successUnseen successOverall success
Extractor-onlySolver-only97.0%79.0%92.30%
Extractor-onlyCo-evolution97.0%78.1%92.06%
Co-evolutionSolver-only98.3%81.9%94.02%
Co-evolutionCo-evolution98.6%88.6%95.99%

For either fixed solver, replacing Extractor-only with the jointly trained extractor improves performance. With the jointly trained extractor held fixed, the jointly trained solver also uses its skills more effectively than Solver-only. The complete Co-evolution model achieves the best results and outperforms the combination of separately trained extractors and solvers. These comparisons identify improvements in both capabilities and show their combined benefit. We have added the complete results and analysis to Appendix E.2 and Table 12 (p. 21, lines 1080–1093).


W2/Q4: Comparisons with Skill1 and Complementary RL

We reproduced Skill1 under our experimental setting to directly compare methods that jointly develop experience and policy. The ALFWorld results are:

MethodEvaluation settingSeen successUnseen successOverall success
Skill1w/o skills94.2%49.7%82.6%
Evolving-RLw/o skills97.4%81.1%93.1%
Skill1w/ skills91.6%61.1%83.6%
Evolving-RLw/ skills98.6%88.6%96.0%

Evolving-RL outperforms Skill1 with and without skills, especially on unseen tasks, demonstrating gains in both experience reuse and the underlying policy.

A key difference is the supervision for skill extraction. Skill1 uses a proxy reward based on the current trajectory's outcome relative to the skill library's utility trend (Section 3.2, Eq. 7). Evolving-RL directly measures each skill's average performance across related downstream tasks. Candidates from the same source are evaluated on the same task set for a fair comparison, and these evaluation trajectories also train the solver.

CRL is independent concurrent work. It initially explored shared weights, but switched to separate actor and extractor parameters after encountering instability attributed to parameter conflicts (Appendix B.2). We investigated similar instability through the relationship between noise in finite-sample skill evaluation and increasing extractor entropy. Reward filtering for abnormal outputs and extractor entropy control enabled stable joint training, as analyzed in Appendix A.2 and Figure 4 (p. 15). The revised discussion explicitly identifies CRL’s shared-parameter instability (p. 15, lines 797–802).

This allows one model to support both capabilities, avoiding separate extractor parameters and optimizer states during training and the overhead of maintaining and coordinating two models at deployment.


Q5: Inference costs with and without skills

We measured how skill injection changes inference costs during target-task solving on ALFWorld. The comparison reports average actions, cumulative input and output tokens, and generation-request time for this stage.

Evaluation settingAverage actionsCumulative input tokensCumulative output tokensGeneration-request time (s)
w/o skills13.3043,028.48195.4410.98
w/ skills12.5344,395.04211.6512.36

Skill injection improves task success while reducing the number of interaction actions. The additional guidance increases input and output token usage and generation-request time, but these increases are modest and represent an acceptable overhead. We have added the table and its measurement scope to Appendix E.1.1 and Table 10 (p. 19, lines 1009–1020).


Q6: Training details for “Solver-only (Static Skills)” in Figure 1(b)

In Figure 1(b), “Static Skills” means that the skills are generated in advance and held fixed throughout training. The Solver-only model is then updated using only the solver loss: we remove the extractor loss and keep all other experimental settings identical to Evolving-RL.

This gives a control in which the solver learns to use a fixed set of skills. Evolving-RL generates skills throughout training and jointly optimizes extraction and utilization, allowing both capabilities to develop together. We have added these training details to Appendix E.2 (p. 20, lines 1062–1064) to make the distinction explicit.

Dear Reviewer 4ZYV,

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve the quality of our work.

We would like to kindly follow up on our previous rebuttal and ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications.

Thank you again for your time, consideration, and valuable feedback.

Best regards, The Authors

Reviewer_bYHROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

To address existing agents’ limited ability to learn from past experience, Evolving-RL jointly trains experience extraction and utilization through reinforcement learning. The model extracts reusable textual skills from task trajectories and evaluates them on related tasks. The resulting rewards improve both skill extraction and task execution within a shared model, leading to stronger generalization even without skill guidance at test time.

Critical strengths

The framework efficiently reuses downstream evaluation trajectories to train both experience extraction and utilization, aligning skill generation with practical task performance.

The experiments include objective ablations, skill-relevance tests, and cross-policy skill transfer, providing useful evidence beyond the main performance comparisons.

Critical weaknesses
  1. The authors distinguish Evolving-RL from Complementary RL through a shared model for experience extraction and utilization, and skill evaluation across multiple related tasks. However, the paper neither includes Complementary RL as a baseline nor explains its omission. It therefore remains unclear whether using a single policy to extract and use experience offers advantages over two separate policies, such as better performance or lower computational cost. The architectural difference alone does not establish the superiority of either approach. [1] Complementary RL: Towards Efficient Experience-Driven Agent Learning

  2. The paper allows experience collection on the Mind2Web test set, but does not clarify whether skills extracted from a test task are excluded when evaluating that same task. Without this separation, the reported gains may reflect learning from previous attempts on the same task rather than transferring experience across tasks.

  3. ALFWorld has already reached near-ceiling performance in several researches, limiting its ability to distinguish between strong methods. For example, Skill-DisCo[2] reports 99.3% success on the official unseen split and shares the goal of distilling reusable experience from past interactions to support adaptation to new tasks at deployment time. This result motivates evaluation on additional, more challenging environments to establish the broader effectiveness of the proposed approach. [2] Skill-DisCo: Distilling and Compiling Agent Traces into Reusable Procedural Skills

  4. On Mind2Web, the paper computes action accuracy only over actions executed before termination at the first error, whereas the original benchmark evaluates every annotated step independently. This makes its action accuracy difficult to compare with existing results, while the overall task success rate remains low at 1.99%. Moreover, on the cross-website split, skill injection leaves action accuracy essentially unchanged (35.15% to 35.14%) and reduces task success from 3.39% to 1.81%, leaving the benefit of experience reuse on unseen websites unclear.

  5. This two benchmarks provide limited resolution for assessing the method: ALFWorld is near saturation in several settings, making strong methods difficult to distinguish, while Mind2Web yields very few fully completed tasks, making task-level gains harder to assess reliably. Given the low success rate on Mind2Web, the authors could evaluate Evolving-RL with a stronger, more recent LLM rather than relying solely on the roughly two-year-old Qwen2.5-7B.

Questions for authors

see Weaknesses

4 comments

We sincerely thank you for the careful reading and for highlighting the value of trajectory reuse and our ablation and transfer experiments. Your comments raise three central questions: what the unified design contributes, what information the evaluation protocol permits, and how well the benchmarks distinguish between methods. We address these below, drawing on the original controlled experiments and new Mind2Web results with Qwen3.5-9B Base.


W1: Comparison with Complementary RL and the benefits of a unified model

CRL was developed independently and concurrently with Evolving-RL and was made public on arXiv only when our research was largely complete. This is why the original submission discusses its relationship to our method but does not include it as an experimental baseline.

The practical advantage of our unified design is that one set of trained parameters supports both experience extraction and utilization. During training, this avoids maintaining separate trainable parameters and optimizer states for a dedicated extractor. At deployment, the same model executes tasks and extracts or revises experience from new interactions, eliminating the need to load, maintain, and coordinate a separate extractor model. These benefits are especially relevant to agents that continue to acquire experience during use.

Realizing these benefits requires stable joint optimization. CRL also initially explored training its actor and extractor with a single set of shared weights. As described in its Appendix B.2, this approach encountered training instability, which the authors attributed to parameter conflicts; they subsequently switched to separate parameter sets. We encountered similar instability and investigated the relationship between noise in finite-sample skill evaluation and increasing extractor entropy. Filtering rewards for abnormal outputs and controlling extractor entropy enabled stable joint training within a shared policy. Appendix A and Figure 4 of our original submission present this analysis and compare reward, KL, and entropy curves with and without these controls. Identifying and addressing this instability so that both capabilities can be trained within one model is itself a contribution of our work.


W2: Is the target task excluded from the source of its test-time skill on Mind2Web?

Yes—the target is excluded before its skill source is retrieved. This exclusion was already part of the original protocol. The revised manuscript retains the explicit statement in Section 4.1 (p. 6, lines 310–312): “Mind2Web uses other official test-split tasks with the target excluded. No target is used as its own source.” The same requirement appears in Appendix E.1.1 (p. 18, lines 928–935 and 959–968), with the source pools and information boundaries summarized in Table 9 (p. 19). The skill-conditioned Base Model, GRPO, and Evolving-RL all follow this protocol.


W3: ALFWorld saturation and the ability to distinguish between methods

Skill-DisCo's 99.3% success rate provides useful context for assessing ALFWorld, although its model and evaluation setting differ from ours. It uses GPT-4o, CodeAct, and compiled executable skills on the official unseen-environment split. Our main experiments use Qwen2.5-7B-Instruct with textual skills and additionally hold out entire task types during training. The benchmark's ability to distinguish methods is therefore best assessed through comparisons under the same conditions.

Our task-type split makes the evaluation more challenging by requiring transfer across both environments and task types: Look and Pick2 are excluded from policy optimization and training-time skill evaluation. With related methods evaluated under the same conditions, Evolving-RL substantially outperforms both GRPO and SkillRL in Table 1. These controlled comparisons demonstrate that our ALFWorld setting remains informative for distinguishing methods.

The original submission also includes ScienceWorld, where Table 2 shows clear gains over GRPO under the same skill-conditioned protocol. This provides an additional environment that distinguishes the methods beyond ALFWorld.

W4: The Mind2Web action-accuracy metric and skill benefits on cross-website tasks

We chose this metric to evaluate sequential task execution, which is the capability our agentic RL training aims to improve. Our model learns through agent interactions rather than supervised fine-tuning on Mind2Web's expert trajectories. Evaluating from the initial state therefore tests whether the agent can reach and correctly execute successive steps. By comparison, the official independent-step metric supplies each annotated state regardless of whether the agent could reach it. Our action accuracy complements task success by measuring correctness during attempted execution, including on tasks that are not fully completed.

Within this setting, all methods and baselines follow the same sequential protocol, ensuring a fair comparison. Comparisons with other Mind2Web studies are also valuable, but they require accounting for differences in experimental setup. For example, some studies represent webpages as HTML, while others use accessibility trees. The same model can achieve different scores under these settings, so results reported across papers may not be directly comparable.

At the same time, we agree that Qwen2.5-7B-Instruct's limited performance on Mind2Web makes it harder to distinguish methods by task success, and that skill injection reduces CW task success in the original results. Following your suggestion, we therefore added experiments with the stronger Qwen3.5-9B Base model, as reported in W5. With this backbone, skill injection improves both CW action accuracy and task success rate.


W5: Evaluation with a newer, stronger backbone

Following your suggestion, we evaluated GRPO and Evolving-RL with Qwen3.5-9B Base using the same training settings, evaluation protocol, and metrics as in the paper. Results are averaged over five evaluations per trained checkpoint (%, AA: action accuracy; SR: task success rate).

MethodCT AACW AACD AACT SRCW SRCD SR
GRPO (w/o skills)41.9139.8634.383.183.393.62
GRPO (w/ skills)42.1339.1633.933.593.413.51
Evolving-RL (w/o skills)43.2943.4832.604.374.523.18
Evolving-RL (w/ skills)44.7349.2936.224.376.783.84

Under the same skill-conditioned protocol, Evolving-RL outperforms GRPO on both metrics across all three splits, with particularly clear gains on CW. Within Evolving-RL, skill injection improves action accuracy on every split. The CW results thus directly demonstrate the benefit of experience reuse on unseen websites with the stronger backbone.

These results extend our findings to a newer, stronger backbone. We have included the results in Table 14 and the evaluation details in Appendix E.4 (p. 23, lines 1217–1231).

Based on the authors’ response, I have decided to raise my score from 1 to 2.

Thank you for taking the time to read our rebuttal and reconsider your assessment. We appreciate your thoughtful feedback and are glad that our clarifications addressed your concerns.

Comments on the paper

3 comments

Dear Reviewers,

We sincerely thank all reviewers for your time, careful reading, and constructive feedback, which have helped us strengthen our paper.

We thank Reviewer bYHR for revisiting our responses and updating the assessment, and Reviewer 8SKj for the positive assessment and recognition of our experimental controls and training design. We appreciate your thoughtful consideration of our work.

As the discussion deadline approaches, we would be grateful if Reviewers 4ZYV and 8SKj could revisit our responses when your schedule permits and let us know whether the additional experiments and clarifications address your remaining concerns. These additions include controlled extractor–solver comparisons, the requested 3B GRPO baseline, and evaluations examining the information used for test-time skill extraction. We hope the updated evidence will be helpful in your final assessment.

Thank you again for your valuable feedback and consideration. We would be happy to clarify any outstanding questions.

Best regards,
The Authors

Dear Area Chair,

Thank you very much for your time and effort in overseeing the review process for our submission #29010 (Evolving-RL).

We have provided detailed responses to the reviewers’ concerns, added experiments and analyses, and posted follow-up comments. Reviewer bYHR has revisited our responses and updated the assessment. We have not yet received follow-up feedback from Reviewers 4ZYV and 8SKj and would appreciate the opportunity to understand whether any concerns remain.

As the discussion period draws to a close, we would be grateful if you could kindly remind these two reviewers to revisit our responses, if time permits. We hope to clarify any outstanding questions and help ensure that the additional evidence can be considered in their final assessments.

We fully respect the reviewers’ judgments and sincerely appreciate your time, consideration, and assistance.

Best regards,
The Authors

AuthorsOpenReview ↗

For AC, SAC and PC: Summary of Rebuttal Process

Dear Area Chairs, Senior Area Chairs, and Program Chairs,

As the discussion concludes, we thank the reviewers for their careful feedback and you for overseeing our submission. We propose Evolving-RL, a framework that improves both experience extraction and experience utilization in self-evolving agents. By reusing downstream task trajectories, it efficiently aligns the quality of extracted experience with success on actual tasks while allowing the two capabilities to reinforce each other during training. The paper includes extensive experiments, comparisons with other approaches, and in-depth analyses of our design. During rebuttal, we added controlled experiments and clarified our evaluation protocol and claims in response to the reviews.

1. Reviewer bYHR (Rating: 1 → 2)

Main concerns: The reviewer asked what advantages our shared-model design offers over the separate-model design of the concurrent work Complementary RL and raised a concern about potential target-task information leakage in evaluation. The reviewer also questioned the benchmark setting and evaluation metric, noting that the original backbone’s limited performance on Mind2Web makes methods difficult to distinguish, and suggested evaluation with a stronger, more recent model.

Our response: We explained the practical benefits of a shared model and how our training design addresses the instability of joint optimization. The suspected target-task leakage was already excluded by the evaluation protocol stated in the original submission: a target task cannot serve as its own skill source. Our rebuttal clarified this existing safeguard; it did not change the protocol. We also explained the scope of our benchmark comparisons and sequential evaluation metric, and added Mind2Web experiments with Qwen3.5-9B Base. The new results provide clearer evidence for our method and for the benefit of experience reuse on unseen websites with a stronger backbone.

Reviewer’s follow-up: The reviewer revisited our responses and raised their rating. We appreciate their thoughtful reassessment.

2. Reviewer 4ZYV (Rating: 2)

Main concerns: The reviewer questioned whether externally supplied outcome labels introduce additional information during test-time skill extraction and requested controls separating the contributions of the extractor and solver. The reviewer also asked about the interaction between the two training objectives, comparisons with concurrent work, inference costs, and the Static Skills baseline.

Our response: We added an experiment in which the model assesses source-trajectory outcomes itself, showing that externally supplied result labels are not necessary for the observed transfer benefit. We evaluated controlled extractor–solver combinations to examine the contribution of each jointly trained capability, supplemented by analyses of model updates and agent behavior. We also added a direct experimental comparison with the concurrent work Skill1, explained how our design differs from the concurrent work Complementary RL, reported inference costs, and specified how the Static Skills control is trained.

3. Reviewer 8SKj (Rating: 3)

Main concerns: The reviewer observed that improvement in the underlying policy accounts for a substantial part of the headline gain and highlighted the missing skill-conditioned GRPO baseline as important for assessing our comparison. The reviewer also questioned the use of other test-split tasks as Mind2Web skill sources and asked for clarification of training-task diversity and differing measurements across experiments.

Our response: We clarified the distinction between policy improvement and test-time experience reuse, pointing to controlled skill comparisons and test-time refinement analyses that examine the latter more directly. We added the requested skill-conditioned GRPO baseline, confirming our advantage when both methods receive skill guidance. To address the separate concern about test-split skill sources, we also evaluated Mind2Web using skills extracted only from training tasks; the overall benefit over using no skills persists in this setting. Finally, we clarified how tasks are grouped during training.

Overall, we have addressed the reviewers’ main concerns through additional controlled experiments. It clarifies that the target-task leakage raised by Reviewer bYHR was already excluded in the original paper, while the separate question of using other test-split tasks as experience sources was addressed with an additional training-source experiment. Reviewer bYHR has reconsidered their assessment; Reviewers 4ZYV and 8SKj have not responded to our rebuttal or follow-up invitations. We respectfully ask that the additional experiments and clarifications be considered in the final assessment.

Thank you for your time and thoughtful consideration.

Best regards,
The Authors of Submission #29010

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.