- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Evolving-RL trains one policy to do two jobs: extract reusable skills from experience and use them. For each source task, the policy solves the task and then writes candidate skills from its trajectory. Each candidate is scored by the mean reward the same policy gets when solving retrieved related tasks with that skill (Eq. 1–3), and the score trains the extractor. The same rollouts also train the solver (Eq. 4–5). Appendix A adds stability fixes and a bound on how reliably rollouts can rank skills. Experiments with Qwen2.5-7B and 3B cover ALFWorld with two held-out task types, Mind2Web and ScienceWorld. The headline result is 88.6% on unseen ALFWorld. Ablations, a token-matched control and five seeds back this up.
Critical strengths
S1. The training signal for extraction is simple and well posed. Every candidate skill is scored on the same task set, so the extractor's GRPO group is a fair comparison. The same rollouts are then reused for the solver at the same budget as GRPO (Table 5). This idea is easy to reuse, and it is the main reason I lean positive.
S2. The main comparisons are better controlled than is usual in this area. Co-evolution beats solver-only training on unseen types (Table 3), and the gap holds over five seeds (Table 7). Fig. 1(a) fixes the solver and the source trajectories and still finds the trained skills more useful. The token-matched GRPO run (App. B.1) rules out the simplest compute explanation.
Critical weaknesses
W1. The headline gain is mostly a better policy, not test-time self-evolution. On unseen ALFWorld, 36.5 of the 44-point gap over GRPO is already there with no skills (81.1 vs. 33.7, Table 1). Injecting extracted skills adds about 7.5 points, measured on 35 tasks. The result is interesting, but it is not the "extract and reuse experience at test time" story the abstract leads with. On ScienceWorld OOD the no-skill policy is actually worse than GRPO (9.39 vs. 11.34). Table 4 (3B) has no GRPO (w/ skills) row, so L465–466 compares a with-skills method against a no-skills baseline. On 7B, skill injection alone gave GRPO +10.9 points. I would like the headline split into policy gain and test-time skill gain, plus the missing 3B row.
W2. The OOD and Mind2Web evidence is thinner than the text suggests. ALFWorld "unseen" is two types and 35 tasks, ScienceWorld OOD is a single random split, and the baselines are single runs. On Mind2Web, test-time skills come from other test tasks. Since rollouts stop at the first wrong action, a source trace reveals the gold prefix of a sibling task. This is transductive use of the test split. The metric is also non-standard, and success rates are around 2%. In addition, Fig. 1(a) (7B: 44.0 / 48.8) does not match Table 1 (45.5 / 44.5). A second held-out split, seeds for the baselines, and a Mind2Web run with training-split skill sources would address most of this.
Questions for authors
- [Clarification] Does the ALFWorld training retrieval pool exclude all Look and Pick2 instances? Look reaches 100% with no skills, against 53.3% for GRPO, which surprised me.
- [Clarification] Can you add GRPO (w/ skills) for 3B in Table 4? If Evolving-RL still clearly wins, W1 matters much less to me.
- [Clarification] What differs between Fig. 1(a) and Table 1 for the 7B base model?
- [Clarification] How is the GRPO baseline grouped (tasks × rollouts per update)? Evolving-RL's solver sees 64 distinct tasks per update, and that diversity alone could explain part of the unseen gap.
- [Suggestion] A Mind2Web run where skills are extracted only from training tasks, as on the other two benchmarks, would remove the transductive concern.
3 comments
Q3: Why do the results in Figure 1(a) and Table 1 differ?
The reported values come from independent experiments with different purposes; we did not reuse the same measurements. Although we evaluated the base model five times in each experiment, small variations during inference can still accumulate across multi-step trajectories, resulting in differences in the reported success rates.
Q4: How are GRPO tasks and rollouts grouped?
GRPO uses 64 distinct tasks per update. Evolving-RL retrieves 4 downstream tasks for each of 16 source tasks, yielding at most 64 distinct downstream tasks because retrievals may overlap. Its solver therefore sees no more distinct tasks per update than GRPO, so greater task diversity cannot explain the gains.
W2/Q5: Extracting Mind2Web skills only from training tasks
Following your suggestion, we evaluated Evolving-RL using only training-task trajectories for skill extraction. Below, we compare this new setting with the original results from Table 2 under the same evaluation protocol.
Action accuracy (%):
| Skill source | CT | CW | CD | Overall |
|---|---|---|---|---|
| No skills | 35.32 | 35.15 | 24.67 | 28.05 |
| test tasks | 41.99 | 35.14 | 26.97 | 30.87 |
| Training tasks | 38.20 | 34.70 | 26.33 | 29.67 |
Task success rate (%):
| Skill source | CT | CW | CD | Overall |
|---|---|---|---|---|
| No skills | 1.59 | 3.39 | 1.21 | 1.57 |
| test tasks | 2.38 | 1.81 | 1.91 | 1.99 |
| Training tasks | 1.98 | 2.82 | 1.64 | 1.86 |
Training-task skills still improve both overall metrics over no skills. Their slightly lower gains than test-task skills are consistent with weaker source–target matching across splits. We have added the training-source experiment’s overall results to the existing Mind2Web setting in Appendix E.3 and clarified the distinction from the official action-accuracy metric (p. 21, lines 1129–1134).
Dear Reviewer 8SKj
Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve the quality of our work.
We would like to kindly follow up on our previous rebuttal and ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications.
Thank you again for your time, consideration, and valuable feedback.
Best regards, The Authors
We sincerely thank you for the careful reading and for recognizing our controlled skill evaluation, trajectory reuse, and budget matching. We address the distinction between policy improvement and test-time experience use first, then clarify the experimental controls. We also provide the requested GRPO (w/ skills) results for the 3B model and a Mind2Web experiment using only training-task experience.
W1: How can self-evolution capability be evaluated separately?
Your distinction between a stronger policy and better use of experience is central to evaluating self-evolution. Training experience extraction and utilization also strengthens the shared policy, so final task success alone cannot fully separate these contributions. We also recognized this issue and therefore included the controlled experiments in Figure 1(a) and Figure 3(b) in the original submission to examine skill extraction and test-time refinement more directly.
Figure 1(a) changes only the skill extractor, holding the solver, source trajectories, feedback, and target tasks fixed. The larger gains from Evolving-RL's skills thus demonstrate a better ability to turn the same experience into useful guidance. Figure 3(b) examines the next step: refining that guidance through interaction. Both models start with the same trajectories and feedback and iteratively refine their skills with model parameters fixed. Evolving-RL improves task performance through this process, demonstrating self-evolution through experience refinement at test time.
Q1: Does the training retrieval pool exclude Look and Pick2?
Yes. All Look and Pick2 instances are excluded from the training retrieval pool. Source tasks, downstream retrieval, and skill evaluation during training use only Pick, Heat, Cool, and Clean, so neither held-out type contributes to parameter updates.
The strong no-skill result on Look can be understood through the basic capabilities it shares with training task types: search, navigation, object pickup, and state assessment. These capabilities can transfer even though Look itself is held out. We examined this explanation further by reproducing the concurrent work Skill1 under the same task-type split:
Skill1 also achieves relatively strong performance on Look despite not being trained on that type, consistent with this explanation.
Q2: Additional GRPO (w/ skills) results for the 3B model
We added the missing GRPO (w/ skills) results for Qwen2.5-3B-Instruct so that both methods can be compared with skill guidance. The tables place these new results alongside GRPO (w/o skills) and Evolving-RL (w/ skills) from the original submission.
ALFWorld task success rate (%):
Mind2Web action accuracy (%):
ScienceWorld environment score:
Evolving-RL retains its advantage over GRPO across all three environments when both methods receive skills, and also outperforms GRPO without skills. We have added the new results to Table 4 (p. 9) and updated the corresponding comparisons in Section 4.3 (p. 9, lines 464–471).