- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces AlphaSkill, a reinforcement learning framework which jointly trains LLMs to develop self-evolving skills whilst simultaneously learning to solve a reasoning task. The core contribution of AlphaSkill is the self-evolving skill mechanism used to improve task performance. AlphaSkill is evaluated across virtual world, multi-hop and mathematical environments which suggest the advantage of the AlphaSkill over GRPO, skill-augmented RL methods and domain-specific RLVR methods. The effect of individual components is analysed through several ablation studies.
Critical strengths
- The evaluation suite is comprehensive. The main experiments cover nine benchmarks across three domains and two models, with comparisons against GRPO, skill-augmented RL and domain-specific baselines. This provides evidence of the efficacy of AgentSkill over other approaches.
- The approach is well-motivated and has a simple design. Hand-written rules do not scale and skills extracted by a fixed model can fall out of step with the policy. Training a Reasoner and SkillsCreator jointly addresses these problems with a simple approach.
Critical weaknesses
- My main concern with the training approach is that there is no robust mechanism to stop the curator outputting skills that leak answer-specific information in the prompt rather than teaching generalisable strategies. The only safeguard seems to be the instruction in the curator prompt. A skill which leaks answers information would very likely lead to G1 < G2 . This behaviour would be reinforced in the curator via SFT. The curator is therefore optimised for whatever passes same-task validation rather than for generality. Whilst there is some evidence in Figure 4 that some skills do transfer, it is an inherent weakness when G2 is evaluated on a question the curator has already seen.
- Whilst the evaluation suite is comprehensive, it is difficult to establish the advantage of AlphaSkill over competing methods without confidence intervals. This is made especially important as many of the evaluation benchmarks used are small (AIME2025 30, Bamboogle 125, ALFWorld 140) and many reported differences are small and look like they could be within statistical noise.
Questions for authors
- Can you provide confidence intervals in Figure 2 and Figure 3?
- Can the authors train a variant where G2 uses a different question from the same task to G1, and evaluate its performance against the original method?
3 comments
Rebuttal (2/2)
[W2 & Q1] Can you provide confidence intervals?
Thanks for the question. We have additionally conducted three independent evaluation runs with sampling seeds 7, 42, and 123 for each result and report the corresponding standard deviations in Table R3 and Table R4. The evaluation results show minimal fluctuation across different seeds, confirming that AlphaSkill's improvements are not attributable to statistical noise. The additional results have been updated in Appendix B of our revised manuscript.
Table R3. Extended version of Table 1 in the main paper, with standard deviations added.
| Model | Method | ALFWorld | Webshop | NQ | HotpotQA | Bamboogle | MATH500 | Olympiad | Minerva | AIME25 |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | Initial | 13.3 ± 0.3 | 7.5 ± 0.3 | 12.7 ± 0.9 | 15.8 ± 0.4 | 15.2 ± 0.7 | 76.5 ± 0.4 | 39.5 ± 0.3 | 28.7 ± 0.3 | 5.6 ± 1.6 |
| GRPO | 85.2 ± 0.7 | 68.9 ± 0.2 | 40.3 ± 0.2 | 37.8 ± 0.3 | 39.5 ± 0.4 | 79.3 ± 0.2 | 41.1 ± 0.2 | 31.4 ± 0.2 | 10.0 ± 2.7 | |
| AlphaSkill | 96.7 ± 0.3 | 76.2 ± 0.7 | 49.2 ± 0.7 | 45.0 ± 0.6 | 44.3 ± 0.8 | 82.5 ± 0.2 | 43.6 ± 0.1 | 34.7 ± 0.3 | 10.0 ± 5.4 | |
| Qwen3-4B-Instruct-2507 | Initial | 25.0 ± 0.6 | 18.9 ± 0.4 | 19.3 ± 0.3 | 19.3 ± 0.4 | 14.7 ± 1.0 | 88.5 ± 0.3 | 56.7 ± 0.3 | 42.2 ± 0.2 | 37.8 ± 3.1 |
| GRPO | 93.3 ± 0.7 | 74.5 ± 0.2 | 42.4 ± 1.0 | 40.1 ± 0.3 | 40.3 ± 0.8 | 89.4 ± 0.5 | 58.2 ± 0.1 | 43.4 ± 0.6 | 40.0 ± 5.4 | |
| AlphaSkill | 98.8 ± 0.9 | 79.1 ± 0.2 | 51.0 ± 0.6 | 44.5 ± 0.2 | 46.4 ± 1.3 | 89.1 ± 0.4 | 60.2 ± 0.1 | 45.5 ± 0.3 | 41.1 ± 1.6 |
Table R4. Extended version of the baseline comparison in the main paper, with standard deviations added.
| Method | ALFWorld | Webshop | Method | NQ | HotpotQA | Bamboogle | Method | MATH500 | Minerva | Olympiad | AIME25 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Initial | 13.3 ± 0.3 | 7.5 ± 0.3 | Initial | 12.7 ± 0.9 | 15.8 ± 0.4 | 15.2 ± 0.7 | Initial | 76.5 ± 0.4 | 28.7 ± 0.3 | 39.5 ± 0.3 | 5.6 ± 1.6 |
| Reflexion | 43.1 ± 0.3 | 28.9 ± 0.5 | ZeroSearch | 43.7 ± 0.1 | 34.5 ± 0.7 | 28.0 ± 0.0 | Fewshot | 77.1 ± 0.4 | 29.2 ± 0.2 | 40.0 ± 0.2 | 5.6 ± 4.2 |
| SkillRL | 90.2 ± 0.9 | 72.9 ± 0.6 | SkillRL | 46.3 ± 0.5 | 42.6 ± 0.5 | 73.1 ± 0.4 | AbsoluteZero | 78.3 ± 0.1 | 33.9 ± 0.2 | 42.0 ± 0.3 | 8.9 ± 1.6 |
| Skill0 | 89.5 ± 0.7 | 73.2 ± 0.7 | Skill0 | 42.3 ± 0.6 | 39.5 ± 0.4 | 66.9 ± 0.4 | Critique-GRPO | 80.7 ± 0.3 | 33.2 ± 0.5 | 42.5 ± 0.3 | 12.2 ± 3.1 |
| AlphaSkill | 96.7 ± 0.3 | 76.2 ± 0.7 | Ours | 49.2 ± 0.7 | 45.0 ± 0.6 | 44.3 ± 0.8 | AlphaSkill | 82.5 ± 0.2 | 34.7 ± 0.3 | 43.6 ± 0.1 | 10.0 ± 5.4 |
[Q2] Can the authors train a variant where G2 uses a different question from the same task to G1, and evaluate its performance against the original method?
Thanks for the valuable suggestion. As suggested, we have provided the results of this variant in Table R2 in response to W1. As shown in Table R2, this variant remains competitive and even achieves higher scores on some benchmarks. We greatly appreciate the reviewer for giving us a new perspective, which helped us strengthen the empirical validation of our method. The additional results have been updated in Appendix B of our revised manuscript.
Dear Reviewer 29ZB,
Thanks again for your insightful review. As the discussion period is coming to a close (less than one day), we would like to briefly summarize the additional experiments addressing your main concerns:
- Skills content audit. Most skills are classified as containing no answer-specific information, indicating that the curated skills remain general.
- Training with skills from different tasks. We train a variant in which each guided rollout group receives skills curated from another unguided rollout group with different tasks. This cross-task variant remains effective, demonstrating that the training benefits do not stem from potential answer-specific guidance on the tasks that produced it.
These results have been updated in our revised manuscript, with full experimental details provided in the rebuttal above. We would be very grateful if you could revisit our rebuttal and let us know whether any points need further clarification.
Best regards,
Authors of AlphaSkill
Rebuttal (1/2)
We sincerely appreciate the reviewer's time and effort in reviewing our manuscript. Our responses to your concerns are as follows.
[W1] My main concern with the training approach is that there is no robust mechanism to stop the curator outputting skills that leak answer-specific information in the prompt rather than teaching generalisable strategies.
Thanks for the insightful comments. We address it through an audit of the accumulated skills, discussion of their role during training, and a cross-task training experiment.
(1) Auditing answer-specific information in the skill library after training. We inspected each skill in the final libraries using DeepSeek-V4.1-flash to judge whether it explicitly contained the answer of a specific question. As shown in Table R1, over 90% of skills are classified as containing no direct answer information, indicating that the curated skill library remains general and is not dominated by explicit answer information to specific questions for passing same-task validation. We also observed a lower incidence of explicit answer information for a stronger backbone model with stronger instruction-following capability. We further visualized the high-frequency terms appearing in the skill library in Figure 8 of the paper.
Table R1. The percentage (%) of skills without answer-specific information in the content.
(2) Deeper discussion on how skills take effect during training. We would like to discuss the implications when a skill may contain answer-specific information. First, the evaluation is conducted on held-out tasks, so any answer-specific information of a training instance contained in the skills does not reveal answers to the evaluation instances. Second, we believe that the training gains primarily arise from learning higher-quality trajectories under skill guidance (i.e., reasoning) and the model’s improved understanding/knowledge of the environment (i.e., skills internalization). More broadly, many agentic tasks like ALFWorld and Webshop are verified through changes in the environment state, rather than by matching a reference answer as in mathematics. Solving these tasks requires interacting with the environment and changing its state through successive actions. Even when skills contain a task-specific procedure, the agent must ground that procedure in its observations and execute a sequence of valid actions to reach the expected final state. Such task-specific guidance can therefore serve as scaffolding that reduces redundant exploration to help the policy sample efficient reasoning and acting trajectories for more effectively learning helpful behavioral patterns.
(3) Training with skills from a different task. To more thoroughly rule out the potential influence of answer-specific information in the skills, we train a variant using Qwen3-4B-Instruct-2507 in which each guided rollout group G2 receives skills curated from another unguided rollout group G1 with different tasks. For ALFWorld, we exchange skills between distinct tasks of the same task type. For the other environments, we randomly pair training tasks and exchange their curated skills for guided rollouts. As shown in Table R2, this cross-task variant remains effective and even achieves higher performance than AlphaSkill on some benchmarks. These results provide empirical evidence that the training benefits largely do not stem from potential answer-specific guidance on the tasks that produced it.
Table R2.
Overall, our empirical results indicate that explicit answer-specific information is rare, while the cross-task experiment provides further evidence that skill guidance remains useful beyond its source tasks. When task-specific information does occur, its effect on policy learning may not be negative, since it can still contribute useful training experience. Moreover, the main evaluations use held-out tasks without skill injection, so this information does not directly supply answers during evaluation. The additional results and discussion have been updated in Appendix B of our revised manuscript.