acceptodds
Under review as a conference paper at ICLR 2027

SkillRein: Reusable Agent Skills Generation with Reinforcement Learning

Abstract

Recursive self-improvement calls for agents that improve not only task execution but also the mechanisms that drive future improvement. Motivated by this, we ask whether language agents can learn to turn their own execution experience into effective, reusable guidance. We introduce SkillRein, a reinforcement learning framework that trains a dedicated proposer to generate natural-language skills while keeping the task executor frozen. For each batch of related tasks, the proposer receives selected initial trajectories collected without skills and generates a shared skill set. The executor then reattempts the same tasks using these skills, with task outcomes and skill-use feedback jointly providing the reward for training the proposer. This directly optimizes the skill-generation policy for downstream execution performance. Across four search task categories, a trained 4B proposer raises overall success from 30.2% to 35.6% with a Qwen3-4B executor and from 18.6% to 30.1% with a Qwen3-8B executor. It also outperforms three general-purpose language-model proposers in overall success with both executors. These results show that reinforcement learning can improve an agent’s ability to generate useful guidance from execution experience, supporting learned skill generation as a building block for agent self-improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, reinforcement learning, agent skill, recursive self-improvement, trajectory learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FyPpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SkillRein trains a Qwen3-4B skill proposer with RL while the executor stays frozen. The executor first solves a batch of 50 related tasks without skills, the proposer reads selected successes and failures and writes one shared skill set, and the executor solves the batch again with it. The reward is R=L+λ(2q−1)R = L + \lambda(2q-1), where LL averages the outcome change over tasks that cite each used skill and qq is skill coverage. On search QA, success rises from 30.9% to 36.5%, while the untrained proposer and five large models lower it. The paper also reports transfer to Qwen3-8B and a WebShop gain from 3.3% to 17.7%.

Critical strengths
  1. The formulation and reward are simple and well targeted, and the reward correlates with downstream success change (Spearman 0.78).
  2. The learned behavior is stable across seeds (35.8% to 36.5%) and easy to interpret. Learned skills lead to 1.5 searches and 12% unanswered questions, versus 2.4 to 2.6 searches and 60% to 64% with large-model skills.
Critical weaknesses
  1. The main evaluation batches are sampled from the training question pool, and the shown examples are also scored. Generalization is therefore not measured.
  2. The transfer claim is not supported by the released checkpoint. The main text reports 30.1% on Qwen3-8B, above the large models, but Appendix E reports 29.2% with the released checkpoint versus 31.4% to 34.5% for the large models.
  3. Baselines are weak. There is no hand-written skill baseline, no prompt-optimization baseline, and no related RL method. The reward relies on self-reported skill use, and the reward terms are not ablated.
Questions for authors
  1. Do the Table 1 batches come from the training pool, and what are the results on official test splits, including unseen datasets?
  2. What are the results if the shown examples are excluded from scoring?
  3. Which Qwen3-8B result is correct, and how does it change the transfer claim?
  4. How does a simple hand-written "answer within budget" skill perform under the same wrapper?
  5. How often does the executor report skills it does not follow?

Suggestions for future work:

  1. Ablate the coverage term and per-skill weighting, and strengthen the WebShop baselines.
Reviewer_paoGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SkillRein, a method that trains a skill proposer for large language model (LLM) agents using reinforcement learning (RL). SkillRein improves the performance of LLM agents by generating useful skills to guide an agent. More specifically, SkillRein trains the skill proposer on a set of successful and unsuccessful experiences of the executor using Group-in-Group Policy Optimization (GiGPO).

Critical strengths
  • S1. This paper proposes an interesting approach that trains a skill proposer using reinforcement learning (RL). It also provides experiment results on comparison between SkillRein and No Skills.
Critical weaknesses
  • W1. The problem formulation does not seem to be clearly stated. Although this paper proposes a method for training the skill proposer using reinforcement learning (RL), it does not provide a problem formulation based on Markov decision process (MDP).

  • W2. The main contribution does not seem to be clearly provided. SkillRein uses Group-in-Group Policy Optimization (GiGPO) as a main RL algorithm, and this paper proposes outcome-and-use objective for RL training. However, GiGPO is not clearly explain in the paper, and it is simply described in the Appendix. Therefore, it is somewhat hard to find the main contribution compared to GiGPO.

  • W3. The comparison to related works seems somewhat limited. This paper mainly provides experiment results on comparison between SkillRein and No Skills.

Questions for authors
  • Q1. Is there any reason to select Group-in-Group Policy Optimization (GiGPO) for a reinforcement learning algorithm?
Reviewer_pYjDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses RL to train an LLM to propose skills to a frozen simple multi-turn agent doing textual search and web-shopping tasks. The proposer synthesizes feedback from multiple agent trajectories from a batch of tasks and feeds them back to the agent as skills (just a prompt) for improving on the same tasks. This method beats multiple baselines including proposers made with stronger LLMs.

Critical strengths

The paper is well organized, well written and clearly describes the methodology, and SkillRein clearly outperforms the baselines.

Critical weaknesses

The method is very simple and not very novel: a basic RL algorithm with a basic mechanism for doing skills with prompts. Also, the proposer is not very efficient at test time: to be used, it requires 1) to infer on a large batch (it cannot infer on a single task) and 2) to run the executor twice.

On the empirical side, the baselines are suspiciously bad and have a performance below the simple executor without skills. It makes their results less impressive. More problematically, the search experiments seem to be evaluated on the train set and not on a held-out test set. It's especially problematic because the proposer has access to evaluation feedback as input. Other secondary weaknesses:

Presentation

  • Fig. 1. The caption does not explain most of the symbols used in the figure.
  • Algorithm 1, line 3. “For each group g and candidate k = 1, …, K do” merges two loops into one line. Please write them as two nested loops.

Recursive self-improvement framing

  • The introduction states: “Because the proposer learns only from the executor’s trajectories and pass rates in the environment, the same loop can keep refining skill generation as the executor grows stronger or the environment changes, turning the mechanism of adaptation itself into a target of recursive self-improvement” (l.083–086). Could you clarify why you consider this recursive self-improvement? To me it looks like standard RL with one component frozen.
  • The only part that resembles recursive self-improvement is the proposed next step of “feeding skill-conditioned experience into subsequent proposal rounds” (l.479–480). There, the skill-generation mechanism would receive feedback on its own generated skills. However, this has nothing to do with the RL training, which seems to be the core of the proposal.

Evaluation

  • No standard deviations. The evaluation only requires inference, so repeating it is cheap. Please report standard deviations for Table 3, ideally for all of Table 1, before and after training.
Questions for authors

Clarification: why is the 8B executor worse than the 4B executor? It's very surprising when no skills are used.

Future work: could you evaluate your search results on a dedicated test set unseen during training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.