acceptodds
Under review as a conference paper at ICLR 2027

Skill-R1: Agent Skill Evolution via Reinforcement Learning

Abstract

Agentic large language models increasingly rely on skills, reusable natural-language procedures that guide planning, action, and tool use. Current practice improves skills through prompt engineering or by fine-tuning the task LLM, both of which are costly, model-specific, and often infeasible for closed-source models. Effective skill optimization is moreover recurrent rather than one-shot: a useful skill must improve rollouts under the current conditioning, and a useful revision must turn observed outcomes into a better skill for the next round. We propose Skill-R1, a reinforcement learning framework that addresses skill optimization as a bi-level problem coupling within-generation execution quality with across generation skill improvement. Skill-R1 trains a lightweight skill generator conditioned on the task context, prior rollouts, and verified outcomes to produce skills that steer a frozen task LLM, preserving compatibility with both open and closed-source models. The generator is optimized via a bi-level group-relative policy optimization objective: an intra-generation advantage compares skill–rollout pairs within each generation, and an inter-generation advantage rewards population-level improvement across generations. On agent benchmarks with verifiable rewards, Skill-R1 yields substantial gains over prompt evolution, and Best-of-N sampling methodologies, with the largest improvements on complex multi-step reasoning benchmarks.

Primary area:
reinforcement learning
Keywords:
Skill Evolution, Agentic AI, LLMs, Reinforcement Learning, Agent Skills

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PmA5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Skill-R1 keeps the task model frozen and instead trains a small 4B editor to improve the natural-language skills that model uses. For each task instance, the editor revises a population of KK skills over GG generations, conditioned on earlier (skill, rollout, reward) triples. The main technical contribution is a "bi-level" GRPO advantage. It combines the usual within-generation term with λ\lambda times the change in mean population reward from one generation to the next (Eq. 9–10). Evaluation covers 50 GAIA tasks and 65 tasks each from WebWalker and Geometry3k. Grok-4.20 and GPT-4o-mini serve as task models, and GPT-4o-mini also tests whether edited skills transfer. On GAIA, accuracy rises from 36% to 62% with Grok and from 8% to 40% with GPT-4o-mini.

Critical strengths

S1. The setting is practical. Only a small editor is trained, so the approach works with closed task models, and editing costs under 1% of training tokens (Table 6). Many people running skill-based agents cannot fine-tune the executor, so this matters.

S2. The GAIA gains are large and transfer across task models: 20/50 vs. 4/50 with GPT-4o-mini and 31/50 vs. 18/50 with Grok. These differences are unlikely to be noise. The case study in App. A.2 also helps. The untrained editor pastes unrelated task content into a skill, and the trained one does not.

Critical weaknesses

W1. The main methodological contribution is never isolated. The claimed novelty is the inter-generation term. Yet no experiment compares λ=0\lambda=0 with λ>0\lambda>0 at matched GG. The Table 4 ablation ("Intra-gen, G=1G=1") changes the objective and the number of generations at the same time, and possibly the number of evaluation attempts too. Its 18-point gap could therefore come from multi-round revision or from extra attempts. In Table 5, all three λ\lambda values give exactly 50.8% on Geometry3k. I read that as "no measurable effect", not as evidence that the method is insensitive to λ. Fig. 2 also shows generations 1–3 converging to the warm start late in training, with Gen 2 below it. A λ=0\lambda=0 vs. λ>0\lambda>0 run on GAIA, with G=3G=3 and the same protocol, is what I need to see.

W2. Methods are not evaluated under the same protocol. App. A.1 says Skill-R1 gets a writable copy of the skill directory while baselines share a read-only one. If the bank keeps changing during evaluation, Skill-R1 learns across the test set and the baselines cannot. Skill-R1 is also scored as any-correct over 3 attempts, while Best-of-N appears to return a single majority answer. Its per-level GAIA counts match ReAct+skills and Direct exactly, which looks odd. Outside GAIA the margins are 2–3 problems out of 65, and "best variant" is chosen on the test set from two editors. Please report results with a frozen bank, the same attempt budget and scoring for all methods, and confirm that the 20 distillation tasks do not overlap the GAIA test set.

Questions for authors
  1. [Clarification] At test time, does Skill-R1 write edited skills back to the bank? What is the accuracy with the bank frozen?
  2. [Clarification] What is GAIA accuracy for G=3G=3, λ=0\lambda=0? Without this I cannot credit any gain to the inter-generation term.
  3. [Clarification] How many attempts does "Intra-gen (G=1G=1)" get in Table 4, and how is Best-of-N scored?
  4. [Clarification] Are the 20 distillation tasks and the 115 training tasks disjoint from the 50 GAIA test tasks?
  5. [Suggestion] A feedback-driven prompt optimizer such as GEPA or TextGrad would be a stronger baseline than EvoPrompt, which falls below the no-skill baseline in several settings.
Reviewer_oHD6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Skill-R1, which trains a lightweight skill generator (Qwen3-4B / Qwen3.5-4B) to iteratively revise natural-language skills for a frozen task LLM (Grok-4.20 / GPT-4o-mini). Over G generations, the generator samples K skills conditioned on the instance and the accumulated history of (skill, rollout, reward) triples. It is trained with a “bi-level” GRPO advantage that adds an intra-generation group-relative term and an inter-generation term r̄_g − r̄_{g−1}. Experiments on subsets of GAIA (50 tasks), WebWalker (65), and Geometry3k (65) report gains over no-skill, static-skill, EvoPrompt, and majority-vote baselines.

Critical strengths

The setting is practical: only a small editor is trained and the task model stays frozen, making the approach compatible with closed-source models. The gain on GAIA with GPT-4o-mini (4/50 → 16–20/50) is substantial.

Critical weaknesses

The core contribution is not validated. The inter-generation term, the paper’s main novelty, is never isolated: the key ablation (G=3, λ=0) is missing. Table 5 shows identical held-out accuracy (33/65) across all tested λ, which suggests the term has little effect.

The loss in Eq. (11) uses only per-generation advantages and ignores the discount γ from Eq. (7), so it does not optimize the claimed recurrent objective. Closely related prior work is neither cited nor compared, notably Retroformer (ICLR 2024), which trains a small model with policy gradient to refine prompts for a frozen actor, and Training-Free GRPO.

The experiments are weak and the evaluation is not fair. Scale: only small subsets (50–65 tasks per benchmark), a single run, and no variance. Apart from GAIA with GPT-4o-mini, most gains are within a few tasks. Metric: the pass@3 metric is an oracle, while Best-of-N is scored by majority vote. Asymmetry: Skill-R1 accumulates skills across test tasks, whereas baselines use a read-only bank. Confounded ablation: the Table 4 ablation confounds the training objective with the number of test-time attempts.

Questions for authors
  1. Can you report the G=3, λ=0 ablation and explain why performance is unchanged across λ?

  2. Can you report per-generation test accuracy, results over multiple seeds, and results without test-time skill accumulation?

  3. How does Skill-R1 compare with Retroformer and Training-Free GRPO under a matched budget?

  4. Can the method extend to full benchmarks and standard agentic benchmarks (e.g., WebArena, ALFWorld)?

Reviewer_5ZGJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Skill-R1 trains a small skill editor to revise external procedural instructions while keeping the task LLM frozen. Its GRPO objective combines within-generation relative rewards with changes in population-average reward across generations. Experiments on subsets of GAIA, WebWalker, and Geometry3k evaluate two Qwen editor variants with Grok-4.20 and GPT-4o-mini, reporting particularly substantial gains on GAIA.

Critical strengths
  1. Training an external editor while freezing the executor offers a practical way to improve agents using closed-source task models. The separation between skill adaptation and task execution is useful.
  2. The GAIA improvements are substantial: the best variant raises accuracy from the strongest baseline's 8% to 40% with GPT-4o-mini and from 36% to 62% with Grok-4.20 (Table 1). Transfer of the Grok-trained editor to GPT-4o-mini provides encouraging evidence of its utility across task models.
Critical weaknesses
  1. The distinctive contribution of the inter-generation objective is insufficiently established. Table 4 changes G from 1 to 3, confounding the objective with revision opportunities and training rollout budget; Table 5 omits λ=0\lambda=0. Equations (8)–(10) reduce to Ai=ri+(λ−1)rˉg−λrˉg−1A_{i} = r_{i} + (\lambda-1)\bar{r}_{g} - \lambda\bar{r}_{g-1}. Conditional on history and under the stated i.i.d. sampling, the unclipped on-policy expected policy-gradient term is only a rescaling of its λ=0\lambda=0 counterpart. Clipping and variance effects may matter, but the claimed cross-generation credit assignment and its connection to Equation (7) need stronger justification.
  2. The evaluation provides limited evidence of superiority over relevant alternatives. Tests contain only 50, 65, and 65 tasks, with two closely related 4B editors, two closed-source executors, and no reported uncertainty or repeated-seed results. The best Grok gains over the strongest baseline on WebWalker and Geometry3k amount to only two and three additional correct tasks. Recent reflective alternatives such as GEPA [1] and Memento-Skills [2], the latter discussed in Section 6.1, are absent from the comparisons. Section 4.1 also leaves unclear how its any-attempt scoring rule applies to the majority-answer sampling baseline.

References:

[1] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, ICLR 2026

[2] Memento-Skills: Let Agents Design Agents

Questions for authors
  1. Evidence request: How does λ=0 perform with G=3, K=4 and matched training/evaluation budgets? What benefit of the inter-generation term remains beyond gradient rescaling?
  2. Clarification: Is Best-of-N scored on its majority answer or on whether any of its 12 rollouts succeeds? Can the comparisons use a common answer-selection rule and comparable compute budgets?
  3. Evidence request: Would the improvements persist against a recent reflective prompt/skill baseline under matched budgets, and across repeated seeds, particularly for the small gains in Tables 2–3?
  4. Clarification: After removing correctness labels and final answers, what feedback reaches the editor at evaluation? What history-length limits or truncation rules are used? Section 5.3 attributes degradation at larger K to longer context, but the handling of accumulated history is unspecified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.