acceptodds
Under review as a conference paper at ICLR 2027

SkillEvolver: Skill Learning as a Meta-Skill

Abstract

Agent skills, folders of instructions and scripts that an agent loads at inference time, are today static artifacts: authored once, by human curation or by one-shot generation from the model's own knowledge, and never improved by use. Methods that do learn skills from execution traces learn them from many instances of a task distribution. We study online skill learning, in which a single new task arrives and only a handful of trials can be afforded before its skill must ship. We propose SkillEvolver, a meta-skill, itself loaded like any other skill, that drives any CLI-agent to author, deploy and refine a skill for that task; it learns the skill's prose and code rather than model weights, so the result drops into any agent without retraining. Two design choices make a handful of trials enough: each trial follows a distinct strategy the agent writes itself, so that a small batch explores different approaches rather than resampling one, and, when a first batch leaves failures, the distilled skill is refined from how fresh agents fail while using it. On SkillsBench, skills learned by SkillEvolver outperform the human-curated skills in overall resolution rate on all three models we evaluate, raising the resolution rate to 57.9% on DeepSeek-V4-Flash (13.3 points absolute improvement), 62.2% on GLM-5.2 (7.2 points absolute improvement) and 65.5% on Claude Opus 4.6 (10.8 points absolute improvement). Even when all four trials of a first batch fail, a second batch solves the training task 64.4% of the time after distillation, against 29.7% when drawn blind, and the loop spends 6.57 exploration rollouts per task on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Skill Evolution, Meta-skill

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mS5KOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposes SkillEvolver, which lets agents create and improve skills through a few rounds of trial and error. it generates different strategies, compares successful and failed attempts, and updates skills based on the results. experiments on SkillsBench show improvements over human-written skills across three models.

Critical strengths

the experimental improvements over human-written skills are interesting, and the strategy diversification ablation shows some useful results

Critical weaknesses
  1. i find the novelty quite weak. generating skills from execution traces, comparing successes and failures, and refining skills through repeated testing have already been explored by SkillGen, EvoSkill, Anthropic's skill-creator and other recent work. the main difference here seems to be asking agents to try different strategies on a single task. i dont see enough new ideas to justify another framework built around essentially the same loop.
  2. more importantly, the paper doesnt compare against any of these closely related methods. beating static human-written skills says very little about whether SkillEvolver is better than existing automated skill creation methods. the authors argue that prior methods use different settings, but this is not a convincing reason to skip the most important comparisons. also, the claimed single-task setting still requires an LLM-generated training variant with verifier feedback, which makes the practical motivation less convincing.
Questions for authors
  1. what exactly is the technical novelty compared with SkillGen, EvoSkill and Anthropic's skill-creator, beyond strategy diversification and the single-task setting?
  2. why not adapt these existing methods to the same trial budget? without such comparisons, how can we tell whether the proposed method actually offers anything new?
Reviewer_ofHPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SkillEvolver packages skill learning as a meta-skill that a CLI agent can load. For a new task, it writes distinct strategies, runs a small number of trials on a generated training variant, and synthesizes the resulting experience into a skill containing instructions and code. Fresh agents then use the candidate skill, and their failures inform further revisions; an independent-session Auditor checks the artifact. Model parameters remain fixed. Evaluation on SkillsBench compares no skills, human-curated skills, and learned skills across three models, with component ablations, a fixed-rollout training comparison, and repeated full-pipeline experiments for one model.

Critical strengths
  1. The few-trial learning setting is practically useful, and strategy diversification has direct empirical support. Explicitly assigning different strategies creates distinct execution paths from one training instance, while deployment to fresh agents exposes shortcomings of the resulting artifact. Removing strategy injection reduces held-out performance by approximately 11 percentage points, with a paired interval excluding zero. This provides stronger support for the design than successful examples alone (Section 3; Section 4.3; Table 2).

  2. The main evaluation reports meaningful controls and uncertainty. No-skill and curated-skill baselines are rerun within each model and harness configuration. Learned skills outperform curated skills in aggregate on all three models, with paired intervals excluding zero for two. Three independent DeepSeek repetitions cover the entire learning and evaluation pipeline. The paper also distinguishes training-task rescue rates from held-out accuracy and acknowledges that the Auditor and failure-trace ablations are not statistically significant, rather than presenting every component as an established contribution (Tables 1–2; Appendices F and K).

Critical weaknesses
  1. Direct comparisons with alternative automatic skill learners or simpler feedback-based revision methods are missing. The main comparisons are against no skills and human-curated skills. Appendix L explains the different data requirements of existing methods, but this does not fully replace an empirical comparison using the same training variant, feedback, and exploration budget. Removing strategy injection is also not equivalent to independently implementing those methods. The results establish benefits over the reported baselines, but less clearly establish the advantage of the complete meta-skill workflow over simpler generation-and-revision procedures.

  2. Evidence for reusability and evaluation robustness remains limited. One generated training variant and one held-out instance per task test some instance variation, but provide limited evidence of broader within-task generalization. Appendix F also excludes rollouts reaching the harness time limit, and GLM conditions cover different numbers of tasks. If timeouts depend on method behavior, exclusion can affect the comparison. Results on the common task set and a sensitivity analysis treating non-infrastructure timeouts as failures would strengthen the evaluation (Appendices D and F).

Questions for authors
  1. How does a simple baseline that synthesizes a skill from trajectories and then revises it from failures perform with the same training variant, verifier feedback, and exploration budget? Please report any available adaptations of related methods to help establish the additional value of the complete workflow.

  2. Which original task materials are available when generating each training variant? Does generation access the original instance's answer or solution procedure, and what information may be retained? Can the same learned skill be evaluated on multiple independent held-out variants to test broader procedural reuse?

  3. Can you report timeout and missing-task counts and causes by condition, results on the common task set, and a sensitivity analysis counting method-related timeouts as failures? Do these choices change the conclusions of Table 1?

The paper offers a clear and useful implementation of single-task, few-trial skill learning. The main results and strategy-diversification ablation are persuasive, and repeated experiments and uncertainty reporting improve confidence in the findings. Missing automatic skill-learning baselines and limitations in held-out variants and timeout handling restrict the scope of the contribution, but the available evidence supports weak acceptance.

Reviewer_n7BCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the online learning of reusable agent skills from a single task with a handful of trials afforded without updating model weights. The proposed method, SkillEvolver, is implemented as a meta-skill that guides a CLI agent through strategy-diversified exploration, trajectory-based skill synthesis, and deployment-grounded refinement. The experiments compare the system performance with the learned skills, no skill, and human-curated skill on SkillsBench. The improvement over the compared baselines looks promising. However, I have several concerns about the positioning and novelty relative to earlier skill-learning methods, the value of the proposed mechanisms supported by current experiments, and the robustness of the evaluation protocol.

Critical strengths
  1. Given the background of the common use of skills, tools, and MCP in modern coding agents, the studied problem of learning external skills for CLI agents under a limited budget is practically useful. Besides, evaluating the learned skills with a fresh downstream agent is also an appropriate design, as it decouples the influence of context for learning the skills from the use of the skills and also facilitates evaluating the same library of skills with various agents.
  2. The performance improvement over the compared baselines on SkillsBench across three different LLMs looks promising. Specifically, the proposed method achieves 13.3, 7.2, and 10.8 gains in resolution rate against human-curated skills with DeepSeek-V4-Flash, GLM-5.2 under OpenHands, and Opus 4.6 under Claude Code harness, respectively.
  3. The ablation evidence is also somewhat meaningful. Removing strategy injection results in a −11.0 performance loss on 73 investigated tasks, demonstrating the effectiveness of the proposed strategy-diversified exploration.
Critical weaknesses
  1. The positioning and novelty of this work need to be carefully inspected and reconsidered. First, several closely related prior skill-learning works are not investigated, making many claims in the paper and the novelty of the work overstated. For instance, the claim that agentic skills "never improved by use" is obviously overstated. There is a lot of previous work studying improving agent skills in use [1,2,3,4]. LearnAct [1] already revises executable action functions or their usage guidance based on execution errors, while SkillWeaver [2] explicitly tests and debugs synthesized skill APIs in its skill-honing stage. These approaches differ from the present framework, but they are substantive antecedents to improving executable skills through use. Besides, the claim "Methods that do learn skills from execution traces learn them from many instances of a task distribution" does not hold for all previous work neither. PSN [3] explicitly studies an online framework that constructs, optimizes, and refactors reusable programmatic skills as tasks arrive, using execution traces for failure-driven repair. It learns a network of skills from scratch as new tasks arrive. Therefore, formulating online skill learning is also not something novel, which is listed as the first contribution of this work. The authors are encouraged to discuss skill-learning prior work, carefully check and narrow down the claims in the paper, and position this work properly.

  2. The controlled experiment in Table 2 shows that distillation improves subsequent success on the training variant. However, it does not compare the final skills’ held-out performance. Therefore, it remains unclear whether the iterative procedure produces more transferable skills than collecting eight rollouts and distilling once. Extending the existing comparison to the final skills on the same held-out instances would strengthen this claim.

  3. The current cost comparison with baselines is insufficient. Both the proposed skill learning (involves exploration, synthesis, auditing, and refinement) and the skill cost tokens. The authors are encouraged to report detailed token costs for both skill learning and using against the compared baselines, which could be helpful for understanding both the detailed improvement and cost of the designed modules.

  4. The evidence for robustness and broader use of the proposed method is also limited. The evaluation covers only 1 benchmark with 87 tasks (60 for Opus), and each skill is mainly tested on one held-out instance. Repeating execution on that instance does not establish generalization across different inputs. Besides, two ablation study experiments only run on 36 tasks.

[1] Zhao et al., Empowering Large Language Model Agents through Action Learning, Feb 2024

[2] Zheng et al., SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills, Apr 2025

[3] Shi et al., Evolving Programmatic Skill Networks, Jan 2026

[4] Ye et al., Meta Context Engineering via Agentic Skill Evolution, Jan 2026

Questions for authors

Can the same-strategy control in Appendix J be extended to final skill performance to separate the benefits of explicit planning from those of strategy diversification? Please refer to the weaknesses for more.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.