acceptodds
Under review as a conference paper at ICLR 2027

ASD: Automated Skill Discovery through Agent Directed Optimization

Abstract

Large language model (LLM) systems are increasingly equipped with skills to improve task performance. Recent methods automate the development of these skills from execution feedback, but the optimization process itself typically follows human-designed rules. We investigate whether an agent can instead design and direct the skill discovery process. We introduce Automated Skill Discovery (ASD), a framework in which a discovery agent develops a skill library for a target model by autonomously designing and adapting the optimization procedure. In ASD, the target model selects and loads skills as needed for each problem, reducing the input context by approximately 69% compared with including all skills in the model's context. We evaluate ASD on eight open-weight models across five benchmarks spanning question answering, mathematics, and software engineering. ASD improves test success over the base model by 16.2 points on average and outperforms the human-designed optimization baselines on each benchmark. We further investigate ASD in a mixed-task setting, where a single library developed for each target model supports four tasks, improving test success over the base model by 14.3 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, agent, skill discovery, auto-research

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ovsVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the author proposes a framework, Automatic Skill Discovery (ASD), to explore letting LLMs compose the skill bank by self-decision with no human intervention. Within the framework, the method only defines some basic data structures for skill storages and some actions for the LLMs to take leaving all other decisions to LLMs for deciding. To validate the effectiveness of the method, the paper apply the ASD with 3 task open sourced models as task execution model on 5 different task datasets, uniformlly use the Claude Opus 4.8 as the optimizer. From the comparison with the other 2 baseline prompt optimization methods, the paper claims the advanatges of the method and additional analysis is conducted to inteprete the feature of the optimized skills.

Critical strengths
  1. The idea of the paper is well-presented and easy to follow.

  2. The experiment was conducted with a 3 backbone models, providing a view of the generalization of the method.

  3. Additional analysis of the optimized process and features of the generated skill bank helps to understand the features of the method.

Critical weaknesses
  1. The proposed framework is largely heuristic, with no theoretical guarantees regarding its optimization outcomes. Since nearly all decisions are made autonomously by the LLM, the effectiveness of the framework heavily depends on the capabilities of the optimizer model. Consequently, the observed performance improvements may primarily stem from the optimizer's inherent capabilities, making it a potential bottleneck for the overall framework. This raises a fundamental question: Why not directly use the optimizer model to perform the downstream tasks?

  2. Since the optimization process relies entirely on LLM-driven decisions, the stability and reproducibility of the resulting performance remain uncertain. Although the authors state that experiments are repeated three times and average results are reported, it is unclear whether the entire optimization pipeline is independently executed three times or whether the same optimized skill bank is evaluated three times. To assess the robustness of the proposed method, not only the mean value but also the variance should be reported.

  3. The reported performance value may be problematic. The reported baseline performance raises concerns about the validity of the experimental comparisons. Specifically, the paper includes GEPA as a baseline with Qwen3-8B as the backbone model. However, the performance reported under this setting appears substantially lower than the corresponding results presented in the original GEPA paper. This discrepancy raises questions about whether the baselines have been properly implemented and evaluated under comparable experimental settings.

  4. The paper claims that all methods are compared under the same optimization budget. However, the budget calculation appears to account only for rollouts on the training and validation datasets, excluding additional LLM calls involved in intermediate decision-making. Given that the proposed framework relies heavily on LLM-driven decisions throughout the optimization process, these additional calls may introduce substantial computational and financial overhead, which may be unfair to the other baseline method.

  5. The paper only chooses two prompt optimization methods for comparison; more advanced skill discovery and construction methods should be included as baselines.

Questions for authors
  1. What's the variance of the independent runs of the method? Can the performance gain pass the statistical significance test?

  2. Why is the performance not matching the results reported in other papers? Please detail the settings and explain the cause of those contradictions

  3. Can you calculate the budget based on the api calls or the token consumption, and then do the experiment and compare the performance?

  4. Can you add more baseline methods for comparison?

Reviewer_VWPiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ASD, a framework in which a discovery agent develops a skill library for a target model and autonomously determines training-example selection, skill revisions, and validation timing. The authors evaluate ASD on eight open-weight models across five benchmarks, reporting overall improvements over the base models and optimization baselines such as GEPA and SkillOpt. They also investigate a shared library for multiple tasks and the context efficiency of on-demand skill loading.

Critical strengths
  • The research question is practically relevant: example selection, revision procedures, and validation scheduling are meaningful factors in effective skill discovery.
  • The evaluation spans multiple model families and task types, including mathematics, question answering, and software engineering, supporting applicability across different settings.
  • The paper provides skill-library examples, skill-use trajectories, and optimization-process visualizations that make the framework easy to follow.
  • The paper includes several useful ablations, including leaf-skill removal, independently optimized meta-only guidance, and comparisons between discovery models of different capabilities.
Critical weaknesses
  1. The contribution of autonomous optimization is not sufficiently isolated. ASD combines a multi-skill library, on-demand skill access, and an agent-directed optimization strategy. GEPA and SkillOpt primarily optimize a single prompt and therefore do not match ASD’s structure. Consequently, the current experiments cannot disentangle the gains from autonomous decision-making from those provided by the multi-skill library and access mechanism. A structure-matched baseline should use the same library representation and list_skills/read_skill tools, while following predefined sampling, revision, and validation procedures.

  2. Details of the discovery agent’s decisions and implementation are insufficient. Figure 5 shows only the number of examples requested per evaluation and the timing of validation requests. It does not provide the selected example IDs or distributions, selection rationales, before-and-after skill diffs, or the reasoning behind early stopping.

  3. The cost accounting is incomplete. ASD uses target-model rollout counts as the default optimization budget, and Figure 4 compares the validation progress of the three methods under this measure. However, the paper does not report token consumption, monetary costs, or wall-clock time for the discovery/optimization models and target models. Since optimizer overhead and individual rollout lengths may differ across methods, equal rollout budgets do not imply equal total costs. Complete cost statistics and comparisons under matched total costs are needed to assess ASD’s end-to-end cost efficiency.

  4. The baseline comparison is incomplete. The paper discusses EvoSkill and Trace2Skill but experimentally compares only against GEPA and SkillOpt. EvoSkill also creates and revises structured skill libraries using execution feedback. The authors should include EvoSkill and Trace2Skill as baselines or explain why they are unsuitable for this setting.

Questions for authors
  1. Under the same skill-library representation, access tools, discovery and target models, feedback access, and budget, could you compare ASD with an optimizer that follows predefined sampling, revision, and validation procedures? Could you further control example selection and validation scheduling separately to assess the contribution of each autonomous decision component?

  2. Which training examples were selected at each step in Figure 5? Please provide a complete discovery-agent trajectory, including example-selection rationales, skill revision diffs, validation-timing decisions, and stopping rationales.

  3. Appendix D specifies a SWE-bench validation set of 60 examples, whereas Figure 5 shows validation requests of 60 examples for ASD and SkillOpt but 50 for GEPA. What accounts for this difference?

  4. Beyond target-model rollout counts, what are the discovery model’s token consumption, monetary costs, and wall-clock time? Does ASD remain more efficient than GEPA and SkillOpt after accounting for these costs?

  5. Please correct the following grammatical errors: Page 2, approximately line 91: “The skill library improve test success…” → “The skill library improves test success…” Page 9, approximately line 474: “ASD can also develops…” → “ASD can also develop…”

  6. Add experimental comparisons with EvoSkill and Trace2Skill.

Reviewer_4gsYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Automated Skill Discovery (ASD), a framework in which a discovery agent develops skill libraries for a frozen target model. The discovery agent edits skills, selects training examples, and schedules validation evaluations within a target-model rollout budget. Each library consists of an always-on meta-skill and optional leaf skills retrieved by the target model during execution.

Experiments evaluate eight open-weight models across five benchmarks, reporting an average improvement of 16.2 percentage points over the seed baseline. Additional experiments examine a shared library across four tasks, leaf-skill contributions, discovery-model strength, and on-demand loading, which reduces reported input context by approximately 69% across five evaluated configurations.

Critical strengths
  1. Broad empirical coverage with substantial reported improvements. The experiments cover eight target models from three families and tasks spanning scientific reasoning, factual question answering, mathematics, and software engineering. ASD achieves higher benchmark-level mean performance than both optimization baselines. The Mixed-4 results further demonstrate that a single library can support several task types. This breadth supports the practical usefulness of the overall system.

  2. The library ablations go beyond simply deleting useful content. Table 5 includes separately optimized meta-only guidance in addition to removing leaves from an existing library. Full libraries outperform both alternatives in the four examined configurations. Together with the comparison against preloading all leaves, these experiments provide meaningful evidence for organizing guidance into selectively accessible components.

Critical weaknesses
  1. The central claim about agent-directed optimization is not isolated by the experimental design. ASD simultaneously introduces autonomous experimental scheduling, a structured skill library, and on-demand retrieval, whereas GEPA and SkillOpt optimize a single prompt. Consequently, the main comparison changes both the optimization procedure and the representation being optimized. Figure 5 demonstrates that ASD chooses different evaluation schedules, but does not demonstrate that these choices improve discovery. Table 5 helps assess library structure, yet does not compare autonomous scheduling against a fixed schedule while holding the discovery model, library representation, feedback access, and budget constant.

  2. This omission is particularly important given existing work on structured skill discovery and agentic optimization from stored execution histories, such as EvoSkill and Meta-Harness. Both are cited but absent from the experiments. The discovery-agent prompt and optimizer-specific baseline model configurations are also insufficiently documented. The results therefore establish that the complete ASD system can outperform the chosen implementations, but do not establish that autonomously directing the optimization process is the decisive contribution.

  3. The evaluation leaves the source and robustness of the reported gains unresolved. Some seed results are strikingly low: Qwen3-8B and Mistral-Nemo-12B achieve only 4.0% and 2.3% on GPQA, respectively, far below uniform guessing over four options. Appendix C also instructs the seed model to browse and read skills despite initializing an empty leaf library. Meanwhile, the discovered guidance contains extensive instructions for submission formatting, avoiding truncated responses, constructing shell commands, and executing Python. These observations raise a concrete alternative explanation: a substantial fraction of the gains may come from repairing interaction and submission failures. Such repairs are useful, but their contribution must be separated from improved substantive problem solving before interpreting the headline gains.

  4. The uncertainty analysis is also inadequate. Section 4.1 reports three test evaluations of each selected library, rather than three independent discovery runs. This does not measure variation in the optimization process. AIME and HMMT each contain only 30 distinct test problems, and GPQA contains 58; repeated evaluation of those same questions does not increase task diversity. Without independent discovery repetitions and paired uncertainty estimates, the reliability of the optimizer comparison and the smaller ablation improvements remains unclear.

Questions for authors
  1. Additional evidence needed to establish the main contribution: How does ASD perform when the same discovery model edits the same library representation but training-example selection and validation timing follow fixed rules? Please hold feedback access and rollout budget constant. What optimizer models and configurations were used for GEPA and SkillOpt, and were they given the same opportunity to exploit execution histories?

  2. Clarification and a baseline control: What fraction of seed and optimized failures comes from incorrect reasoning, invalid tool calls, timeouts, or missing/malformed final submissions? How much improvement remains over a no-skill baseline equipped with correct submission instructions and basic code-execution guidance, without instructions to retrieve from an empty library? This would directly clarify what ASD is improving.

  3. Robustness evidence: How many independent discovery runs produced each reported result? Can you report variability across discovery runs and paired confidence intervals for the main comparisons and Table 5? Please distinguish variation from skill discovery from variation in repeated execution of one fixed library.

  4. Efficiency clarification: What exactly is counted in the approximately 69% context reduction: skill-body tokens, the entire input context, or cumulative input tokens across all model calls? Are skill descriptions, retrieval calls, and repeated conversation history included? Please also report discovery-agent tokens and total inference cost, since equal target-rollout budgets do not imply equal computational expenditure.

These questions concern the validity and interpretation of the current claims, rather than extensions to additional application domains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.