acceptodds
Under review as a conference paper at ICLR 2027

Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions

Abstract

Skills are external artifacts that Large Language Models (LLMs) consume at inference time to improve their performance on specialized domains by incorporating relevant procedural and domain knowledge. Expert-authored skills are expensive to produce, and the resulting artifacts are not optimized for the specific model that consumes them, whose failure modes can vary with version, scale and training. In addition, emerging tasks may fall outside the scope of existing skill libraries, creating a need to develop new skills before curated training data become available. Recent works have explored automated skill optimization through reflection, but they require a curated, in-distribution training set, which users might not always have. To address these limitations, we present **Prompt2Skill**, a framework that builds skills from natural-language task description alone. From the prompt, the system derives a task specification, discovers or synthesizes datasets, and refines the skill in a closed loop of reflective editing. Across four domains spanning question answering, reading comprehension, spreadsheet manipulation, and mathematical reasoning, Prompt2Skill consistently outperforms the direct prompting baseline, achieving an average improvement of **10.8** across open-source and frontier models.

Primary area:
generative models
Keywords:
Large Language Models, Harness, Skills

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bYaxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Prompt2Skill, which automatically builds and optimizes LLM skills from natural-language task descriptions. It combines proxy data acquisition with reflection-based skill refinement and evaluates the resulting skills across four tasks and five models.

Critical strengths
  1. The framework retrieves or generates task-specific data without requiring user-provided training examples. This makes skill optimization more practical for tasks without curated datasets.

  2. Prompt2Skill improves performance on several tasks, particularly SQuAD, where it outperforms direct prompting across all five models. The matched-budget comparison with SkillOpt also provides useful evidence for the optimization procedure.

Critical weaknesses
  1. The framework combines data acquisition, reflective editing, and skill selection, but the experiments do not isolate their individual effects. The comparison with SkillOpt covers only two tasks, with improvements of 2.92 and 1.20 percentage points in exact match. It remains unclear how much the proposed pipeline improves over existing skill optimization methods using the same automatically acquired data.

  2. 3/4 benchmarks mainly evaluate task-specific text instructions, while improvements on SpreadsheetBench, which involves actual tool execution, are inconsistent. For example, Prompt2Skill performs slightly worse than direct prompting on SpreadsheetBench with both Claude Haiku 4.5 and GPT-5.5. The current results provide stronger evidence for optimizing task-specific prompts than for constructing effective skills across different types of agent tasks.

Questions for authors
  1. Can the authors separately evaluate the contributions of automatic data acquisition, reflective editing, and final skill selection under the same evaluation budget?

  2. Can the authors extend the matched-budget comparison to AIME and SpreadsheetBench and report results across multiple independent optimization runs?

  3. Why does the LLM-generated baseline use Qwen3-32B while Prompt2Skill uses GPT-5.5 for reflection? How does the method compare with a GPT-5.5-generated skill under a comparable generation and evaluation budget?

  4. How much of the optimization data comes from retrieved datasets versus generated examples? How does performance change when relevant labeled datasets are unavailable or the retrieved data differ substantially from the target task?

  5. The paper describes accepted edits as statistically significant, while Appendix D states that the threshold is a heuristic without multiple-comparison correction. Can the authors clarify this claim and evaluate whether stricter acceptance criteria change the results?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer bYax - Part 1

We thank the reviewer for the detailed review and constructive feedback, and we provide the feedback as follows:

W1(a)/Q1. Isolation of Contribution. To directly isolate the contributions of the components of the proposed P2S method, we conduct additional experiment to evaluate the individual component. Specifically, we design the following variant to the P2S for evaluation:

(1) Description only synthesis + Skill Agent: This variant removes the data agent's retrieval stage, and thus isolate the effectiveness of the proxy data construction from retrieval.

(2) GPT-5.5 / Qwen3-32B skill from 10 examples of the real dataset. This differs from the LLM-generated baseline: here GPT-5.5 or Qwen3-32B is given 10 actual examples from the benchmark and writes the skill once. This variant isolates the contribution of execution feedback and iterative refinement, compared with writing a skill in one pass from real examples. It simulates a user who collects a few examples by hand and asks an LLM to write the skill.

(3) Last incumbent instead of final skill selection. This variant exports the last accepted skill instead of choosing among the retained skills on the frozen selection set. It isolates the contribution of the final selection step.

ConditionSearchQA (%)SQuAD (%)
Full Prompt2Skill77.8282.53
Description only synthesis + Skill Agent68.5779.10
GPT-5.5 skill from 10 examples63.0081.50
Qwen3-32B skill from the same 10 examples70.3661.70
Last incumbent instead of final skill selection64.9382.53

As we can observe, removing any component hurts performance, so every component contributes. Removing retrieval costs 9.25 points on SearchQA and 3.43 on SQuAD. Writing the skill once from 10 real examples, without the refinement loop, is weaker and less stable across authoring models. Skipping the final selection step costs 12.89 points on SearchQA. The full Prompt2Skill remains best on both tasks.

W1(b)/Q2. Comparing to SkillOpt on AIME and SpreadsheetBench

We agree that the comparison strengthens the case for our optimization procedure. The table below compares SkillOpt and P2S on AIME and SpreadsheetBench, using Qwen3-32B as the target model and the same optimization budget for both methods.

MethodAIMESpreadsheetBench
SkillOpt0.200.21
P2S0.220.23

P2S outperforms SkillOpt on both benchmarks. We attribute this to the reflect and select design of P2S. SkillOpt accepts an edit whenever it strictly improves a held out validation score. Because that same score is reused across rounds, small gains that do not hold on new items can still be accepted. P2S instead compares each edit with the current skill on a fresh batch that was not used to write the edit. It then chooses the final skill once, on a frozen selection set that refinement never sees. This design is especially important when optimizing on proxy data. SkillOpt was designed for a curated training set drawn from the target task. Proxy items are retrieved or synthesized instead, so they can be noisy and differ from the real task. Testing every edit on fresh items, and choosing the final skill on a separate frozen set, keeps only edits that carry over beyond the items used to write them.

AuthorsOpenReview ↗

Reply to Reviewer bYax - Part 2

Q3. GPT-5.5 Authored Skills. We thank the reviewer for raising this point. Since P2S uses GPT-5.5 as its reflector, we agree that a skill authored by GPT-5.5 is a fairer baseline. We report the results below, with Qwen3-32B as the target model:

BenchmarkQwen authored baselineGPT-5.5 authored baselineExisting P2S
SearchQA60.81 ± 0.2357.26 ± 0.1177.76 ± 1.88
SQuAD63.17 ± 0.2157.10 ± 0.1782.47 ± 0.90
AIME17.78 ± 1.2719.17 ± 0.8422.22 ± 1.74
SpreadsheetBench17.90 ± 0.2515.40 ± 0.4923.33 ± 1.94

As the table shows, a stronger authoring model does not produce a better skill. The GPT-5.5 authored skill is no better than the Qwen authored one: it is lower on three of the four benchmarks and higher only on AIME. P2S outperforms both on every benchmark. This shows that the gains of P2S do not come from using a stronger model to write the skill. They come from refining the skill with feedback from the target model's own failures on the proxy data.

Q4. Retrieval vs. Synthetic Data

How much data is retrieved and how much is synthesized depends on the task. For SearchQA, SQuAD, and AIME, retrieval succeeds, and P2S mainly adapts the retrieved items to the format the user requested. For SpreadsheetBench, no suitable dataset exists in the data sources, so P2S synthesizes over 90% of the optimization data.

To keep retrieved sources close to the request, the Data Agent checks every source before it enters the pool (Section 3.2). It can mark a source as irrelevant and skip it. It compares converted items with the task specification on language, answer style, task type, and whether each input contains every required part; for example, a QA source without passages is rejected for passage based QA. It drops any source on which the seed equipped model fails every sampled item. Beyond these checks, the final selection step can fall back to the initial skill when the proxy data do not support useful edits.

Although it is rare, we acknowledge that relevant labeled data may not exist for some tasks. In Appendix B (Figure 4), we study two such tasks where the Data Agent found no matching labeled dataset: temporal expression normalization and Japanese to Python code generation (MCoNaLa-ja). Working from the task description and three demonstrations, P2S still improves over the plain Qwen3-32B baseline (76.12 vs 69.51 on Temporal, 38.73 vs 37.99 on MCoNaLa-ja). It also remains competitive with Prompt2Model, which fine tunes the model on up to 1,000 generated examples, even though P2S updates no weights.

We see this setting as a promising future direction. Instead of relying only on existing datasets and synthesis, the Data Agent could collect raw data itself, for example by scraping Wikipedia or other web sources through a coding agent harness, and then build labeled items from that material. This is beyond the scope of this paper, and we leave it to future work.

Q5. Statistical Significance of Accepted Edits.

We thank the reviewer and will correct the wording. Accepted edits pass a heuristic paired screening threshold (continuity corrected z ≥ 1), not a 5% significance test, and no correction for multiple comparisons is applied. In the revision, we will replace "statistically significant paired comparison" in Section 1 with "paired screening test" and state the threshold in the main text.

To check whether a stricter rule changes the results, we reran Qwen3-32B on SQuAD with an exact one sided paired test at p ≤ 0.05. The stricter rule accepts fewer edits (3 to 1), and the gain over the starting skill falls from +3.2 to +2.4 points. Refinement therefore still helps under a conventional significance test, and keeps most of its gain. The looser screen accepts additional useful edits that the strict test rejects, while weak edits it lets through are still filtered by the final selection on the frozen set (Theorem 1). We will add this comparison to the appendix.

We will update the PDF with the additional results and the proposed changes, highlighted in blue. We hope these responses address the reviewer's concerns, and we would be glad to clarify anything further.

Thank you for the additional experiments and detailed clarifications, which have addressed most of my concerns. However, I still have some concerns about the lack of results across multiple independent runs and the limited evidence for effectiveness on tool-execution tasks. I would be happy to reconsider my score if these concerns can be further addressed.

Reviewer_ZsZxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Prompt2Skill, a framework that automatically creates and optimizes LLM skills from natural language task descriptions. The core idea is to retrieve and generate task-relevant data from sources such as Hugging Face, and then use an iterative skill optimization process to produce skills. The paper evaluates Prompt2Skill on four benchmarks and shows that the proposed framework generally improves performance.

Critical strengths

The framework is useful, especially for general users who are not familiar with AI techniques. Users only need to describe their tasks in natural language, and the system can automatically collect data and optimize a skill.

Critical weaknesses
  1. I think the paper could better focus on its main contribution. Compared with previous optimization methods, such as GEPA and SkillOpt, the skill optimization process is largely similar. They follow a similar workflow: execute tasks, collect failures, reflect on the failures, revise the skill, and decide whether to accept the revision. And the performance improvements over SkillOpt are also relatively limited as shown in Table 2. In my opinion, the more interesting contribution is automatically discovering and constructing useful data from natural language descriptions. However, the analysis of this component is relatively limited. I would suggest focusing more on how the framework retrieves, selects, and validates data, and how these decisions affect the final skill performance. This would better highlight what makes Prompt2Skill different from existing optimization methods.

  2. Another important concern is that the automatically retrieved or generated data may not match the distribution of the user's actual tasks. The paper introduces several validation steps to check task compatibility and answerability. However, it is unclear whether these checks are enough to ensure that the collected data matches the difficulty and reasoning requirements of the target task. For example, suppose a user wants to solve frontier-level mathematical reasoning problems, but the system retrieves a relatively simple dataset such as GSM8K. Would the framework accept this dataset? If so, could optimizing skills on these easier problems negatively affect performance on the actual target tasks?

Questions for authors
  1. How does the framework balance dataset diversity and the number of examples from each dataset? For example, for a reading comprehension task, would it be better to retrieve 10 datasets with 100 examples each, or 2 datasets with 500 examples each?

  2. Since automatic data acquisition is a key contribution of this paper, I think it would be useful to evaluate the quality of retrieved datasets beyond the final skill performance. For example, given a user's natural language task description, how can we measure whether the retrieved datasets are actually suitable for the target task? Could the authors consider intermediate metrics such as task relevance, difficulty alignment, etc.? Any good thoughts on this?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer ZsZx - Part 1

We thank the reviewer for the review and for recognizing that the framework is useful for general users. We address the weaknesses and questions as follows.

Weaknesses

W1. Main contribution and analysis of the data component. We appreciate the reviewer's comments on contribution focus. The main contribution of Prompt2Skill lies in its integrated pipeline, which automates both data construction and skill refinement so that a user only needs to describe the task. Although the refinement loop shares the general workflow of prior work (SkillOpt, GEPA), these methods assume a curated training set drawn from the target task. In our setting, the optimization data are retrieved or synthesized proxies, which can be noisy, mismatched in difficulty, or different from the real task. The refinement process is therefore designed to take advantage of proxy data while guarding against its errors. An edit is accepted only if it beats the current skill on a fresh batch of proxy items that was not used to write it, which filters out edits that fit the quirks of a few items. The final skill is then chosen once on a frozen selection set, with the initial skill kept as a fallback, so later edits that do not carry over can be discarded.

The following table compares the performance of P2S to SkillOpt with the same retrieved data. As we can observe, P2S outperforms SkillOpt on all four benchmarks by consistent margins. Removing final selection costs 12.89 points on SearchQA, and in W2 below, final selection stops every harmful AIME skill learned from easier data from shipping. We agree that the data component deserves deeper analysis. In the revision, we will add a new section on how the Data Agent retrieves, selects, and validates data, and how these choices affect the final skill, including the new results from the rebuttal.

MethodSearchQA EMSQuAD EMSQuAD F1AIME AccSpreadsheetBench Acc
SkillOpt45.7981.3090.5020.021.0
P2S48.7182.5091.1122.023.0

W2. Distribution of the retrieved dataset. We appreciate the reviewer for raising this concern. Our checks cover task type, answer format, language, and whether the solver can answer at all, but not difficulty. We specifically tested the example the reviewer described, optimizing a skill for answering olympiad level math questions. Since difficulty is not part of the checks, the framework would accept GSM8K. Raw GSM8K was rejected only because of its format, and after format processing, GSM8K passed every check. MATH (all levels) and OpenMathInstruct-2 were also accepted. We recognize that difficulty is an important dimension of data quality, and we study its effect below.

To test whether easier data hurts, we optimized AIME skills on pools of different difficulty:

Table R1. AIME accuracy (%) after optimizing on data of different difficulty. Prompt only baseline: 15.8.

Optimization dataSkill after refinementFinal selected skill
GSM8K17.218.9
GSM8K, failed items only0.015.8
MATH levels 1 and 25.815.8
MATH level 521.119.4
NuminaMath-1.521.118.6

As the table shows, optimizing on easier data can fail to transfer to the target task (1st and 3rd row), while optimizing on items too hard can lead to no learning signal (2nd row) since there is no lesson to be learned from rollouts. However, this is exactly the mismatch the refinement process is designed to guard against (W1): the final selection step, which keeps the plain prompt as a fallback, stopped every harmful skill from shipping (15.8 in both cases), at the cost of being slightly conservative on the harder pools (19.4 and 18.6 vs 21.1). We discuss more potential ways of leveraging the retrieved task difficulty in the following response to question 2 (Q2).

AuthorsOpenReview ↗

Reply to Reviewer ZsZx - Part 2

Questions

Q1. Dataset diversity vs. number of examples per dataset. In Prompt2Skill, dataset diversity depends on the sources available for the task, and the Data Agent decides it. The agent chooses which retrieval tools to call and what queries to issue (Section 3.2), and the items from sources that pass validation make up the optimization pool. As a result, neither the number of sources nor the number of items per source is fixed in advance. A task with many relevant datasets can draw on several sources, while a task with few relies on one source or on synthesis. In our main experiments, this led to one or two sources per task (Appendix C, Table 3). To understand how this choice affects the final skill, we tested it directly on SQuAD with a fixed budget of 1,000 items drawn from 10 relevant sources:

We tested this directly on SQuAD with a fixed budget of 1,000 items drawn from 10 relevant sources:

Table R2. SQuAD accuracy (%) for different source mixes. Prompt only baseline: 76.9.

MixMean ± SD
1 source × 1,000 items81.9 ± 2.4
2 sources × 500 items82.3 ± 1.4
10 sources × 100 items82.4 ± 2.6

As we can observe, the framework is robust to how the budget is split. Mixing sources barely changes the average score. This supports letting the agent decide based on the available sources. When several validated sources exist, combining a few of them also lowers the risk of relying on one poor source.

Q2. Measuring the quality of retrieved datasets. Currently, automatic data acquisition relies on the checks performed by the Data Agent: task relevance, answer format, language, input completeness, and whether the solver can answer at all (Section 3.2). Beyond these checks, the suitability of the data is reflected in the final skill performance. We agree that intermediate metrics would make data acquisition easier to evaluate, and we see difficulty alignment as one promising direction. As shown in Table R1, refining on harder data (MATH level 5 and olympiad level problems) can lead to better skills than refining on overly easy data.

This suggests a natural extension toward a self improving loop. As the skill gets stronger, the Data Agent could retrieve or generate harder items that target the skill's remaining failures, which in turn leads to a stronger skill. So instead of a one shot evaluation on the data quality, the framework can update its skills online with new data and skills involved in each update. Such a loop must ensure that the skill does not regress on the target task and still works well with the base model, for example by keeping the paired acceptance test and the final selection with the plain prompt as a fallback. We recognize this as an important future direction and will discuss it in the revision.

Thank you for the detailed response and the new experiments. I also find the self-improving loop interesting. However, after reading the response, I still have concerns about two parts of the paper.

  1. From the response, the main difference is that SkillOpt uses a fixed dev set to accept edits, while Prompt2Skill draws a new dev batch each round. This seems like a small change to the existing workflow rather than a new optimization method. The gains over SkillOpt with the same data are also small. I am also unsure whether the authors use the same number of dev data points. Prompt2Skill draws new items every round, so it may see much more distinct data than SkillOpt’s single fixed dev set. If so, the gains may come from more and fresher data rather than from the method itself. I would like the authors to clarify this. Based on this rebuttal, I still believe that the retrieval part is the main novelty of this paper, and my rating will be mainly for this part.

  2. For the related dataset retrieval, the W2 results suggest that the retrieval part is still not well explored. For example, generating skills on MATH levels 1–2 does not help. The agent retrieves and rewrites data, but it does not judge whether the data (e.g., MATH levels 1–2) is actually useful. It relies fully on the dev set. However, in real use, the user only has a natural language description, so the dev set is also retrieved data. A skill that works well on this dev set may still not work well on the real task. I think this is not an easy problem, and the current version does not address it well.

In summary, I still view retrieving useful and relevant data from a natural language description as the main contribution of this paper, and it has not been well explored in this version. I will keep my score.

Dear Reviewer ZsZx,

Thank you for the follow up and for the constructive discussion. To clarify the reviewer's question: as described in Section 4 (Table 2), both methods use the same proxy data pool, the same initial skill, and the same cap of 7,400 solver evaluations per task, so neither method has access to more data. The only difference is how the pool is partitioned: SkillOpt accepts edits on a fixed dev split, while P2S draws a new acceptance batch from the same pool in each round. We will make this explicit in the revision. In the revision, we will add our analysis of how data difficulty affects the final skill, and discuss this limitation explicitly. Thank you again for your time and valuable feedback.

Best regards,

Authors

Reviewer_h43HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Prompt2Skill builds an inference-time "skill" (i.e., a SKILL.md file) from a natural-language task description alone, with no user data. It has two agents:

  1. The "Data Agent": It infers a task spec and a proxy metric, retrieves data from Hugging Face and Wikipedia, then adapts or synthesizes items and validates them (answerability and consistency screens).

  2. The "Skill Agent": It proposes reflective edits from the target model's own failures. It accepts an edit only if it wins a paired sign-test-style comparison on a fresh batch, then picks the final skill once on a frozen selection set.

Critical strengths
  1. It builds skills with no curated training set, a step beyond SkillOpt and Trace2Skill, which need one.

  2. The experiment has broad coverage with 5 target models, including open 1B–32B and frontier ones, and 4 different domains. Large gains on SQuAD for every model; for example, Haiku 4.5 goes from 0.42 to 0.79.

Critical weaknesses

The paper demonstrated the effectiveness of the method in experiments. However, it is unclear why the inference-time built-up skill can lead to the performance gain. Suggestions: Add detailed ablations and analysis on what are included in the skills and why they would be beneficial for the model to performance better.

Questions for authors

Which datasets were retrieved for each benchmark? Do they include the benchmark's own train split?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer h43H

We thank the reviewer for the positive feedback, and we address the weaknesses and questions below.

W1. How does the skills improve performance?

To answer this, we analyzed the content of all 19 final skills (Appendix F.5) against the Direct prompt. We split each skill into single instructions and assigned each instruction one of eight types using GPT-6.1 Sol. What gets added depends strongly on the task, as shown in the following table:

Table R1. Share of instructions by type (%), pooled over the five target models.

Type of instructionSearchQASQuADAIMESpreadsheet
Answer form and granularity3950714
Reading the input (what is asked, answer type)209119
Solving procedure20123710
Verification and recovery751729
Tool and library knowledge00030

As the table shows, what the skills add depends on the task. For text question answering (SearchQA and SQuAD), most added instructions set the answer form and granularity. SearchQA skills also spend a fifth of their instructions on reading the clue and a fifth on solving procedure. For math reasoning (AIME), the skills mostly add solving procedures and verification steps. For tool heavy tasks such as spreadsheet manipulation, the skills mostly add tool and library knowledge, such as openpyxl code templates, together with verification checklists.

Q1. Which datasets were retrieved for each benchmark, and do they include the benchmark's own train split?

No. No benchmark's train split was used, either as input or as retrieved data. We list the sources of the retrieved dataset in Appendix C, Table 3:

BenchmarkRetrieved sourcesOwn train split used?Notes
SearchQAopenaccess-ai-collective/jeopardyNoA separate collection of Jeopardy clues.
SQuADallenai/sciq; hotpotqa/hotpot_qaNoBoth are different dataset from SQuAD. SciQ and hotpotQA have a different domain from SQuAD's Wikipedia passages
AIMEnvidia/OpenMathInstruct-2; SuperSecureHuman/competition_math_hf_datasetNocompetition_math_hf_dataset was released in 2021, and OpenMathInstruct-2 is built from additional GSM8K training problems
SpreadsheetBenchSyntheticNoN/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.