- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Prompt2Skill, which automatically builds and optimizes LLM skills from natural-language task descriptions. It combines proxy data acquisition with reflection-based skill refinement and evaluates the resulting skills across four tasks and five models.
Critical strengths
-
The framework retrieves or generates task-specific data without requiring user-provided training examples. This makes skill optimization more practical for tasks without curated datasets.
-
Prompt2Skill improves performance on several tasks, particularly SQuAD, where it outperforms direct prompting across all five models. The matched-budget comparison with SkillOpt also provides useful evidence for the optimization procedure.
Critical weaknesses
-
The framework combines data acquisition, reflective editing, and skill selection, but the experiments do not isolate their individual effects. The comparison with SkillOpt covers only two tasks, with improvements of 2.92 and 1.20 percentage points in exact match. It remains unclear how much the proposed pipeline improves over existing skill optimization methods using the same automatically acquired data.
-
3/4 benchmarks mainly evaluate task-specific text instructions, while improvements on SpreadsheetBench, which involves actual tool execution, are inconsistent. For example, Prompt2Skill performs slightly worse than direct prompting on SpreadsheetBench with both Claude Haiku 4.5 and GPT-5.5. The current results provide stronger evidence for optimizing task-specific prompts than for constructing effective skills across different types of agent tasks.
Questions for authors
-
Can the authors separately evaluate the contributions of automatic data acquisition, reflective editing, and final skill selection under the same evaluation budget?
-
Can the authors extend the matched-budget comparison to AIME and SpreadsheetBench and report results across multiple independent optimization runs?
-
Why does the LLM-generated baseline use Qwen3-32B while Prompt2Skill uses GPT-5.5 for reflection? How does the method compare with a GPT-5.5-generated skill under a comparable generation and evaluation budget?
-
How much of the optimization data comes from retrieved datasets versus generated examples? How does performance change when relevant labeled datasets are unavailable or the retrieved data differ substantially from the target task?
-
The paper describes accepted edits as statistically significant, while Appendix D states that the threshold is a heuristic without multiple-comparison correction. Can the authors clarify this claim and evaluate whether stricter acceptance criteria change the results?
6 comments
Reply to Reviewer bYax - Part 2
Q3. GPT-5.5 Authored Skills. We thank the reviewer for raising this point. Since P2S uses GPT-5.5 as its reflector, we agree that a skill authored by GPT-5.5 is a fairer baseline. We report the results below, with Qwen3-32B as the target model:
| Benchmark | Qwen authored baseline | GPT-5.5 authored baseline | Existing P2S |
|---|---|---|---|
| SearchQA | 60.81 ± 0.23 | 57.26 ± 0.11 | 77.76 ± 1.88 |
| SQuAD | 63.17 ± 0.21 | 57.10 ± 0.17 | 82.47 ± 0.90 |
| AIME | 17.78 ± 1.27 | 19.17 ± 0.84 | 22.22 ± 1.74 |
| SpreadsheetBench | 17.90 ± 0.25 | 15.40 ± 0.49 | 23.33 ± 1.94 |
As the table shows, a stronger authoring model does not produce a better skill. The GPT-5.5 authored skill is no better than the Qwen authored one: it is lower on three of the four benchmarks and higher only on AIME. P2S outperforms both on every benchmark. This shows that the gains of P2S do not come from using a stronger model to write the skill. They come from refining the skill with feedback from the target model's own failures on the proxy data.
Q4. Retrieval vs. Synthetic Data
How much data is retrieved and how much is synthesized depends on the task. For SearchQA, SQuAD, and AIME, retrieval succeeds, and P2S mainly adapts the retrieved items to the format the user requested. For SpreadsheetBench, no suitable dataset exists in the data sources, so P2S synthesizes over 90% of the optimization data.
To keep retrieved sources close to the request, the Data Agent checks every source before it enters the pool (Section 3.2). It can mark a source as irrelevant and skip it. It compares converted items with the task specification on language, answer style, task type, and whether each input contains every required part; for example, a QA source without passages is rejected for passage based QA. It drops any source on which the seed equipped model fails every sampled item. Beyond these checks, the final selection step can fall back to the initial skill when the proxy data do not support useful edits.
Although it is rare, we acknowledge that relevant labeled data may not exist for some tasks. In Appendix B (Figure 4), we study two such tasks where the Data Agent found no matching labeled dataset: temporal expression normalization and Japanese to Python code generation (MCoNaLa-ja). Working from the task description and three demonstrations, P2S still improves over the plain Qwen3-32B baseline (76.12 vs 69.51 on Temporal, 38.73 vs 37.99 on MCoNaLa-ja). It also remains competitive with Prompt2Model, which fine tunes the model on up to 1,000 generated examples, even though P2S updates no weights.
We see this setting as a promising future direction. Instead of relying only on existing datasets and synthesis, the Data Agent could collect raw data itself, for example by scraping Wikipedia or other web sources through a coding agent harness, and then build labeled items from that material. This is beyond the scope of this paper, and we leave it to future work.
Q5. Statistical Significance of Accepted Edits.
We thank the reviewer and will correct the wording. Accepted edits pass a heuristic paired screening threshold (continuity corrected z ≥ 1), not a 5% significance test, and no correction for multiple comparisons is applied. In the revision, we will replace "statistically significant paired comparison" in Section 1 with "paired screening test" and state the threshold in the main text.
To check whether a stricter rule changes the results, we reran Qwen3-32B on SQuAD with an exact one sided paired test at p ≤ 0.05. The stricter rule accepts fewer edits (3 to 1), and the gain over the starting skill falls from +3.2 to +2.4 points. Refinement therefore still helps under a conventional significance test, and keeps most of its gain. The looser screen accepts additional useful edits that the strict test rejects, while weak edits it lets through are still filtered by the final selection on the frozen set (Theorem 1). We will add this comparison to the appendix.
We will update the PDF with the additional results and the proposed changes, highlighted in blue. We hope these responses address the reviewer's concerns, and we would be glad to clarify anything further.
Thank you for the additional experiments and detailed clarifications, which have addressed most of my concerns. However, I still have some concerns about the lack of results across multiple independent runs and the limited evidence for effectiveness on tool-execution tasks. I would be happy to reconsider my score if these concerns can be further addressed.
Reply to Reviewer bYax - Part 1
We thank the reviewer for the detailed review and constructive feedback, and we provide the feedback as follows:
W1(a)/Q1. Isolation of Contribution. To directly isolate the contributions of the components of the proposed P2S method, we conduct additional experiment to evaluate the individual component. Specifically, we design the following variant to the P2S for evaluation:
(1) Description only synthesis + Skill Agent: This variant removes the data agent's retrieval stage, and thus isolate the effectiveness of the proxy data construction from retrieval.
(2) GPT-5.5 / Qwen3-32B skill from 10 examples of the real dataset. This differs from the LLM-generated baseline: here GPT-5.5 or Qwen3-32B is given 10 actual examples from the benchmark and writes the skill once. This variant isolates the contribution of execution feedback and iterative refinement, compared with writing a skill in one pass from real examples. It simulates a user who collects a few examples by hand and asks an LLM to write the skill.
(3) Last incumbent instead of final skill selection. This variant exports the last accepted skill instead of choosing among the retained skills on the frozen selection set. It isolates the contribution of the final selection step.
As we can observe, removing any component hurts performance, so every component contributes. Removing retrieval costs 9.25 points on SearchQA and 3.43 on SQuAD. Writing the skill once from 10 real examples, without the refinement loop, is weaker and less stable across authoring models. Skipping the final selection step costs 12.89 points on SearchQA. The full Prompt2Skill remains best on both tasks.
W1(b)/Q2. Comparing to SkillOpt on AIME and SpreadsheetBench
We agree that the comparison strengthens the case for our optimization procedure. The table below compares SkillOpt and P2S on AIME and SpreadsheetBench, using Qwen3-32B as the target model and the same optimization budget for both methods.
P2S outperforms SkillOpt on both benchmarks. We attribute this to the reflect and select design of P2S. SkillOpt accepts an edit whenever it strictly improves a held out validation score. Because that same score is reused across rounds, small gains that do not hold on new items can still be accepted. P2S instead compares each edit with the current skill on a fresh batch that was not used to write the edit. It then chooses the final skill once, on a frozen selection set that refinement never sees. This design is especially important when optimizing on proxy data. SkillOpt was designed for a curated training set drawn from the target task. Proxy items are retrieved or synthesized instead, so they can be noisy and differ from the real task. Testing every edit on fresh items, and choosing the final skill on a separate frozen set, keeps only edits that carry over beyond the items used to write them.