- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SkillCraft, a benchmark of 126 tasks across 21 task families. It studies whether agents can create, reuse, and combine executable skills. Experiments with eight models show that skills often improve success and reduce token use, while deeper skill hierarchys can be less reliable.
Critical strengths
- The benchmark makes skill creation and reuse directly observable. Its controlled task difficulty helps study when reusable code benefits agents.
- The experiments cover multiple models, skill hierarchies, and transfer settings. They also show that successful skill execution does not always lead to task success.
Critical weaknesses
- Transfer mainly occurs within existing task families, and Hard-to-Hard transfer repeats the same tasks. This provides limited evidence of generalization to new workflows.
- The DirectExec baseline is useful but covers only 48 tasks. Token comparisons use jointly solved subsets, making overall efficiency and the cost of creating skills harder to assess.
Questions for authors
- How well do skills transfer to held-out workflow structures rather than repeated tasks or task families?
- Can you extend DirectExec to the full benchmark and report total costs across all tasks, including skill creation?
We thank the reviewer tLKB for the constructive suggestions on transfer and baselines. We are glad that the reviewer finds that the benchmark makes skill creation and reuse directly observable, that the controlled difficulty helps study when reusable code benefits agents, and that the experiments cover multiple models, hierarchies and transfer settings. Here we answer all the questions and hope they can address the concerns.
Q1: How well do skills transfer to held-out workflow structures?
Thank you for your suggestion! We add a leave-families-out evaluation using skills from 16 families and 30 tasks from five held-out families with disjoint APIs. We compare Claude-4.5-Sonnet with an empty library against a foreign skill library. Both conditions allow new skill creation and share the same tools and base instructions; the foreign-library condition additionally exposes skill descriptions, input/output schemas, and invocation examples. The results are presented below:
Exposure to skills from other families helps the agent construct skills for held-out workflows: skill creation increases from 10/30 to 24/30 tasks, with 50% fewer tokens and unchanged task success. This benefit occurs through the provided skill information, without directly executing foreign skills. We follow your advice and add in the paper in line 1201-1212.
Q2: Can you extend DirectExec to the full benchmark and report total costs across all tasks, including skill creation?
Thank you for your suggestion! We extend the baseline evaluation and report full-trajectory costs below.
Expanded evaluation. We rerun all three modes on 30 tasks covering all 21 families: each family's hard task plus nine tasks with the largest prior disagreement across models and modes. The rebuttal budget limits coverage to this subset rather than all 126 tasks.
GPT uses the same 27 tasks across modes after excluding three API-interrupted tasks. Costs include successful and unsuccessful trajectories. These results are consistent with Appendix D.1: constructing parameterized, reusable skills introduces overhead in single-task execution. The value of the resulting procedures also depends on whether they remain useful beyond the task in which they were created. Our cross-difficulty and cross-model analyses test this reuse without modifying the skills, while the hierarchy analysis examines the reliability of composing them into larger workflows. Together, these analyses characterize both the cost of skill construction and the transferability and composition reliability of the acquired skills. We follow your advice and add in the paper in line 877-883.
Full-benchmark costs. We aggregate all available trajectories, including failed tasks, skill creation, repairs, and fallback:
GLM and Gemini lack one and two Skill records, respectively. Skill Mode reduces total tokens by 42–70% relative to Normal, including the above overheads. We follow your advice and add in the paper in line 1217-1221.