- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SkillGenBench, a benchmark of 211 tasks for evaluating skill generation from code repositories and documents. It includes task-conditioned and task-agnostic generation, in which skills are created before downstream tasks are revealed. Generated skills are evaluated through downstream task execution, with additional static checks and human analysis. Experiments compare five generation methods across six model backbones and analyze common failure modes.
Critical strengths
-
The paper studies a useful problem. It focuses on generating skills from raw repositories and documents, rather than only evaluating provided skills. Separating generation from execution makes it easier to study the quality of skill-generation methods.
-
The benchmark includes varied tasks and useful analyses. It covers 211 tasks and evaluates several generation methods and model backbones. Human task review and failure analysis help assess task quality and understand where generated skills fail.
Critical weaknesses
-
The main table does not compare methods on the same task sets. Task coverage varies across cells, and the results combine two executor configurations. Although the paper provides some matched comparisons, the main table does not support a clear and fair comparison between methods. Also, skill-generation failures are excluded from the reported scores. Failing to produce a valid skill is itself a failure of the generation pipeline. Excluding these cases can overestimate performance and favor methods that produce valid skills on fewer tasks.
-
The benchmark provides limited evidence of skill generalization. With few tasks per source, a skill may perform well by recording task-specific rather than reusable information. Although hidden tests can verify correctness, the paper does not sufficiently demonstrate that the generated skills can help with general tasks from the same source.
Questions for authors
- Could you report method comparisons on the same task sets, separately for each executor configuration?
- Why are invalid or missing skill packages excluded rather than counted as generation failures? How would including these failures affect the results and rankings?
- How do you check whether generated skills capture reusable procedures rather than task-specific information? With so few tasks per source, a skill could perform well by reward hacking. Could you test the same frozen skills on new tasks and varied inputs to rule out this possibility?
4 comments
Response to Reviewer RUQ7: Method comparisons on shared task sets, rankings with generation failures, and frozen-skill reuse (Part 2)
W1, Q2: How including generation failures changes scores and rankings
Both scores and the ordering of some methods change; we have recalculated them using fixed task denominators. In the original 24-task configuration, each method is aggregated over five available generator backbones, giving 120 planned skill–task combinations:
| Method | Success conditional on valid packages | End-to-end success including combinations without valid packages |
|---|---|---|
| Naive Prompt | 84/117 = 71.8% | 84/120 = 70.0% |
| EvoSkill | 47/73 = 64.4% | 47/120 = 39.2% |
| SkillNet | 61/83 = 73.5% | 61/120 = 50.8% |
| SkillCreator | 76/103 = 73.8% | 76/120 = 63.3% |
| SkillSeekers | 98/116 = 84.5% | 98/120 = 81.7% |
Naive consequently moves from below SkillCreator and SkillNet to above both; SkillSeekers remains highest in this aggregate. Overall success changes from 366/492 = 74.4% to 366/600 = 61.0%. These combinations share tasks and sources; 120 is not the number of independent source samples.
The new matched-backbone experiment also uses end-to-end accounting: all 52 downstream failures associated with 13 GLM-5.3 generation-protocol failures are retained. We will use this accounting for generation-pipeline comparisons while retaining success conditional on valid packages for diagnosis.
W2, Q3: Frozen-skill reuse, new-input tests, and checks for evaluation exploitation
We have added three complementary tests: frozen packages on different tasks, fixed submitted programs on varied inputs, and interventions on rule content.
First, generators see only sources, not new tasks or tests, and artifacts are frozen before being used on new tasks from the same source. In the four-source experiment with three tasks per package, 9/12 Naive packages and 11/12 SkillNet packages pass all three tasks; in the experiment with six different sources and two tasks per package, the respective counts are 9/12 and 8/12. Task-level results in the latter experiment are 20/24 for Naive, 19/24 for SkillNet, 5/24 for no materials, and 24/24 for both the full source and summary. The fraction of packages passing all tasks directly answers whether the same materials serve different programming requirements, beyond high scores on the original tasks.
Second, we keep 312 existing submitted programs fixed and add 100 inputs per program, completing 31,200 offline evaluations. Of the 257 programs that originally passed, 248 continue to pass all new inputs (96.5%), while 9 show a decline. Submitted programs remain unchanged throughout, directly testing whether generated programs handle varied inputs.
Third, generators do not see target tasks in the controlled cross-encoding experiment with private rules. Each condition has 48 evaluations; Naive- and SkillNet-generated materials yield 47 and 36 successes, respectively, while the control receiving only target encodings and public computational contracts, without private mapping rules, yields 0 successes. Preserving the material structure while permuting only the rules causes outputs in 288/288 hidden batches to follow the permuted rules. This directly links source content to execution behavior, adding evidence of information use that hidden tests alone do not provide.
The three tests measure coverage of different tasks by the same frozen artifact, fixed programs' ability to handle new inputs, and the effect of source rules on execution behavior, respectively. In controlled transfer, target encodings and computational contracts are supplied, and the plain summary also achieves 48/48. Rule interventions provide evidence of content use, while packages passing multiple tasks provide evidence of reuse. The revision will organize the generalization analysis around these three measured properties, connecting high scores to specific observable behavior.
Thank you for encouraging us to present fair comparisons and reuse evidence directly. End-to-end denominators, source mappings, and frozen tests will be central to the corresponding analyses in the revision.
Response to the Authors
Thank you for the detailed response and the additional experiments. My first two concerns are addressed.
My concern about generalization is only partly resolved. The new reuse results come from supplementary experiments rather than from the 211-task benchmark itself, in which most sources still have only one task. In addition, method rankings vary considerably across the new experiments, and a plain summary often matches or exceeds the skill methods, so it remains unclear what the benchmark reliably measures about skill generation.
More importantly, the rebuttal adds a large body of new experiments that are central to the paper's claims. Across the responses to all reviewers, the rebuttal is effectively used as the main text. Meanwhile, the submitted main text does not even reach the page limit. The submission reads as an incomplete draft rather than a finished paper, which reflects a lack of care in its preparation and places an unreasonable burden on reviewers. I encourage the authors to complete the paper and submit it to a future venue. I therefore maintain my score.
Response to Reviewer RUQ7: Manuscript Revision and Evidence Organization
Thank you for your specific suggestions on fair comparison and skill reuse. We have completed a revised manuscript with 9 pages of main text, organized around generation reliability, reuse of frozen artifacts, the sources of failure, and the quality–cost tradeoffs of reuse.
The Introduction and Section 3 first clarify the relationships among sources, generated artifacts, and downstream tasks, distinguishing the original 211-task resource from the new multi-task evaluations. Section 4.2 then compares methods on the same tasks, budgets, and scoring rules within each execution configuration. It reports both generation reliability and downstream utility, retaining generation failures in the end-to-end denominator: the aggregate success rate in the original 24-task matrix changes from 74.4% to 61.0%, and Naive's ordering relative to some other methods changes accordingly. This directly demonstrates why generation reliability is an essential part of pipeline evaluation.
Section 4.3 examines whether the generated artifacts are reusable: evaluation tasks are withheld during generation, and each package remains frozen while serving tasks with different program requirements. For example, in the four-source study with three tasks per package, 11/12 SkillNet packages pass all three tasks. Appendix N separately reports new-input tests on fixed programs: of the 257 programs that originally passed, 248 pass all 100 additional inputs assigned to each program (96.5%), separating cross-task reuse from input robustness.
Building on these results, Section 4.4 examines the causes of failure. In the controlled cross-encoding study, inlining existing auxiliary rules at the entry point of fixed packages increases success from 36/48 to 42/48, recovering 6 cases with no regressions. This shows how the evaluation distinguishes rule retention from rule accessibility.
Finally, Section 4.5 connects task coverage to total cost, including generation overhead, with full-source, plain-summary, and retrieval controls. Method comparison, reuse validation, failure diagnosis, and quality–cost analysis thus form a coherent argument. The key results are now included in the main text, with supplementary experimental settings, stratified results, and robustness analyses in the appendices.
Thank you again for helping us clarify the connection between the benchmark's contributions and its supporting evidence. We hope that the revised manuscript and additional evidence will merit your support.
Response to Reviewer RUQ7: Method comparisons on shared task sets, rankings with generation failures, and frozen-skill reuse (Part 1)
Thank you for recognizing the research question and task construction effort. To address your three central questions, we have added matched comparisons, end-to-end denominator recalculation, and tests of frozen skills on new tasks and inputs. Including generation failures does change rankings, and the same frozen skill has succeeded on multiple new tasks. We provide direct evidence for each point below.
W1, Q1: Fair comparisons on the same task set and execution configuration
We have added comparisons on the same task set under the same execution configuration, incorporating generation reliability into end-to-end evaluation. Evaluation of skill-generation pipelines includes three complementary metrics: valid-package generation rate measures whether generation produces packages satisfying the interface and completion protocol; task success conditional on valid packages measures the downstream utility of generated artifacts; and end-to-end task success includes generation failures with all planned tasks in the denominator. These metrics capture generation reliability, artifact utility, and actual task-completion ability, respectively.
The original 187-task and 24-task results are reported separately under their two historical configurations, and executor-specific results on the original 20-task shared set are retained. The supplementary experiment fixes sources, tasks, execution budgets, and scoring rules within each configuration: eight sources and 16 tasks are crossed with two generator backbones and two executor backbones, with three independently generated packages per source–method–generator-backbone combination.
The table includes all five methods. Each cell is the number of successes from the same 48 planned evaluations, with generation failures retained in the denominator. Thus, every method is compared on the same task set, without changing the evaluation subset according to the number of valid packages.
Differences during generation also enter these scores. Across this batch's generation records, which include five skill methods and the summary baseline, GPT-5.5 completes 144/144 generations and GLM-5.3 completes 131/144; all 52 downstream failures associated with the latter's 13 generation-protocol failures are included. The full source, plain summary, and no-material conditions use the same tasks as controls. For example, under GLM-5.3 → GLM-5.3, the full source achieves 43/48, while the summary and SkillSeekers both achieve 41/48. SkillSeekers has the highest success count among skill methods across all four combinations on these sources, a within-configuration result obtained under a uniform protocol.
In the larger experiment with 16 sources and 96 contracts, eight conditions use three generations and two executions per package. The leading skill methods in the three generation repeats are SkillCreator, SkillSeekers, and Naive, respectively. This shows why method comparisons should report sources, generated artifacts, and execution configurations, and how repeated experiments reveal ranking sensitivity to these conditions.
Natural specification/implementation tasks provide another complete comparison of native methods. Across eight existing families and 32 tasks, each source direction uses three existing generated packages, yielding 96 evaluations per method. All methods use the same GPT-5.5 execution configuration and retain their native entry points:
On this task set, SkillCreator and SkillNet achieve 68/96 and 67/96, respectively. The leaders across the three generation repeats are SkillNet, SkillCreator, and a tie between EvoSkill and SkillCreator, respectively. All methods retain fixed denominators, with generation failures and service statuses handled under the same rules.
These additions strengthen the evaluation of skill-generation pipelines: jointly measuring generation reliability and downstream utility under the same task and execution conditions, with end-to-end success reflecting actual task-completion ability. The recalculation below shows how including generation failures changes scores and rankings.