- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-Skills-Bench, a skill-centric benchmark for evaluating the effectiveness of agent skills in software engineering. It covers 98 skills paired with real GitHub repositories, generating 980 requirements and execution-based verifiers for evaluation across four agent–model configurations. Results from 3,920 paired comparisons show minimal overall improvement (76.59% to 76.93%), with only four skills significantly helpful and two harmful.
Critical strengths
- The benchmark provides a reasonably broad evaluation of SWE skills, covering 98 skills across six SWE-related categories.
- The empirical findings are insightful and challenge the assumption that more procedural guidance necessarily improves agent performance.
- The case studies provide interpretable examples of both beneficial and harmful skill usage.
Critical weaknesses
- The experimental analysis can be deeper, and the paper falls short of a systematic investigation into skill effectiveness and the mechanisms behind it. Results are drawn largely from aggregate performance numbers and a small number of case studies. More ablation studies would substantially strengthen the claims. For example, ablating different components of skill content can help tease apart the effects of procedural knowledge and concrete templates. The authors could also carry out further analysis to examine how performance gains vary with skill length, content quality, task difficulty, and task relevance.
- The benchmark construction and evaluation lack sufficient validation. The paper uses LLMs to generate tasks and tests based on skills without adequately verifying their reliability. Differences in task difficulty, complexity, and testing criteria also limit comparability across skills. Moreover, the primary metric is the average verifier pass ratio rather than the complete task success rate. Passing some tests does not necessarily indicate task completion, weakening the credibility of the experimental conclusions.
- The paper provides limited details on how skill–repository pairs are selected. Although the authors manually curate these pairs, they do not elaborate on their selection and matching rules. Another concern is that each skill is evaluated using only one repository, which limits the generalizability of their conclusions.
Questions for authors
- The evaluation is limited to relatively weak models. Could the authors evaluate more recent state-of-the-art models to examine whether the findings generalize to stronger agents?
- What distinguishes the four Golden Skills from the remaining skills? Why these skills are effective while others are not? The paper only provides a detailed case study of viper-configuration, without systematically analyzing the other Golden Skills or identifying their shared characteristics.
7 comments
Response to Reviewer 5SkC(Part 2)
3. (W2) Are the LLM-generated requirements and verifiers reliable?
We agree that automatic generation needs direct validation. We therefore added a full-scale automatic audit, a human audit of a sampled subset, and an execution test. The table summarizes the results, and the details follow.
| Validation | Scope | Result |
|---|---|---|
| Automatic audit | 980 pairs, 16,924 checks | 95.7% of the 4,652 referenced file paths are named in the requirement. All 41 dependency checks match requested packages |
| Human audit | 100 pairs, two annotators | 90 and 95 pairs pass all five dimensions. Each dimension passes in at least 96 pairs |
| Execution test | 3,920 pairs and 479 requirement instances on unmodified repositories | Both implementations pass every check in 1,313 pairs. Unmodified repositories pass 11.7% of the checks, against 77.2% for the agent |
An automatic audit was run on all 980 requirement-verifier pairs to check that no verifier relies on a file or dependency that its requirement never mentions. We extracted the file paths from the source of every check and looked them up in the requirement text. The 4.3% of paths that are not named are almost all paths that are not deliverables: test drivers that the verifier itself writes, test inputs, build outputs, existing repository files, and alternative locations that a check also accepts.
A human audit was then run on 100 pairs to check what the automatic audit cannot. The sample covers requirements R1-R5 of 20 skills: eight diagnostic skills (the four Golden skills, the two Harmful skills, and the two case-study skills vitest-unit-testing and github-actions-templates) and 12 randomly selected skills. Two annotators, each with three years of software-engineering experience, read the skill, the requirement, and the verifier source of every pair and labeled five dimensions independently: requirement clarity, leakage of the skill procedure, coverage of the stated criteria, undeclared constraints, and acceptance of reasonable alternative implementations.
An execution test was then run to check whether the verifiers accept different correct implementations and reject incomplete ones. The main experiment contains two independent implementations of every requirement, one without and one with the skill, and both pass every check in 1,313 of the 3,920 pairs. We also ran the verifiers on unmodified repositories, in which nothing is implemented, for requirements R1-R5 of all 98 skills. They pass only 11.7% of the checks on average, against 77.2% for the agent without the skill on the same requirements.
The audit protocol and results are now in Section 4.3 and Appendices B.2 and B.3 (highlighted).
4. (W2) Do differences in task difficulty, complexity, and testing criteria limit comparability across skills?
The benchmark does not compare absolute scores across skills. Each effect is the difference between two runs on the same requirement, verifier, repository, and agent configuration, so difficulty, complexity, and testing criteria are fixed within every comparison.
These differences could still matter for the aggregate, which pools skills with different tasks. If the effect depended on the kind of task, the average would reflect the task mix. A stratified analysis was run on all 3,920 pairs to check this. We split the skills into thirds by difficulty and the requirements into thirds by complexity, and computed the mean gain of each third in points.
| Stratified by | Low third | Middle third | High third |
|---|---|---|---|
| Skill difficulty (pass ratio without the skill) | +0.98 | +0.14 | -0.10 |
| Complexity (acceptance criteria per requirement) | +0.32 | +0.75 | -0.05 |
| Complexity (verifier checks per requirement) | +0.65 | +0.50 | -0.12 |
Every third lies within one point of zero. Across the four types of checks, the gain ranges from -0.54 to +0.17 points. The aggregate result therefore does not depend on the task mix. These analyses are now in Appendix E (Stratified Analyses, highlighted).
Response to Reviewer 5SkC (Part 3)
5. (W2) Do the conclusions hold when success requires passing every check?
Yes. Under complete task success, the effect of skills remains small.
A complete-success analysis was run on all 3,920 pairs to check whether the pass ratio hides a different picture. We counted a run as successful only if it passes every check of its verifier, and compared the success rate with and without the skill in each configuration with the exact McNemar test. Values are (without skill → with skill) followed by the difference in points. Lost / gained counts the pairs that change from success to failure and from failure to success with the skill.
| Configuration | Complete success rate | Lost / gained pairs | McNemar p |
|---|---|---|---|
| Codex + GPT-5.4-mini | (39.39% → 39.59%) +0.20 | 55 / 57 | 0.925 |
| Codex + Kimi-K2.6 | (39.69% → 42.04%) +2.35 | 56 / 79 | 0.058 |
| Claude Code + Kimi-K2.6 | (42.35% → 43.27%) +0.92 | 62 / 71 | 0.488 |
| Claude Code + Haiku 4.5 | (35.41% → 37.14%) +1.73 | 51 / 68 | 0.142 |
The change is not significant in any configuration, and the skill leaves the outcome unchanged in 3,421 of the 3,920 pairs (87%). The conclusion that the effect of skills is small therefore holds under both metrics. We keep the pass ratio as the primary metric because it also records partial progress. Complete success is now reported alongside it in Section 4.2 (Finding 1) and Appendix E (highlighted).
6. (W3) How were the skill-repository pairs selected and matched?
The 98 pairs were constructed in four stages.
- Sourcing. We collected candidate skills from public skill marketplaces in six software-engineering categories.
- Admission. A skill was admitted only if it met four criteria. It describes an executable and objectively testable operation. It targets a concrete tool or workflow. It has evidence of adoption, namely at least 100 deduplicated installs on skills.sh (93 skills, median 3,300) or a highly starred source repository (the other five, each above 45,000 GitHub stars). It is not a duplicate of an admitted skill.
- Repository matching. An LLM proposed candidate repositories and an author confirmed each match. A project-specific skill was paired with its own project. A general skill was paired with the repository of its target tool or with a widely used repository that uses the tool.
- Environment validation. Each repository was pinned to a fixed commit and checked for a feasible local environment.
The 98 skills map to 91 repositories with a median of 25.2K GitHub stars. The selection criteria and matching rule are now described in Appendix B.1 (highlighted).
7. (W3) Does using one repository per skill limit generality?
A skill is not meant to transfer across arbitrary repositories. It encodes a fixed procedure for a specific tool or project, so it applies only where that tool or project is used. For example, implementing-jsc-classes-zig describes how to create JavaScript classes with Bun's Zig bindings generator and has no use outside the Bun codebase, and viper-configuration applies only to Go projects that load their configuration with Viper.
We therefore paired each skill with the repository where it is meant to be used: its own project for a project-specific skill (Bun in the first example), and the repository of its target tool or a widely used project built on that tool for a general skill (Hugo in the second). The conclusions do not rest on one repository. They span 98 skills and 91 repositories, with ten requirements and four agent configurations per skill. Because the repository is an input to the pipeline, a user can also rerun the evaluation on any other repository in which a skill will be deployed.
Response to Reviewer 5SkC (Part 1)
We thank the reviewer for asking what makes a skill useful and how robust the benchmark findings are to task construction, evaluation metrics, repository choice, and stronger models. We added targeted analyses for each point and clarify the limits of the resulting claims below.
1. (W1) Which part of a skill drives its effect: procedural knowledge or concrete templates?
The two components play different roles. Where a helpful skill contains code examples, they are indispensable. In the harmful skills, the harm comes from the surrounding prose. Reference files add nothing.
A component ablation was run on six skills to separate the effect of procedural knowledge from that of concrete templates. The ablation covers the six diagnostic skills whose
SKILL.mdseparates into code examples and prose: Viper, FastAPI, and Testify (Golden), Asyncio and Grafana (Harmful), and GitHub Actions from the case studies. The split does not apply to the other two of the eight diagnostic skills (listed in item 3).junit5-testingstates its rules in prose without code blocks, andvitest-unit-testingkeeps most of its content in reference files. We split each skill into code examples (code blocks, sample configurations, and templates), prose (the remaining text ofSKILL.md, which carries the procedural guidance), and reference files (accompanying files, present in three of the six skills). We removed one component at a time and reran requirements R1-R5 under Codex + GPT-5.4-mini. Values are pass ratios in percent, with the change from the full skill in parentheses.viper-configurationfastapi-patternsgo-testify-patternsasyncio-patternsgrafana-dashboardsgithub-actions-templatesThe table separates two cases. For the three Golden skills, removing the code examples lowers the score in every case (by 2.38 to 14.66 points), and removing the prose lowers it in two. This matches Finding 2: a skill helps by supplying concrete library usage that the agent would otherwise omit. For
grafana-dashboards, removing the prose raises the score by 10.32 points, while removing the code changes it by only 0.32. This matches Finding 3: guidance that the task does not need interferes with the agent. Removing the reference files never lowers a score. The guideline for writing skills is therefore to give the concrete API usage that the task needs and to leave out the surrounding material. Finding 3 adds one condition: the examples should show how to call the API and should not fix paths or values that a task may override. This analysis is now in Section 4.2 (Findings 2 and 3) and Appendix C (highlighted).2. (W1) How does the gain vary with skill length, content quality, task difficulty, and task relevance?
None of these factors has a clear relationship with the gain.
A factor analysis was run to check what the gain depends on. For each factor, we sorted the skills (or the requirements, for task relevance) by its value, split them into thirds, and computed the mean gain of each third in points. We also computed the Spearman correlation between the factor and the gain. If a factor mattered, the gain would rise or fall steadily from the low third to the high third.
SKILL.md)SKILL.md)No row shows such a trend. Every third lies within about one point of zero, and no correlation in the table exceeds 0.18 in magnitude. For task difficulty, the low third contains the hardest skills, with pass ratios of 21% to 75% without the skill, and even these gain only 0.98 points. Longer skills and skills with more code do not gain more, and requirements that name more of the skill's identifiers do not gain more either.
Content quality has no objective measure that applies to all 98 skills. As an exploratory analysis, we scored each skill with an LLM judge, and the score has a weak association with the gain (rho = 0.26). These analyses are now in Appendices C and E (highlighted), with a summary in Section 4.2 (Finding 1).