acceptodds
Under review as a conference paper at ICLR 2027

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

Abstract

Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear, and no standard procedure tests an individual skill before installation. We present SWE-Skills-Bench, a skill-centric, requirement-driven benchmark that measures the extent to which individual agent skills improve performance on real-world software engineering (SWE) tasks. We start from 98 public SWE skills, each paired with a real GitHub repository pinned at a fixed commit. For each skill, our pipeline (i) reverse-engineers ten requirements specific to that repository, (ii) turns each requirement's acceptance criteria into deterministic, execution-based verifiers, and (iii) builds a reproducible container environment. Under each of four agent–model configurations, we run every task once without the skill and once with it, yielding 3,920 paired evaluations. For each requirement, we compute a paired delta, i.e., the change in outcome when the skill is added. We then aggregate these deltas across a skill's ten requirements into a per-skill utility profile, and use a statistical test to classify each skill as Golden (significantly helpful), Ineffective (no significant effect), or Harmful (significantly detrimental). Our results show that skills provide limited benefits: 92 of 98 skills have no statistically significant effect, and adding skills raises the average pass ratio only from to . The few helpful skills supply procedures that the agent would otherwise omit (gains of up to points). In contrast, harmful skills anchor the agent on their own templates and override the task's explicit instructions (losses of up to points). Moreover, a skill's utility depends on the agent harness and model: 40 of 98 skills flip the sign of their effect between two harnesses running the same model. These findings suggest that skill utility is an empirical, configuration-dependent property rather than an intrinsic one, and SWE-Skills-Bench provides a testbed for testing skills before deployment. Our code is available at https://anonymous.4open.science/r/SWE-Skills-Bench.

Primary area:
datasets and benchmarks
Keywords:
Agent Skills, Software Engineering, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5SkCOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SWE-Skills-Bench, a skill-centric benchmark for evaluating the effectiveness of agent skills in software engineering. It covers 98 skills paired with real GitHub repositories, generating 980 requirements and execution-based verifiers for evaluation across four agent–model configurations. Results from 3,920 paired comparisons show minimal overall improvement (76.59% to 76.93%), with only four skills significantly helpful and two harmful.

Critical strengths
  1. The benchmark provides a reasonably broad evaluation of SWE skills, covering 98 skills across six SWE-related categories.
  2. The empirical findings are insightful and challenge the assumption that more procedural guidance necessarily improves agent performance.
  3. The case studies provide interpretable examples of both beneficial and harmful skill usage.
Critical weaknesses
  1. The experimental analysis can be deeper, and the paper falls short of a systematic investigation into skill effectiveness and the mechanisms behind it. Results are drawn largely from aggregate performance numbers and a small number of case studies. More ablation studies would substantially strengthen the claims. For example, ablating different components of skill content can help tease apart the effects of procedural knowledge and concrete templates. The authors could also carry out further analysis to examine how performance gains vary with skill length, content quality, task difficulty, and task relevance.
  2. The benchmark construction and evaluation lack sufficient validation. The paper uses LLMs to generate tasks and tests based on skills without adequately verifying their reliability. Differences in task difficulty, complexity, and testing criteria also limit comparability across skills. Moreover, the primary metric is the average verifier pass ratio rather than the complete task success rate. Passing some tests does not necessarily indicate task completion, weakening the credibility of the experimental conclusions.
  3. The paper provides limited details on how skill–repository pairs are selected. Although the authors manually curate these pairs, they do not elaborate on their selection and matching rules. Another concern is that each skill is evaluated using only one repository, which limits the generalizability of their conclusions.
Questions for authors
  1. The evaluation is limited to relatively weak models. Could the authors evaluate more recent state-of-the-art models to examine whether the findings generalize to stronger agents?
  2. What distinguishes the four Golden Skills from the remaining skills? Why these skills are effective while others are not? The paper only provides a detailed case study of viper-configuration, without systematically analyzing the other Golden Skills or identifying their shared characteristics.
7 comments
AuthorsOpenReview ↗

Response to Reviewer 5SkC (Part 1)

We thank the reviewer for asking what makes a skill useful and how robust the benchmark findings are to task construction, evaluation metrics, repository choice, and stronger models. We added targeted analyses for each point and clarify the limits of the resulting claims below.

1. (W1) Which part of a skill drives its effect: procedural knowledge or concrete templates?

The two components play different roles. Where a helpful skill contains code examples, they are indispensable. In the harmful skills, the harm comes from the surrounding prose. Reference files add nothing.

A component ablation was run on six skills to separate the effect of procedural knowledge from that of concrete templates. The ablation covers the six diagnostic skills whose SKILL.md separates into code examples and prose: Viper, FastAPI, and Testify (Golden), Asyncio and Grafana (Harmful), and GitHub Actions from the case studies. The split does not apply to the other two of the eight diagnostic skills (listed in item 3). junit5-testing states its rules in prose without code blocks, and vitest-unit-testing keeps most of its content in reference files. We split each skill into code examples (code blocks, sample configurations, and templates), prose (the remaining text of SKILL.md, which carries the procedural guidance), and reference files (accompanying files, present in three of the six skills). We removed one component at a time and reran requirements R1-R5 under Codex + GPT-5.4-mini. Values are pass ratios in percent, with the change from the full skill in parentheses.

SkillNo skillFull skillWithout codeWithout proseWithout references
viper-configuration74.5880.0069.34 (-10.66)81.34 (+1.34)81.34 (+1.34)
fastapi-patterns13.2621.2618.88 (-2.38)17.54 (-3.72)23.70 (+2.44)
go-testify-patterns85.0085.0070.34 (-14.66)70.72 (-14.28)85.00 (0.00)
asyncio-patterns94.3892.9492.84 (-0.10)94.38 (+1.44)n/a
grafana-dashboards49.2848.9649.28 (+0.32)59.28 (+10.32)n/a
github-actions-templates89.0089.0089.00 (0.00)89.00 (0.00)n/a

The table separates two cases. For the three Golden skills, removing the code examples lowers the score in every case (by 2.38 to 14.66 points), and removing the prose lowers it in two. This matches Finding 2: a skill helps by supplying concrete library usage that the agent would otherwise omit. For grafana-dashboards, removing the prose raises the score by 10.32 points, while removing the code changes it by only 0.32. This matches Finding 3: guidance that the task does not need interferes with the agent. Removing the reference files never lowers a score. The guideline for writing skills is therefore to give the concrete API usage that the task needs and to leave out the surrounding material. Finding 3 adds one condition: the examples should show how to call the API and should not fix paths or values that a task may override. This analysis is now in Section 4.2 (Findings 2 and 3) and Appendix C (highlighted).

2. (W1) How does the gain vary with skill length, content quality, task difficulty, and task relevance?

None of these factors has a clear relationship with the gain.

A factor analysis was run to check what the gain depends on. For each factor, we sorted the skills (or the requirements, for task relevance) by its value, split them into thirds, and computed the mean gain of each third in points. We also computed the Spearman correlation between the factor and the gain. If a factor mattered, the gain would rise or fall steadily from the low third to the high third.

Factor (measure)Low thirdMiddle thirdHigh thirdSpearman rho
Task difficulty (pass ratio without the skill)+0.98+0.14-0.10-0.046
Skill length (words in SKILL.md)-0.30+0.67+0.640.18
Code proportion (share of code in SKILL.md)+0.53+1.05-0.50-0.06
Task relevance (share of the skill's distinctive identifiers named in the requirement)-0.06+1.18-0.09-0.015

No row shows such a trend. Every third lies within about one point of zero, and no correlation in the table exceeds 0.18 in magnitude. For task difficulty, the low third contains the hardest skills, with pass ratios of 21% to 75% without the skill, and even these gain only 0.98 points. Longer skills and skills with more code do not gain more, and requirements that name more of the skill's identifiers do not gain more either.

Content quality has no objective measure that applies to all 98 skills. As an exploratory analysis, we scored each skill with an LLM judge, and the score has a weak association with the gain (rho = 0.26). These analyses are now in Appendices C and E (highlighted), with a summary in Section 4.2 (Finding 1).

AuthorsOpenReview ↗

Response to Reviewer 5SkC(Part 2)

3. (W2) Are the LLM-generated requirements and verifiers reliable?

We agree that automatic generation needs direct validation. We therefore added a full-scale automatic audit, a human audit of a sampled subset, and an execution test. The table summarizes the results, and the details follow.

ValidationScopeResult
Automatic audit980 pairs, 16,924 checks95.7% of the 4,652 referenced file paths are named in the requirement. All 41 dependency checks match requested packages
Human audit100 pairs, two annotators90 and 95 pairs pass all five dimensions. Each dimension passes in at least 96 pairs
Execution test3,920 pairs and 479 requirement instances on unmodified repositoriesBoth implementations pass every check in 1,313 pairs. Unmodified repositories pass 11.7% of the checks, against 77.2% for the agent

An automatic audit was run on all 980 requirement-verifier pairs to check that no verifier relies on a file or dependency that its requirement never mentions. We extracted the file paths from the source of every check and looked them up in the requirement text. The 4.3% of paths that are not named are almost all paths that are not deliverables: test drivers that the verifier itself writes, test inputs, build outputs, existing repository files, and alternative locations that a check also accepts.

A human audit was then run on 100 pairs to check what the automatic audit cannot. The sample covers requirements R1-R5 of 20 skills: eight diagnostic skills (the four Golden skills, the two Harmful skills, and the two case-study skills vitest-unit-testing and github-actions-templates) and 12 randomly selected skills. Two annotators, each with three years of software-engineering experience, read the skill, the requirement, and the verifier source of every pair and labeled five dimensions independently: requirement clarity, leakage of the skill procedure, coverage of the stated criteria, undeclared constraints, and acceptance of reasonable alternative implementations.

An execution test was then run to check whether the verifiers accept different correct implementations and reject incomplete ones. The main experiment contains two independent implementations of every requirement, one without and one with the skill, and both pass every check in 1,313 of the 3,920 pairs. We also ran the verifiers on unmodified repositories, in which nothing is implemented, for requirements R1-R5 of all 98 skills. They pass only 11.7% of the checks on average, against 77.2% for the agent without the skill on the same requirements.

The audit protocol and results are now in Section 4.3 and Appendices B.2 and B.3 (highlighted).

4. (W2) Do differences in task difficulty, complexity, and testing criteria limit comparability across skills?

The benchmark does not compare absolute scores across skills. Each effect is the difference between two runs on the same requirement, verifier, repository, and agent configuration, so difficulty, complexity, and testing criteria are fixed within every comparison.

These differences could still matter for the aggregate, which pools skills with different tasks. If the effect depended on the kind of task, the average would reflect the task mix. A stratified analysis was run on all 3,920 pairs to check this. We split the skills into thirds by difficulty and the requirements into thirds by complexity, and computed the mean gain of each third in points.

Stratified byLow thirdMiddle thirdHigh third
Skill difficulty (pass ratio without the skill)+0.98+0.14-0.10
Complexity (acceptance criteria per requirement)+0.32+0.75-0.05
Complexity (verifier checks per requirement)+0.65+0.50-0.12

Every third lies within one point of zero. Across the four types of checks, the gain ranges from -0.54 to +0.17 points. The aggregate result therefore does not depend on the task mix. These analyses are now in Appendix E (Stratified Analyses, highlighted).

AuthorsOpenReview ↗

Response to Reviewer 5SkC (Part 3)

5. (W2) Do the conclusions hold when success requires passing every check?

Yes. Under complete task success, the effect of skills remains small.

A complete-success analysis was run on all 3,920 pairs to check whether the pass ratio hides a different picture. We counted a run as successful only if it passes every check of its verifier, and compared the success rate with and without the skill in each configuration with the exact McNemar test. Values are (without skill → with skill) followed by the difference in points. Lost / gained counts the pairs that change from success to failure and from failure to success with the skill.

ConfigurationComplete success rateLost / gained pairsMcNemar p
Codex + GPT-5.4-mini(39.39% → 39.59%) +0.2055 / 570.925
Codex + Kimi-K2.6(39.69% → 42.04%) +2.3556 / 790.058
Claude Code + Kimi-K2.6(42.35% → 43.27%) +0.9262 / 710.488
Claude Code + Haiku 4.5(35.41% → 37.14%) +1.7351 / 680.142

The change is not significant in any configuration, and the skill leaves the outcome unchanged in 3,421 of the 3,920 pairs (87%). The conclusion that the effect of skills is small therefore holds under both metrics. We keep the pass ratio as the primary metric because it also records partial progress. Complete success is now reported alongside it in Section 4.2 (Finding 1) and Appendix E (highlighted).

6. (W3) How were the skill-repository pairs selected and matched?

The 98 pairs were constructed in four stages.

  1. Sourcing. We collected candidate skills from public skill marketplaces in six software-engineering categories.
  2. Admission. A skill was admitted only if it met four criteria. It describes an executable and objectively testable operation. It targets a concrete tool or workflow. It has evidence of adoption, namely at least 100 deduplicated installs on skills.sh (93 skills, median 3,300) or a highly starred source repository (the other five, each above 45,000 GitHub stars). It is not a duplicate of an admitted skill.
  3. Repository matching. An LLM proposed candidate repositories and an author confirmed each match. A project-specific skill was paired with its own project. A general skill was paired with the repository of its target tool or with a widely used repository that uses the tool.
  4. Environment validation. Each repository was pinned to a fixed commit and checked for a feasible local environment.

The 98 skills map to 91 repositories with a median of 25.2K GitHub stars. The selection criteria and matching rule are now described in Appendix B.1 (highlighted).

7. (W3) Does using one repository per skill limit generality?

A skill is not meant to transfer across arbitrary repositories. It encodes a fixed procedure for a specific tool or project, so it applies only where that tool or project is used. For example, implementing-jsc-classes-zig describes how to create JavaScript classes with Bun's Zig bindings generator and has no use outside the Bun codebase, and viper-configuration applies only to Go projects that load their configuration with Viper.

We therefore paired each skill with the repository where it is meant to be used: its own project for a project-specific skill (Bun in the first example), and the repository of its target tool or a widely used project built on that tool for a general skill (Hugo in the second). The conclusions do not rest on one repository. They span 98 skills and 91 repositories, with ten requirements and four agent configurations per skill. Because the repository is an input to the pipeline, a user can also rerun the evaluation on any other repository in which a skill will be deployed.

Reviewer_r1mQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a skill-centric benchmark for measuring the marginal benefit of procedural skill documents in software-engineering agents. It pairs 98 public skills with fixed-commit repositories, synthesizes ten requirements per skill and executable verifiers, and evaluates four agent–model configurations with and without each skill. Across 3,920 paired evaluations, the mean verifier pass ratio changes from 76.59% to 76.93%. An exact paired sign-flip test identifies four overall helpful and two harmful skills; the other 92 have no detected directional effect. Trajectory examples illustrate useful procedural guidance and interference from conflicting templates, while cross-configuration comparisons suggest that observed utility varies with the agent setup.

Critical strengths

1.A useful intervention-centered evaluation design. Holding the repository, requirement, verifier, and agent configuration fixed while changing skill access is a sensible way to study marginal utility. The 980 requirements across 98 skills and four configurations make this a substantial empirical resource. This contribution matters because successful task completion alone does not establish that an installed skill was beneficial. 2.Concrete diagnostics beyond aggregate scores. The Viper, GitHub Actions, and Vitest examples connect observed changes to inspectable agent actions: library use, misplaced deliverables, and dependency installation (Figures 4–6). These cases make the benchmark potentially useful for improving skill content and agent workflows, rather than merely ranking skills by an average score.

Critical weaknesses

1.The validity of the generated evaluation targets is insufficiently established. Both requirements and tests are generated automatically, with requirements derived from the skill being evaluated. Section 5 explicitly leaves human verifier audits to future work. Prompts requesting behavioral checks do not establish that tests accept valid alternative solutions, reject incorrect ones, or encode only declared requirements. This matters because pass-ratio differences may reflect implementation preferences or test artifacts. For example, the Viper case rewards adoption of a particular library; it needs to be clear whether that dependency is an explicit requirement and whether any otherwise valid implementation would be accepted. Skill-derived tasks and one repository per skill also limit what the results establish about independently occurring engineering requests. The missing verifier validation is the more immediate issue: it affects the interpretation of the measured intervention itself.

2.The skill classifications and configuration-dependence claims need stronger statistical support. Each experimental cell has one execution per condition, and each skill is tested on only ten requirements. Section 3.4 uses p < 0.05, but the paper does not report a multiple-testing adjustment across the 98 overall assessments and 392 configuration-specific assessments. Under a complete null, 98 valid tests at this threshold have an upper bound of 4.9 on the expected number of false positives; this does not establish that the six discoveries are false, but makes their unadjusted count insufficient evidence of reliable categories. The alternative tests in Table 4 reuse the same runs and do not resolve multiplicity or run-to-run stability. Likewise, 40 observed sign reversals and weak correlations can partly reflect noisy estimates near zero. I appreciate the explicit qualifications that “Ineffective” means non-detection and Figure 7 does not establish significant interactions. Nevertheless, the evidence is not yet strong enough to support dependable skill classification or stable harness-specific effects, which are central promised uses of the benchmark.

Questions for authors

1.Please report the exact p-values and effect uncertainty for the overall and configuration-specific classifications, together with a stated multiple-testing treatment. Which helpful/harmful conclusions remain supported? Can repeated runs on a small, explicitly selected diagnostic subset distinguish stable effects and harness differences from execution noise? This would directly affect my assessment of Weakness 1.

2.What checks, beyond the generation prompts, have been performed on requirement–test consistency, known-correct solutions, behaviorally equivalent alternatives, and deliberately incorrect implementations? Please report the audited sample and outcomes. For the Viper example, is use of spf13/viper explicitly required in the task given to both conditions, or is it an additional verifier constraint? This would determine whether the reported gains measure requirement satisfaction as intended.

6 comments
AuthorsOpenReview ↗

Response to Reviewer r1mQ (Part 1)

We thank the reviewer for focusing on verifier validity, statistical uncertainty, and the scope of the conclusions. These concerns led us to add a human audit, alternative-solution tests, corrected significance analyses, and repeated runs.

1. (W1, Q2) What checks were performed on requirement–test consistency, known-correct solutions, equivalent alternatives, and incorrect implementations?

Beyond the generation prompts, we tested each of the four properties directly. The table maps each property to the check we ran, and the details follow.

PropertyCheckResult
Requirement–test consistencyAutomatic audit of all 980 pairs (16,924 checks)95.7% of the 4,652 referenced file paths are named in the requirement. All 41 dependency checks match requested packages
Requirement–test consistencyHuman audit of 100 pairs by two annotators90 and 95 pairs pass all five dimensions. Neither annotator found an undeclared constraint
Known-correct solutionsReference solutions written independently by ourselves, on a subset of the human-audit sample (20 implementations of 10 requirements)All 20 pass every check
Equivalent alternativesTwo independent implementations of every requirement (3,920 pairs)Both pass every check in 1,313 pairs
Incorrect implementationsUnmodified repositories (479 requirement instances)11.7% of the checks pass, against 77.2% for the agent

The automatic audit extracts the file paths from the source of every check and looks them up in the requirement text. The 4.3% of paths that are not named are almost all paths that are not deliverables, such as test inputs, build outputs, and existing repository files. In the human audit, two annotators read the skill, the requirement, and the verifier source of 100 pairs and labeled five dimensions independently. The sample covers requirements R1-R5 of 20 skills: the eight diagnostic skills of item 5 and 12 randomly selected skills. Each found that 96 of the 100 pairs accept reasonable equivalent implementations.

The reference solutions cover the first requirement of the 20 audited skills, restricted to the 10 that at least one run of the main experiment passes in full, so that a full pass is attainable. We wrote two implementations of each. The unmodified repositories cover requirements R1-R5 of all 98 skills, and the 77.2% is the pass ratio of the agent without the skill on the same requirements. The protocol and results are now in Section 4.3 and Appendices B.2 and B.3 (highlighted).

2. (W1, Q2) Is spf13/viper an explicit requirement, and would another valid implementation be accepted?

The task explicitly requires spf13/viper, and both conditions receive exactly the same requirement. The dependency check therefore enforces a stated constraint and not a hidden preference. Most of the gain comes from functional behavior and not from this check.

An ablation was run on the ten Viper requirements to check how much of the gain comes from the dependency checks. Each verifier contains one dependency check and several behavioral checks for source precedence, environment-variable mapping, nested decoding, optional configuration files, and isolation between calls. We removed the ten dependency checks and recomputed the paired effect on the remaining 138 functional checks.

Checks scoredNumber of checksPass ratio (without → with skill)Exact sign-flip p
All checks148(59.61% → 77.30%) +17.690.0039
Dependency checks removed138(60.30% → 75.65%) +15.340.0078

A second check was run to test whether an implementation without Viper is accepted by the functional checks. In the main experiment, three runs without the skill implement their own configuration loader without Viper, and all three pass every functional check. They lose only the dependency check, which enforces the library that the task asks for. The functional checks therefore test behavior and not the library.

The ablation is now in Appendix B.5 (Check Types and the Viper Dependency Ablation, highlighted).

AuthorsOpenReview ↗

Response to Reviewer r1mQ (Part 2)

3. (W1) Do skill-derived tasks and one repository per skill limit what the results establish?

The results establish whether a skill helps on tasks within its own declared scope, in a repository where it applies. This is the question a user faces before installing a skill. They are not an estimate over an independent distribution of naturally occurring requests, and Section 5 states this scope.

Skill-derived tasks make the setting favorable to the skill, because every requirement is relevant to it by construction. Even in this setting, the aggregate gain is only +0.34 points (95% CI [-0.43, +1.12]). The near-zero result is therefore conservative for requests that are less closely related to the skill.

The aggregate result does not rest on one repository. It spans 98 skills and 91 repositories, with ten requirements and four agent configurations per skill. The result for an individual skill remains specific to its paired repository. Because the repository is an input to the pipeline, a user can rerun the evaluation on the repository in which the skill will be deployed. Related discussions are added to the revised Section 5.

4. (W2, Q1) What are the exact p-values and effect uncertainty under a multiple-testing treatment, and which helpful/harmful conclusions remain supported?

None of the per-skill labels remains significant after correction. The aggregate conclusion that the average effect is small is unaffected.

A multiple-testing analysis was run on all per-skill assessments to check which labels survive correction. We apply the exact two-sided paired sign-flip test to the ten requirement-level deltas of each skill. For the overall tests, each requirement delta is first averaged across the four configurations. We treat the 98 overall tests and the 392 configuration-specific tests as two separate families and apply Benjamini-Hochberg correction within each. The 95% CI of each effect is a t interval over the same ten deltas. Six of the 98 skills have unadjusted p < 0.05, and none remains significant after correction. The table lists these six, which are the four Golden and the two Harmful skills.

SkillEffect95% CIExact pBH q
viper-configuration+17.69[5.6, 29.8]0.00390.383
junit5-testing+3.90[1.7, 6.1]0.00780.383
grafana-dashboards-2.96[-5.0, -0.9]0.01560.510
go-testify-patterns+4.88[-1.2, 11.0]0.03130.613
asyncio-patterns-16.96[-31.0, -2.9]0.03130.613
fastapi-patterns+3.23[0.7, 5.8]0.04690.681

The same holds for the 392 configuration-specific tests. Six have unadjusted p < 0.05, one has p < 0.01, and none survives correction. The "Golden" and "Harmful" labels therefore mark candidate skills identified by the unadjusted criterion, and they are not statistically confirmed categories. The revised paper states where the labels are defined that they use the unadjusted criterion (Section 3.4, highlighted).

The aggregate conclusion is a single estimate and not one of 98 per-skill discoveries. Across the 98 skills, the mean effect is +0.34 points with a 95% CI of [-0.43, +1.12]. The exact p-values and q-values of all 98 overall tests are now listed next to the full results in Figures 9 and 10, and the confidence intervals and the configuration-specific results are in Appendix E (highlighted), with a summary in Section 4.2 (Finding 1).

AuthorsOpenReview ↗

Response to Reviewer r1mQ (Part 3)

5. (W2, Q1) Do repeated runs distinguish stable effects and harness differences from execution noise?

Yes. Four positive effects recur in every rerun, three near-zero effects stay near zero, and the large loss of asyncio-patterns does not recur because it came from runs in which the repository package failed to import. A large configuration-specific effect is also stable, while small differences between harnesses stay within two points of zero.

A repeated-run experiment was conducted on an explicitly selected diagnostic subset to check which effects are stable. The subset contains the six candidate skills of the previous answer and two case-study skills (vitest-unit-testing and github-actions-templates), with requirements R1-R5 of each. We reran both conditions twice under Codex + GPT-5.4-mini. Gains are in points, and the last column gives the mean ± standard deviation over the three runs.

SkillOriginal runRerun 1Rerun 2Mean ± std
viper-configuration+6.68+5.42+4.10+5.40 ± 1.29
vitest-unit-testing+56.40+41.74+56.40+51.51 ± 8.46
junit5-testing+5.30+1.82+1.68+2.93 ± 2.05
fastapi-patterns+9.60+1.34+8.00+6.31 ± 4.38
go-testify-patterns0.000.000.000.00 ± 0.00
grafana-dashboards+0.72-0.320.00+0.13 ± 0.53
github-actions-templates0.000.000.000.00 ± 0.00
asyncio-patterns-31.10-1.440.00-10.85 ± 17.55

The first four skills are positive in all three runs. The next three stay within one point of zero. The loss of asyncio-patterns disappears. We traced the original loss to whole-run environment failures. Across the four configurations, the repository package could not be imported in nine of the 36 runs with the skill and in one of the 36 runs without it. These failures did not recur in the reruns.

A second experiment was run to check harness differences. We evaluated the same subset three times under both Codex and Claude Code with one model, DeepSeek-V4.1-Flash-1017. The mean gain is -0.07 points under Codex and +0.20 under Claude Code, and 41 of the 48 skill-harness-repeat cells are within two points of zero. viper-configuration is positive in all six runs under both harnesses.

Repeated runs therefore separate the two cases. The gain of vitest-unit-testing under Codex + GPT-5.4-mini recurs in both reruns (+41.74 and +56.40) and is absent with the stronger model, which already passes 98.7-100% of the checks without the skill. This is a configuration-specific effect and not noise. Differences of a few points between harnesses are within the variation between runs. These results are now in Section 4.2 (Finding 4) and Appendix D (Repeated Runs on the Diagnostic Subset and Stronger-Model Evaluation, highlighted).

Reviewer_dceBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes an automatic framework for turning a public software engineering "skill" into an assessment of its counter-factual utility and applies this to 98 public skills to form SWE-skills bench. The automatic pipeline works by using models to map skills to requirements, and then construct synthetic "RL environments" with verifiers based on these requirements. These tasks are then executed with and without the skill. The paper empirically finds that the measured utility of skills is model and harness-dependent, but a large number of skills do not make a meaningful difference to model performance, or in some cases even deteriorate performance.

Critical strengths

In my view, the primary strength of the work is the analysis of the utility of existing skills, including the findings that skills are sensitive to model and harness, and that some skill usage can be detrimental.

The other major contribution is the automatic task creation pipeline. While this pipeline is reasonably sophisticated with multiple stages (deriving requirement, docker environment construction, verifiers, paired evaluation setting), I have some major reservation about its validity, and I am skeptical of automatic pipelines having a high enough quality assurance to make good evaluation benchmarks.

Critical weaknesses

There are three points of weaknesses:

A. I am confused about the benchmark framing as SWE-skill-bench. Is SWE-skill-bench the set of 98 curated instances? Is it a pipeline to measure the utility of any skill? Is it an evaluation benchmark for how well models use skills? It seems to be framed as a skill-centric benchmark, but does that mean we should optimize our skills for this benchmark? In my view, the awkward framing present a major problem for the paper and what follow up work would look like.

B. There is almost no evidence in the paper beyond providing prompts and a few examples that the automatic evaluation pipeline is sound. I'm not even able to find details about which models are used to extract requirements and build the evaluation tasks. There are various quality issues I am worried about, for instance, what happens when the evaluation prompt simply restates all the important information in the skill, at which point the skill becomes redundant. Verifiers can also be brittle and prefer specific . Given the large amount of quality issues that have been surfaced even in popular public benchmarks like terminal bench 4, I would be surprised if this automatic pipeline does not have similar issues. The limitations admit to the lack of human review, but since the automatic evaluation is one of the core contributions of the paper, I would look for plenty of evidence that it produces high-quality assessments.

C. A major motivation for skill writing is not necessarily task performance, but also efficiency in terms of tokens, latency, and cost. I'm surprised that this aspect is completely ignored by the paper.

Questions for authors

Section 3.4 states that "we initialize two copies of the same container state", and "we invoke skill s explicitly at the start of the prompt". However, if the skill is present as a file on disk, most models have automatic skill discovery tendencies, and harnesses may show brief summaries of each skill to the model. Is this possibility controlled for in the paired evaluation setting? Are you confident that the skills are not used in the "no-skill" setting.

6 comments
AuthorsOpenReview ↗

Response to Reviewer dceB (Part1)

We thank the reviewer for identifying several places where the scope and validation of SWE-Skills-Bench were unclear. We address each point below and revise the paper accordingly.

Q1. What is SWE-Skills-Bench: the 98 instances, a pipeline, or a benchmark? Should skills be optimized for it?

SWE-Skills-Bench is primarily an evaluation pipeline for skills. Given a skill and corresponding repository, it generates in-scope requirements and verifiers, constructs an executable environment, and compares the same agent with and without the skill. The 98 skills are a reference instantiation of this pipeline, comprising 980 requirements, 980 verifiers, and 3,920 paired comparisons across four agent configurations.

The object being evaluated is the skill, not the model. The 98 instances can also serve as a fixed set for comparing how different agent configurations use skills, but this is a secondary use. Skills should therefore not be optimized for these 98 instances. Skill authors should run the pipeline on the skill and repositories in which it will be deployed. We make this distinction explicit in the revised paper. Please see the highlighted text in Section 3.1 (Overview and Problem Formulation). The same framing appears in the contribution list of Section 1 ("Automated skill-centric evaluation") and in Section 5 (Discussion).

Q2. How sound is the automatic evaluation pipeline?

We validated the pipeline directly in three ways: a full-scale automatic audit, a human audit of a sampled subset, and an execution test. All three support its soundness. The table summarizes the results. Details follow.

ValidationScopeResult
Automatic audit980 pairs, 16,924 checks95.7% of the 4,652 referenced file paths are named in the requirement. All 41 dependency checks match requested packages
Human audit100 pairs, two annotators90 and 95 pairs pass all five dimensions. Each dimension passes in at least 96 pairs
Execution test3,920 pairs and 479 requirement instances on unmodified repositoriesBoth implementations pass every check in 1,313 pairs. Unmodified repositories pass 11.7% of the checks, against 77.2% for the agent

An automatic audit was run on all 980 requirement–verifier pairs to check whether each verifier tests only what its requirement states, i.e., that it does not rely on files or dependencies that the requirement never mentions. Each verifier is a pytest file, and each test function in it is one check (16,924 total). We parsed the syntax tree of every check, extracted the file paths in its string constants, and looked them up in the requirement text. We also selected the checks that read a dependency file and look for a particular package. Every requirement includes explicit acceptance criteria, 95.7% of the 4,652 file paths referenced by verifier checks are also named in the requirement, and all 41 dependency-specific checks enforce packages explicitly requested by the corresponding requirement. The remaining 4.3% are almost all paths that are not deliverables: test drivers that the verifier itself writes, test inputs, build outputs, existing repository files, and alternative locations that a check also accepts (for example, ci.yaml in addition to the ci.yml named in the requirement).

A human audit was then run on a sample of 100 pairs to check what the automatic audit cannot: whether the requirements are clear and solvable, and whether the checks are semantically aligned with them. The sample covers requirements R1-R5 for 20 skills: 8 diagnostic skills (four Golden, two Harmful, and two case-study skills vitest-unit-testing, github-actions-templates) and 12 drawn randomly. Two annotators, each with three years of SWE experience, read the skill, requirement, and verifier source of every pair and labeled it independently as pass or fail on five dimensions: requirement clarity, leakage of the skill procedure, coverage of the stated criteria, undeclared constraints, and acceptance of reasonable alternative implementations. They judged 90/100 and 95/100 pairs to pass all five dimensions. Every individual dimension passed in at least 96/100 pairs.

We then ran an execution test to check what neither audit can show directly: whether the verifiers accept different correct implementations and reject incomplete ones. The main experiment contains two independent implementations of every requirement (without and with the skill), and in 1,313 of the 3,920 pairs both pass every check. We also ran the verifiers on the unmodified repositories, in which nothing is implemented, for requirements R1-R5 of all 98 skills (479 instances). They pass only 11.7% of the checks on average, compared with 77.2% for the agent without the skill on the same requirements.

The audit protocol and results are now in Section 4.3 (Benchmark Quality) and Appendices B.2 and B.3 (highlighted), and the statement in Section 5 that human validation is future work has been removed.

AuthorsOpenReview ↗

Response to Reviewer dceB (Part 2)

Q3. Which models are used to construct and evaluate the tasks?

Claude Opus 4.6, accessed through Claude Code CLI v2.1.32, generates both the requirements and the verifiers. The evaluation configurations are:

StageHarnessModel
Requirement generationClaude Code CLI v2.1.32Claude Opus 4.6
Verifier generationClaude Code CLI v2.1.32Claude Opus 4.6
Evaluation 1Codex CLI 0.144.1GPT-5.4-mini
Evaluation 2Codex CLI 0.144.1Kimi-K2.6
Evaluation 3Claude Code 2.1.168Kimi-K2.6
Evaluation 4Claude Code 2.1.168Claude Haiku 4.5

The exact models, harnesses, and versions are now reported in the revised paper. Please see Appendix B.1 and its table "Construction and evaluation configurations" (highlighted).

Q4. Do the requirements restate the skill and thereby make it redundant?

No. By design, the skill defines which tasks are relevant, and each task's acceptance criteria define what counts as success. We generate repository-specific requirements within the skill's intended scope and instruct the generator to omit the skill's procedural guidance (see the requirement-generation prompt in Appendix F).

An overlap analysis was run on all 980 requirements to check whether a requirement repeats the content of its skill. For each requirement, we computed the share of its five-word sequences that also occur in its skill, and the same share against the skill's code blocks and templates only. On average, 0.27% of a requirement's five-word sequences occur in its skill (median 0.00%), and 737 of the 980 requirements share nothing with the skill's code or templates. We also computed the share of the skill's distinctive APIs, commands, and configuration keys that the requirement names. This identifier overlap is higher (5.65% on average) because a requirement must often name the tool or interface it concerns.

The remaining overlap is not associated with the measured skill effect. We sorted the requirements by identifier overlap and split them into thirds. The mean effects are -0.06, +1.18, and -0.09 points, and the Spearman correlation between overlap and the effect is -0.015. This analysis is now in Appendix B.4 (Requirement–Skill Overlap and Skill Isolation), with a summary in Section 4.3.

Q5. Are the verifiers brittle or tied to a particular implementation?

No. By design, the verifier generator receives only the requirement and repository metadata, and its instructions require it to derive each executable check from a stated acceptance criterion. It sees neither the skill nor a reference solution, so it cannot favor the implementation that the skill suggests. Three results confirm this.

First, the human audit of Q2 judged implementation flexibility directly. Each annotator found that 96 of the 100 audited pairs accept reasonable equivalent implementations. Second, the main experiment contains two independently written implementations of every requirement, one without and one with the skill, and both pass every check in 1,313 of the 3,920 pairs. Third, checks that demand a specific library are rare. We selected the checks that read a dependency file and look for a particular package. There are 41 of them among 16,924 checks, and all correspond to packages that the task explicitly requests.

The skill effect is also stable across kinds of checks. We classified every check by its source code and recomputed the effect for the four main types on the 3,136 pairs that keep check-level logs. The effect is 0.00 points for checks that run the agent's code, -0.54 for checks that parse a produced artifact, +0.17 for checks that inspect file content, and +0.10 for checks that a required file exists. The near-zero aggregate effect is therefore not driven by one kind of check. These results are now in Appendices B.2, B.3, and B.5 (highlighted).

AuthorsOpenReview ↗

Response to Reviewer dceB (Part 3)

Q6. Do skills improve efficiency?

Not uniformly. Their effect depends on the harness: skills increase tokens, steps, and estimated cost under Codex but reduce them under Claude Code.

An efficiency analysis was run on the paired runs of the main experiment to check whether skills save work. For each configuration, we compared the runs with and without the skill on total tokens (input and output, including cached input), wall-clock time, tool calls, and model turns. We also estimated the cost of a run from its token counts at public API list prices, with cached input billed at the cached rate. All values in the table are means per requirement, shown as (without skill → with skill) followed by the relative change.

MetricCodex(GPT-5.4-mini)Codex(Kimi-K2.6)Claude Code(Kimi-K2.6)Claude Code(Haiku 4.5)
Total tokens (M)(1.81 → 1.96) +8.1%(1.32 → 1.43) +8.2%(2.48 → 2.33) -5.9%(2.31 → 2.01) -13.1%
Time (min)(6.27 → 6.76) +7.7%(10.90→10.84) -0.6%(9.22 → 9.07) -1.7%(4.00 → 3.63) -9.1%
Tool calls(41.5 → 43.1) +3.8%(48.4 → 50.9) +5.2%(43.0 → 40.8) -5.1%(38.9 → 33.4) -14.2%
Turns(26.4 → 28.2) +6.5%(40.9 → 42.3) +3.4%(31.7 → 30.5) -4.0%(39.1 → 33.3) -14.8%
Estimated cost (USD)(0.28 → 0.30) +5.5%(0.35 → 0.36) +3.2%(0.53 → 0.49) -7.7%(0.37 → 0.34) -8.8%

The prices in USD per million tokens are 0.75 input, 0.075 cached input, and 4.50 output for GPT-5.4-mini, 0.95, 0.16, and 4.00 for Kimi-K2.6, and 1.00 input, 1.25 cache write, 0.10 cache read, and 5.00 output for Haiku 4.5.

The tool calls explain the difference between the harnesses. Under Claude Code + Haiku 4.5, calls that explore the repository fall from 15.4 to 12.2 per run, so the skill replaces exploration. Under Codex + GPT-5.4-mini, exploration stays at about 25 calls, while running, testing, and installing rise from 9.8 to 11.2. Efficiency must therefore be read together with task success: a skill may save work, or it may add useful work that improves the result. This analysis is now in Section 4.2 (Finding 4) and Appendix D.1 (highlighted).

Q7. Is the no-skill condition genuinely free of the target skill?

Yes. In the no-skill condition, no skill file exists in the environment, so the harness has nothing to discover or summarize. The initial containers contain no skills. After cloning a repository, we remove any skill directories that it ships with. The target skill is installed only in the with-skill condition.

A trajectory audit was run on all 3,919 available no-skill trajectories to check whether the skill still reaches the agent. We searched each trajectory for the name of its target skill and checked whether the agent loaded or read that skill. We found no such instance. These controls and the audit are described in Section 3.3, and now also in Section 4.3 and Appendix B.4 (highlighted).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.