acceptodds
Under review as a conference paper at ICLR 2027

SkillPrism: Taxonomy-Guided Evaluation and Cross-Skill Optimization of Agent Skills

Abstract

Agent skills extend the domain-specific capabilities of LLM agents. As the open skill ecosystem grows in scale and heterogeneity, evaluating skill utility and understanding how it varies across skill characteristics remain challenging. Existing benchmarks typically focus on selected domains or task sets, limiting their coverage of the broader skill ecosystem. We introduce SkillPrism-Bench, a systematic benchmark grounded in a multidimensional taxonomy that characterizes skills by artifact form, verification support, execution dependency, and control flow. The taxonomy guides skill sampling and task construction, enabling structured analysis of skill utility. Experiments across multiple models show that providing skills improves the average pass rate by 13.9 percentage points relative to the no-skill baseline, with distinct utility patterns across skill categories. Building on this taxonomy, we further introduce SkillPrism-Opt, which distills transferable editing strategies from source-skill trajectories and uses the target skill's taxonomy attributes to guide a one-step rewrite, without access to its test tasks or execution feedback. In multi-model evaluation on held-out skills, SkillPrism-Opt improves average pass@1 by 3.3 percentage points over original skills and by 5.2 percentage points over general rewriting. Together, our results provide a taxonomy and empirical foundation for systematic skill evaluation and demonstrate that cross-skill experience transfer can improve unseen skills without target-specific execution feedback.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent skills, skill evaluation, skill taxonomy, cross-skill optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fxcQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SkillPrism-Bench, a benchmark of 75 public agent skills and 300 capability-grounded tasks organized by artifact form, verification support, execution dependency, and control flow. It evaluates paired skill availability across six models. The paper also proposes SkillPrism-Opt, which consolidates measured source-skill edits into reusable editing strategies and rewrites held-out skills without accessing their test tasks or execution feedback.

Critical strengths

The benchmark offers a useful structural perspective beyond domain-level aggregation. The explicit F/V/E/C definitions, selected-skill roster, and capability-grounded construction procedure make the intended coverage inspectable, and the combination of rule-based and rubric-based tasks accommodates different kinds of skill outputs.

Critical weaknesses

W1. The evaluation is self-referential and does not resolve the limitation the paper attributes to prior work. The introduction criticizes existing benchmarks for organizing coverage around declared capabilities (lines 51–53). Yet SkillPrism-Bench builds each task from the evaluated skill's own declared capabilities, and the references and rubrics are derived from these tasks. The skill under evaluation therefore defines the standard it is graded against. A positive gain shows that the skill can get the agent to do what the skill itself claims, not that the skill is useful or well designed. This undermines the category-level design recommendations and the claim that rewritten skills are better. The existing safeguards do not address this:

  • Generating references and rubrics without access to the skill prevents direct answer leakage. However, the task requirements already come from the skill.
  • Removing claims that are answerable without the skill keeps exactly the skill-specific content. It also drops the cases where the skill adds nothing.
  • The taxonomy changes which skills are sampled. It does not change the fact that each skill is graded against its own claims.

W2. There is no human annotation or validation. The V/E/C labels, tasks, references, and rubrics are all LLM-generated, and many tasks are graded by an LLM judge. The paper reports no inter-annotator, LLM–human, or human–judge agreement, and no expert review of tasks. For a benchmark paper, and for a gain of only few points, this is a substantial gap.

Questions for authors
  1. What exactly does the benchmark measure: the skill's quality, the model's ability to use skills, or whether a skill can transmit its own claims? Which decision is it meant to inform, for example model selection, skill screening, or skill authoring?
  2. Could you audit a sample of claims and rubric items for correctness independently of the skill? What fraction of rubric items encode conventions that are not stated in the task?
  3. Do the main category differences and the optimization gain persist on tasks that were not built from the skills, such as taxonomy-blind or externally defined tasks?
  4. Which models were used in the paired pilots? How many tasks in each category were adjusted, and in which direction? Do the results hold on tasks that were never adjusted?
  5. What are the inter-annotator and LLM–human agreement scores for the V/E/C labels on a sampled subset?
  6. What is the human–judge agreement on rubric-graded tasks, and how sensitive are the results to the 0.8 threshold? Does the optimization gain hold on the rule-based subset alone?
4 comments
AuthorsOpenReview ↗

Response to fxcQ (Part 1): Evaluation goals, scoring criteria, and tasks grounded in real materials

Thank you for recognizing the taxonomy’s structural perspective and the transparency of our benchmark construction. We clarify the evaluation goal, explain claim filtering, and report audit and real-material results below.

Q1. What the benchmark measures, diagnoses, and helps optimize

We measure how much providing a skill improves task performance for a specified execution model and agent harness within the skill’s intended uses. Paired evaluations with and without the skill reveal gains, repaired failures, and newly introduced failures. The taxonomy helps diagnose how these outcomes vary across skill structures and models, identifying where skills provide useful guidance and where execution needs improvement. We demonstrate this across six source models with OpenHands held fixed.

On source skills, we inspect execution failures, revise instructions, and test whether the revisions help. SkillPrism-Opt consolidates this experience into editing strategies that can be reused on unseen skills without collecting their task execution feedback. As the benchmark and experience library grow, the aim is to reduce the repeated rollout and revision needed to adapt each new skill to its execution setting.

W1 & Q2. Task grounding, claim filtering, and independent audit

Claim filtering focuses evaluation on capabilities where skill guidance could contribute. Tasks unrelated to a skill’s capabilities, or resolved by general knowledge alone, offer little insight into how models use its guidance. We therefore exclude claims answerable from the request alone, bundled-script outputs, or general model knowledge. Retained claims identify potential contributions; paired execution measures whether providing the skill yields gains or regressions. Pilots calibrate task difficulty (Q4); final task selection uses coverage and diversity, not measured gains (Appendix A.4).

Our introduction identifies a gap in structural coverage, not a flaw in capability-grounded task construction. Claims establish the skill–task match, while task requirements, material facts, and conditions necessary for correctness provide grounds for judging completion.

We randomly sampled one task per skill, covering all 75 skills. Using task inputs and independently verifiable domain facts, we checked whether each claim applied in its setting and whether the task exercised that capability. Among these tasks, 38 used rubric grading. We audited all 451 criteria, including scoring items and mandatory failure conditions:

Basis of the criterionCriteriaShare (%)
G1. Explicit in the task or supplied materials14532.15
G2. Unstated but necessary for a correct solution23852.77
G3. Additional standards, including conventions or preferences6815.08

Appendix A.4, Table 9 illustrates all three categories with one task requesting research notes on Hacker News articles:

  • G1: Include both article summaries and discussion syntheses, as requested.
  • G2: When an article cannot be retrieved, state the information gap rather than present speculation in comments as confirmed facts.
  • G3: Place the article summary before the discussion synthesis—an additional organizational convention.

Thus, 84.92% of criteria are grounded in task requirements or necessary for correctness. G3 covers additional conventions or preferences, including those concerning domain practice or presentation; it does not mean all such criteria are skill-specific rules. We added this audit and its examples to Appendix A.4.

Q3. Task independence and gains on real materials

We further examined whether gains occur on tasks grounded in external facts and materials.

We analyzed tasks using public code, documents, packages, and real data. Source tasks use open-source code and tests, incident reports, and upstream configurations; held-out tasks use public app repositories, Debian/OpenWrt packages, and vulnerability records. Their code behavior, records, and operational constraints provide evidence for judging completion independently of skill claims.

Six models evaluated 64 tasks from 16 source skills; three models ran each of 29 tasks from 8 held-out skills three times. Gains compare Original with no-skill on source tasks and Opt with Original on held-out tasks:

Evaluation metricSkillsTasksNo-skill (%)Original (%)Opt (%)Gain (pp)
Source pass@1166447.9258.85—+10.94
Held-out pass@1829—60.9263.98+3.07
Held-out pass@3829—70.1177.01+6.90

All six source models improve, as do mean held-out pass@1 and pass@3. Gains on public real materials support the benchmark’s practical testing and adaptation uses. From existing evaluation results, we added rates and gains for both non-generated and synthetic materials to Appendix D.4, Table 23 (p. 26).

AuthorsOpenReview ↗

Response to fxcQ (Part 2): Human audits, task calibration, and scoring sensitivity

To address W2, we added quantitative human audits of skill labels, capability claims and scoring criteria, and output grading. Part 1, Q2 reports the claim and criterion audits; below we report label agreement, task calibration, and scoring reliability, addressing Q5, Q4, and Q6, respectively.

W2 & Q5. LLM–human and inter-annotator agreement on V/E/C labels

Two annotators used the paper’s taxonomy definitions and skill packages to review all 225 V/E/C labels for the 75 skills. F is determined by rules over the resource types in each package.

ComparisonLabels comparedExact matchesAgreement (%)
Annotator 1–LLM22520591.11
Annotator 2–LLM22520691.56
Annotator 1–Annotator 222519586.67

Overall inter-annotator κ is 0.775. Appendix A.2, Table 5 reports results separately for V, E, and C.

Q4. Which models were used in the paired pilots, how were tasks adjusted, and do results hold on unadjusted tasks?

The paired pilots used a single-turn, tool-free setting, comparing responses with and without the original skill. Most pilots used Claude-Sonnet-4.6; some tasks used Qwen3.7-Plus or DeepSeek-V4-Flash. When both conditions passed, the original task offered little opportunity to reveal whether the skill helped, so we prioritized these tasks for hardening. Their initial ease also helps explain why the calibrated group’s final no-skill pass rate remained higher.

Of the 300 tasks, 147 had materials or constraints adjusted following the pilots: 142 were only made harder, two had adjustments in both directions, and three received requirement clarifications or feasibility fixes. No task was only made easier. Appendix D.4, Table 22 reports counts and directions by skill category.

Both gains remain positive on the other 153 uncalibrated tasks. The source comparison measures the mean success-rate gain from providing the original skill over no-skill; the held-out comparison measures the mean pass@1 gain of Opt over Original:

Evaluation splitCalibration statusTasksGain (pp)
SourceCalibrated96+6.42
SourceUncalibrated104+20.83
Held-outCalibrated51+4.79
Held-outUncalibrated49+1.81

We added calibration counts (Table 22) and a breakdown of existing evaluation results (Table 21) to Appendix D.4; the latter also reports each group’s success rates. Source results aggregate six models; held-out results aggregate three models with three runs per task.

Q6. What is the human–judge agreement on rubric-graded tasks?

We sampled 30 rubric tasks from 30 distinct skills, reviewing two outputs per task. Human annotators used the same task-specific rubrics as the Claude-Sonnet-4.6 LLM judge, rating individual items as Yes, Partial, or No. We then compared human and automated pass/fail decisions using the same item weights, mandatory failure conditions, and 0.80 threshold.

Judgment typeComparisonsExact matchesAgreement (%)
Rubric item rating (Yes / Partial / No)56551891.68
Final evaluation outcome (Pass / Fail)605795.00

Results are reported in Appendix D.5, Table 24.

Q6. How sensitive are results to the 0.80 threshold, and does optimization help on rule-only tasks?

Holding model outputs, item scores, and mandatory failure decisions fixed, we varied the pass threshold from 0.60 to 0.95 across all 159 rubric tasks. Absolute pass rates declined in every condition as the threshold increased. However, at all eight thresholds examined, source Original consistently outperformed no-skill, and held-out Opt consistently outperformed Original (Figure 6).

Optimization gains also occur on tasks graded solely by executable checks. Using the default rubric threshold of 0.80, the results for the two evaluator subsets are:

Evaluator typeHeld-out tasksΔpass@1 (pp)Δpass@3 (pp)
Rule-based40+4.44+5.00
Rubric-based60+2.59+5.00

Thus, the optimization gain holds on the rule-only subset. Using existing evaluation outputs, we added threshold sensitivity (Figure 6) and results by evaluator type (Table 25) to Appendix D.5.

Thanks for the response. The added analyses addressed some of my concerns, but the main issues remain unresolved. So I will keep my rating.

First, the response still lacks validation on tasks defined independently of the evaluated skills. Although real materials allow correctness to be checked against external facts, task requirements remain derived from each skill's own claims, with requirements solvable by general capabilities filtered out, and the response does not report whether the category differences persist on these tasks. The measured gains therefore do not show whether a skill helps on an independently specified task, such as accurately extracting target information from given web pages.

Second, the response narrows the benchmark's purpose to supporting skill rewriting, but the main evaluation only contrasts providing versus withholding a skill. Even for rewriting, tasks are fixed by the original skill's claims, and the optimizer only enforces stricter adherence to existing rules. The benchmark can therefore reward more faithful execution of the original, but cannot show that the skill itself has improved. I acknowledge the held-out comparison between rewritten and original versions, but it is insufficient in principle: since all versions are evaluated on tasks derived from the original skill's claims, it can only reflect better delivery of those claims, not whether the skill's content or strategy has improved. If the original strategy is suboptimal, stricter adherence may even amplify it.

Finally, the revised results weaken rather than support the central claims. Once all paired runs are included, some category advantages narrow or even reverse. The appendix acknowledges that whether changing a skill attribute improves performance remains a hypothesis, yet the conclusion still claims an empirical foundation for skill design. The optimization gains concentrate on tasks hardened during pilot calibration and become marginal on uncalibrated tasks, and removing either the experience library or the taxonomy guidance yields no significant difference on the primary metric.

Thank you for taking the time to explain your remaining concerns. We agree on the value of independently specified tasks. Where we see things differently is what the present benchmark measures and what rewriting can change.

1. Why construct tasks from skills?

Given an existing skill, developers need to know how much different models benefit from it in a given harness. We therefore construct a task distribution around its declared capabilities to evaluate the work it is intended to support.

The task requirements are informed by the skill’s claims, so they are not independently specified in the sense you describe. Your proposed evaluation starts from user requests and then selects suitable skills, as in SkillsBench. Constructing tasks around the skill gives developers a common basis for comparing models’ gains and examining how those gains vary across skill structures. The Introduction and Related Work explain this choice. The benchmark’s value is to establish where providing skills improves model performance within their intended uses, and where it fails to do so.

2. Benchmark evaluation and skill rewriting

The benchmark measures models’ gains from skills. SkillPrism-Opt uses the results to learn edits transferable to unseen skills.

(1) What does the benchmark measure? Given a model and an agent harness, how much does providing a skill change the model's ability to complete the tasks the skill is intended for? Let J(m,h,U;D)J(m,h,U;\mathcal D) denote the expected pass rate of model mm in harness hh using skill version UU on task distribution D\mathcal D. For skill SS and its intended task distribution DS\mathcal D_S, the gain is

Δuse(m,h;S,DS)=J(m,h,S;DS)−J(m,h,∅;DS),\Delta_{\mathrm{use}}(m,h;S,\mathcal D_S)=J(m,h,S;\mathcal D_S)-J(m,h,\varnothing;\mathcal D_S),

where ∅\varnothing denotes no skill. We hold OpenHands fixed and compare six models, each on the same tasks with and without the skill. This shows which models benefit more, which tasks they solve only with the skill, and where it introduces errors on tasks they could already solve. Grouping by taxonomy lets us compare how the models perform with different types of skills (§§5.2–5.3).

(2) Is rewriting limited to helping the model follow the original skill more faithfully? No. The editor can revise the procedure itself. We compare the expected pass rates of the original skill SS and its rewrite S~\widetilde S with the model, harness, and task distribution held fixed.

Δrewrite(m,h;S,DS)=J(m,h,S~;DS)−J(m,h,S;DS).\Delta_{\mathrm{rewrite}}(m,h;S,\mathcal D_S)=J(m,h,\widetilde S;\mathcal D_S)-J(m,h,S;\mathcal D_S).

We use a skill’s claims to choose what to test. For example, the XLSX skill supports linking data across sheets, so one task asks the model to build a linked workbook from campaign data. We check the values and formula references in the submitted workbook. Even when a task closely matches the claim, the original instructions may omit a step needed to complete it. When a source trace reveals such an omission, the editor can add the missing step and rerun the same model on the same task to test whether the revised procedure works better.

3. Additional analyses

(1) Category findings and conclusions about skill design. Including all paired runs changes the scope of some category findings. Section 5.3 now limits the script-backed advantage to runs where the agent accesses the skill, while the lower regression rate associated with static verification support holds in both analyses. Figure 4 and Appendix D.1 show which patterns persist and where they vary across models and tasks. These results provide evidence about how models benefit from different skill structures. We will revise the Conclusion to reflect these findings and their scope more precisely.

(2) Does the rewriting gain persist on uncalibrated tasks? Yes. When increasing difficulty, we prioritized tasks the model could solve both with and without the skill in the pilot. We revised materials or constraints on 51 held-out tasks and left 49 unchanged. On those 49, rewriting raised pass@1 from 48.98% to 50.79% (+1.81 pp). The gain was larger on calibrated tasks (+4.79 pp) and positive in both groups (Table 21). Across all 100 tasks, the gain was 3.33 pp (paired 95% CI [1.11, 5.67]).

(3) What do the ablations show? Removing the source experience library lowers pass@1 for all three execution models, by 2.67 pp on average. Keeping the same library and removing only the target skill's taxonomy attributes and guidance also lowers pass@1 for all three models, by 2.11 pp on average. The full method retains its advantage in both comparisons when any one held-out skill is omitted (Tables 2 and 20). The ablations suggest that source execution feedback helps the editor learn useful edits and that a new skill’s taxonomy attributes help it choose which of those edits to apply.

Reviewer_uczCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SkillPrism-Bench, a benchmark of 75 public skills and 300 tasks organized along four taxonomy dimensions: artifact form, verification support, execution dependency, and control flow. Through paired evaluations with and without skills, the benchmark measures how skills affect pass rates, failure repairs, regressions, skill access frequency, and execution cost. Building on these source-skill executions, the authors propose SkillPrism-Opt, an optimizer that distills measured source edits into conditional editing strategies and applies them to rewrite held-out target skills using only their package contents and taxonomy attributes, without accessing target test tasks or execution feedback. Multi-model experiments report a 13.9 percentage points source-task pass rate gain from providing original skills and a 3.33 percentage-point held-out pass@1 improvement from SkillPrism-Opt over the original skills.

Critical strengths
  1. Section 3 and Appendix A make the benchmark coverage transparent by detailing the four-dimensional taxonomy, joint sampling strata, capability claim extraction, and evaluator construction. Because tasks are linked to explicit skill attributes, the paired evaluations offer more explanatory value than an aggregate pass rate alone, showing where skills repair failures, where they cause regressions, and which skill categories drive these outcomes. Consequently, even though the 75 skills are deliberately curated, the benchmark enables structured comparisons across distinct skill characteristics.

  2. SkillPrism-Opt improves held-out skills without accessing their test tasks or execution feedback, which addresses the practical cost of maintaining large skill collections where per-skill optimization loops are expensive. By strictly separating source and target skills, the evaluation tests whether distilled editing experience transfers beyond the skills that produced it. Moreover, evaluating a single rewritten version of each target skill across three execution models demonstrates that the refined instructions generalize across agent backbones.

  3. Table 15 supports the main held-out pass@1 improvement of 3.33 percentage points over original skills with a 95% skill-cluster bootstrap interval spanning 1.11 to 5.67 percentage points. Resampling whole skills respects task clustering within each skill, while the leave-one-skill-out range of 2.89 to 3.70 percentage points confirms that no single favorable skill drives the aggregate gain. Combined with the explicit reporting of skill-level regressions in Table 14, these analyses provide a credible basis for concluding that the complete optimizer helps on average without implying that every target skill benefits.

Critical weaknesses
  1. The category-level analysis in Section 5.3 cannot isolate the independent effect of individual skill properties such as scripts, verification support, or adaptive control flow. Because the analysis conditions on runs where the agent chooses to access a skill, the evaluated task population shifts across both models and taxonomy categories. In addition, the marginal categories in Table 11 differ in domain, baseline difficulty, and co-occurring taxonomy attributes, and some strata are very small, such as the asset-backed source category with only one skill. While these comparisons reveal useful descriptive associations, they do not establish that adding a specific structural property to a skill will directly produce the observed gain.

  2. Although the benchmark construction pipeline includes sensible quality checks, the paper reports no quantitative audit metrics such as sample sizes, disagreement rates, or evaluator error rates. This omission is consequential because an LLM labels three of the four taxonomy dimensions, 207 of the 300 tasks rely on synthetic materials, 159 tasks rely on rubric evaluators, and paired pilot runs guide task difficulty revisions. While testing evaluators on defective and semantically equivalent outputs helps catch obvious judging flaws, the absence of quantitative audit results leaves uncertainty about how sensitive the reported gains are to task generation, calibration, and evaluation choices.

Questions for authors
  1. With the experience library, editor, rewrite constraints, and target packages held fixed, how does held-out performance change when taxonomy labels are removed or shuffled? Please use this comparison to separate the contribution of taxonomy guidance from the contribution of the source-derived editing strategies.

  2. Do the main category-level findings in Section 5.3 persist when evaluated across all assigned model-task pairs instead of only the runs where the agent accessed the skill? Furthermore, how sensitive are these patterns to model choice, domain, baseline pass rate, and correlated taxonomy attributes, and which design recommendations should remain descriptive hypotheses given the small sample sizes in certain categories?

3 comments
AuthorsOpenReview ↗

Response to uczC (Part 1): Category-level findings and taxonomy ablation

Thank you for recognizing how the paired evaluations reveal where skills help or cause regressions, and for your positive assessment of the held-out transfer evaluation. Your suggestions helped us clarify the scope of the category-level findings. This comment addresses W1 & Q2 and the taxonomy ablation in Q1; Part 2 reports the quantitative audits and sensitivity analyses requested in W2.

W1 & Q2. Do the category-level findings hold across all evaluations and within different strata?

We expanded Figure 4 with all 1,200 model–task pairs, retaining the original results for the 817 pairs in which the agent accessed the skill. Extending the analysis to all pairs yields the following observations:

  • Artifact form: Script- and reference-backed skills yield larger gains than instruction-only skills in the accessed subset; across all pairs, the three groups show similar gains.
  • Verification support: In both populations, static criteria have the lowest regression rate, followed by executable checks and then no explicit support.
  • Environment interaction: File operations, command execution, and service calls have higher gains and repair rates than context-only processing in both populations.
  • Workflow: Relative to single generation, adaptive workflows have larger gains and predefined workflows have lower regression rates in both populations.

We further examined sensitivity to model, domain, and baseline difficulty, defined by how many of the six models solved each task without the skill. Holding domain and difficulty fixed within each model, scripts and references each yield larger gains than instructions alone in 5/6 models. However, reference-backed gains exceed instruction-only gains in business operations but are lower in software engineering. Thus, similar overall gains can mask differences across models and domains.

To examine co-occurring attributes, we separately matched on execution model and the other three taxonomy dimensions. With artifact form, execution dependency, and control flow held fixed, static criteria still have a 7.9 pp lower regression rate than no explicit support. This supports the verification finding beyond the marginal comparison.

The revised §5.3 presents these findings as group-level observations, with possible explanations. Whether adding a property to the same skill improves performance remains a design hypothesis. In Appendix D.1, Table 13 adds all-pair results; new Tables 14, 16, and 17 report stratified and attribute-matched analyses, including skill, task, and comparable sample counts. Table 15 retains the existing mixed-resource analysis by model.

Q1. With the experience library fixed, how much does taxonomy guidance contribute?

We added w/o Tax., removing only explicit target F/V/E/C labels and their guidance while fixing the shared experience library, editor, target packages, rewrite procedure, and constraints. Both conditions retain the same domain information and strategy content. We evaluate the same 25 held-out skills and 100 tasks in the same harness, with three runs per model, task, and condition. Cells show pass@1 / pass@3 (%).

Execution modelOriginal skillw/o Tax.Full method
Qwen3.7-Plus48.00 / 61.0049.67 / 61.0050.67 / 68.00
DeepSeek-V4-Flash-073159.67 / 73.0059.33 / 70.0062.67 / 75.00
GLM-5.260.33 / 72.0062.67 / 72.0064.67 / 78.00
Mean56.00 / 68.6757.22 / 67.6759.33 / 73.67

The full method exceeds w/o Tax. on both metrics for all three models. With the same experience library, taxonomy provides additional guidance for judging which strategies apply. We added the w/o Tax. row to the existing §5.4, Table 2, and its skill-level paired resampling intervals to Appendix D.3, Table 20.

AuthorsOpenReview ↗

Response to uczC (Part 2): Human audits and evaluation sensitivity

We added quantitative human audits (Tables 5, 9, and 24) and examined how the reported gains vary with task construction, calibration, and scoring choices. Below we address W2.

Human audit of skill labels

Two annotators each reviewed all 225 V/E/C labels for the 75 skills using the taxonomy definitions and skill packages.

ComparisonLabels comparedExact matchesAgreement (%)
Annotator 1–LLM22520591.11
Annotator 2–LLM22520691.56
Annotator 1–Annotator 222519586.67

Overall inter-annotator κ is 0.775. Appendix A.2, Table 5 reports the results by dimension.

Audit of capability claims and rubric criteria

We randomly sampled one task per skill, covering all 75 skills. Using task inputs and independently verifiable domain facts, we checked whether each claim applied in that setting and whether the task exercised the claimed capability.

Among these tasks, 38 used rubric grading. We examined all 451 criteria, including scoring items and mandatory failure conditions, to identify their basis:

Basis of the criterionCriteriaShare (%)
G1. Explicit in the task or supplied materials14532.15
G2. Unstated but necessary for a correct solution23852.77
G3. Additional standards, including conventions or preferences6815.08

Most criteria are grounded in explicit task requirements or requirements necessary for correctness. Appendix A.4, Table 9 illustrates the three categories with examples.

Human–judge agreement on rubric-graded tasks

We sampled 30 rubric tasks from 30 distinct skills and reviewed two outputs per task. Human reviewers used the same task-specific rubrics as the Claude-Sonnet-4.6 LLM judge, rating individual items as Yes, Partial, or No. We then compared human and automated pass/fail decisions using the same item weights, mandatory failure conditions, and 0.80 threshold.

Judgment typeComparisonsExact matchesAgreement (%)
Rubric item rating (Yes / Partial / No)56551891.68
Final evaluation outcome (Pass / Fail)605795.00

These results are reported in Appendix D.5, Table 24.

Sensitivity to construction and evaluation choices

The revision also adds three sensitivity analyses using existing evaluation outputs:

  • Input materials: On held-out tasks using non-generated and synthetic materials, optimization improves pass@1 by 3.07 and 3.44 pp, respectively (29 and 71 tasks; Appendix D.4, Table 23, p. 26, newly added analysis of existing evaluation results).
  • Task calibration: Providing skills improves success by 20.83 pp on the 104 uncalibrated source tasks. Optimization improves pass@1 by 1.81 pp on the 49 uncalibrated held-out tasks. Thus, both gains persist on tasks left unchanged by the pilots. Appendix D.4, Table 21 reports results for both calibrated and uncalibrated groups.
  • Scoring choices: Across all 159 rubric tasks, gains from providing skills on source tasks and optimizing held-out skills remain positive at all eight thresholds from 0.60 to 0.95 (Figure 6). Optimization also improves held-out pass@1 by 4.44 pp on rule-only tasks and 2.59 pp on rubric tasks (Appendix D.5, Table 25).

Thank you for the additional analyses and human audits. The evaluation across all assigned model-task pairs and the revised interpretation of category differences address my concern about the original analysis. The human audits and sensitivity checks also strengthen confidence in the benchmark, while the new taxonomy ablation provides more direct evidence that taxonomy guidance contributes beyond the shared editing experience.

These additions address my main concerns and strengthen my support for acceptance. I will maintain my rating of 3. The benchmark and the transfer of editing experience are useful contributions, although the evaluation on 25 held-out skills still leaves some uncertainty about how broadly the optimization gains apply. I consider the responses sufficient to conclude this discussion and have no further questions.

Reviewer_yq6FOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SkillPrism, a benchmark and optimization framework for LLM agent skills. The key idea is to learn editing strategies from execution feedback on source skills and reuse them to improve unseen skills without testing them. The results are promising, with an average pass@1 improvement of 3.3 percentage points over the original skills.

Critical strengths

The paper addresses a practical problem, and the benchmark construction is reasonably thorough. I also like the idea of grounding editing strategies in measured outcomes and explicitly preserving existing functionality. The held-out evaluation provides some evidence that these strategies can transfer across skills.

Critical weaknesses

My main concern is that the contribution of the proposed taxonomy is not clearly established. The ablation without the experience library does not show whether taxonomy-guided retrieval is actually necessary. An experience-based editor without taxonomy information would be a more informative comparison.

I am also not fully convinced that the learned experience offers substantial benefits over general skill-authoring guidelines. Many of the strategies appear broadly applicable, and several are supported by only one source skill. A stronger baseline using comparable generic editing rules would help clarify what is gained from learning these strategies through execution feedback.

The benchmark also has some limitations. Tasks are derived from the capabilities declared by the evaluated skills, and many use synthetic inputs. This makes the results useful for measuring skill-aligned performance, but less conclusive about real-world utility. Finally, the average gains are modest, and 17 of 75 held-out skill–model combinations regress relative to the original skills. More analysis of these failures would be useful, especially given that the method cannot validate its edits on target tasks.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to yq6F: Taxonomy ablation, the value of execution feedback, and practical utility

Thank you for recognizing the practical need to evaluate and improve agent skills, and for your positive assessment of our benchmark construction. Following your comments, we address the taxonomy’s contribution, the added value of execution feedback, the benchmark’s practical utility, and regressions on held-out skills.

W1. Taxonomy ablation

We added w/o Tax., removing only explicit target taxonomy attributes and guidance while fixing the shared experience library, editor, and rewrite procedure. Evaluation uses the same 25 held-out skills and 100 tasks, with three runs per model, task, and condition. Cells show pass@1 / pass@3 (%).

Execution modelOriginal skillw/o Tax.Full method
Qwen3.7-Plus48.00 / 61.0049.67 / 61.0050.67 / 68.00
DeepSeek-V4-Flash-073159.67 / 73.0059.33 / 70.0062.67 / 75.00
GLM-5.260.33 / 72.0062.67 / 72.0064.67 / 78.00
Mean56.00 / 68.6757.22 / 67.6759.33 / 73.67

These results suggest that taxonomy helps the editor identify applicable experience. We added the w/o Tax. row to the existing §5.4, Table 2, and its intervals to Appendix D.3, Table 20.

W2. The added value of experience learned from execution feedback over general guidelines

Our general rewriting baseline uses Anthropic’s official skill-authoring guidelines and the same editor as the full method. These cover usage descriptions, concise instructions, and ordered workflows, now clarified in §5.1. Across held-out tasks, the full method improves pass@1 and pass@3 over this baseline by 5.22 and 4.33 pp.

Execution feedback grounds adaptation to a particular model and harness. It identifies where general principles fail in practice and tests how specific the instructions must be to correct those errors, even when the resulting rules sound familiar.

For example, DeepSeek-V4-Flash-0731 in OpenHands filled a report’s root-cause field with a suspected cause despite acknowledging that it was unconfirmed. We required unconfirmed causes to remain unknown and suspicions to be recorded separately. The same model then passed the source task.

This experience guided a held-out skill’s rewrite to preserve estimate sources and unknown values. In one general-rewriting run, the same model recognized that a reference value described a different object, yet suggested entering it as an estimate. The estimate label did not establish that it applied to the current object. In one successful full-rewrite run, the model stated that the evidence could not determine the current object’s value.

The transferable rule is to check whether the evidence supports a value for the object being recorded before filling the field. Execution feedback revealed that acknowledging uncertainty in prose did not ensure this check at the point of entry, and helped us place the constraint where the error occurred.

W3. The benchmark’s practical value and reuse of adaptation experience across skills

SkillPrism addresses a practical need: determining whether a skill helps a given execution model and harness perform its intended work. Its pipeline constructs tasks and rubrics from skill capabilities, allowing newly added skills to undergo systematic evaluation.

The benchmark measures how supplying a skill changes execution performance, while the taxonomy reveals differences across skill types and models to guide targeted adaptation. We demonstrate this across six source models with OpenHands held fixed (§5.3). To assess utility on real-world materials, we analyzed 64 source tasks grounded in public repositories and documents. Providing skills improves average success by 10.94 pp, with positive gains in all six models (new analysis in Appendix D.4, Table 23).

Our optimization method illustrates reuse of benchmark experience. As coverage grows across models and harnesses, we aim to let developers specify a target model and harness and adapt a new skill in one rewrite, without target execution feedback or repeated testing.

W4. Regressions on held-out skills

We examined the 17 of 75 held-out skill–model combinations that regressed. In an advertising task, a successful original-skill run recommended pausing a campaign to stop spending, since permanent deletion was unsupported. The rewrite retained this restriction, yet the same model read the skill and recommended deletion in all three runs, one explicitly “as the director ordered.” This illustrates a limit of rewriting without target feedback. The editor can preserve a constraint, but cannot verify whether the model will apply it when a user request conflicts with platform capabilities. We added this case to Appendix E. Appendix D.2, Table 19 summarizes skill-level outcomes.

Reviewer_yq6FOpenReview ↗

Reply from reviewer

Thank you for the thoughtful response and additional experiments. I appreciate the authors' efforts to address my concerns.

The new taxonomy ablation and transfer analysis provide useful evidence supporting the proposed method. However, I still have some reservations about the relatively modest improvements, the independent contribution of the learned experience, and the regressions observed on held-out skills.

Overall, the rebuttal has improved my assessment of the paper, but I remain somewhat uncertain about the significance and reliability of the contributions. I therefore lean toward maintaining my current rating.

Thank you again for the detailed clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.