- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
SkillIF is a benchmark for whether agents adhere to the skill specifications they are given. It pairs 96 GitHub-sourced skills (ten domains) with LLM-generated executable tasks, derives 2,194 constraints in three categories (Deliverable / Procedure / Restraint, nine subcategories), and scores each constraint on a recorded execution by a deterministic script check conjoined with an LLM judge (DeepSeek V4 Flash). Eleven model–harness configurations (Claude Code, Codex) are evaluated, one execution per skill (1,056 runs). Findings: Procedure is the weakest category in every configuration; explicit prohibitions are followed more consistently than scope and stopping requirements; 298 of 717 task-successful runs satisfy fewer than half of their constraints. Validation includes three judging passes, a substitute judge, a 600-item manual re-read, and a partial release (11 of 96 skills, result CSVs, judge and reproduction scripts).
Critical strengths
-
The reported numbers are reliable and the judge is validated more carefully than in most benchmark papers. All 143 cells of Table 1, Fig. 9, Tables 12–15 and the A.4 count chain reproduce exactly from the released
constraints.csv/categories.csv; the substitute-judge agreement (Kendall τ = 0.745) and the Wilson intervals recompute. This matters because it means the issues below are about what is measured, not whether the arithmetic is right — the fixes are experimental, not editorial. -
The question is the right one. Separating "task judge accepts the output" from "the agent followed the skill's process requirements," with a paired run-level analysis (Sec. 4.3), gives the community a measurable handle on a failure mode that success-rate benchmarks hide. If the measurement problems were fixed, the benchmark would be valuable.
Critical weaknesses
-
The released artifacts show the benchmark does not measure what the paper says it measures. (a) Sec. 3.2 (p.3) says normalization preserves the requirements and the App. B.3 prompt (p.22) says "Do NOT add new requirements," yet every released
SKILL.mdopens with an added "## Interaction policy" (do not ask, do not request confirmation) and the forced system prompt inrunner_retrigger.pyL34–50 states "PRE-AUTHORIZED FOR THIS RUN (the user has already answered every gate the skill may ask) … Ignore any skill instruction that tells you to stop and wait for user confirmation or input" — while termination, boundary and step constraints are still scored. (b) Contrary to the "assignment distinct from the skill" design (p.2–3),053-go-to-market-plan/SKILL.mdL297–403 contains a worked plan for "DevAnalytics," the company in that skill'stask/task.txt(constraints C25/C29 referencetask.txt/DevAnalytics);006-capture-triage/SKILL.mdnames the task's daily note (L37, L198) and logs "Version 2.3 (2026-08-17): Made routing fully autonomous for headless runs" (L365); three of the 96 "GitHub" skills (348/349/350-*-constrained) are undisclosed author-made variants of 082/034/010. None of this is visible from the paper. It undercuts the two design premises on which the adherence numbers rest. -
The judge's rule for prohibitions contradicts the paper's own semantics, and the headline Restraint contrast likely measures that rule. Sec. 4.2 (p.7): "a prohibition may be satisfied by the absence of an action."
judge.pyL289–296, rule 4: for negative checks, "yes" still requires "proof the agent did the correct thing, not just the absence of a visible violation"; rules 5/7 default to "no." With 90.1% of negative judgments LLM-dependent (A.4) and 41.8% of negative failures typed "Content missing" (Fig. 9), the finding that prohibitions are followed more consistently than boundary/termination requirements cannot be separated from the judging rule. The judge also imports a missingevidence_extractor.pyon which rule 7 hinges, so the verdicts cannot be audited.
Questions for authors
Clarification (answers would change my assessment):
- How many of the 2,194 constraints concern gates, confirmations, setup or stop-and-wait behaviour that the forced prompt instructs agents to skip, and what are the category scores with those constraints excluded?
- How many of the 96
SKILL.mdfiles were edited beyond the App. B normalization, and did task generation permit reusing examples contained in the skill? What are the results with task-coupled pairs removed? - What are the negative/boundary/termination scores when a prohibition is judged satisfied by the absence of the forbidden action, i.e., under the Sec. 4.2 semantics?
judge.py --taskinjectsSKILL.mdinto the task-judge prompt (L58, L302), whereas Sec. 3.4 says the task judge runs "without access to the skill specification." Which was used for the 717 task-success verdicts?- Sec. 3.4 says constraints without a verdict are excluded from numerator and denominator;
judge.pyL414–432 instead falls back to the script verdict with a fixed denominator (44 units are script-only). Which is correct?
Suggestions (do not affect my assessment): release the evidence extractor, all 96 skills, the task rubrics and execution records; state the harness asymmetry (--disallowedTools WebFetch vs. -s danger-full-access); distinguish execution failures (e.g., 12.5 s, 123-character output, scored 0/43) from non-adherence; remove the invisible draw.io interface text embedded in Fig. 1.
6 comments
Response to Reviewer C5EY — Part 2: Skill Construction
Weakness 1(b) & Question 2: Modifications, examples, and task relationships
Modification scope. All 96 skills received a common autonomous-execution policy. Three constrained variants also contain author-added workflow requirements: we developed them from mobile-games, client-intake, and server-management, adding specified process records, structured fields, resource-reading sequences, and verification records.
These variants, their corresponding originals, and the other skills underwent the same screening and construction process: normalization and rewriting, task construction, executability checks, constraint extraction, and scoring. They are evaluated as skill versions under the same protocol. We will explicitly identify their provenance and additions, distinguishing these additions from structural normalization.
Use of existing examples. The complete DevAnalytics example was already present in the upstream open-source version of go-to-market-plan. We subsequently constructed the task from the complete skill package, including that example. The shared business context follows from this construction order.
SkillIF measures adherence to the supplied specification. Templates and examples are available guidance, and reusing them is permitted when the result fits the task inputs and skill requirements. Originality is not a scoring criterion.
An example can help produce a deliverable, but it does not establish that the required process occurred in the current run. Go-to-market-plan still requires reading the business context, assessing GTM readiness, analyzing the strategy, and performing a final self-check. These requirements are assessed separately.
The marketing activities in the example are an action plan for the user; the skill explicitly states that the agent does not carry out those activities on the user’s behalf. This distinction separates the proposed marketing actions from the analysis and verification the agent must perform during evaluation.
Task and skill have different roles. The task supplies the concrete goal and inputs, while the skill supplies requirements for performing that kind of work. This distinction does not require a task to use a different company name or business context from a skill’s example.
Similarly, the dates and filenames in capture-triage identify inputs to be processed; they do not provide the processing results. Its unattended-execution adaptations concern the setting described in Part 1. We will clarify these task–skill relationships in the construction description.
Effect on the main observations. We recomputed scores from the submitted constraint-level data after excluding the identified cases. Scores are macro-averaged over skills within each configuration, then averaged equally across the 11 configurations.
| Evaluation set | Skills | Overall | Deliverable | Procedure | Restraint |
|---|---|---|---|---|---|
| Original set | 96 | 50.18% | 52.24% | 43.68% | 50.23% |
| Excluding go-to-market-plan and capture-triage | 94 | 50.40% | 52.30% | 44.08% | 50.48% |
| Excluding the three constrained variants | 93 | 50.29% | 52.43% | 43.34% | 50.42% |
| Excluding all five skills | 91 | 50.51% | 52.50% | 43.74% | 50.69% |
In all three exclusion settings, Procedure remains the lowest-scoring category in every configuration. Negative also remains higher than Boundary and Termination in every configuration. These recurring mean-score patterns therefore do not depend on the five identified skills.
Response to Reviewer C5EY — Part 3: Judging Prohibitions
Weakness 2 & Question 3: Sensitivity to the prohibition rule
We agree that, for a pure prohibition, a sufficiently complete relevant record showing that the prohibited action did not occur should support compliance. No additional action should be required unless the skill specifies it. This differs from a retrieval step returning no relevant evidence.
Paired rule comparison. To test the prompt’s effect directly, both arms used the currently available DeepSeek V4.1 Flash, identical frozen execution evidence, and the original judging pipeline. Only Rule 4, concerning prohibitions, was changed; mechanical verdicts and all other scoring settings were held fixed.
The revised rule accepts compliance when the relevant record is sufficient and the prohibited action did not occur. Explicit additional obligations remain in force, and insufficient evidence is not automatically treated as a pass.
The comparison covered all 11 configurations and the 94 skills with applicable requirements. The LLM components for Negative, Boundary, and Termination were rejudged in both arms.
| Metric | Revised rule − original rule | 95% CI for the difference |
|---|---|---|
| Negative | +3.84% | [1.58%, 6.17%] |
| Boundary | +0.55% | [−1.54%, 2.65%] |
| Termination | +2.54% | [−0.65%, 5.93%] |
| Restraint | +2.43% | [0.70%, 4.21%] |
Differences use skill-macro averages within each configuration, followed by equal weighting of the 11 configurations. Confidence intervals use a paired bootstrap clustered by skill.
The revised rule increases the Negative score, but Negative remains higher than both Boundary and Termination in all 11 configurations under both rules. The wording affects absolute scores, while the reported category mean pattern remains present under the interpretation you suggested.
How the existing checks divide the requirements. In capture-triage, moving the files, checking their final location, and prohibiting deletion have separate scoring items:
| Constraint | What is checked | Subcategory |
|---|---|---|
| C22 | Whether processed files are moved into Processed | Step |
| C23 | Whether all five files are in Processed at the end | Evidence |
| C24 | No deletion of original files; the text also describes preserving them by moving them | Negative |
The deterministic component of C24 uses a negative match: it passes when rm is not matched in the tool log, without requiring an affirmative action. The final verdict also includes an LLM component. The comparison above therefore holds mechanical checks fixed and directly tests the LLM prompt.
Interpreting Content missing within Negative. Negative requirements do not always prohibit a tool action. For example, “Do not give generic advice; each strategy must include a concrete plan tailored to the current product” requires examining whether the output contains the specified detail.
Whether Content missing is appropriate depends on the actual requirement; the label alone does not establish a contradiction. We will clarify these judgment conditions and provide further information on evidence sources and extraction where disclosure is permitted.
Response to Reviewer C5EY — Part 1: Execution Setting
Thank you for examining the supplementary materials and confirming that the main numerical results are reproducible. Your concerns center on whether the execution setting, skill construction, and judging rules support the intended measurement. We have revisited these points and added skill-exclusion analyses and a judging-rule comparison.
Weakness 1(a) & Question 1: Unattended execution and stopping requirements
SkillIF runs without live user participation after a task starts. Given the supplied task materials and authorizations, we found no requirement among the reviewed items that necessitated a live question or wait for confirmation prohibited by the execution prompt.
Scope of the check. Our additional review covered the submitted roster of 96 skills and 2,194 final scoring constraints. We searched the scoring specifications and examined candidates concerning confirmation, missing inputs, authorization, and stopping conditions against the skill text and task instructions.
During construction, we had already conducted at least three compatibility reviews. We supplied task materials and advance authorization, revised skill instructions, and removed inapplicable scoring criteria where necessary. The external prompt prevents repeated confirmation requests from interrupting evaluation; it is part of the execution configuration, not a source from which scoring constraints are extracted.
Why stopping and process requirements remain applicable. Termination, Boundary, and Step cover stopping conditions, scope limits, and actions or their ordering. These requirements can often be satisfied without live interaction, such as specifying when an experiment should stop or completing a check before proceeding.
Two cases illustrate how the task conditions matter:
go-to-market-plan: C25 permits strategy generation when sufficient information is available or the user authorizes inference from the supplied materials. The task provides that authorization, so another confirmation is unnecessary. Reading the business context, assessing GTM readiness, analyzing the strategy, and checking the deliverable must still be supported by execution evidence. Authorization does not replace these steps.
sprint-planning: Team-member comments were supplied to the executor and were available for judging. The requirement checks whether the plan records those existing comments; it does not require contacting team members and awaiting new replies during the run. These materials were omitted from the public supplement because they could reveal sensitive information or author identities.
The public supplement therefore does not contain every input or piece of evidence used in the original evaluation. We will make the execution conditions, compatibility checks, and redaction scope more explicit so that readers can interpret the requirements in their actual task context.