acceptodds
Under review as a conference paper at ICLR 2027

SkillIF: Do Agents Follow Their Skills?

Abstract

Agent skills specify what an agent should produce, how it should perform the work, and which limits it should respect. Evaluating their use therefore requires examining whether agents follow these requirements during execution. We introduce SkillIF, a benchmark for systematically studying adherence to skill specifications. SkillIF pairs 96 curated skills across ten domains with executable tasks and derives 2,194 assessable constraints from their specifications. A shared taxonomy enables comparisons of the same requirement types across different skills, while deterministic checks and LLM judgments assess each constraint against execution evidence. Together, these components reveal which requirements agents repeatedly fail to satisfy and how those failures appear in their actions and outputs. Across combinations of language models and execution harnesses, procedural requirements emerge as a shared weakness, while explicit prohibitions are followed more consistently than scope limits and stopping requirements. Substantial violations also remain in runs accepted by the task judge. Analysis of successful runs with low adherence distinguishes missing content and unexecuted steps from outputs that fulfill requirements only partly or in a different form. SkillIF provides a systematic account of how agents follow reusable specifications and identifies concrete targets for evaluating and improving skill adherence.

Primary area:
datasets and benchmarks
Keywords:
Agent Skills, Instruction Following, Skill Adherence, LLM Agents, Benchmarking, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C5EYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

SkillIF is a benchmark for whether agents adhere to the skill specifications they are given. It pairs 96 GitHub-sourced skills (ten domains) with LLM-generated executable tasks, derives 2,194 constraints in three categories (Deliverable / Procedure / Restraint, nine subcategories), and scores each constraint on a recorded execution by a deterministic script check conjoined with an LLM judge (DeepSeek V4 Flash). Eleven model–harness configurations (Claude Code, Codex) are evaluated, one execution per skill (1,056 runs). Findings: Procedure is the weakest category in every configuration; explicit prohibitions are followed more consistently than scope and stopping requirements; 298 of 717 task-successful runs satisfy fewer than half of their constraints. Validation includes three judging passes, a substitute judge, a 600-item manual re-read, and a partial release (11 of 96 skills, result CSVs, judge and reproduction scripts).

Critical strengths
  1. The reported numbers are reliable and the judge is validated more carefully than in most benchmark papers. All 143 cells of Table 1, Fig. 9, Tables 12–15 and the A.4 count chain reproduce exactly from the released constraints.csv/categories.csv; the substitute-judge agreement (Kendall τ = 0.745) and the Wilson intervals recompute. This matters because it means the issues below are about what is measured, not whether the arithmetic is right — the fixes are experimental, not editorial.

  2. The question is the right one. Separating "task judge accepts the output" from "the agent followed the skill's process requirements," with a paired run-level analysis (Sec. 4.3), gives the community a measurable handle on a failure mode that success-rate benchmarks hide. If the measurement problems were fixed, the benchmark would be valuable.

Critical weaknesses
  1. The released artifacts show the benchmark does not measure what the paper says it measures. (a) Sec. 3.2 (p.3) says normalization preserves the requirements and the App. B.3 prompt (p.22) says "Do NOT add new requirements," yet every released SKILL.md opens with an added "## Interaction policy" (do not ask, do not request confirmation) and the forced system prompt in runner_retrigger.py L34–50 states "PRE-AUTHORIZED FOR THIS RUN (the user has already answered every gate the skill may ask) … Ignore any skill instruction that tells you to stop and wait for user confirmation or input" — while termination, boundary and step constraints are still scored. (b) Contrary to the "assignment distinct from the skill" design (p.2–3), 053-go-to-market-plan/SKILL.md L297–403 contains a worked plan for "DevAnalytics," the company in that skill's task/task.txt (constraints C25/C29 reference task.txt/DevAnalytics); 006-capture-triage/SKILL.md names the task's daily note (L37, L198) and logs "Version 2.3 (2026-08-17): Made routing fully autonomous for headless runs" (L365); three of the 96 "GitHub" skills (348/349/350-*-constrained) are undisclosed author-made variants of 082/034/010. None of this is visible from the paper. It undercuts the two design premises on which the adherence numbers rest.

  2. The judge's rule for prohibitions contradicts the paper's own semantics, and the headline Restraint contrast likely measures that rule. Sec. 4.2 (p.7): "a prohibition may be satisfied by the absence of an action." judge.py L289–296, rule 4: for negative checks, "yes" still requires "proof the agent did the correct thing, not just the absence of a visible violation"; rules 5/7 default to "no." With 90.1% of negative judgments LLM-dependent (A.4) and 41.8% of negative failures typed "Content missing" (Fig. 9), the finding that prohibitions are followed more consistently than boundary/termination requirements cannot be separated from the judging rule. The judge also imports a missing evidence_extractor.py on which rule 7 hinges, so the verdicts cannot be audited.

Questions for authors

Clarification (answers would change my assessment):

  1. How many of the 2,194 constraints concern gates, confirmations, setup or stop-and-wait behaviour that the forced prompt instructs agents to skip, and what are the category scores with those constraints excluded?
  2. How many of the 96 SKILL.md files were edited beyond the App. B normalization, and did task generation permit reusing examples contained in the skill? What are the results with task-coupled pairs removed?
  3. What are the negative/boundary/termination scores when a prohibition is judged satisfied by the absence of the forbidden action, i.e., under the Sec. 4.2 semantics?
  4. judge.py --task injects SKILL.md into the task-judge prompt (L58, L302), whereas Sec. 3.4 says the task judge runs "without access to the skill specification." Which was used for the 717 task-success verdicts?
  5. Sec. 3.4 says constraints without a verdict are excluded from numerator and denominator; judge.py L414–432 instead falls back to the script verdict with a fixed denominator (44 units are script-only). Which is correct?

Suggestions (do not affect my assessment): release the evidence extractor, all 96 skills, the task rubrics and execution records; state the harness asymmetry (--disallowedTools WebFetch vs. -s danger-full-access); distinguish execution failures (e.g., 12.5 s, 123-character output, scored 0/43) from non-adherence; remove the invisible draw.io interface text embedded in Fig. 1.

6 comments
AuthorsOpenReview ↗

Response to Reviewer C5EY — Part 1: Execution Setting

Thank you for examining the supplementary materials and confirming that the main numerical results are reproducible. Your concerns center on whether the execution setting, skill construction, and judging rules support the intended measurement. We have revisited these points and added skill-exclusion analyses and a judging-rule comparison.

Weakness 1(a) & Question 1: Unattended execution and stopping requirements

SkillIF runs without live user participation after a task starts. Given the supplied task materials and authorizations, we found no requirement among the reviewed items that necessitated a live question or wait for confirmation prohibited by the execution prompt.

Scope of the check. Our additional review covered the submitted roster of 96 skills and 2,194 final scoring constraints. We searched the scoring specifications and examined candidates concerning confirmation, missing inputs, authorization, and stopping conditions against the skill text and task instructions.

During construction, we had already conducted at least three compatibility reviews. We supplied task materials and advance authorization, revised skill instructions, and removed inapplicable scoring criteria where necessary. The external prompt prevents repeated confirmation requests from interrupting evaluation; it is part of the execution configuration, not a source from which scoring constraints are extracted.

Why stopping and process requirements remain applicable. Termination, Boundary, and Step cover stopping conditions, scope limits, and actions or their ordering. These requirements can often be satisfied without live interaction, such as specifying when an experiment should stop or completing a check before proceeding.

Two cases illustrate how the task conditions matter:

  • go-to-market-plan: C25 permits strategy generation when sufficient information is available or the user authorizes inference from the supplied materials. The task provides that authorization, so another confirmation is unnecessary. Reading the business context, assessing GTM readiness, analyzing the strategy, and checking the deliverable must still be supported by execution evidence. Authorization does not replace these steps.

  • sprint-planning: Team-member comments were supplied to the executor and were available for judging. The requirement checks whether the plan records those existing comments; it does not require contacting team members and awaiting new replies during the run. These materials were omitted from the public supplement because they could reveal sensitive information or author identities.

The public supplement therefore does not contain every input or piece of evidence used in the original evaluation. We will make the execution conditions, compatibility checks, and redaction scope more explicit so that readers can interpret the requirements in their actual task context.

AuthorsOpenReview ↗

Response to Reviewer C5EY — Part 2: Skill Construction

Weakness 1(b) & Question 2: Modifications, examples, and task relationships

Modification scope. All 96 skills received a common autonomous-execution policy. Three constrained variants also contain author-added workflow requirements: we developed them from mobile-games, client-intake, and server-management, adding specified process records, structured fields, resource-reading sequences, and verification records.

These variants, their corresponding originals, and the other skills underwent the same screening and construction process: normalization and rewriting, task construction, executability checks, constraint extraction, and scoring. They are evaluated as skill versions under the same protocol. We will explicitly identify their provenance and additions, distinguishing these additions from structural normalization.

Use of existing examples. The complete DevAnalytics example was already present in the upstream open-source version of go-to-market-plan. We subsequently constructed the task from the complete skill package, including that example. The shared business context follows from this construction order.

SkillIF measures adherence to the supplied specification. Templates and examples are available guidance, and reusing them is permitted when the result fits the task inputs and skill requirements. Originality is not a scoring criterion.

An example can help produce a deliverable, but it does not establish that the required process occurred in the current run. Go-to-market-plan still requires reading the business context, assessing GTM readiness, analyzing the strategy, and performing a final self-check. These requirements are assessed separately.

The marketing activities in the example are an action plan for the user; the skill explicitly states that the agent does not carry out those activities on the user’s behalf. This distinction separates the proposed marketing actions from the analysis and verification the agent must perform during evaluation.

Task and skill have different roles. The task supplies the concrete goal and inputs, while the skill supplies requirements for performing that kind of work. This distinction does not require a task to use a different company name or business context from a skill’s example.

Similarly, the dates and filenames in capture-triage identify inputs to be processed; they do not provide the processing results. Its unattended-execution adaptations concern the setting described in Part 1. We will clarify these task–skill relationships in the construction description.

Effect on the main observations. We recomputed scores from the submitted constraint-level data after excluding the identified cases. Scores are macro-averaged over skills within each configuration, then averaged equally across the 11 configurations.

Evaluation setSkillsOverallDeliverableProcedureRestraint
Original set9650.18%52.24%43.68%50.23%
Excluding go-to-market-plan and capture-triage9450.40%52.30%44.08%50.48%
Excluding the three constrained variants9350.29%52.43%43.34%50.42%
Excluding all five skills9150.51%52.50%43.74%50.69%

In all three exclusion settings, Procedure remains the lowest-scoring category in every configuration. Negative also remains higher than Boundary and Termination in every configuration. These recurring mean-score patterns therefore do not depend on the five identified skills.

AuthorsOpenReview ↗

Response to Reviewer C5EY — Part 3: Judging Prohibitions

Weakness 2 & Question 3: Sensitivity to the prohibition rule

We agree that, for a pure prohibition, a sufficiently complete relevant record showing that the prohibited action did not occur should support compliance. No additional action should be required unless the skill specifies it. This differs from a retrieval step returning no relevant evidence.

Paired rule comparison. To test the prompt’s effect directly, both arms used the currently available DeepSeek V4.1 Flash, identical frozen execution evidence, and the original judging pipeline. Only Rule 4, concerning prohibitions, was changed; mechanical verdicts and all other scoring settings were held fixed.

The revised rule accepts compliance when the relevant record is sufficient and the prohibited action did not occur. Explicit additional obligations remain in force, and insufficient evidence is not automatically treated as a pass.

The comparison covered all 11 configurations and the 94 skills with applicable requirements. The LLM components for Negative, Boundary, and Termination were rejudged in both arms.

MetricRevised rule − original rule95% CI for the difference
Negative+3.84%[1.58%, 6.17%]
Boundary+0.55%[−1.54%, 2.65%]
Termination+2.54%[−0.65%, 5.93%]
Restraint+2.43%[0.70%, 4.21%]

Differences use skill-macro averages within each configuration, followed by equal weighting of the 11 configurations. Confidence intervals use a paired bootstrap clustered by skill.

The revised rule increases the Negative score, but Negative remains higher than both Boundary and Termination in all 11 configurations under both rules. The wording affects absolute scores, while the reported category mean pattern remains present under the interpretation you suggested.

How the existing checks divide the requirements. In capture-triage, moving the files, checking their final location, and prohibiting deletion have separate scoring items:

ConstraintWhat is checkedSubcategory
C22Whether processed files are moved into ProcessedStep
C23Whether all five files are in Processed at the endEvidence
C24No deletion of original files; the text also describes preserving them by moving themNegative

The deterministic component of C24 uses a negative match: it passes when rm is not matched in the tool log, without requiring an affirmative action. The final verdict also includes an LLM component. The comparison above therefore holds mechanical checks fixed and directly tests the LLM prompt.

Interpreting Content missing within Negative. Negative requirements do not always prohibit a tool action. For example, “Do not give generic advice; each strategy must include a concrete plan tailored to the current product” requires examining whether the output contains the specified detail.

Whether Content missing is appropriate depends on the actual requirement; the label alone does not establish a contradiction. We will clarify these judgment conditions and provide further information on evidence sources and extraction where disclosure is permitted.

Reviewer_WmtKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new benchmark for evaluating AI agents. Overall, the study is well-designed with complete experiments, and focuses on an important research topic. However, some details can be improved. Technically, the paper presents "SkillIF," a benchmark designed to assess whether an agent adheres to "reusable skill specifications." The benchmark comprises 96 curated skills, which are from GitHub repositories and cover ten distinct domains. SkillIF treats "adherence to skill specifications" as an independent evaluation metric, distinct from "task completion." Through experiments across 11 "model–harness" configurations, the study proves that "process adherence" is a common weakness among AI agents.

Critical strengths
  1. The paper proposes a systematic evaluation framework applicable to a wide range of agent configurations. The benchmark includes 96 "skill–task" pairs, each equipped with an executable task and relevant input materials. Task instructions clearly define specific objectives, while process requirements are integrated into the skill specifications, enabling the independent assessment of "task completion" versus "skill adherence."

  2. The paper highlights the limitations of relying solely on "task success rate" as an evaluation metric. For instance, when tasked with creating a "go-to-market plan," an agent might satisfy all 7 task-level criteria while meeting only 2 out of 16 specified skill constraints.

Critical weaknesses
  1. Each skill is mapped to only one task, and every "skill–task" pair is executed only once for each "model–harness" configuration.

  2. The current adherence metric treats "deviations from a prescribed format" and "substantive omissions in the process" as equivalent failures; this ambiguity makes the practical implications of a "low total score" somewhat unclear.

Questions for authors

Is there a clear distinction between a process "not being executed at all" and a situation where an evaluator is unable to make a judgment due to "insufficient evidence"?

2 comments
AuthorsOpenReview ↗

Response to Reviewer WmtK

Thank you for your positive assessment of SkillIF’s design and research question. We appreciate your suggestions and have added experiments and clarifications to address each concern.

Weakness 1: Task design and execution variability

Each task is constructed from the complete skill package and checked through trial runs. Task instructions provide objectives and inputs, while skill specifications define workflows and behavioral requirements. This lets us assess task completion and skill adherence separately on the same execution. We agree that additional tasks would help test sensitivity to different inputs, and plan to add such instances with comparable skill requirements in future open-source releases.

To measure variation across executions, we ran all 96 tasks independently three times with GLM5.3Flash, totaling 288 executions. Overall scores were 57.59%, 57.22%, and 57.28%. We calculated the sample SD across the three scores for each task: the mean was 8.79% (skill-bootstrap 95% CI: [7.59%, 10.02%]), and the median was 7.22%. Thus, individual task scores varied, while this configuration’s benchmark-wide means were close and Procedure remained the lowest-scoring category in every round.

Weakness 2: Interpreting low adherence scores

Your concern highlights the distinction between satisfying a requirement and the practical consequences of violating it. Overall measures the fraction of assessed requirements satisfied within each skill, averaged across skills. Equal scoring weight does not imply equal practical impact. We therefore also report category scores and failure forms to show which requirements were missed and how. These distinctions matter because, for example, a format error may affect readability or prevent downstream processing.

To examine how much wording failures contribute to low scores, we conducted a sensitivity analysis. Format is a requirement type, whereas Wrong wording is a failure form that can occur across categories; this analysis waives the latter. In a fixed cohort of 298 task-successful runs originally below 50% adherence, we recoded all 990 Wrong wording failures as passes, keeping other verdicts, scoring denominators, and cohort membership unchanged.

Adherence scoreOriginalWrong wording waived
Overall34.29%48.76%
Deliverable35.60%51.92%
Procedure29.86%38.94%
Restraint39.15%49.75%

Scores are averaged over skills within each configuration, then equally across the 11 configurations. Scores rose substantially, but Procedure remained lowest, and 3,357 other failed judgments remained (77.2% of the original failures). In this cohort, task success therefore still coexists with unmet skill requirements beyond wording mismatches.

Question: Non-execution and insufficient evidence

We agree that these situations need to be distinguished. An absent verification report violates a requirement to provide that report. Missing execution logs, however, do not establish that an action never occurred. The original Step never ran definition refers to the absence of evidence of the prescribed step in the execution record (Appendix E, Figure 9). Appendix A.3 likewise limits judgments to available records. We will make this scope clearer, including the fact that the seven failure forms do not separately label insufficient evidence.

To check whether the classifications were supported by the records, we sampled 40 Step never ran judgments across 30 skills and all 11 configurations. After the final review, both annotators supported 39 original classifications (97.5%) and identified execution evidence in one case (2.5%). Final-review agreement was 100% (Cohen’s κ = 1.000).

That case concerned the requirement to assess GTM readiness item by item before deciding whether to ask questions. The agent’s recorded reasoning assessed pricing, the competitive landscape, and distribution, but the original automated check searched only tool logs and missed this evidence. The skill also did not specify whether the assessment had to be presented to the user or how its completion should be documented.

In future skill versions, we will clarify whether such assessments must be shown to the user and which checks or missing information should be reported. We will also update the checks to examine the appropriate records and correct confirmed evaluation errors.

We hope these results and clarifications address your concerns, and appreciate your thoughtful feedback.

Thanks for your response. I maintain my rating.

Reviewer_a2ymOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces SkillIF, a benchmark for studying adherence to reusable skill specifications that evaluates task completion and skill adherence separately on the same agent execution. The authors pair 96 skills across ten domains with executable tasks, derive 2,194 assessable constraints organized under a shared 3-category, 9-subcategory taxonomy, and judge each constraint with deterministic checks and an LLM judge, enabling comparison across skills and diagnosis of individual unmet requirements. The evaluation across eleven model-harness configurations identifies recurring weaknesses (procedural requirements are weakest everywhere) and examines successful runs with low adherence through a failure-form analysis. Overall, the paper addresses a timely and well-motivated problem, but would strongly benefit from a larger evaluation set and stronger validation of the benchmark and evaluation, including statistical analysis of the main results and human validation of the automatically constructed constraint set.

Critical strengths
  1. Timely and interesting problem scope: The authors study an important question for agents: adherence to skills, which is an important requirement separate from task completion.
  2. Useful taxonomy for cross-skill comparison. The authors classify constraints under an intuitive and useful 3-category, 9-subcategory taxonomy (Deliverable: content/format/coverage; Procedure: step/evidence/tool; Restraint: negative/boundary/termination).
  3. Thorough reliability reporting. The paper uses three independent judging passes, a substitute judge, a blinded two-rater manual re-read of 600 judgments with a third adjudicator (κ = 0.84 on failure form), split-half reliability over skills, subsampling curves for the two central findings, and leave-one-out analysis of the harness comparison. The paper is also discusses which findings are stable and which are not.
Critical weaknesses
  1. The constraint set is derived with a model and is not human-validated. Skill screening, normalization, task generation, constraint extraction, subcategory labelling, and deduplication (1,090 of 3,284 candidates dropped) are all LLM-driven. The manual re-read in Appendix A.7 validates verdicts given a constraint, not whether the extracted constraints are faithful to the skill or correctly labelled, and the AI Use Statement's note that constraints "were checked against their source skills" gives no numbers. Since the headline Procedure-versus-Deliverable finding is a claim about those labels, human validation with reported agreement is needed.
  2. The benchmark is small, and no main comparison is tested for significance. Each skill has exactly one task and one execution per configuration, giving 96 instances per configuration and thin per-subcategory coverage (termination: 38 constraints in 34 skills; tool: 47 in 33). Uncertainty is reported in the appendix (skill-bootstrap intervals of about ±4 points in Figure 4a, standard errors in A.5, clustered CIs in F.3) but is never applied to the comparisons the main text draws: Table 1 gives point estimates with no intervals, the six leading configurations span about 2 points, and no test is reported for configuration rankings, harness differences, or domain orderings. Run-to-run execution variance is not estimated at all. The authors should enlarge the effective test set and report significance for the main results.
  3. The judge is one of the executors, permitting self- or same-family preference. DeepSeek V4 Flash under Claude Code is the top configuration under the DeepSeek judge (54.7). Under the Qwen3.8-Flash substitute, DeepSeek drops to 51.7 and Qwen3.8-Max becomes the top configuration (53.5, the smallest shift of any row). The authors note the judge is blind to configuration identity and that tiers are unchanged, but the pattern is still what judge-family preference would look like. A third judge from outside the executor set, or a per-executor breakdown of the shift, would resolve this.
Questions for authors
  1. How were constraint extraction and subcategory labelling validated? Please report agreement with human annotators on a sample of the 2,194 constraints, including whether dropped candidates should have been dropped.

  2. Please add confidence intervals to Table 1 and paired significance tests (e.g., permutation or sign tests paired over skills) for the configuration rankings, the harness differences in Figure 3a, and the domain and subcategory orderings. Which of the comparisons discussed in Section 4.2 survive?

  3. Can the authors re-judge with a model from outside the executor set, or report the judge-substitution shift separately for DeepSeek and Qwen executors versus the others, to address the preference pattern in Table 5?

  4. What are the Overall and per-category scores under LLM-only judging, and with wrong-wording failures recoded as passes?

6 comments
AuthorsOpenReview ↗

Response to Reviewer a2ym — Part 1: Human Validation of Constraint Construction

Thank you for your constructive comments. We have added human validation, statistical comparisons, repeated executions, an external judge, and scoring sensitivity analyses. We report the main results below and will incorporate the necessary clarifications into the manuscript.

Weakness 1 & Question 1: Human validation of constraint construction

The new audit supports the reproducibility of the broad classification: inter-annotator agreement was 97.0%, with κ = 0.952. It also directly examines source fidelity and removal decisions, beyond the execution-verdict review in Appendix A.7.

Human supervision was already part of requirement identification and classification; the original manuscript did not sufficiently quantify it. Construction also included skill and task trial runs and rule-based decomposition. The added audit provides the requested quantitative evidence.

Retained constraints. We sampled 300 retained constraints from 92 skills and 100 removed candidates from 58 skills, covering 95 distinct skills. Sampling used a fixed seed and nine-subcategory stratification, with higher sampling fractions for smaller subcategories. Two annotators consulted the complete normalized skill packages, resources, and shared definitions. The results are their revised judgments after clarification to consider the full skill context; they are not adjudicated gold labels.

CriterionAnnotator AAnnotator B
Supported by source specification288 of 300 (96.0%)295 of 300 (98.3%)
Matches original broad category296 of 300 (98.7%)295 of 300 (98.3%)
Matches original subcategory283 of 300 (94.3%)278 of 300 (92.7%)
Suitable as an independent scoring item296 of 300 (98.7%)297 of 300 (99.0%)

Agreement with original labels measures correspondence, not gold-standard accuracy. Inter-annotator agreement was:

CriterionAgreementCohen's κ
Source fidelity97.0% (291 of 300)0.462
Three broad categories97.0% (291 of 300)0.952
Nine subcategories89.0% (267 of 300)0.870

Fidelity judgments concentrated on “supported,” accounting for the difference between raw agreement and κ. These percentages describe the stratified audit sample without population weighting.

Of 33 subcategory disagreements, 24 stayed within the same broad category. Relative to the original labels, A changed two Procedure labels to Deliverable; B changed three Deliverable labels to Procedure and one in the reverse direction. This directly addresses the classification underlying the Procedure–Deliverable comparison.

Removed candidates. The 1,090 removals from 3,284 candidates addressed both semantic duplication and excessive granularity. Since constraints are equally weighted within a skill, splitting one requirement into many separately scored details can give it disproportionate weight. The selection stage therefore considered whether a candidate warranted independent scoring; specificity or brevity alone was not a reason for removal.

AssessmentAnnotator AAnnotator B
Removal justified: semantic duplication1218
Removal justified: too fine-grained for separate assessment6971
Retention recommended1911
Total reviewed100100

The annotators endorsed 81% and 89% of sampled removals, with 70 candidates endorsed by both. These results provide direct human support for most sampled removal decisions. Most endorsed removals concerned granularity, explaining why the discarded set should not be described solely as duplicates.

AuthorsOpenReview ↗

Response to Reviewer a2ym — Part 2: Statistical Support and Execution Variability

Weakness 2 & Question 2: Statistical support and execution variability

We added intervals, paired tests, and 288 new executions. Specific configuration and requirement differences are supported; the leading configurations' fine ordering remains unresolved.

Table 1 and configuration comparisons. Overall CIs use 10,000 skill-bootstrap resamples. Tests are two-sided, skill-paired permutations with Holm correction within each stated family. CIs are pointwise; the statistical unit is the skill.

ModelHarnessOverall (%)95% CI
Claude Opus 5Codex53.0[48.88, 56.75]
Claude Opus 5Claude Code53.1[49.13, 56.87]
DeepSeek V4 FlashCodex48.7[44.57, 52.85]
DeepSeek V4 FlashClaude Code54.7[50.86, 58.37]
GLM 5.2Codex54.3[50.49, 58.12]
GLM 5.2Claude Code53.3[49.41, 57.12]
GPT-5.6 LunaCodex48.8[44.91, 52.67]
GPT-5.6 LunaClaude Code49.3[45.49, 53.14]
Grok 4.5Codex40.7[36.61, 44.57]
Grok 4.5Claude Code42.1[38.08, 46.20]
Qwen3.8-MaxClaude Code54.0[50.15, 57.84]

Across all 55 configuration pairs, 12 differences survived correction: each of the six leading configurations exceeded each Grok configuration. No pair within the leading six survived. Thus, the evidence supports larger differences, not a precise ranking within this closely scored group.

Figure 3a. DeepSeek V4 Flash scored 5.97% higher under Claude Code, 95% CI [1.69, 10.24], Holm-adjusted p = 0.0393 across five same-model harness comparisons. The other four were not significant after correction. This supports the model-specific observation. The difference did not survive the separate 55-pair correction; we distinguish the testing families.

Category comparisons. Procedure was lowest in all 11 configurations and Procedure−Deliverable remained negative after matching skills. The four comparisons used 94 skills for Procedure−Deliverable, 92 for Procedure−Restraint, 62 for Negative−Boundary, and 30 for Negative−Termination. After correcting all 44 tests, three were significant:

Configuration and comparisonDifference (%)95% CIAdjusted p
GPT-5.6 Luna & Codex: Procedure - Deliverable-12.39[-18.85, -5.71]0.0202
Grok 4.5 & Codex: Procedure - Deliverable-15.45[-22.94, -8.08]0.0073
Grok 4.5 & Codex: Negative - Boundary+24.78[13.87, 35.83]0.0035

Procedure−Restraint was negative in all configurations; Negative−Boundary was positive in all, and Negative−Termination in ten. No additional comparisons survived the 44-test correction. We distinguish these recurring mean patterns from individually significant differences.

Subcategories. Averaging configurations within each skill, five differences survived correction across all 36 pairs:

ComparisonPaired skillsDifference (%)95% CIAdjusted p
Content - Step83+11.39[5.59, 17.13]0.0066
Content - Evidence74+13.35[7.69, 19.02]0.0018
Format - Evidence67+13.35[5.75, 20.89]0.0358
Negative - Step77+15.94[9.06, 22.92]0.0021
Negative - Evidence68+17.96[9.63, 26.12]0.0037

These identify Step and Evidence as specific procedural weaknesses. The domain figure is descriptive, not a difficulty ranking: different skills and requirement compositions offer no natural one-to-one pairing.

Scale and task design. Each skill has a task constructed to exercise its workflow, with identical inputs across configurations. Appendix A.8 sampled 500 subsets per size: Procedure remained lowest in 99% of 50-skill subsets and every sampled subset at tested sizes of 60 or more. The success–adherence gap stayed within 5% of its full-set value in 98% of 50-skill subsets. This supports stability across subsets of the existing collection.

Repeated executions. GLM5.3Flash independently executed all 96 tasks three times:

Adherence (%)Run 1Run 2Run 3
Overall57.5957.2257.28
Deliverable59.0757.8357.77
Procedure54.9855.4954.89
Restraint58.2659.1659.77

The mean within-task sample SD was 8.79%, with skill-bootstrap 95% CI [7.59, 10.02]; the median was 7.22% and the 90th percentile 17.91%. Overall round means differed by at most 0.36% before rounding, and Procedure was lowest each time. These results directly quantify task-level variation and show recurrence of the aggregate pattern.

Averaging executions within skills gave Procedure−Deliverable = −3.43%, 95% CI [−9.68, 2.76]: the direction recurs, but this additional category difference is not significant. Repeated executions measure variation on fixed tasks; they do not add distinct task instances.

Score changes are absolute differences, not relative percentage changes.

AuthorsOpenReview ↗

Response to Reviewer a2ym — Part 3: Full Re-evaluation with an External Judge

Weakness 3 & Question 3: Re-evaluation with an external judge

Following your suggestion, we used Kimi K3.1 Pro, outside every evaluated executor family, to re-evaluate all 11 configurations across all 96 skills. The same six configurations remained the leading group, and DeepSeek & Claude Code again had the highest Overall point estimate.

We agree that hiding configuration identities alone cannot exclude output-style preferences. This new comparison retains the constraints, execution evidence, evidence processing, deterministic checks, and other settings, replacing only the hybrid evaluator's LLM judge. The execution records are unchanged.

ExecutorHarnessDeepSeek judgeQwen judgeKimi judge
Claude Opus 5Codex53.050.054.6
GLM 5.2Codex54.348.855.5
DeepSeek V4 FlashCodex48.746.151.0
GPT-5.6 LunaCodex48.845.950.9
Grok 4.5Codex40.739.041.7
Qwen3.8-MaxClaude Code54.053.556.0
DeepSeek V4 FlashClaude Code54.751.756.2
GLM 5.2Claude Code53.350.953.9
Claude Opus 5Claude Code53.150.754.1
GPT-5.6 LunaClaude Code49.347.350.6
Grok 4.5Claude Code42.138.943.1
Mean across configurations—50.247.551.6

Kimi increased mean Overall by 1.41%, calculated before rounding. The same six leading configurations and two lowest-scoring Grok configurations were retained. 48 of 55 configuration pairs kept their order, with Kendall's τ = 0.745.

For the configuration you highlighted, DeepSeek V4 Flash & Claude Code scored 56.2% under Kimi, followed by Qwen3.8-Max at 56.0%. Its highest point estimate therefore recurs under a judge with no executor-family match, providing evidence against an explanation based solely on self-judging. We do not interpret the 0.2% gap as significant superiority.

Absolute scores do depend on judge choice: Qwen lowers all configuration means, whereas Kimi raises them. Kimi and the original judge disagreed on 13.0% of jointly scored constraint judgments. The external comparison therefore supports the broad configuration grouping while documenting changes in absolute scores and fine ordering. It directly supplies the external-judge check requested in your review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.