acceptodds
Under review as a conference paper at ICLR 2027

SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications

Abstract

Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability. Conducting such an evaluation demands agentic tasks that reconcile broad domain coverage with deep procedural complexity. Existing paradigms face a fundamental trade-off: expert-curated benchmarks achieve procedural depth but suffer from domain diversity bottlenecks, whereas crowdsourced benchmarks span diverse scenarios but lack execution rigor. To resolve this tension, we present , which leverages agent skills as unifying execution blueprints across three co-designed stages: (1) Resource Preparation uses domain-balanced skills to guide artifact retrieval, establishing broad workflow coverage; (2) Skill-driven Task Synthesis employs skills as procedural anchors to partition artifact roles, synthesize naturalistic task instructions, and package sandboxed scenarios for deep procedural execution; and (3) Holistic Evaluation Design translates skill-embedded requirements into fine-grained completion and process rubrics to enable strict requirement-level scoring. Comprising 842 sandboxed tasks across five major professional domains, reveals a steep capability hierarchy among eight mainstream agents, led by Opus 4.8 ( strict completion). Requiring all constrains satisfaction uncovers severe capability bottlenecks across all model tiers that remain masked under individual rubric pass rates ( to vs. to ). Crucially, under the external guidance from skills, top-tier agents obtain larger gains than low-tier agents (i.e., for Opus 4.8 vs. for Haiku 4.5). Finally, agents satisfy process rubrics far less frequently than completion rubrics (under for Haiku 4.5), proving that completion-only checks overestimate real-world agent reliability.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Evaluation, Agent Skills, Long-Horizon Agents, Agentic Benchmarks, Task Synthesis, Rubric-Based Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MVWNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SkillForgeBench to evaluate how reliably LLM agents follow outcome and procedural requirements in complex professional tasks. It addresses a trade-off in existing benchmarks between domain coverage and procedural depth.

The authors build 842 sandboxed tasks across five domains using a three-stage pipeline. They first curate agent skills and retrieve real-world materials, then use these resources to construct executable tasks. Finally, they develop completion and process rubrics by comparing agent executions with and without skill guidance, followed by human expert review.

The authors evaluate eight models using a shared OpenHands framework, with rubrics scored by an LLM judge. While average completion rubric pass rates range from 70.0% to 85.7%, strict task completion rates are much lower, at 12.6% to 40.9%. The authors report that stronger models gain more from skill guidance under strict scoring and that satisfying process requirements remains particularly challenging. An independent human audit reports high task and rubric quality, with 95.9% of paired ratings differing by no more than one point.

Critical strengths

The paper uses agent skills as "unifying blueprints across the task lifecycle". Figure 1 illustrates this with a citation verification task that separates editable files, read-only context, and withheld reference evidence. The pipeline produces 842 executable tasks across five domains, and the independent human audit gives task quality ratings of 4.56 and 4.71 out of 5 on a 100-task sample (Table 3). Appendix E also provides detailed construction prompts, making the pipeline easier to inspect and reproduce.

Separating completion and process rubrics allows the benchmark to evaluate both final outputs and adherence to required procedural steps. With skill access, Gemini 3.6 Flash achieves 18.4% strict completion but only 5.11% strict process compliance, while Opus 4.8 reaches 40.9% and 27.7%. These results support the authors' argument that "completion-only checks overestimate real-world agent reliability." The breakdown into six rubric categories also helps identify specific failure modes.

Critical weaknesses

The benchmark relies on an LLM judge to evaluate final workspaces and execution trajectories, but the reliability of its scoring is not sufficiently established. Table 3 assesses task and rubric quality without comparing the judge's pass/fail decisions against human evaluations. The paper also does not specify the evaluation judge model.

Figure 2 reports larger absolute gains from skill guidance for stronger models, with 19.2 percentage points for Opus 4.8 and 5.8 for Haiku 4.5. However, loose-score gains are similar across models, ranging from 5.1 to 7.4 points, and the relative strict gains for Opus and Haiku are also close at 89% and 86%. Since strict scoring requires all criteria to pass, baseline performance and the number of criteria may contribute to these differences. A baseline-adjusted analysis would help clarify whether stronger models benefit more from skill guidance, as the authors conclude.

Questions for authors

Could the authors clarify which judge model was used for the main evaluation? Have the judge’s pass/fail decisions been compared with human judgments on a sample of agent executions, particularly for process rubrics?

Could the authors clarify whether stronger models still benefit more from skill guidance after accounting for baseline performance and the number of criteria per task? A comparison using relative gains or baseline-adjusted strict scores would help clarify the findings in Figure 2.

3 comments
AuthorsOpenReview ↗

Response to Reviewer MVWN (1/1)

We sincerely thank the reviewer for the careful reading and the constructive feedback. We are glad that the reviewer found value in using agent skills as unifying blueprints across the task lifecycle, in the 842 executable tasks with detailed construction prompts, and in the separate completion and process rubrics that evaluate both final outputs and required procedural steps.

Q1: Human agreement with LLM-as-judge.

Could the authors clarify which judge model was used for the main evaluation? Have the judge’s pass/fail decisions been compared with human judgments on a sample of agent executions, particularly for process rubrics?

Our judge for the main evaluation is GPT-5.5 with xhigh reasoning effort, run in Codex. As detailed in General Response 1 (GR1), the judge agrees with human judgments on 96% of decisions, and changing the judge model does not change model rankings.

Q2: Accounting for Checklist Length and Baseline Performance

Could the authors clarify whether stronger models still benefit more from skill guidance after accounting for baseline performance and the number of criteria per task? A comparison using relative gains or baseline-adjusted strict scores would help clarify the findings in Figure 2.

Assume each criterion passes independently with probability pp, where pp is the loose score without skills. Under this assumption, the expected strict score of a task with nn criteria is pnp^n. With skills, the loose score rises to p+Δp+\Delta, and the expected strict score becomes (p+Δ)n(p+\Delta)^n. We define the expected strict gain as

G^=1T∑t=1T[(p+Δ)nt−pnt],\hat G = \frac{1}{T}\sum_{t=1}^{T}\left[(p+\Delta)^{n_t} - p^{n_t}\right],

where TT is the number of tasks and ntn_t is the number of criteria in task tt. G^\hat G depends only on the loose score pp, the loose gain Δ\Delta and the number of criteria per task.

In practice, the measured strict scores ScompNSS_{\mathrm{comp}}^{\mathrm{NS}} and ScompWSS_{\mathrm{comp}}^{\mathrm{WS}} differ from these expected values. We write the actual strict gain as G=ScompWS−ScompNSG = S_{\mathrm{comp}}^{\mathrm{WS}} - S_{\mathrm{comp}}^{\mathrm{NS}} and define the excess strict gain as

E=G−G^.E = G - \hat G.

EE is the part of the actual strict gain that the loose scores do not predict. E>0E > 0 measures the extent to which skills help the model more than mathematics expects.

We compute EE for all 8 models:

ModelScompNSS_{\mathrm{comp}}^{\mathrm{NS}}GGG^\hat GEE
Opus 4.821.7+19.2+12.2+7.0
GLM-5.220.0+16.8+12.5+4.3
Sonnet 4.615.0+13.5+9.2+4.3
Hy314.0+13.3+9.6+3.7
Kimi K2.611.2+8.9+6.8+2.1
Gemini 3.6 Flash8.9+9.5+8.8+0.7
Haiku 4.56.8+5.8+3.7+2.1
Qwen3.5-397B5.3+8.0+3.8+4.2

The correlation with ScompNSS_{\mathrm{comp}}^{\mathrm{NS}} is r=0.70r = 0.70 for the excess gain EE. This result shows a strong correlation, although rr drops from 0.970.97 for the raw gain GG. Therefore, after accounting for the baseline and the number of criteria, stronger models still gain more from skills, which means they use skills better.

In the revised paper, we have added the human-agreement and judge-sensitivity results to Appendix E (Tables 8–10).

thanks for sharing. i keep score 3

Thank you so much for your thoughtful review! Your feedback helped us improve the experiments in our paper. We truly appreciate the time you devoted to reviewing our work.

Reviewer_PoeeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

SkillForgeBench is a benchmark dataset comprising 842 sandboxed agent tasks, each derived from a unique agent skill across five domains. Its construction process utilizes skill retrieval to identify real-world artifacts, assign roles to them within the task environment, synthesize instructions, and derive task completion and process evaluation criteria based on the performance of a reference agent with and without access to the skill. The scoring criteria were subsequently calibrated by human experts. The reported results show there is a significant gap exists between partial and full compliance with constraints. For more powerful models, skill access leads to a greater absolute improvement in strict task completion, and there is a discrepancy between result compliance and process compliance.

Critical strengths
  1. The paper constructs a benchmark comprising 842 tasks that integrates skills, real-world file resources, and an executable environment, thereby offering significant value in terms of data resources.
  2. It designs process-level and outcome-level evaluations and compares performance with and without skills; the evaluation metrics are designed to be quite comprehensive.
Critical weaknesses
  1. Although the paper combines process-level and outcome-level rubrics, I remain concerned about the reliability of the evaluation. It relies almost entirely on LLM judges, without clearly explaining the use of rule-based or execution-based checks, even for requirements that can be directly checked using rules or tests. This may affect scoring consistency, especially under strict scoring, where a single incorrectly judged criterion can cause an entire task to fail.
  2. The paper emphasizes evaluation across multiple domains, but only reports overall results. Neither the main text nor the appendix provides results or analysis for individual domains. This makes it difficult to understand how models perform across domains and whether the main findings hold across them, leaving the paper’s motivation for broad domain coverage insufficiently supported.
  3. The advantages over related benchmarks, such as Claw-Eval, SkillsBench, and Agentic Skills at Scale, are not clearly demonstrated. Does the proposed pipeline produce higher-quality tasks, or mainly cover a wider range of domains? Concrete task examples and direct comparisons would help clarify this distinction. From the current presentation, it is difficult to see how the generated tasks reflect real user needs and what specific benefits the pipeline provides over existing approaches.
Questions for authors
  1. Why does the evaluation rely almost entirely on LLM judges, including for requirements that can be checked using rules or tests? Can the authors compare judge decisions with human judgments or execution-based checks on a representative subset, and examine whether judging errors affect strict scores or model rankings?
  2. Can the authors report model performance and gains from skill access for each domain? Do the main findings hold across domains, or are they mainly driven by particular domains?
  3. Can the authors provide concrete task examples and comparisons with Claw-Eval, SkillsBench, and Agentic Skills at Scale to clarify the pipeline’s advantages? Does it produce more realistic or higher-quality tasks, or mainly broader domain coverage? How do these examples reflect real user needs? I would be willing to raise my rating if these examples clearly demonstrate the benchmark’s advantages or unique value compared with prior work.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Poee (1/3)

We sincerely thank the reviewer for the careful reading and the constructive feedback. We are glad that the reviewer found value in the 842-task resource and in the combined process-level and outcome-level evaluation. Q1, Q2, and Q3 correspond to W1, W2, and W3, so our answers to the questions also address the weaknesses. This comment answers Q1 and Q2. The next comment answers Q3. The final comment provides the full example task.

Q1: LLM-as-judge sensitivity

Why does the evaluation rely almost entirely on LLM judges, including for requirements that can be checked using rules or tests? Can the authors compare judge decisions with human judgments or execution-based checks on a representative subset, and examine whether judging errors affect strict scores or model rankings?

To keep scores comparable across tasks and domains, we use LLM judges even for criteria that rules or tests can check. In a study of 50 tasks, the judge agrees with human judgments on 96% of decisions. Across both process and completion rubrics, strict scores from the judge closely align with human judgments, and model rankings remain identical under an alternative judge model family (Sonnet 5.5).

Comparable scores across tasks and domains require LLM judges for every criterion. Three of our five domains (Business, Creative, and Knowledge) contain many writing and analysis tasks. Rules or tests cannot score these tasks, so these tasks need LLM judges. If rules or tests were applied to the criteria that they can check, the scoring method would differ between tasks and between domains. A score difference could then come from the scoring method rather than from the agent.

We acknowledge the inherent limitations of LLM-as-judge, so we carefully tune the judge prompts to make judging stable and robust. As detailed in General Response 1 (GR1), the judge agrees with human judgments on 96% of decisions, and changing the judge model does not change model rankings.

Q2: Findings hold across domains

Can the authors report model performance and gains from skill access for each domain? Do the main findings hold across domains, or are they mainly driven by particular domains?

We compute every Section 4 analysis within each of the five skill domains (Technology 257, Business 211, Creative 136, Harness Behavior 125, Knowledge 113 tasks), and the model rankings and main findings hold in all five, so they are not driven by any particular domain. We have added the strict and loose per-domain scores and skill gains of all eight models to Appendix G (Tables 14–16) of the revised paper.

Rankings are stable across domains. Absolute scores vary with domain difficulty. For example, Opus 4.8 reaches 24.9% strict ScompWSS_{\text{comp}}^{\text{WS}} in Technology and 62.8% in Knowledge. The model ranking, however, changes little. Kendall's τ\tau between each domain's strict ranking and the overall ranking is 0.79–0.93, so only 1–3 of 28 model pairs swap. In every domain, Opus 4.8 and GLM-5.2 are the top two models, and Qwen3.5-397B and Haiku 4.5 are the bottom two.

The main findings hold in all five domains.

  1. Holistic constraint satisfaction. In every domain, strict scoring widens the relative gap between models: the top model scores 3.56–4.10× the bottom model under strict ScompS_{\text{comp}}, versus 1.17–1.29× under loose ScompS_{\text{comp}}.
  2. Skill grounding. Skills raise ScompS_{\text{comp}} for every model in every domain, and under strict scoring the top two models gain 4.9–16.8 pp more than the bottom two (95% task-bootstrap intervals exclude zero), whereas under loose scoring the two groups gain about equally (−0.2 to +4.0 pp).
  3. Process vs. completion. The five lower-ranked models score lower on loose SprocS_{\text{proc}} than on loose ScompS_{\text{comp}} in every domain except Harness Behavior, where Kimi K2.6 and Haiku 4.5 show no gap (+0.8 and +0.3 pp), and strict SprocS_{\text{proc}} is below strict ScompS_{\text{comp}} in 37 of the 40 model–domain pairs.
  4. Efficiency. In every domain, the mean number of steps is only weakly correlated with strict ScompS_{\text{comp}} across models (Pearson rr = 0.09–0.29), so longer trajectories do not reliably yield higher scores.
  5. Release date. In every domain, all three scores remain positively correlated with release date.
  6. Rubric categories. In every domain, Opus 4.8, Hy3, and Gemini 3.6 Flash differ little on Inspectable Deliverable but more on each process category.

In the revised paper, we have added the human-agreement and judge-sensitivity results to Appendix E (Tables 8–10) and the per-domain scores and skill gains to Appendix G (Tables 14–16).

AuthorsOpenReview ↗

Response to Reviewer Poee (2/3)

Q3: Methodological Advances over Prior Work (Claw-Eval, SkillsBench, and ASS)

Can the authors provide concrete task examples and comparisons with Claw-Eval, SkillsBench, and Agentic Skills at Scale to clarify the pipeline’s advantages? Does it produce more realistic or higher-quality tasks, or mainly broader domain coverage? How do these examples reflect real user needs? I would be willing to raise my rating if these examples clearly demonstrate the benchmark’s advantages or unique value compared with prior work.

SkillForgeBench can expand domain coverage efficiently, because every new skill written by human experts can automatically originate a new task. In addition, SkillForgeBench produces more realistic tasks by searching realistic environments. We show an example at the end about how our pipeline achieves this.

Related benchmarks lack realistic environments or scalable coverage.

  • Claw-Eval has a different focus: it tests whether an agent completes everyday workflows through mock services, such as email triage. The environments are simple.
  • SkillsBench tests whether skills help agents on tasks that experts write by hand. Due to costly expert writing, SkillsBench has only 87 tasks.
  • Agentic Skills at Scale (ASS) is concurrent work that also synthesizes tasks from skills. ASS does not search real user queries. Its environments are shallower and do not reflect the computer of a real user at work.

Our environments are more varied. We compare our 842 tasks with all tasks of ASS (1,110), Claw-Eval (300), and SkillsBench (87) by counting the input files given to the agent.

Measure (mean / median)OursASSClaw-EvalSkillsBench
Files per task172.9 / 101.6 / 11.8 / 18.4 / 2
File types per task9.5 / 41.0 / 10.9 / 12.0 / 2
Size per task13.7 MiB / 3.4 MiB19.5 KiB / 1.1 KiB9.3 MiB / 7.1 KiB5.8 MiB / 139.3 KiB
Tasks with no input files0.0%36.3%25.0%4.6%

Our prompts state more requirements. Keyword rules count two kinds of requirements: What requirements on the deliverable and How requirements on the way of working.

Measure (mean)OursASSClaw-EvalSkillsBench
What requirement types (0–2)0.980.930.421.24
How requirement types (0–5)3.101.751.001.43
How requirement hits per 100 words3.741.772.031.25

Example: how one skill becomes a realistic task.

nickel-concentration-risk-analyzer is a skill teaching how to analyze nickel supply risk. The procedure combines production data from several sources, measures how much of world supply one country controls, and estimates how a government policy would change world supply.

We retrieve authoritative real-world documents, such as official government statistics and industry research reports. The task instruction requires the agent to consolidate complex, multi-source evidence into an institutional-grade research report. This realistic environment and instruction demand real-world analytical rigor.

Specifically, the work of a commodities analyst centers on evaluating supply risks by combining market concentration data with policy interpretation. Our task thus requires computing market concentration indices like HHI, simulating multi-layer policy shocks, and verifying industry supply claims. Real data and professional workflow together construct the realistic work of analysts.

Furthermore, our tasks enforce realistic user constraints, such as resolving data imperfections during analysis. We do not intentionally search low-quality data. Rather, data imperfection is an inherent feature of the real world, which our search faithfully preserves. For example, when government statistics end in 2023 while industry reports cover 2024 and 2025, our instruction imposes a realistic constraint: the agent must cross-check the mismatched periods and flag inconsistencies with explicit warnings. We provide the complete worked task card in Appendix H.1 (Figure 10) of the revised paper.

In the revised paper, we have added this example task to Appendix H.1 (Figure 10) and its comparison with the closest tasks in Claw-Eval, SkillsBench, and ASS to Appendix H.2 (Table 17).

References

  • A Framework for Evaluating Agentic Skills at Scale, arXiv'26
  • Towards Trustworthy Evaluation of Autonomous Agents, arXiv'26
  • Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv'26
AuthorsOpenReview ↗

Response to Reviewer Poee (3/3)

Example task: nickel-concentration-risk-analyzer

Skill. Teaches how to analyze nickel supply risk. The procedure combines production data from several sources, measures how much of world supply one country controls, and estimates how a government policy would change world supply.

Searched files (7 files, 7.9 MiB):

  • Government statistics (USGS yearbook), ending in 2023
  • Industry report (INSG market page), covering 2024–2025 and describing Indonesia’s policy
  • Mining company reports (PT Vale), one annual and two quarterly
  • Encyclopedia page on nickel (Wikipedia)

Instruction given to the agent (verbatim):

I have the downloaded nickel evidence bundle under resources/ and need it turned into usable analysis artifacts. Please create data/normalized_nickel_supply.json, data/policy_scenarios.json, output/nickel_concentration_analysis.json, output/nickel_concentration_report.md, output/indonesia_rkab_scenario_impact.json, output/indonesia_rkab_scenario_report.md, output/source_validation_report.json, and output/source_validation_report.md.

 

Use the USGS historical workbook at myb1-2023-nicke-ERT.xlsx as the Tier 0 baseline for country production, cross-check the supply-demand and RKAB/export-rule context against the INSG pages at req-insg-nickel-balance-html-003.html and req-indonesia-rkab-policy-html-005.html, and treat req-spglobal-public-anchor-html-006.html as a tertiary public fallback for the Indonesia share claim rather than a paid-source anchor.

 

I want the analysis to be unit-first and supply-type clear, with the baseline in mined nickel content, not ore wet tonnes, refined production, NPI product tonnes, or matte tonnes mixed together. The concentration output should give Indonesia country share, CR1/CR3/CR5, HHI, market-structure interpretation, a recent time series, source IDs, confidence, and warnings where the downloaded sources do not line up cleanly.

 

For the Indonesia policy-risk piece, model an RKAB/export-rule scenario from the available public context with policy leverage, execution probability, and the hard cut / half success / soft landing layers, but make the missing numeric quota-cut input explicit instead of inventing a 250 Mt vs 379 Mt assumption.

 

Also use the PT Vale annual and quarterly PDFs in vale_ar_2025/, vale_q1_2024/, and vale_q3_2024/ as Tier 1 mine-level anchors for Indonesia product context and source validation, while being clear that one issuer is not enough for a full Mine_CR5.

 

Finally, validate the common ‘Indonesia controls about 60% of global nickel supply’ claim with a source chain, cross-validation, confidence, caveats, and a recommended wording that states the correct year, unit, and supply scope.

In the revised paper, we have added this task, with its instruction and rubrics, to Appendix H.1 (Figure 10).

Reviewer_goS1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SkillForgeBench, a benchmark of 842 sandboxed tasks across five domains. Its construction pipeline uses agent skills to guide retrieval of real artifacts, synthesize executable tasks, and derive completion and process rubrics from paired reference runs with and without skills. The authors evaluate eight models in a common OpenHands harness, reporting both average criterion satisfaction and strict success requiring every criterion in the relevant rubric set to pass. The reported results show a substantial gap between these metrics and larger strict completion gains from skills for stronger models. Completion and process are scored separately.

Critical strengths
  • Evaluating procedural constraints alongside final outputs addresses an important limitation of assessing agents only through the deliverables they produce.

-The 842 executable tasks provide a substantial resource spanning multiple professional domains. Real artifacts and explicit separation of editable targets, read-only context, and withheld evidence give the scenarios concrete grounding.

Critical weaknesses
  • The positioning overstates some differences from prior work. The overlap with Shaposhnikov et al. work deserves a more detailed comparison than the feature table provides. In Section 2.2, the characterization of Terminal-World as treating skills merely as topic seeds appears too broad: that work explicitly derives task instructions, environments, and execution guidance from skill content. The distinction between training-data synthesis and evaluation is meaningful, but does not establish that using procedural skill content as a shared specification is itself new. Consequently, the novelty of the paper is moderate, which resides primarily in integration, benchmark construction, and validation. Several central ingredients have close precedents: SkillsBench already compares agents with and without skills and uses all-or-nothing verification. Shaposhnikov et al. work generates approximately 1,000 skill-derived tasks with separate completion and instruction-following rubrics; and Terminal-World uses skills to jointly derive tasks, environments, and execution guidance. The more distinctive contribution appears to be the combination of retrieved artifacts, explicit artifact roles, contrastive rubric construction, and expert calibration. This is a potentially valuable methodological contribution, however the authors should provide direct evidence that these choices produce better tests of procedural reliability than existing approaches.

  • A large gap between average criterion satisfaction and all-criteria success is partly expected mathematically. For example, ten independent checks each passed with 90% probability yield only about 35% probability of passing all ten. Strict scores therefore depend on checklist length, difficulty, redundancy, and dependence. The larger strict gains from skills for stronger models may also be amplified by this nonlinear aggregation. The analysis should establish what remains after accounting for these effects.

  • The benefits of the proposed construction components are not isolated. The reported quality audit supports the plausibility of the resulting tasks and rubrics, but does not establish which pipeline components improve them. Matched ablations should compare retrieval against synthesized inputs, skill-guided scenario design against simpler generation, and contrastive rubric generation against rubrics generated directly from the task and skill. Evaluation should hold the skill pool, execution budgets, and judging protocol fixed.

  • Rubric quality and judge accuracy require separate validation. Table 3 assesses task and rubric quality, including observability and relevance. It does not directly demonstrate agreement between human and LLM pass/fail judgments on evaluated agent runs. A human audit of actual scoring decisions, plus judge sensitivity analysis, would strengthen confidence in the leaderboard. This matters particularly under strict aggregation, where one incorrect failure label can change the entire task result.

  • Process compliance needs a clearer operational definition. The benchmark should distinguish mandatory constraints from advisory workflow preferences and accept valid alternative procedures when the requirement permits them. For example, running a required validation is substantively important, whereas reproducing the reference agent's exact command may be unnecessary. Concrete cases would help show that process scores measure meaningful compliance and that judging recognizes equivalent implementations.

Questions for authors
  1. What is the specific methodological advance overShaposhnikov et al. and Terminal-World? Could you provide a direct comparison and clarify the characterization of Terminal-World in Section 2.2?

  2. Can you report matched ablations showing the contribution of authentic artifact retrieval, skill-guided scenario design, contrastive rubric generation, and human calibration?

  3. What are the distributions of completion and process criterion counts per task? Do the reported gaps and rankings persist after matching checklist length and accounting for criterion difficulty and dependence?

  4. Do stronger models still obtain larger skill benefits after accounting for baseline performance and the nonlinear conversion from criterion pass rates to strict task success?

  5. How were candidate requirements selected from the two reference runs? How do you prevent incidental differences, reference-model limitations, or a single run's errors from shaping the rubric?

  6. What is human agreement with the evaluation judge on actual agent-run pass/fail labels? How sensitive are strict scores and model rankings to the judge model or prompt?

  7. How do you distinguish required process constraints from optional recommendations, and how do you verify that valid alternative workflows receive credit?

  8. Can you evaluate a subset with independently written user requests and provide a complete worked task showing retrieved inputs, withheld evidence, reference runs, and final rubric?

7 comments
AuthorsOpenReview ↗

Response to Reviewer goS1 (1/5)

We sincerely thank the reviewer for the thorough review and many constructive suggestions. We are glad that the reviewer valued evaluating procedural constraints alongside final outputs and our grounded 842-task resource. We have conducted extensive experiments to address every concern. Questions Q1–Q8 correspond to weaknesses W1–W5. This comment answers Q1. The next four comments answer Q2–Q8.

Q1: Methodological Advances over Prior Work (ASS & Terminal-World)

What is the specific methodological advance over Shaposhnikov et al. and Terminal-World? Could you provide a direct comparison and clarify the characterization of Terminal-World in Section 2.2?

ASSTerminal-WorldSkillForgeBench
PurposeEvaluationTraining dataEvaluation
MeasuresSkill-use abilityOutput correctnessReliability under user constraints
EnvironmentsFit to the target skill; often small and syntheticBuilt for executable testsReal file collections
InstructionsCentered on the skillCentered on testable outputsModeled on real user queries
Process constraintsFewNot a focusCentral

All three works derive tasks from skills, but they measure different things. ASS measures whether an agent can use a skill. Terminal-World generates training data. SkillForgeBench measures reliability: whether an agent satisfies every constraint a real user states, on both the deliverable and the process. This difference in purpose leads to different data synthesis pipelines.

SkillForgeBench uses realistic environments and refers to realistic user queries. Because ASS evaluates skill-using ability, its priority is making environments and files relevant to the target skill. Consequently, it contains many purely synthetic, simple environments. In addition, SkillForgeBench leverages the procedural steps of skills to specify both final outcomes and process constraints. Terminal-World mainly uses executable tests to check task outputs, and detailed process constraints are not its focus.

These different synthesis pipelines lead to two concrete differences in the resulting benchmark data:

1. Environments are larger and more varied. We compare all 842 SkillForgeBench tasks and 1,110 ASS tasks with the aggregate statistics reported for Terminal-World's 5,723 environments. We measure the workspace files that agents can operate on directly.

Measure (mean / median unless noted)SkillForgeBenchASSTerminal-World
Input files per task172.89 / 101.55 / 12.25 / —
File types per task9.46 / 40.98 / 1≤2.25 / —
Distinct file types across all tasks (total)70658104
Input size per task13.67 MiB / 3.37 MiB19.5 KiB / 1.1 KiB—
Tasks with no input files0.0%36.3%—

For Terminal-World, “—” means not reported, and ≤2.25 is an upper bound: the number of file types cannot exceed the number of files. SkillForgeBench contains 706 distinct file types, compared with 104 across Terminal-World's environments. ASS has no input files in 36.3% of tasks and a median input size of only 1.1 KiB. These results support our claim that agents face larger and more varied collections of input files.

2. Instructions carry more process constraints. We apply the same keyword rules to all SkillForgeBench and ASS instructions. What requirements specify the deliverable (format/structure and quantity/length), while How requirements specify the way of working (prohibitions, evidence/verification, order/priority, style/tone, and audience/purpose). Terminal-World's full task instructions are not publicly available. Its single appendix example is insufficient for dataset-level statistics, so we exclude Terminal-World from this table.

Measure (mean)SkillForgeBenchASS
What requirement types (0–2)0.980.93
How requirement types (0–5)3.101.75
How requirement hits per 100 words3.741.77

The two benchmarks have similar What coverage, but SkillForgeBench instructions specify more How requirements, both in the number of types and in hits per 100 words. These keyword counts measure explicit requirement coverage rather than task quality directly; they support our emphasis on user constraints beyond the final deliverable.

Clarification of Terminal-World in Section 2.2. In the revised paper, we have rewritten Section 2.2 to clarify that Terminal-World does not use skills to introduce process constraints into tasks. Its skill procedures serve only as guidelines for the teacher model to generate tasks, so in the task itself, skills define only the required final outcome checked by executable tests.

References

  • A Framework for Evaluating Agentic Skills at Scale, arXiv'26
  • Terminal-World: Scaling Terminal-Agent Environments via Agent Skills, arXiv'26
AuthorsOpenReview ↗

Response to Reviewer goS1 (2/5)

Q2: Ablation Study

Can you report matched ablations showing the contribution of authentic artifact retrieval, skill-guided scenario design, contrastive rubric generation, and human calibration?

We design a modular pipeline with four swappable stages mentioned in Q2, allowing each component to be isolated and replaced while holding the rest of the pipeline fixed. Next, we conduct a matched ablation study across 50 skills and derive the following conclusions.

  • S0: Original pipeline.
  • A: Authentic artifact retrieval prevents task environments from collapsing into artificially small and simple files.
  • B: Skill-guided scenario design grounds instructions in real workspace files and prevents unnatural requirements.
  • C: Contrastive rubric generation makes rubrics specific.
  • D: Human calibration ensures high-quality criteria.

In series A, an LLM agent writes all workspace files directly from skill requirements without web search. Across all the tasks, environments generated purely by the LLM are less complex in terms of size, distinct types, number of files, etc. In addition, human experts observe that files perfectly match the seed skill's requirements, making the task unrealistic. These two features divide the purely generated environment from reality, where files are more complex and varied.

Measure (median unless noted)S0Series A
Environment size (KiB)2,39339
Distinct file types (total)11236
Median file size (KiB)17.93.0
Tasks with ≥100 files24%0%

In series B, we remove the planning layer (the file change summary agent and requirement list compilation). The compose agent writes instructions directly from downloaded workspace files. Environments in Series B are byte-identical to S0 across all 50 tasks. Removing scenario planning shortens instructions (median 126.5 words vs. 176 words). Moreover, removing planning breaks semantic grounding. A paired review of all 50 task pairs reveals that 18 instructions in Series B (36%) make unnatural demands on workspace files.

In series C, we replace the contrastive rubric generation module with direct rubric construction, which composes rubrics directly based on the environment and instruction. Human experts compare 100 pairs of rubrics (50 for completion and 50 for process) generated in S0 and Series C. To prevent bias, the comparison removes all the identity information of rubrics, so human experts can only see the contents of rubrics. In addition, which rubric appears as the first in each pair is random with a fixed seed. The results in the following table show that S0's rubrics are less vague in 73 of 100 pairs (73%). In addition, we count the number of words per criterion. We find that criteria in Series C are about 17% longer than in S0. Together, these results show that Series C's rubrics use more words but produce more vague criteria, proving the necessity of contrastive rubric generation.

S0 less vagueSeries C less vague
completion3515
process3812
per rubric or per criterionS0Series C
words per criterion129.9151.9

In series D, we re-analyze the modified criteria in these 50 tasks (9.5%). We conclude that the main weakness of each is one of the following:

  • superficial requirements, like formatting details.
  • ambiguous or vague, making the grading harder.
  • expression flaws, such as imprecise description of the behavior, that might be misleading.
AuthorsOpenReview ↗

Response to Reviewer goS1 (3/5)

Q3 & Q4: Accounting for Checklist Length, Baseline Performance, and Nonlinear Aggregation

Q3: What are the distributions of completion and process criterion counts per task? Do the reported gaps and rankings persist after matching checklist length and accounting for criterion difficulty and dependence?

Q4: Do stronger models still obtain larger skill benefits after accounting for baseline performance and the nonlinear conversion from criterion pass rates to strict task success?

Criteria per task (842 tasks)

MeanMedianIQRRange
Completion7.1586–91–11
Process7.9387–91–13

Tasks per length bin:

Bin1–56–78910+
Completion186159191198108
Process6415530423089

Strict scores after matching criteria length and difficulty

Here, we report τ = Kendall rank correlation between the bin's model ranking and the overall ranking under strict scoring. Model scores in each bin are reported in Appendix F (Tables 11–13) of the revised paper.

τn=1–5n=6–7n=8n=9n=10+
Comp. w/0.790.930.860.931.00
Comp. w/o0.710.790.930.860.93
Proc. w/0.570.861.000.930.93

After matching checklist length, every bin's model ranking remains almost unchanged. In addition, we argue that since longer criteria length measures more constraints, the results after matching criteria length can reflect accounting for task difficulty. Thus, model rankings generally persist under each level of task difficulty.

Skill gains after accounting for baseline performance and the nonlinear conversion

Assume each criterion passes independently with probability pp, where pp is the loose score without skills. Under this assumption, the expected strict score of a task with nn criteria is pnp^n. With skills, the loose score rises to p+Δp+\Delta, and the expected strict score becomes (p+Δ)n(p+\Delta)^n. We define the expected strict gain as

G^=1T∑t=1T[(p+Δ)nt−pnt],\hat G = \frac{1}{T}\sum_{t=1}^{T}\left[(p+\Delta)^{n_t} - p^{n_t}\right],

where TT is the number of tasks and ntn_t is the number of criteria in task tt. G^\hat G depends only on the loose score pp, the loose gain Δ\Delta and the number of criteria per task.

In practice, the measured strict scores ScompNSS_{\mathrm{comp}}^{\mathrm{NS}} and ScompWSS_{\mathrm{comp}}^{\mathrm{WS}} differ from these expected values. We write the actual strict gain as G=ScompWS−ScompNSG = S_{\mathrm{comp}}^{\mathrm{WS}} - S_{\mathrm{comp}}^{\mathrm{NS}} and define the excess strict gain as

E=G−G^.E = G - \hat G.

EE is the part of the actual strict gain that the loose scores do not predict. E>0E > 0 measures the extent to which skills help the model more than mathematics expects.

We compute EE for all 8 models:

ModelScompNSS_{\mathrm{comp}}^{\mathrm{NS}}GGG^\hat GEE
Opus 4.821.7+19.2+12.2+7.0
GLM-5.220.0+16.8+12.5+4.3
Sonnet 4.615.0+13.5+9.2+4.3
Hy314.0+13.3+9.6+3.7
Kimi K2.611.2+8.9+6.8+2.1
Gemini 3.6 Flash8.9+9.5+8.8+0.7
Haiku 4.56.8+5.8+3.7+2.1
Qwen3.5-397B5.3+8.0+3.8+4.2

The correlation with ScompNSS_{\mathrm{comp}}^{\mathrm{NS}} is r=0.70r = 0.70 for the excess gain EE. This result shows a strong correlation, although rr drops from 0.970.97 for the raw gain GG. Therefore, after accounting for baseline performance and the nonlinear conversion, stronger models still gain more from skills, which means they use skills better.


Strict scoring is standard in agent evaluation and naturally weights the difficulty of baseline improvement. Binary pass/fail evaluation is widely adopted across agent benchmarks (e.g., Terminal-Bench 2.0 and SWE-bench), where partially satisfied tasks typically offer zero practical utility. Furthermore, strict scoring inherently accounts for diminishing headroom at higher baseline performance. For example, consider Model A with a baseline loose score of 80%80\% and Model B with 60%60\%. For the same loose-score gain Δ\Delta, Model A mathematically achieves a higher expected strict gain G^\hat G than Model B.

This difference is reasonable and expected. Originally, Model A has much less room for improvement than Model B. Under this condition, if Model A achieves the same per-criterion improvement Δ\Delta as Model B, Model A's improvement is more difficult and more valuable. Strict scoring captures this difficulty and properly reflects it as a higher strict gain for Model A.

In the revised paper, we have added the strict scores of all eight models within each checklist-length bin to Appendix F (Tables 11–13).

Comments on the paper

1 comment
AuthorsOpenReview ↗

GR1: Human agreement with LLM-as-judge and judging sensitivity

First, we sincerely thank all reviewers for their constructive comments on improving this paper. Below, we list a general response targeting some shared questions.

GR1: Human agreement with LLM-as-judge and judging sensitivity

Our judge for the main evaluation is GPT-5.5 with xhigh reasoning effort running in Codex, which is one of the strongest frontier agents.

The judge agrees with human judgments on 96% of decisions (Cohen's κ = 0.90–0.91). We compared two independent runs of the judge (judge-1, judge-2) with human judgments on the executions of Opus 4.8 for 50 tasks. In total, the comparison covers 941 decisions.

PairDisagreement [95% CI]Cohen's κ [95% CI]
judge-1 vs judge-23.29% [2.06, 4.76]0.907 [0.872, 0.939]
judge-1 vs human3.61% [2.22, 5.39]0.901 [0.862, 0.935]
judge-2 vs human3.40% [2.19, 4.85]0.905 [0.871, 0.935]

All 95% CIs are bootstrapped over tasks (2,000 resamples, seed 0).

Across process and completion rubrics, strict scores from the judge closely match human judgments. Across all evaluated settings, human strict scores remain close to at least one judge run. For completion without skills (w/o), human judgments match judge-1 (J1) exactly (28.0%) and differ from judge-2 (J2) by only 2.0 pp (30.0%). For completion with skills (w/), human judgments stay within 2.0 pp of both runs (54.0% vs 52.0% and 56.0%). On process rubrics with skills, although the difference from J1 is larger than in completion (6.0 pp, 26.0% vs 20.0%), J2 matches human judgments exactly (20.0%).

ModelSettingStrict J1Strict J2Strict human
Opus 4.8completion · w/o28.030.028.0
Opus 4.8completion · w/52.056.054.0
Opus 4.8process · w/26.020.020.0

Scores are percentages across 50 evaluated tasks.

Sensitivity to the judge model

To test sensitivity to the judge model, we independently judge those 50 tasks with Sonnet 5.5 in Claude Code, which comes from a completely different model family. The tables below show that model rankings do not change under two judge models, although absolute scores judged by Sonnet are systematically lower. These results suggest that our rubrics are robust under completely different judge models. Specifically, across all six evaluated settings (loose and strict scores for completion w/o, completion w/, and process w/), both judges yield the identical ranking: Opus 4.8 > GLM-5.2 > Gemini 3.6 Flash.

GPT-5.5 as judge:

ScoreSettingOpus 4.8GLM-5.2Gemini 3.6 Flash
Loosecompletion · w/o78.473.171.5
Loosecompletion · w/82.279.074.4
Looseprocess · w/86.380.973.6
Strictcompletion · w/o28.026.016.0
Strictcompletion · w/52.042.020.0
Strictprocess · w/26.024.010.0

Sonnet as judge:

ScoreSettingOpus 4.8GLM-5.2Gemini 3.6 Flash
Loosecompletion · w/o67.965.860.7
Loosecompletion · w/78.073.466.2
Looseprocess · w/80.179.667.4
Strictcompletion · w/o12.010.02.0
Strictcompletion · w/30.028.08.0
Strictprocess · w/8.06.04.0

We have added these results to Appendix E (Tables 8–10) of the revised paper.

Minor corrections

We also fixed typos and inconsistencies in the paper. The appendix now uses the same rubric-category names as the main text, model names are spelled consistently (e.g., Hy3, GLM-5.2), and the skill-gain statistics in Appendix A now match the main results. All revisions are marked in blue in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.