- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SkillForgeBench to evaluate how reliably LLM agents follow outcome and procedural requirements in complex professional tasks. It addresses a trade-off in existing benchmarks between domain coverage and procedural depth.
The authors build 842 sandboxed tasks across five domains using a three-stage pipeline. They first curate agent skills and retrieve real-world materials, then use these resources to construct executable tasks. Finally, they develop completion and process rubrics by comparing agent executions with and without skill guidance, followed by human expert review.
The authors evaluate eight models using a shared OpenHands framework, with rubrics scored by an LLM judge. While average completion rubric pass rates range from 70.0% to 85.7%, strict task completion rates are much lower, at 12.6% to 40.9%. The authors report that stronger models gain more from skill guidance under strict scoring and that satisfying process requirements remains particularly challenging. An independent human audit reports high task and rubric quality, with 95.9% of paired ratings differing by no more than one point.
Critical strengths
The paper uses agent skills as "unifying blueprints across the task lifecycle". Figure 1 illustrates this with a citation verification task that separates editable files, read-only context, and withheld reference evidence. The pipeline produces 842 executable tasks across five domains, and the independent human audit gives task quality ratings of 4.56 and 4.71 out of 5 on a 100-task sample (Table 3). Appendix E also provides detailed construction prompts, making the pipeline easier to inspect and reproduce.
Separating completion and process rubrics allows the benchmark to evaluate both final outputs and adherence to required procedural steps. With skill access, Gemini 3.6 Flash achieves 18.4% strict completion but only 5.11% strict process compliance, while Opus 4.8 reaches 40.9% and 27.7%. These results support the authors' argument that "completion-only checks overestimate real-world agent reliability." The breakdown into six rubric categories also helps identify specific failure modes.
Critical weaknesses
The benchmark relies on an LLM judge to evaluate final workspaces and execution trajectories, but the reliability of its scoring is not sufficiently established. Table 3 assesses task and rubric quality without comparing the judge's pass/fail decisions against human evaluations. The paper also does not specify the evaluation judge model.
Figure 2 reports larger absolute gains from skill guidance for stronger models, with 19.2 percentage points for Opus 4.8 and 5.8 for Haiku 4.5. However, loose-score gains are similar across models, ranging from 5.1 to 7.4 points, and the relative strict gains for Opus and Haiku are also close at 89% and 86%. Since strict scoring requires all criteria to pass, baseline performance and the number of criteria may contribute to these differences. A baseline-adjusted analysis would help clarify whether stronger models benefit more from skill guidance, as the authors conclude.
Questions for authors
Could the authors clarify which judge model was used for the main evaluation? Have the judge’s pass/fail decisions been compared with human judgments on a sample of agent executions, particularly for process rubrics?
Could the authors clarify whether stronger models still benefit more from skill guidance after accounting for baseline performance and the number of criteria per task? A comparison using relative gains or baseline-adjusted strict scores would help clarify the findings in Figure 2.
3 comments
thanks for sharing. i keep score 3
Thank you so much for your thoughtful review! Your feedback helped us improve the experiments in our paper. We truly appreciate the time you devoted to reviewing our work.
Response to Reviewer MVWN (1/1)
We sincerely thank the reviewer for the careful reading and the constructive feedback. We are glad that the reviewer found value in using agent skills as unifying blueprints across the task lifecycle, in the 842 executable tasks with detailed construction prompts, and in the separate completion and process rubrics that evaluate both final outputs and required procedural steps.
Q1: Human agreement with LLM-as-judge.
Our judge for the main evaluation is GPT-5.5 with xhigh reasoning effort, run in Codex. As detailed in General Response 1 (GR1), the judge agrees with human judgments on 96% of decisions, and changing the judge model does not change model rankings.
Q2: Accounting for Checklist Length and Baseline Performance
Assume each criterion passes independently with probability p, where p is the loose score without skills. Under this assumption, the expected strict score of a task with n criteria is pn. With skills, the loose score rises to p+Δ, and the expected strict score becomes (p+Δ)n. We define the expected strict gain as
G^=T1∑t=1T[(p+Δ)nt−pnt],
where T is the number of tasks and nt is the number of criteria in task t. G^ depends only on the loose score p, the loose gain Δ and the number of criteria per task.
In practice, the measured strict scores ScompNS and ScompWS differ from these expected values. We write the actual strict gain as G=ScompWS−ScompNS and define the excess strict gain as
E=G−G^.
E is the part of the actual strict gain that the loose scores do not predict. E>0 measures the extent to which skills help the model more than mathematics expects.
We compute E for all 8 models:
The correlation with ScompNS is r=0.70 for the excess gain E. This result shows a strong correlation, although r drops from 0.97 for the raw gain G. Therefore, after accounting for the baseline and the number of criteria, stronger models still gain more from skills, which means they use skills better.
In the revised paper, we have added the human-agreement and judge-sensitivity results to Appendix E (Tables 8–10).