Published as a conference paper at NeurIPS 2026

SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

Xiangyi Li, Yimin Liu, Wenbo Chen, Shenghan Zheng, Yifeng He, Xiaokun Chen, Bingran You, Haotian Shen, Yubo Li, Kyoung Whan Choe, Binxu Li, Shuyi Wang, Chris Kong, Xinyi Liu, Jiankai Sun, Runhui Wang, Xuanqing Liu, Jiachen Li, Xin Lan, Yuanli Wang, Xuandong Zhao, Yueqian Lin, Wengao Ye, Junwei He, Di Wang, Roey B Chaim, Songlin Li, Hanwen Xing, Yue Zhang, Yipeng Gao, Yijiang Li, Ze Ma, Liqiang Jing, Qunhong Zeng, Tianyu Wang, Kaixin Li, Yiqi Xue, Zonglin Di, Yizhuo He, Yuchen Tian, Yixuan Gao, Hejia Geng, Bo Chen, Litong Liu, Sikai Cheng, Zelin Tan, Jiajun Bao, Junwei Zhou, Shuaicheng Tong, Shuwen Xu, Penghao Jiang, Terry Yue Zhuo, Tinghan Ye, Chujun Tao, Qi Qi, Miao Li, Longtai Liao, Zhikang Dong, Chang Shi, Steven Dillmann, Han-chung Lee

[PDF][NeurIPS][OpenReview][arXiv]

Abstract

Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a paired benchmark of 84 tasks across 8 expertise-heavy domains; every task runs under three matched conditions (no Skills, curated Skills, self-generated Skills) on 7 agent–model configurations and 7,326 trajectories with deterministic verifiers. Curated Skills raise pass rate by 16.4 pp on average, but the lift is heterogeneous: per-domain $Δ$ spans +7.2 to +24.8pp and 18 of 84 tasks regress. Self-generated Skills net -1.3pp, indicating that current models cannot author the procedural knowledge they benefit from consuming. SkillsBench establishes paired evaluation as standard practice for measuring agent augmentations.

Related papers