- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SkillNet-Gym collects skills from ClawHub/SkillHub, applies LLM quality filtering and deduplication (8,159 → 2,225), and builds a directed skill graph. Edges come from LLM-extracted pre/post scenarios and are LLM-verified (53,904 edges). The pipeline samples DAG subgraphs (single, chain, fan-out, fan-in, diamond; 1–5 skills) and has Claude Code execute them on input files. From the resulting exploration report it synthesizes an instruction, a reference solution (solve.sh), tests and a Dockerfile, followed by multi-stage LLM and human QC. The result is 159 tasks, split post hoc into Easy (77) and Hard (82). Eight models are evaluated on Claude Code or Codex under No Skill, Official Skill, Skill-Creator, and retrieval/orchestration settings. Reported findings: skills help hard tasks more, one-shot skill construction loses information, agents retrieve and orchestrate poorly in a 2,000-skill pool, and specialized retrievers beat agents. A failure taxonomy is also given.
Critical strengths
- A single executable, Docker-based protocol covers skill construction, provided-skill efficacy, and skill retrieval/orchestration with deterministic tests. That is broader than retrieval-only or efficacy-only skill benchmarks.
- Some observations are useful for the community: the retrieval-vs-orchestration difficulty inversion, the failure taxonomy (knowledge bias 59.6% vs instruction override 40.4%), and specialized retrievers outperforming agentic self-retrieval in large pools.
Critical weaknesses
- The central "dynamic benchmark" claim is not evaluated, and the key empirical claim depends on a post-hoc split. Only one 159-task snapshot is built. There is no experiment that regenerates tasks from a later skill snapshot, compares difficulty or model rankings across versions, or tests drift or contamination resistance. As evaluated, this is a static benchmark built by an automated pipeline. Separately, Easy/Hard is defined "using post hoc empirical performance as a calibration signal". If Hard means low success, then "skills help hard tasks more reliably" is partly a selection or regression-to-the-mean effect. Please give the exact split rule and repeat the analysis with a priori difficulty (DAG size/topology) or a held-out-model split.
- Task construction is entangled with the evaluated condition and with specific models, and the statistics are thin. Task synthesis starts from an exploration report produced by Claude Code executing the official skills; an LLM writes the instruction from that report, Claude Code produces the reference solution guided by it, and tests are iterated until that reference passes. Tests may therefore encode skill-specific choices (libraries, thresholds, formats), which favors the Official-Skill condition and may itself cause "instruction override" failures. The difficulty filter removes tasks GLM-5.1 solves 3/3 without skills, and GLM-5.1 is also an evaluated model. With 77/82 tasks and 3 runs, Table 2 reports no CIs, while many conclusions rest on few-point differences or tiny per-domain cells (e.g., +50 pp Geospatial). Some Table 2 numbers also look inconsistent with 3 runs over 77/82 tasks: DeepSeek-V4 Easy 65.28 = 47/72, and Kimi-K2.6 and GLM-5.1 have identical No-Skill values (70.13/26.83).
Questions for authors
Clarifications:
- Can you show at least two benchmark versions built at different times and compare model rankings and difficulty across them?
- What is the exact Easy/Hard assignment rule, and do the main findings hold under a priori difficulty?
- How is Avg@3 computed? Were any tasks or runs excluded for some models (see the DeepSeek-V4 values)? Please confirm the identical Kimi/GLM rows.
- In the composition setting, what interface do agents use to access the 2,000-skill pool? Table 3 compares agents selecting about 3.9 skills with retrievers at top-10 on a binary completeness metric; please match budgets. How are valid alternative skill sets or workflows handled, given an average out-degree of 24?
- What fraction of Skill-Creator source documents are LLM-synthesized surrogates rather than real documentation?
- Will code, tasks and environments be released? The paper states an MIT license but I could not find a link or supplement. Suggestions: audit test leniency and strictness, e.g., check whether independently written correct solutions pass; add a model-family ablation for the construction agent.
6 comments
Response to Reviewer wNVu
W2 & Q3. Clarification and Correction of Table 2 Statistics
Thank you for carefully identifying the numerical inconsistencies in Table 2, particularly the DeepSeek-V4 results. Upon auditing the evaluation records, we identified a specific issue in the DeepSeek-V4 score aggregation: a subset of timed-out runs was inadvertently excluded from the denominator rather than counted as unsuccessful attempts. We apologize for this oversight. Importantly, these runs were attempted; the issue arose from how timeouts were handled during score aggregation, rather than from missing experimental executions. We have corrected the affected DeepSeek-V4 results as follows:
| DeepSeek-V4 setting | Originally reported | Corrected |
|---|---|---|
| No Skill — Easy | 65.28% | 61.04% |
| No Skill — Hard | 23.38% | 21.95% |
| Official Skill — Easy | 69.44% | 64.94% |
| Official Skill — Hard | 29.87% | 28.05% |
Importantly, this correction does not change the qualitative finding for DeepSeek-V4: Official Skills continue to improve performance over No Skill by 3.90 percentage points on Easy tasks and 6.10 percentage points on Hard tasks, respectively. We have also audited the remaining model–condition results and confirmed that no missing runs or analogous denominator omissions were found. Therefore, this was a specific aggregation issue affecting the reported DeepSeek-V4 scores, rather than a systematic inconsistency in the evaluation protocol. We thank the reviewer again for identifying this issue, which has helped us improve the accuracy and transparency of our experimental reporting.
W3 & Q4. Skill Composition: Wild Skill Discovery, Retrieval Budget, and Alternative Workflows
Thank you for raising these important questions. We clarify three aspects of our skill composition evaluation.
- Realistic skill discovery in the wild. The 2,000+ skill pool is designed to evaluate whether agents can effectively discover, select, and load relevant skills from a large, realistic skill repository, rather than relying on task-specific skills provided in advance. This setting is consistent with the motivation of SkillRouter and AgentSkillOS, which also investigate skill discovery and orchestration in large-scale repositories. Our goal is to evaluate not only whether agents can execute a known workflow, but also whether they can identify and organize the required procedural resources in the wild.
- Clarification of the Top-10 retrieval budget. Importantly, the Agent baseline is also evaluated under a Top-10 selection setting, as already stated in Section 4.4. Specifically, agents are instructed to identify up to 10 most relevant skills, rather than being restricted to approximately 3.9 skills. The reported 3.86 average selections in Table 3 reflect the number of skills agents actually choose, not an imposed retrieval budget. Since our synthesized tasks involve 1–5 skills, including workflows requiring more than three, this distinction is important. Both agents and specialized retrievers operate under a Top-10 selection limit, while their actual selection behavior differs.
- Alternative valid workflows. We agree that the reference DAG represents one validated composition rather than the only possible valid workflow. Our retrieval and orchestration completeness metrics are intended as strict diagnostics of reference-workflow recovery, rather than exhaustive measures of all valid compositions. Importantly, we additionally evaluate end-to-end task success in the wild (Table 5), where agents are assessed based on whether their final outputs satisfy task requirements, without requiring exact recovery of the reference skill set or DAG. For example, Kimi-K2.6 decreases from 34.15% to 25.61% on Hard tasks when moving from provided Official Skills to the In-Wild setting. This demonstrates that the challenge of skill discovery and composition also manifests in downstream execution, beyond reference-graph matching.
Response to Reviewer wNVu
W4 & Q5. Authenticity and Provenance of Skill-Creator Source Documents
Thank you for raising this important question. We clarify that authentic documentation is always prioritized in our construction pipeline, and LLM-generated surrogate documents are used only as a fallback when no suitable external documentation can be retrieved.
Specifically, approximately 85.5% of the source documents are retrieved from real-world resources, while only 14.5% are synthesized from their corresponding original skills. Importantly, these synthesized documents are not accepted without verification. They undergo human review to ensure that the content is technically plausible, consistent with the original skill's functionality, and preserves essential procedural information without omitting critical guidance.
Therefore, surrogate documentation represents a limited, quality-controlled fallback mechanism, rather than the primary source of our benchmark. Moreover, the underlying skills themselves are collected from real community repositories, regardless of whether their supporting documentation is retrieved or synthesized.
W5 & Q6. Code, Tasks, and Environment Availability
Thank you for highlighting the missing artifact links. We clarify that the implementation and benchmark data have already been publicly released, rather than being deferred until acceptance.
To preserve the double-blind review process, we have refrained from including links to repositories that may contain identity-revealing information. We acknowledge that this limits independent verification during the review process. To address this concern, we will provide anonymized copies of the code, benchmark tasks, and environment specifications through the conference's permitted supplementary-material or reviewer-only channels, allowing reviewers to inspect the artifacts without compromising anonymity.
We appreciate the reviewer's emphasis on reproducibility and will clarify the artifact availability and access instructions in the revised submission.
Response to Reviewer wNVu
Dear Reviewer wNVu,
We appreciate your detailed, technically informed critique and your recognition of the single executable, Docker-based protocol across multiple skill capabilities, together with the empirical insights on retrieval versus orchestration, agent versus specialized retrievers, and skill-induced failure modes. Because your main concerns concern soundness, we distinguish completed new experiments from clarifications and corrections still required.
W1 & Q1–Q2. No temporal dynamic evaluation; Hard is determined post hoc.
We agree that the submitted 159 tasks represent a single initial snapshot. Beyond the pipeline design, we now provide a updated 20 tasks with updated skills, comparing all three access conditions and both outcome metrics:
These results demonstrate that tasks evaluated with dynamically updated skill resources remain challenging across different models, providing empirical evidence that the benchmark can accommodate skill updates while retaining its ability to assess agent performance.
We also agree that the 77 Easy / 82 Hard stratification is post hoc. This is an intentional design choice, as realized task difficulty cannot be reliably determined from structural properties alone and requires empirical calibration. While skill count (1–5) and DAG topology (five structural types) are specified a priori during task construction and provide measures of compositional complexity, they do not necessarily imply a universal difficulty ordering.
Importantly, we explicitly investigate this relationship in our existing analysis, "Topology-Related Difficulty Is Context Dependent" (Figure 5). For example, among tasks involving 3–5 skills, fan-out tasks are substantially harder than chains in Biomedical Processing (4.2% vs. 73.6% No-Skill success), whereas this pattern does not hold in Document Processing (38.9% vs. 33.3%). These findings demonstrate that task difficulty depends jointly on dependency structure, task semantics, and empirical agent capabilities, rather than topology alone. Therefore, our post-hoc stratification, with domain balancing, is a deliberate choice to capture realized difficulty, while the pre-specified topology analysis in Figure 4(b) provides complementary evidence on how structural complexity influences skill effectiveness. The benchmark's central contribution—evaluating skill construction, retrieval, orchestration, and task execution on a unified executable task set, does not depend on treating this stratification as a causal intervention.
W2. Construction is entangled with official skills/Claude Code; tests might encode the reference implementation.
We agree that construction provenance matters. Official skills are used during DAG-guided exploration to establish that a plausible multi-skill workflow exists; the exploration report then guides instruction and reference-solution synthesis. This makes the tasks grounded in feasible procedural workflows, but does not automatically make the resulting tests implementation-neutral. We will state this source of potential inductive bias explicitly and refrain from claiming that official-skill advantages are unconfounded. Our existing safeguards operate at several levels: skill and edge filtering, a separate instruction-quality rubric, trajectory/process verification for the proposed solution, test-instruction alignment review, candidate rollouts, and independent review by three annotators (κ = 0.834; Appendix D.1). Importantly, we agree that passing the reference solution is necessary but insufficient to establish verifier completeness.
Meanwhile, the existing failure analysis explicitly reports skill-induced regressions, including knowledge bias and instruction override (Figure 3). Therefore, we would like to clarify that injecting skills does not necessarily lead to improved task performance.