acceptodds
Under review as a conference paper at ICLR 2027

SkillNet-Gym: A Dynamic Benchmark for Compositional Skill Learning

Abstract

Large language model (LLM) agents increasingly rely on skills to solve complex, long-horizon tasks. However, existing benchmarks for agent skills are mostly static, manually curated snapshots, making them poorly suited to the continuously evolving nature of real-world skill ecosystems. To address this gap, we introduce SkillNet-Gym, a dynamic benchmark for evaluating compositional skill learning in LLM agents. Grounded in real-world sources, SkillNet-Gym automatically builds a continuously evolving heterogeneous SkillNet, and uses it to synthesize domain-diverse, compositionally complex tasks. These tasks enable unified evaluation for skill construction and skill composition. Evaluating eight frontier models with two widely used agent harnesses reveals that directly compressing documents into skills can cause information loss and agents remain weak at retrieving and orchestrating skills in the wild. These findings show that effective skill use is not a one-shot augmentation mechanism, but a compositional pipeline of interdependent competencies. SkillNet-Gym provides a dynamic testbed for developing, diagnosing, and improving evolving skill centric agents.

Primary area:
datasets and benchmarks
Keywords:
Skills, Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wNVuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SkillNet-Gym collects skills from ClawHub/SkillHub, applies LLM quality filtering and deduplication (8,159 → 2,225), and builds a directed skill graph. Edges come from LLM-extracted pre/post scenarios and are LLM-verified (53,904 edges). The pipeline samples DAG subgraphs (single, chain, fan-out, fan-in, diamond; 1–5 skills) and has Claude Code execute them on input files. From the resulting exploration report it synthesizes an instruction, a reference solution (solve.sh), tests and a Dockerfile, followed by multi-stage LLM and human QC. The result is 159 tasks, split post hoc into Easy (77) and Hard (82). Eight models are evaluated on Claude Code or Codex under No Skill, Official Skill, Skill-Creator, and retrieval/orchestration settings. Reported findings: skills help hard tasks more, one-shot skill construction loses information, agents retrieve and orchestrate poorly in a 2,000-skill pool, and specialized retrievers beat agents. A failure taxonomy is also given.

Critical strengths
  1. A single executable, Docker-based protocol covers skill construction, provided-skill efficacy, and skill retrieval/orchestration with deterministic tests. That is broader than retrieval-only or efficacy-only skill benchmarks.
  2. Some observations are useful for the community: the retrieval-vs-orchestration difficulty inversion, the failure taxonomy (knowledge bias 59.6% vs instruction override 40.4%), and specialized retrievers outperforming agentic self-retrieval in large pools.
Critical weaknesses
  1. The central "dynamic benchmark" claim is not evaluated, and the key empirical claim depends on a post-hoc split. Only one 159-task snapshot is built. There is no experiment that regenerates tasks from a later skill snapshot, compares difficulty or model rankings across versions, or tests drift or contamination resistance. As evaluated, this is a static benchmark built by an automated pipeline. Separately, Easy/Hard is defined "using post hoc empirical performance as a calibration signal". If Hard means low success, then "skills help hard tasks more reliably" is partly a selection or regression-to-the-mean effect. Please give the exact split rule and repeat the analysis with a priori difficulty (DAG size/topology) or a held-out-model split.
  2. Task construction is entangled with the evaluated condition and with specific models, and the statistics are thin. Task synthesis starts from an exploration report produced by Claude Code executing the official skills; an LLM writes the instruction from that report, Claude Code produces the reference solution guided by it, and tests are iterated until that reference passes. Tests may therefore encode skill-specific choices (libraries, thresholds, formats), which favors the Official-Skill condition and may itself cause "instruction override" failures. The difficulty filter removes tasks GLM-5.1 solves 3/3 without skills, and GLM-5.1 is also an evaluated model. With 77/82 tasks and 3 runs, Table 2 reports no CIs, while many conclusions rest on few-point differences or tiny per-domain cells (e.g., +50 pp Geospatial). Some Table 2 numbers also look inconsistent with 3 runs over 77/82 tasks: DeepSeek-V4 Easy 65.28 = 47/72, and Kimi-K2.6 and GLM-5.1 have identical No-Skill values (70.13/26.83).
Questions for authors

Clarifications:

  1. Can you show at least two benchmark versions built at different times and compare model rankings and difficulty across them?
  2. What is the exact Easy/Hard assignment rule, and do the main findings hold under a priori difficulty?
  3. How is Avg@3 computed? Were any tasks or runs excluded for some models (see the DeepSeek-V4 values)? Please confirm the identical Kimi/GLM rows.
  4. In the composition setting, what interface do agents use to access the 2,000-skill pool? Table 3 compares agents selecting about 3.9 skills with retrievers at top-10 on a binary completeness metric; please match budgets. How are valid alternative skill sets or workflows handled, given an average out-degree of 24?
  5. What fraction of Skill-Creator source documents are LLM-synthesized surrogates rather than real documentation?
  6. Will code, tasks and environments be released? The paper states an MIT license but I could not find a link or supplement. Suggestions: audit test leniency and strictness, e.g., check whether independently written correct solutions pass; add a model-family ablation for the construction agent.
6 comments
AuthorsOpenReview ↗

Response to Reviewer wNVu

Dear Reviewer wNVu,

We appreciate your detailed, technically informed critique and your recognition of the single executable, Docker-based protocol across multiple skill capabilities, together with the empirical insights on retrieval versus orchestration, agent versus specialized retrievers, and skill-induced failure modes. Because your main concerns concern soundness, we distinguish completed new experiments from clarifications and corrections still required.

W1 & Q1–Q2. No temporal dynamic evaluation; Hard is determined post hoc.

We agree that the submitted 159 tasks represent a single initial snapshot. Beyond the pipeline design, we now provide a updated 20 tasks with updated skills, comparing all three access conditions and both outcome metrics:

ModelSettingAvg@3Pass@3
Kimi-K2.6No Skills55.00%65.00%
Kimi-K2.6Original Tasks51.67%70.00%
Kimi-K2.6Updated Tasks53.33%60.00%
GLM-5.1No Skills48.33%60.00%
GLM-5.1Original Tasks46.67%55.00%
GLM-5.1Updated Tasks45.00%60.00%
Sonnet 4.6No Skills50.00%60.00%
Sonnet 4.6Original Tasks43.33%55.00%
Sonnet 4.6Updated Tasks43.33%55.00%

These results demonstrate that tasks evaluated with dynamically updated skill resources remain challenging across different models, providing empirical evidence that the benchmark can accommodate skill updates while retaining its ability to assess agent performance.

We also agree that the 77 Easy / 82 Hard stratification is post hoc. This is an intentional design choice, as realized task difficulty cannot be reliably determined from structural properties alone and requires empirical calibration. While skill count (1–5) and DAG topology (five structural types) are specified a priori during task construction and provide measures of compositional complexity, they do not necessarily imply a universal difficulty ordering.

Importantly, we explicitly investigate this relationship in our existing analysis, "Topology-Related Difficulty Is Context Dependent" (Figure 5). For example, among tasks involving 3–5 skills, fan-out tasks are substantially harder than chains in Biomedical Processing (4.2% vs. 73.6% No-Skill success), whereas this pattern does not hold in Document Processing (38.9% vs. 33.3%). These findings demonstrate that task difficulty depends jointly on dependency structure, task semantics, and empirical agent capabilities, rather than topology alone. Therefore, our post-hoc stratification, with domain balancing, is a deliberate choice to capture realized difficulty, while the pre-specified topology analysis in Figure 4(b) provides complementary evidence on how structural complexity influences skill effectiveness. The benchmark's central contribution—evaluating skill construction, retrieval, orchestration, and task execution on a unified executable task set, does not depend on treating this stratification as a causal intervention.

W2. Construction is entangled with official skills/Claude Code; tests might encode the reference implementation.

We agree that construction provenance matters. Official skills are used during DAG-guided exploration to establish that a plausible multi-skill workflow exists; the exploration report then guides instruction and reference-solution synthesis. This makes the tasks grounded in feasible procedural workflows, but does not automatically make the resulting tests implementation-neutral. We will state this source of potential inductive bias explicitly and refrain from claiming that official-skill advantages are unconfounded. Our existing safeguards operate at several levels: skill and edge filtering, a separate instruction-quality rubric, trajectory/process verification for the proposed solution, test-instruction alignment review, candidate rollouts, and independent review by three annotators (κ = 0.834; Appendix D.1). Importantly, we agree that passing the reference solution is necessary but insufficient to establish verifier completeness.

Meanwhile, the existing failure analysis explicitly reports skill-induced regressions, including knowledge bias and instruction override (Figure 3). Therefore, we would like to clarify that injecting skills does not necessarily lead to improved task performance.

AuthorsOpenReview ↗

Response to Reviewer wNVu

W2 & Q3. Clarification and Correction of Table 2 Statistics

Thank you for carefully identifying the numerical inconsistencies in Table 2, particularly the DeepSeek-V4 results. Upon auditing the evaluation records, we identified a specific issue in the DeepSeek-V4 score aggregation: a subset of timed-out runs was inadvertently excluded from the denominator rather than counted as unsuccessful attempts. We apologize for this oversight. Importantly, these runs were attempted; the issue arose from how timeouts were handled during score aggregation, rather than from missing experimental executions. We have corrected the affected DeepSeek-V4 results as follows:

DeepSeek-V4 settingOriginally reportedCorrected
No Skill — Easy65.28%61.04%
No Skill — Hard23.38%21.95%
Official Skill — Easy69.44%64.94%
Official Skill — Hard29.87%28.05%

Importantly, this correction does not change the qualitative finding for DeepSeek-V4: Official Skills continue to improve performance over No Skill by 3.90 percentage points on Easy tasks and 6.10 percentage points on Hard tasks, respectively. We have also audited the remaining model–condition results and confirmed that no missing runs or analogous denominator omissions were found. Therefore, this was a specific aggregation issue affecting the reported DeepSeek-V4 scores, rather than a systematic inconsistency in the evaluation protocol. We thank the reviewer again for identifying this issue, which has helped us improve the accuracy and transparency of our experimental reporting.

W3 & Q4. Skill Composition: Wild Skill Discovery, Retrieval Budget, and Alternative Workflows

Thank you for raising these important questions. We clarify three aspects of our skill composition evaluation.

  1. Realistic skill discovery in the wild. The 2,000+ skill pool is designed to evaluate whether agents can effectively discover, select, and load relevant skills from a large, realistic skill repository, rather than relying on task-specific skills provided in advance. This setting is consistent with the motivation of SkillRouter and AgentSkillOS, which also investigate skill discovery and orchestration in large-scale repositories. Our goal is to evaluate not only whether agents can execute a known workflow, but also whether they can identify and organize the required procedural resources in the wild.
  2. Clarification of the Top-10 retrieval budget. Importantly, the Agent baseline is also evaluated under a Top-10 selection setting, as already stated in Section 4.4. Specifically, agents are instructed to identify up to 10 most relevant skills, rather than being restricted to approximately 3.9 skills. The reported 3.86 average selections in Table 3 reflect the number of skills agents actually choose, not an imposed retrieval budget. Since our synthesized tasks involve 1–5 skills, including workflows requiring more than three, this distinction is important. Both agents and specialized retrievers operate under a Top-10 selection limit, while their actual selection behavior differs.
  3. Alternative valid workflows. We agree that the reference DAG represents one validated composition rather than the only possible valid workflow. Our retrieval and orchestration completeness metrics are intended as strict diagnostics of reference-workflow recovery, rather than exhaustive measures of all valid compositions. Importantly, we additionally evaluate end-to-end task success in the wild (Table 5), where agents are assessed based on whether their final outputs satisfy task requirements, without requiring exact recovery of the reference skill set or DAG. For example, Kimi-K2.6 decreases from 34.15% to 25.61% on Hard tasks when moving from provided Official Skills to the In-Wild setting. This demonstrates that the challenge of skill discovery and composition also manifests in downstream execution, beyond reference-graph matching.
AuthorsOpenReview ↗

Response to Reviewer wNVu

W4 & Q5. Authenticity and Provenance of Skill-Creator Source Documents

Thank you for raising this important question. We clarify that authentic documentation is always prioritized in our construction pipeline, and LLM-generated surrogate documents are used only as a fallback when no suitable external documentation can be retrieved.

Specifically, approximately 85.5% of the source documents are retrieved from real-world resources, while only 14.5% are synthesized from their corresponding original skills. Importantly, these synthesized documents are not accepted without verification. They undergo human review to ensure that the content is technically plausible, consistent with the original skill's functionality, and preserves essential procedural information without omitting critical guidance.

Therefore, surrogate documentation represents a limited, quality-controlled fallback mechanism, rather than the primary source of our benchmark. Moreover, the underlying skills themselves are collected from real community repositories, regardless of whether their supporting documentation is retrieved or synthesized.

W5 & Q6. Code, Tasks, and Environment Availability

Thank you for highlighting the missing artifact links. We clarify that the implementation and benchmark data have already been publicly released, rather than being deferred until acceptance.

To preserve the double-blind review process, we have refrained from including links to repositories that may contain identity-revealing information. We acknowledge that this limits independent verification during the review process. To address this concern, we will provide anonymized copies of the code, benchmark tasks, and environment specifications through the conference's permitted supplementary-material or reviewer-only channels, allowing reviewers to inspect the artifacts without compromising anonymity.

We appreciate the reviewer's emphasis on reproducibility and will clarify the artifact availability and access instructions in the revised submission.

Reviewer_65HDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SkillNet-Gym introduces a benchmark for evaluating how LLM agents create, retrieve, and combine skills. It builds a graph from real-world skills and uses it to generate executable tasks. Experiments across eight models show that skills generally help on harder tasks, but generating useful skills and combining skills from large repositories remain challenging.

Critical strengths

I think the paper addresses an important problem by looking at different stages of skill use, including construction, retrieval, orchestration, and execution, within the same benchmark. This makes it easier to understand where agents struggle, rather than just measuring whether providing skills improves performance. I also like that the authors evaluate agents both when the correct skills are provided and when they have to find them in a large repository. The results highlight a meaningful gap between these two settings.

Another strength is the effort put into building and validating the benchmark. The authors collect over 2,000 real-world skills and use them to generate around 160 executable tasks with different workflow structures, including chains, fan-in, fan-out, and diamonds. They also use several quality-control steps, including execution checks, test refinement, and human evaluation by three annotators. This makes the benchmark more convincing than one built entirely from synthetic instructions, and the analysis of failure cases provides useful insights into when skills help or hurt agent performance.

Critical weaknesses

One of my main concerns is that the paper presents SkillNet-Gym as a dynamic benchmark, but does not actually evaluate this aspect. All experiments are conducted on a fixed set of 159 tasks, so it is unclear how well the framework would handle new skills or changes to existing ones. While the proposed pipeline should support such updates, the paper does not show whether newly generated tasks would remain reliable or have comparable difficulty. An experiment demonstrating how the benchmark changes over time, including the cost of generating and validating new tasks, would strengthen this claim.

I also have some concerns about the skill composition evaluation. The metrics require agents to recover all reference skills and dependencies, even though Table 13 allows skills that are useful but not necessary to complete the task. This means an agent could come up with a different but equally valid workflow and still receive a low score. It is therefore unclear how well these metrics reflect actual composition ability. Evaluating whether the predicted workflows can successfully complete the tasks, or at least showing how composition scores relate to task success, would help address this.

The paper suggests that compressing documents into skills leads to information loss, but I don't think the experiments fully support this explanation. The authors compare Skill-Creator against official skills, without testing what happens when agents use the original documents directly. It is therefore hard to tell whether the lower performance comes from information being lost during compression or simply from differences in the quality of the generated and official skills. A more direct comparison using the same source documents, with and without skill construction, would help clarify this.

Finally, I have some concerns about the experimental setup and reporting. Tasks are filtered using GLM-5.1's performance, and the Easy/Hard split is determined afterward, which could affect the conclusions about task difficulty and skill effectiveness. The main results also lack uncertainty estimates, even though some analyses are based on relatively few tasks. There also seems to be an inconsistency in the reported numbers: the 65.58% score over 77 Easy tasks does not appear consistent with the stated three-run binary evaluation protocol. These details should be clarified to make the results easier to assess.

Questions for authors

.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 65HD

Dear Reviewer 65HD,

Thank you for highlighting the important distinction between skill use and skill discovery in a large repository, and for recognizing the effort behind the real world skill collection, executable task construction, and multi-stage validation. Your comments identify several places where the original framing was stronger than its supporting evidence; we have prioritized them below.

W1. Dynamic is a design claim without an update experiment or cost analysis.

We agree that building the automated pipeline only once is insufficient to demonstrate the dynamic nature of the benchmark. To provide empirical evidence, we sampled 20 tasks from the original benchmark (see the note below for details on the sampled tasks), updated their skills using the latest available skill resources, and evaluated each task over three independent runs. We report both Avg@3 and Pass@3 for each model under different skill-access settings:

ModelSettingAvg@3Pass@3
Kimi-K2.6No Skills55.00%65.00%
Kimi-K2.6Original Tasks51.67%70.00%
Kimi-K2.6Updated Tasks53.33%60.00%
GLM-5.1No Skills48.33%60.00%
GLM-5.1Original Tasks46.67%55.00%
GLM-5.1Updated Tasks45.00%60.00%
Sonnet 4.6No Skills50.00%60.00%
Sonnet 4.6Original Tasks43.33%55.00%
Sonnet 4.6Updated Tasks43.33%55.00%

These results demonstrate that tasks evaluated with dynamically updated skill resources remain challenging across different models, providing empirical evidence that the benchmark can accommodate skill updates while retaining its ability to assess agent performance.

W2. Reference-skill completeness penalizes valid alternative workflows.

We acknowledge that reference skills, dependency edges, and workflows may not fully capture an agent's actual execution process, as agents may follow alternative valid paths to complete a task.

Importantly, Table 5 already reports end-to-end execution results with a repository of over 2,000 skills, where performance is evaluated based on final task success rather than exact matching with the reference skill graph.

For example, on Hard tasks:

  • Kimi-K2.6: Performance drops from 34.15% with Official Skills to 25.61% in the In Wild setting.
  • GLM-5.1: Performance drops from 32.93% with Official Skills to 24.39% in the In Wild setting.

These results provide independent evidence that retrieving and executing skills from a large repository introduces additional challenges, even when agents may succeed through alternative valid workflows. However, our trajectory analysis further reveals that agents rarely attempt to discover or load skills in the In Wild setting.

We therefore argue that directly evaluating an agent's ability to retrieve relevant skills and construct skill workflows provides valuable diagnostic insights that may not be captured by end-to-end task success alone. Accordingly, we view reference-workflow recovery and end-to-end task success as complementary rather than interchangeable measures.

AuthorsOpenReview ↗

Response to Reviewer 65HD

W3. Information loss is not isolated without direct Document-only execution.

We thank the reviewer for this suggestion. We conducted the requested Document-only execution experiment, comparing all four settings on the same 20-task subset, with three runs per task and report Avg@3.

ModelNo SkillOfficial SkillSkill-CreatorDocument-only
Kimi-K2.655.0063.3361.6756.67
GLM-5.146.6758.3351.6756.67
Sonnet 4.650.0055.0053.3358.33
Average50.5658.8955.5657.22

The results support our central observation while clarifying its interpretation. One-shot compression necessarily reduces the amount of source-level detail preserved in a skill. Consistent with this information bottleneck, Document-only outperforms Skill-Creator on average (57.22% vs. 55.56%). However, access to more complete information does not guarantee better execution: GLM and Sonnet benefit from direct documents, whereas Kimi performs better with Skill-Creator.

To understand this discrepancy, we examined execution traces and identified three factors:

  • Reading and planning overhead. On GLM task 149, Document-only delayed execution and caused three timeouts (0/3 vs. 3/3 with Official Skill).
  • Resource discovery and utilization. On Kimi task 136, the agent did not read the provided documents in any of its three runs, illustrating that document availability does not ensure effective usage.
  • Execution and verification differences. On Sonnet task 119, Document-only preserved the required Checkov JSON structure more often, while other differences involved parameter choices or verifier-specific requirements.

These findings highlight an important distinction: one-shot compression introduces an inherent information bottleneck, while the practical value of retaining full documents depends on agents' ability to discover, interpret, and execute them efficiently. We will revise our claim accordingly, distinguishing the inherent loss of source-level detail from experimentally observed differences in task performance, which also reflect interface design, execution overhead, and model-specific behavior.

AuthorsOpenReview ↗

Response to Reviewer 65HD

W4. Post-hoc Easy/Hard split, GLM-5.1 filtering, uncertainty, and Table 2 arithmetic.

Thank you for identifying these issues. We clarify that structural complexity, candidate filtering, and empirical difficulty stratification serve different purposes:

  1. Design-time structural complexity. Skill count (1–5) and DAG topology (single, chain, fan-in, fan-out, and diamond) are determined during task construction, before final model evaluation. Our existing Figure 4(b) analyzes skill effects across these pre-specified structures.
  2. Candidate retention. We exclude candidate tasks that GLM-5.1 solves without skills in all three preliminary trials, to reduce saturation by tasks that do not require external procedural support. This rule was not optimized on the improvement from Official Skills.
  3. Empirically calibrated Easy/Hard stratification. The final 77 Easy / 82 Hard split is intentionally based on post-hoc empirical performance, as realized task difficulty cannot be reliably determined from structural properties alone and requires empirical calibration. While skill count and DAG topology provide a priori measures of compositional complexity, they do not necessarily translate into a universal difficulty ordering. Importantly, we explicitly investigate this relationship in our existing analysis, "Topology-Related Difficulty Is Context Dependent" (Figure 5). For example, among tasks involving 3–5 skills, fan-out tasks are substantially harder than chains in Biomedical Processing (4.2% vs. 73.6% No-Skill success), whereas this pattern does not hold in Document Processing (38.9% vs. 33.3%). These findings demonstrate that task difficulty depends jointly on dependency structure, task semantics, and empirical agent capabilities, rather than topology alone. Therefore, our post-hoc stratification, with domain balancing, is a deliberate choice to capture realized difficulty, while the pre-specified topology analysis in Figure 4(b) provides complementary evidence on how structural complexity influences skill effectiveness. We will clarify the exact stratification procedure in the revision.
  4. We also thank the reviewer for catching the Table 2 arithmetic inconsistency. Our audit isolated a DeepSeek-V4-specific aggregation error: several timed-out attempts were inadvertently omitted from the denominator, although they should have been counted as failures under the binary evaluation protocol. The attempts were executed; the error concerned timeout accounting, not missing experimental runs. We apologize for this oversight and have corrected the four affected entries:
DeepSeek-V4 settingOriginally reportedCorrected
No Skill — Easy65.28%61.04%
No Skill — Hard23.38%21.95%
Official Skill — Easy69.44%64.94%
Official Skill — Hard29.87%28.05%

We have also checked the remaining models and settings and found no missing repetitions or comparable denominator exclusions. In particular, the identical Kimi-K2.6 and GLM-5.1 No-Skill percentages are not attributable to incomplete runs. We appreciate the reviewer's scrutiny, which has improved the precision and transparency of our reporting.

Reviewer_1awfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SkillNet-Gym, a dynamic evaluation benchmark designed to address the static nature of existing LLM agent skill benchmarks. Grounded in real-world community repositories (such as ClawHub and SkillHub), SkillNet-Gym automatically constructs an evolving, heterogeneous Directed Acyclic Graph (DAG) called SkillNet. Using this graph, the system synthesizes domain-diverse, multi-skill tasks featuring structured topologies (e.g., chains, fan-in, fan-out, and diamond patterns) alongside verifiable unit tests and reference solutions. The benchmark evaluates LLM agents across four distinct stages of the skill lifecycle: No Skill, Official Skill, Skill Construction, and Skill Composition. Extensive evaluations across eight frontier models (e.g., GPT-5.4, Claude 4.6 Sonnet) demonstrate that while official skills substantially improve performance on challenging, highly compositional tasks, frontier agents still struggle with retrieving relevant skills and orchestrating complex dependency workflows in open-world settings.

Critical strengths
  1. Dynamic & Community-Grounded Benchmark Design: Unlike traditional static benchmarks that suffer from data drift as skill ecosystems evolve, SkillNet-Gym continuously incorporates real-world community artifacts into a heterogeneous DAG. This allows it to automatically synthesize updated, domain-diverse tasks with multi-stage quality control, ensuring long-term benchmark validity and relevance.

  2. Unified Evaluation Across the Full Skill Lifecycle: The framework expands evaluation beyond simple skill usage by introducing a comprehensive four-stage evaluation methodology. This unified protocol isolates and diagnoses agent performance across direct execution, document-to-skill compression (construction), and large-scale repository retrieval and dependency-aware workflow orchestration (composition).

Critical weaknesses
  1. Dependence on LLM-Driven Benchmark Construction: Although the SkillNet-Gym pipeline includes multi-stage quality control and human cross-validation, the foundational steps,which such as skill collection, scenario extraction, edge verification, task synthesis, and test generation, which rely heavily on LLM-based components. This introduces potential error propagation or subtle inaccuracies in specialized vertical domains.

  2. Restriction to Deterministic Unit Tests: The benchmark currently evaluates only tasks whose outcomes can be deterministically verified using unit tests (e.g., test.sh). It does not cover open-ended, non-deterministic, or highly creative tasks where exact-match or code-based verification fails.

  3. Single-Pass Skill Construction Strategy: The skill construction baseline relies on a single-pass compression/distillation approach (e.g., summarizing documents into SKILL.md). This strategy often causes information loss and limits agent performance compared to iterative or self-evolving skill pipelines.

Questions for authors

see Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1awf

Dear Reviewer 1awf,

Thank you for recognizing both the dynamic, community-grounded benchmark design and the unified evaluation across the skill lifecycle. We appreciate your concerns about LLM-dependent construction, deterministic verification, and single-pass skill distillation. We address them in turn.

W1. LLM-driven construction may propagate subtle errors, especially in specialized domains.

We agree that LLM-based synthesis is not, by itself, a correctness guarantee. Our design therefore uses the LLM primarily to propose and organize candidate structures, followed by several distinct checks. At the skill level, quality filtering and deduplication reduce 8,159 candidates to 2,225 retained skills. Candidate workflow edges are supported by pre/post scenario evidence and separately verified, reducing 255,965 candidate edges to 53,904 retained edges. At the task level, we evaluate instruction clarity and DAG compliance, inspect the reference-solution trajectory with a process verifier, validate executable outputs, and review test alignment against the task instruction. Finally, three independent annotators review the retained tasks, with Fleiss' κ = 0.834 (Appendix D.1). These verification mechanisms help mitigate the risk of error propagation to some extent. We will further clarify the remaining risks and limitations of our validation process in the revised manuscript.

W2. Deterministic tests exclude open-ended and creative tasks.

This is a deliberate scope boundary rather than a claim of coverage. SkillNet-Gym aims to diagnose procedural and compositional competencies where a final artifact can be tested reproducibly. Deterministic tests make it possible to compare many agent models and execution settings under a consistent protocol, and the tasks still span heterogeneous domains and workflows rather than only isolated code exercises. We already note this limitation in Appendix A. We agree that open-ended design, aesthetic judgment, and other non-deterministic outcomes need complementary evaluation methods (e.g., LLM-as-Judge with rubrics). We will clarify that SkillNet-Gym is a benchmark of verifiable compositional skill execution, not a complete benchmark of every form of agent creativity. Broadening evaluation beyond deterministic tests is future work, and should not be conflated with a flaw in the validity of the tasks within the stated scope.

W3. Single-pass Skill-Creator compression may lose information; stronger iterative methods are absent.

Our intention is to diagnose a widely used single-pass construction regime, not to claim that every possible skill-learning algorithm has an intrinsic compression ceiling. To clarify the mechanism, We added a Document-only execution condition alongside No Skill, Official Skill, and Skill-Creator on the same 20-task subset. We report Avg@3.

ModelNo SkillOfficial SkillSkill-CreatorDocument-only
Kimi-K2.655.0063.3361.6756.67
GLM-5.146.6758.3351.6756.67
Sonnet 4.650.0055.0053.3358.33
Average50.5658.8955.5657.22

Averaged across three models, Official Skill achieves 58.89%, Document-only 57.22%, and Skill-Creator 55.56% Avg@3. This supports the view that procedural guidance is useful while a one-pass distillation step may fail to preserve its downstream utility. At the same time, the model dependent ordering and trajectory evidence show that document discovery, loading overhead, and execution decisions also influence these results.

We believe that single-pass skill construction may discard, obscure, or fail to operationalize task-relevant information, with its effectiveness depending on the model and workflow. As an additional cross-check, the existing SkillSeeker comparison in Table 4 also reveals heterogeneous outcomes rather than a single universally superior construction algorithm. Iterative, feedback-driven construction is an important future direction, but is not a prerequisite for establishing the limitations of the single-pass baselines tested here. We thank you again for helping us improve the precision of these claims. We hope these clarifications and additional evidence address your concerns.

AuthorsOpenReview ↗

Kindly Reminder

Dear reviewer 1awf,

We highly appreciate the constructive comments and insightful suggestions you have offered for our work. As the deadline of our discussion period is approaching, in order for us to have sufficient time to address any additional questions you may have, we kindly encourage you to engage in the ongoing discussion and share anyfurther insights or clarifications you may have.

If you find that our response addresses some of your concerns, wouId you kindly consider raising your rating score for our paper?

Thank you very much for your time and consideration. We look forward to hearing from you soon.

Best Regards,

Authors of SkillNet-Gym

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

Global Response

Dear all reviewers,

Thank you for your thoughtful and technically detailed feedback. We appreciate your recognition of (i) the community-grounded, extensible benchmark construction pipeline (1awf), (ii) the unified evaluation of skill construction, discovery, orchestration, and execution (1awf, 65HD), and (iii) the executable Docker-based protocol and useful failure analyses (65HD, wNVu). Your comments have helped us clarify the distinction between what our framework enables, what the current experiments establish, and where further validation is needed.

Common Concerns and Our Responses

1. Dynamic Extensibility: A 20-Task Skill-Update Experiment (65HD, wNVu)

We agree that a one-time evaluation of 159 tasks does not, by itself, demonstrate updating behavior. We therefore sampled 20 tasks from the initial benchmark, refreshed their associated skills using the latest available resources, and evaluated three models under No Skills, Original Skills, and Updated Skills, with three runs per task. We report both Avg@3 (mean trial success) and Pass@3 (fraction of tasks solved at least once):

ModelSettingAvg@3Pass@3
Kimi-K2.6No Skills55.00%65.00%
Kimi-K2.6Original Tasks51.67%70.00%
Kimi-K2.6Updated Tasks53.33%60.00%
GLM-5.1No Skills48.33%60.00%
GLM-5.1Original Tasks46.67%55.00%
GLM-5.1Updated Tasks45.00%60.00%
Sonnet 4.6No Skills50.00%60.00%
Sonnet 4.6Original Tasks43.33%55.00%
Sonnet 4.6Updated Tasks43.33%55.00%

The updated resources remain challenging across all three models. This directly reinforces our central observation that skill utility is conditional on the agent and workflow. The pilot provides empirical evidence that the framework can accommodate and evaluate refreshed skill resources; we will distinguish it from the stronger, not-yet-established claim of longitudinal equivalence across independently regenerated benchmark snapshots.

2. Direct-Document Control: Clarifying One-Shot Skill Construction (1awf, 65HD)

To address the missing document baseline, we added Document-only to the same 20-task evaluation, alongside No Skill, Official Skill, and Skill-Creator:

ModelNo SkillOfficial SkillSkill-CreatorDocument-only
Kimi-K2.655.00%63.33%61.67%56.67%
GLM-5.146.67%58.33%51.67%56.67%
Sonnet 4.650.00%55.00%53.33%58.33%
Average50.56%58.89%55.56%57.22%

Document-only exceeds Skill-Creator on average (57.22% vs. 55.56%), but the ordering varies by model: Kimi performs better with Skill-Creator, while GLM and Sonnet perform better with direct documents.

Trajectory inspection identifies not only potential information bottlenecks, but also differences in resource discovery, planning overhead, execution choices, and verifier requirements. We therefore refine—not discard—our original conclusion: one-shot skill distillation can discard, obscure, or fail to operationalize task-relevant procedural guidance, but observed downstream gaps cannot be attributed exclusively to textual compression.

3. Empirical Difficulty Versus Pre-Specified Compositional Structure (65HD, wNVu)

We distinguish design-time topology, candidate retention, and realized difficulty.

Skill counts (1–5) and DAG patterns (single, chain, fan-in, fan-out, diamond) are defined before downstream model evaluation; Figure 4(b) already analyzes skill effects by these pre-specified structures.

Candidate filtering removes tasks solved by GLM-5.1 without skills in all three preliminary runs to avoid saturation; this was not optimized for gains from Official Skills, though we recognize that it can change the retained difficulty distribution.

The 77 Easy / 82 Hard split is deliberately calibrated post hoc from empirical behavior, with domain balancing, because topology alone does not reliably determine realized task difficulty.

Indeed, our existing "Topology-Related Difficulty Is Context Dependent" analysis (Figure 5) directly examines this relationship: for tasks with 3–5 skills, fan-out versus chain No-Skill success is 4.2% versus 73.6% in Biomedical Processing, but 38.9% versus 33.3% in Document Processing.

AuthorsOpenReview ↗

Global Response

4. Table 2 Accounting: Explicit DeepSeek-V4 Correction (65HD, wNVu)

We thank the reviewers for identifying a reporting error. Our audit found a DeepSeek-V4-specific aggregation issue: some timed-out attempts had been inadvertently omitted from score denominators rather than counted as failures. The attempts were executed; no experimental runs were missing. We apologize and correct the affected Table 2 entries:

DeepSeek-V4 settingOriginally reportedCorrected
No Skill — Easy65.28%61.04%
No Skill — Hard23.38%21.95%
Official Skill — Easy69.44%64.94%
Official Skill — Hard29.87%28.05%

Under the three-run binary protocol, the full denominators are 231 trials for 77 Easy tasks and 246 for 82 Hard tasks, including timeouts as failures. DeepSeek-V4's qualitative result remains: Official Skills improve success by +3.90 percentage points on Easy and +6.10 on Hard. We also audited the other model–condition results and found no missing repetitions or analogous denominator omissions; the identical Kimi/GLM No-Skill values are not attributable to incomplete runs.

5. Composition Validity and Wild-Repository Evaluation (65HD, wNVu)

The 2,000+ skill pool targets genuine discovery and orchestration in a large repository, in the spirit of SkillRouter and AgentSkillOS, rather than only execution with supplied skills.

In Table 3, Top-10 is also the agent's selection limit; the reported 3.86 selected skills on average is the agent's actual output, not a smaller imposed budget. We will make the skill-access/loading interface and this distinction explicit. The comparison reflects different selection behaviors under a common Top-10 limit, not necessarily identical returned-set cardinalities.

We also agree that a reference DAG denotes one validated composition, not the only valid workflow. Consequently, binary orchestration completeness is a strict reference-workflow recovery diagnostic.

It is complemented by the end-to-end In-Wild evaluation in Table 5, which judges final artifacts without exact reference-DAG matching. On Hard tasks, Kimi-K2.6 drops from 34.15% (Official Skill) to 25.61% (In Wild), and GLM-5.1 from 32.93% to 24.39%.

These measurements jointly show that large-pool skill use presents practical challenges while distinguishing structural recovery from final task completion.

6. Construction Reliability, Scope, Provenance, and Artifact Access (1awf, wNVu)

Our construction pipeline couples LLM-driven proposals with skill filtering, edge verification, instruction/test alignment checks, reference-trajectory verification, candidate rollouts, and human review by three annotators (Fleiss' κ = 0.834).

We recognize that official-skill-guided exploration and reference-derived tests can introduce implementation-specific bias: passing a reference solution alone does not establish that every independent correct implementation will be accepted. We will clarify this provenance and examine checks that exceed the instruction contract; the skill-induced regressions already documented in Figure 3 also show that Official Skills do not automatically confer an advantage.

Deterministic verification is a deliberate scope choice for reproducible evaluation of procedural/compositional tasks, not a claim to cover open-ended aesthetic or creative work.

Regarding documentation provenance, 85.5% of Skill-Creator source documents come from real-world resources; 14.5% are LLM-generated fallbacks used only when suitable external documentation is unavailable and human-reviewed for plausibility, functionality, and preservation of critical instructions. The underlying skills themselves originate from community repositories.

Finally, the implementation and benchmark data have already been released on GitHub and Hugging Face, respectively to preserve double-blind review, we will provide anonymized access to code, tasks, and environment specifications through permitted review channels rather than link identity-revealing repositories.

Contribution and Takeaway

Our contribution is not merely a fixed set of 159 tasks. SkillNet-Gym combines a community-grounded, extensible task-construction pipeline with a unified executable protocol for diagnosing the full skill lifecycle.

The supplementary evidence strengthens its central message: neither adding more documentation, providing a skill, nor refreshing that skill guarantees improved execution. Effective agents must discover, select, organize, and apply procedural knowledge in context.

We appreciate the reviewers' feedback, which has helped make these claims more precise, transparent, and reproducible.

Thank you again for your time and constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.