- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SkillGym, an automatic pipeline that transforms community-written agent skills into verifiable training environments for skill-use agents, which comprises three stages: 1) Skill Collection and Curation: skills are crawled from public sources and filtered by completeness, offline executability, and quality, yielding a pool of about 11.9k skills; 2) Skill-Grounded Task Construction: a two-stage builder–reviewer agent system first builds Docker environments and then authors tasks under four reasoning-structure profiles (procedural, abductive, constraint satisfaction, and partial order), each with an instruction, initial workspace, reference solution, and executable verifier, filtered by validity and quality gates; and 3) Trajectory Collection and Training: three teacher models collect successful trajectories across four agent harnesses for supervised fine-tuning. With 6.8k tasks and 19k trajectories, experiments on five models from 2B to 27B show improvements on four skill-use benchmarks, including tasks with skills held out from training.
Critical strengths
- Training agents to use externally provided skills is a practical and timely problem, and turning public skills into executable training environments is a reasonable direction.
- The evaluation covers multiple model families and sizes and several external skill-use benchmarks, and the paper includes a held-out skill split.
Critical weaknesses
-
Limited novelty. Synthesizing verifiable tasks from public skills and distilling teacher trajectories via SFT has already been done by SKT [1], which the paper cites, and profiling large skill corpora and synthesizing tasks for verifiable skills is also explored in [2]. More broadly, LLM-based environment building with build–test–repair loops and fail-before/pass-after validity checks is standard in recent environment-synthesis work [3, 4, 5]. The claimed differences, namely four reasoning-structure profiles and an LLM reviewer, are incremental engineering choices, and the training recipe is plain SFT distillation without any methodological contribution.
-
The task construction process lacks essential details. The main text describes the core pipeline in only a few sentences, and even the appendix provides condensed rather than full prompts. Key information is missing: 1) which models serve as builder and reviewer; 2) the pass and skip rates at each gate, which would explain why only 3,494 of 11,897 curated skills yield tasks; 3) how the test split and held-out skills are selected, and whether tasks from the same skill can appear in both training and held-in test sets; 4) the dataset statistics are inconsistent: the 6,772 tasks include 1,581 reviewer-unresolved tasks, and partial-order tasks are stated to be excluded from these counts, so it is unclear what the final training set contains.
-
Quality control is insufficient for a single cross-domain pipeline. 1) The same pipeline is applied to all 19 domains, although correctness criteria differ substantially across them, and the requirement of value-based verification excludes or reshapes skills with subjective outputs (e.g., writing skills are excluded entirely); 2) quality relies entirely on automatic checks and an LLM reviewer whose transcripts were not stored, and no human or expert audit of tasks or verifiers is reported; 3) skill dependency is not verified per task, unlike SKT [1], which keeps a task only if a run with the skill outperforms a run without it. As a result, Kimi-K3 solves 60.5% of test tasks without skills, with only 3–4 points of gain on abductive and partial-order tasks, suggesting that many tasks are skill-helpful rather than skill-critical.
-
The experiments lack strong baselines. The main comparison is only between each base model and its SFT version. 1) The comparison with SKT copies numbers from its paper under a different harness, rather than training on SKT data under the same setting; 2) there is no comparison with other agentic training data of similar scale (e.g., [3, 4, 5]) or with a simple baseline such as SFT on the same number of teacher trajectories collected without skills, so it is unclear whether the gains come from skill-grounded tasks or simply from distilling stronger teachers; 3) no training-free skill-use methods or RL-based alternatives are compared.
-
The evidence for improved skill use is weak. 1) The SkillGym test set comes from the same pipeline as the training data, while on the human-curated SkillsBench the gains are much smaller, the 2B model degrades, and improvements appear only for skills that provide reference documentation; 2) the largest gain on Skill-Use-Bench comes from the Trigger score (28 to 96), which may mainly reflect learning to open the skill file rather than applying it, as acknowledged by the authors; 3) the SkillGym test set and Skill-Use-Bench are evaluated with a single run, and all data-ablation models fall below the base model on SkillEval, making these conclusions unreliable.
[1] SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation. arXiv:2608.02287, 2026.
[2] SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution. arXiv:2605.18401, 2026.
[3] Endless Terminals: Scaling RL Environments for Terminal Agents. arXiv, 2026.
[4] SETA: Scaling Environments for Terminal Agents. arXiv, 2026.
[5] LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents. arXiv:2605.29559, 2026.
Questions for authors
-
SkillEval, SkillsBench, and Skill-Use-Bench also draw on publicly available skills, some from the same sources (e.g., skills.sh) as SkillGym. Did you check for overlap between the training skills or tasks and the skills used in these external benchmarks, and were overlapping skills removed before training?
-
On the SkillGym test set, success on held-out skills is higher than on held-in skills for both the base and the SFT model. How are the two splits constructed and matched in difficulty and domain? Does this indicate that the held-out split is easier, and if so, how should the claimed generalization to unseen skills be interpreted?
4 comments
Response to Reviewer 8gYd # 2
W3
Quality control is insufficient for a single cross-domain pipeline. 1) The same pipeline is applied to all 19 domains … writing skills are excluded entirely; 2) quality relies entirely on automatic checks and an LLM reviewer whose transcripts were not stored, and no human or expert audit of tasks or verifiers is reported; 3) skill dependency is not verified per task, unlike SKT … many tasks are skill-helpful rather than skill-critical.
-
The process is shared across domains, but the verifier is not. Each verifier is written for its own task from the content of the skill, so the correctness criteria are specific to the task. The checks also behave consistently across the 19 domains. Outside software engineering, between 84% and 92% of built tasks pass the validity checks in every domain, and between 41% and 57% pass review. Writing skills are excluded since their outputs are subjective and would need an LLM judge rather than an executable check, so we left them out to keep every task verifiable by execution. Together with skills about agents themselves, this excludes 9% of the skill pool.
-
In the submitted version, quality is verified mainly by execution and the reviewer model. Every task must pass on its reference solution and fail on an empty attempt, and a reviewer model then examines the verifier. To examine the verifiers directly, we audited the graders of 50 randomly sampled training tasks for problems that make the verdict on the task's outcome unreliable, such as grading that is too strict and checks requirements the task does not explicitly state. For comparison, we applied the same procedure to other datasets. Because of the time limit of the rebuttal, we checked 20 samples from each of them.
| Dataset | Tasks checked | Tasks with a grader problem |
|---|---|---|
| SkillGym (ours) | 50 | 7 (14%) |
| Skill-Use-Bench | 20 | 3 (15%) |
| SkillsBench | 20 | 6 (30%) |
| Endless Terminals | 20 | 6 (30%) |
| SkillEval | 20 | 9 (45%) |
The rate for SkillGym is at the low end of these datasets, including the human-written SkillsBench. Given the sample sizes, we only read this comparison as indicative.
- Making tasks skill-helpful rather than skill-critical is a deliberate choice. Whether a skill is needed depends on the model that attempts the task. Of the 42 test tasks that the 9B base model solves only with the skill, Kimi-K3 solves 35 without it, so a filter run with one model does not hold for another. SKT's filter is an alternative, and we chose instead to keep tasks that a strong model can solve unaided, since they still require a weaker model to consult and apply the skill. The tasks do reward skill use for the models we train. On our test set the skill raises the pass rate of the 9B model by 7.5 points before training and by 16.5 points after it. The revised paper reports the audit in Appendix N.
W4
The experiments lack strong baselines. … 1) The comparison with SKT copies numbers from its paper under a different harness …; 2) there is no comparison with other agentic training data of similar scale (e.g., [3, 4, 5]) or with a simple baseline such as SFT on the same number of teacher trajectories collected without skills …; 3) no training-free skill-use methods or RL-based alternatives are compared.
-
We now compare with SKT under the same setting, as reported under W1.
-
Among the datasets you cite, LiteCoder-Terminal-SFT [5] provides the largest released set of trajectories, so we trained the same base model on it with the same number of tokens (36M) and the same recipe. In Terminus-2, the harness it was collected in, it reaches 25.2 on our test set, against 45.2 for the SkillGym subset and 37.8 for the base model.
We also ran the baseline you describe. For 1010 training tasks, we collected trajectories from the same teacher in the same harness with and without the skill, and trained one model on each set.
| Trained on | Test set | Reads the skill | Skill-Use-Bench SU |
|---|---|---|---|
| Trajectories with skills | 47.3 | 61.7% | 57.0 |
| Trajectories without skills | 46.8 | 19.1% | 13.4 |
| None (base model) | 42.6 | 23.0% | 18.8 |
Both models improve the pass rate over the base model by a similar amount, so at this scale that gain comes from the tasks and the teacher. Skill use, however, is learned only from trajectories collected with skills. The model trained without them reads and follows skills no more than the base model.
- As a training-free baseline, we instructed the base model explicitly to read the relevant skill before acting. As reported under W5, this does not improve it. Training-free methods act at inference time and can be combined with a trained model. RL-based alternatives are time-consuming and outside the scope of this paper, and we leave them to future work.
The revised paper reports the comparison with public data in Appendix F, the baseline without skills in Appendix O, and the prompt-only control in Appendix H.
Response to Reviewer 8gYd # 3
W5
The evidence for improved skill use is weak. 1) The SkillGym test set comes from the same pipeline as the training data, while on the human-curated SkillsBench the gains are much smaller, the 2B model degrades, and improvements appear only for skills that provide reference documentation; 2) the largest gain on Skill-Use-Bench comes from the Trigger score (28 to 96) …; 3) the SkillGym test set and Skill-Use-Bench are evaluated with a single run, and all data-ablation models fall below the base model on SkillEval …
-
Three of our four benchmarks are from other work, and the trained model improves on all of them. On our own test set, half of the tasks use skills that never appear in training, and the gain there (+19.5) is as large as on seen skills. On SkillsBench the gain grows with model size, from +7.6 at 9B to +14.8 at 27B and +23.5 at 122B. The 2B model is the only one of the six that does not improve on it.
-
Consulting a provided skill is itself a capability, and one that prompting does not produce. When we instruct the base model explicitly to read the relevant skill first, it reads the skill no more often (21.9% of test tasks against 23.0%) and does not score higher. The gain is also not limited to consultation. On the tasks where both the base and the trained model read the skill, the trained model still solves more of them.
| Tasks where both models read the skill | Base | SFT |
|---|---|---|
| Our test set, pass rate | 33.7 | 62.0 |
| SkillEval, strict | 26.6 | 36.7 |
| SkillsBench | 38.2 | 36.8 |
On SkillsBench the two models are level on these tasks, and the gain there comes from the trained model consulting skills more often. On Skill-Use-Bench we agree that the gain comes mostly from the Trigger score.
- For the two benchmarks evaluated with a single run, we computed paired comparisons over tasks. On our test set the trained model gains 18.2 points (95% CI 14.0 to 22.5), with 81 tasks solved only by the trained model and 8 only by the base. On Skill-Use-Bench the SU score gains 34.8 points (95% CI 28.4 to 41.1). The data-ablation models are trained on small token-matched subsets and are meant to be compared within each pair. They fall below the base model only on SkillEval's partial-credit pass rate. On its strict metric, where every check must pass, all four are at or above the base (18.3 to 26.3 against 17.3).
Q1
Did you check for overlap between the training skills or tasks and the skills used in these external benchmarks, and were overlapping skills removed before training?
Yes. No benchmark task appears in our training data. Every training task is synthesized by our pipeline from the content of a skill, and the benchmark tasks were written independently by their authors, so the two can share a skill but not a task. We compared our tasks with the benchmark tasks on shared skills, and they differ in instruction, environment and verifier.
At the level of skills, we matched by source, content and name. A small number of benchmark skills also occur in our training set, and these remain in it.
| Benchmark | Skills | Same content as a training skill | Same name only | Tasks affected | Tasks with no overlap |
|---|---|---|---|---|---|
| SkillsBench | 202 | 10 | 10 | 28 of 87 | 59 |
| SkillEval | 186 | 16 | 9 | 23 of 100 | 77 |
| Skill-Use-Bench | 78 | 2 | 2 | 13 of 175 | 162 |
To confirm that these shared skills do not drive our results, we recomputed the gain of the trained model over the base model on the tasks with no overlapping skill.
| Metric | All tasks | Tasks with no overlap |
|---|---|---|
| SkillEval pass rate | +9.0 | +10.3 |
| SkillEval strict | +16.0 | +16.9 |
| Skill-Use-Bench SU | +34.8 | +35.0 |
| SkillsBench | +7.8 | +5.5 |
The gain is unchanged on SkillEval and Skill-Use-Bench, and smaller but still positive on SkillsBench. On our own test set, the 150 held-out skills never appear in training. The revised paper includes both tables in Appendix G.
Response to Reviewer 8gYd # 4
Q2
How are the two splits constructed and matched in difficulty and domain? Does this indicate that the held-out split is easier, and if so, how should the claimed generalization to unseen skills be interpreted?
Skills are assigned to the two sides at random by skill family, with 150 skills on each side. The two sides are matched in task type, with 50 tasks of each of the four types, and in domain, where the counts of the 19 domains differ by at most 3.
To measure whether one side is easier, we used 16 models that were never trained on our data.
| Pass rate | Held-in | Held-out | Difference |
|---|---|---|---|
| Average of 16 models never trained on our data | 53.0 | 57.0 | +4.0 |
| Qwen3.5-9B | 40.0 | 42.5 | +2.5 |
| Qwen3.5-9B with SkillGym SFT | 57.0 | 62.0 | +5.0 |
The held-out side is indeed slightly easier, by about 4 points, and this comes almost entirely from the partial-order tasks. The higher held-out score of the trained model therefore reflects the split rather than an advantage on unseen skills. Generalization should be read from the gain within each side, which is +17.0 on seen skills and +19.5 on unseen skills. Training transfers to unseen skills as well as it does to seen ones. The revised paper states the claim in this form in Section 4.2.
We hope our response could resolve your concerns. And if you still have questions or unresolved concerns, we are happy to answer further.
Response to Reviewer 8gYd # 1
Thank you for your excellent review and we try our best to address them as follows:
W1
We agree that some of the building blocks, such as iterative building and SFT on teacher trajectories, are standard, and we do not claim them as our main contributions. SKT is a close work that appeared around one month before our work, and we regard it as concurrent work. At the time of submission SKT had not released its training data or a checkpoint, which is why the paper could only quote its reported numbers.
Our main contribution is a large open resource and an empirical study of what training on it changes. SkillGym covers 3,494 skills and 6,772 tasks across 19 domains, collects trajectories in three agent harnesses, and is used to train six models from 2B to 122B parameters. Every task must fail on an empty attempt as well as pass on its reference solution, and a reviewer model examines each verifier for defects, the effect of which we measure. The paper also studies questions that prior work on skill-use training leaves open, including what a model learns to do with a skill, whether prompting alone achieves the same, and whether general agent ability is retained. The full corpus will be released. SkillsVote [2] profiles skill libraries and synthesizes tasks to record skill behavior, but does not train models on them.
SKT has since released part of its trajectories, so we can now compare under the same setting. We fine-tuned the same base model with the same recipe on SKT's released data and on a SkillGym subset with the same number of tokens (36M), and evaluated both on our test set in three agent harnesses. SKT's released trajectories were collected in OpenCode, and the SkillGym subset in mini-swe-agent, Terminus-2 and AgentFly.
In OpenCode the SkillGym subset is slightly better than SKT although it contains no OpenCode trajectories, and in the other two harnesses it is ahead by 7 to 10 points. The revised paper adds this comparison in Appendix F.
W2
We answer your four points in order.
Environments are built by DeepSeek-V4-Pro. Tasks are built by DeepSeek-V4-Pro, GLM-5.2 and MiniMax-M3. The reviewer is DeepSeek-V4-Pro for 96.5% of tasks and GLM-5.2 for the rest. Trajectories are produced by Kimi-K3, DeepSeek-V4-Flash and GLM-5.2.
The gap between 11,897 and 3,494 is mostly a sampling decision. Most curated skills are software-engineering skills, so we capped that domain to keep the domains balanced, and 6,395 skills were not attempted. Of the 5,502 skills we attempted, 3,494 produced at least one accepted task. Among the remaining 2,008, 751 were rejected by the reviewer, 714 failed the validity checks, 475 were skipped by the builder, and 68 failed for infrastructure reasons.
Skills are assigned to the two sides at random by skill family. The held-out skills never appear in training. Each side of the test set has 50 tasks for each of the four task types. Tasks from the same skill can appear in both the training set and the held-in test set, and this is by design. Held-in measures new tasks on seen skills, and held-out measures unseen skills. No test task appears in training.
The 6,772 tasks are 5,191 that the reviewer approved and 1,581 that passed every execution check but that the reviewer still considered too strict. A verifier that is too strict can reject a correct solution but cannot accept an incorrect one, so trajectories that pass it are safe to train on. Partial-order tasks were built in a separate run and were omitted from that count. They are part of the training set. The final training set contains 19,070 trajectories from 4,934 tasks, namely the tasks that at least one teacher solved, excluding those of the held-out skills.
The revised paper adds these details in Sections 3.2 and 4.1, replaces Tables 8 and 9 with a single table that follows the tasks from construction to the training set (Appendix K), and gives the full prompts in the supplementary material.