- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SkillGym, a framework for converting human-written agent skills into executable training environments. The pipeline combines manually designed category-specific templates, automated environment construction and refinement, code-based outcome verification, and contrastive executions with and without skills. The resulting resource contains 2,756 environments and 8,364 successful trajectories collected from multiple teacher models and agent harnesses. Supervised fine-tuning of Qwen3.5-35B-A3B on these trajectories improves performance on GDPval-AA v2, Terminal-Bench 2.1, and SkillsBench v1.1.
Critical strengths
-
Substantial resource-construction effort. The dataset covers diverse task categories and packages executable environments, task-specific verifiers, and long-horizon interaction trajectories. This infrastructure could be useful for subsequent agent-training research.
-
Consistent improvements over the original model. Under Claude Code, the trained model improves by 19.10 percentage points on Terminal-Bench and by 28.13 and 14.68 points on SkillsBench with and without skills, respectively. These results show consistent performance gains across the evaluated settings.
Critical weaknesses
-
The central intuition is straightforward: human-written skills contain useful procedures, tasks can be generated to exercise those procedures, and successful executions can provide training supervision. The paper implements this idea through substantial engineering, but offers limited additional insight into when or why this approach is particularly effective. The experiments do not sufficiently substantiate its central claim of internalizing human skills. The improvements without inference-time skill access provide partial support for this claim. However, they do not rule out the alternative explanation that the gains arise from general capability improvements through SFT on high-quality agent trajectories.
-
The contribution of skill-dependent supervision remains unclear. Only 39.2% of the constructed environments satisfy the paper's empirical skill-dependency criterion, while the remaining environments are retained as verifier-passed fallback tasks. This suggests that grounding human-written skills into tasks that demonstrably benefit from those skills is nontrivial. Moreover, only 36.8% of successful trajectories explicitly invoke their corresponding target skills. These statistics leave unclear whether skill-dependent supervision is the main source of the reported improvements.
Questions for authors
The most significant questions are already listed under weaknesses. In addition, I would be interested to get a clarification on the following point:
-
Does training on the Skill-Dep. subset produce greater skill-free transfer than training on the Verifier-Passed subset under matched budgets?
-
The notion of skill internalization is insufficiently defined. How do you define skill internalization, and what evidence distinguishes it from general improvements in agent capability through SFT?
3 comments
2. About the Definition of Skill Internalization and Its Evidence (W1, Q2)
We define skill internalization as the acquisition or strengthening of reusable procedural patterns from human-written skills in model parameters through training, enabling the model to apply them to new tasks without explicitly retrieving or invoking the corresponding skills at inference time. These patterns include task-dependent decisions, execution constraints, verification steps, and recovery strategies. Prior work uses a similar notion: SKILL0 (Lu et al., 2026) also internalizes agent skills into model parameters so that no skill retrieval is needed at inference.
The approach works because it moves the high-quality human knowledge in skills into the model (W1). Human-written skills hold high-quality procedural knowledge, but at inference time they act only as external context, so their benefit depends on retrieval, context length, and faithful instruction following (Section 1). SkillGym instead turns each skill into verified tasks and trains on trajectories that carry the procedure out to completion, including the decisions, checks, and recoveries that the skill text leaves implicit. The same pipeline turns any skill into tasks, so the training data grows with the skill library and is not limited by what fits in the context. Without skills, the trained model already exceeds the base model that has the skills in context: on SkillsBench, 26.81% vs. 23.34% under Claude Code and 21.08% vs. 5.33% under Codex (Table 4).
The approach is most effective on tasks just beyond what a strong agent can do on its own (W1). Under a matched budget, trajectories from tasks that a strong agent cannot complete without the skill but completes with it (Skill-Dep.) train a stronger model than the same number of Verifier-Passed trajectories (Table R1).
Two results show that the gains come from learning the skills' procedures, not from general capability improvements through SFT (W1, Q2).
- The student carries out source-skill procedures with no skill available (Table R2). Internalization is not a mechanism separate from SFT; SFT is how the procedures enter the parameters, and what distinguishes internalization is what is learned. On Terminal-Bench 2.1, where no skill is available and the student neither reads a skill file nor calls the Skill tool, behaviors prescribed by the source skills become far more frequent than in the base: stating a root cause before editing rises from 7.4% to 58.4% of tasks, and checking a requirements list before finishing from 6.2% to 32.6%. On
fix-code-vulnerability, for example, the student located the removed validation through Git history, reproduced the failure, and restored both checks together, whereas the base patched one location at a time. A lexical search finds no verbatim reuse of skill scripts or commands, so what transfers is the procedure, not the text.
Table R2: Procedural behaviors on Terminal-Bench 2.1, extracted from the raw trajectories with heuristic rules (tasks showing the behavior / tasks with saved sessions).
| Behavior | Related source skill | SkillGym-Agent | Base |
|---|---|---|---|
| States a hypothesis or root cause before editing | systematic-debugging | 52/89 (58.4%) | 6/81 (7.4%) |
| Explicitly checks exit codes | verification-before-completion | 44/89 (49.4%) | 2/81 (2.5%) |
| Checks a requirements list before finishing | verification-before-completion | 29/89 (32.6%) | 5/81 (6.2%) |
- SFT on high-quality agent trajectories alone does not reproduce the gains (Table R3). We fine-tuned the same Qwen3.5-35B-A3B checkpoint on 1,869 SWE-Gym (Pan et al., 2025) trajectories from the public SWE-Hero release, a high-quality agent-trajectory corpus not derived from skills, with the same training settings as our single-teacher SkillGym students (1,722 and 1,769 trajectories; Table 5). Both SkillGym students outperform the SWE-Gym student by a wide margin on all three metrics.
Table R3: Same-backbone comparison of skill-derived and non-skill-derived training data under comparable budgets.
| Training data | Terminal-Bench 2.1 | SkillsBench v1.1 | SkillsBench v1.1 (w/o) |
|---|---|---|---|
| None (Base) | 39.33 | 23.34 | 12.13 |
| SWE-Gym (non-skill-derived) | 11.24 | 6.97 | 4.95 |
| SkillGym (DeepSeek V4 Pro subset) | 43.82 | 28.81 | 19.02 |
| SkillGym (GLM-5.2 subset) | 55.06 | 45.50 | 25.10 |
Changes to the paper: In Section 1, we have added the definition of skill internalization, citing SKILL0 (Lu et al., 2026), and the explanation of why and when the approach is effective. We have also added the trajectory analysis with Table R2 as a new Appendix B.5, and Table R3 with its analysis as a new Appendix E.1.
We hope the definition and the new comparisons clarify the role of skill-derived supervision.
Dear Reviewer F9a2,
Thank you again for the time and effort you put into reviewing our work!
We understand that this is a busy period and wanted to briefly follow up on our rebuttal. We have added the requested matched-budget comparison of the Skill-Dep. and Verifier-Passed subsets, clarified our definition of skill internalization, and provided additional trajectory-level analyses and a SWE-Gym comparison.
Whenever you have a chance, we would be grateful for your feedback on whether these additions address your concerns about the contribution of skill-dependent supervision and the evidence for skill internalization. We would be happy to clarify any remaining questions.
Thank you for your time and consideration!
Best, The Authors of Submission 26445
We thank the reviewer for the thoughtful review. We appreciate the recognition of the substantial engineering behind SkillGym and the observation that the improvements without inference-time skill access provide partial support for the internalization claim. The new experiments below strengthen this evidence.
1. About the Contribution of Skill-Dependent Supervision (W2, Q1)
Under a matched budget, training on Skill-Dep. trajectories gives greater skill-free transfer than training on Verifier-Passed trajectories. We randomly sampled 500 of the 935 Skill-Dep. trajectories and 500 of the 834 Verifier-Passed trajectories, all from the same teacher and harness (GLM-5.2 under Claude Code; Table 2), and fine-tuned the same Qwen3.5-35B-A3B checkpoint on each with the same settings. On both evaluations that provide no skills, the Skill-Dep. student is higher: 18.87 vs. 16.74 on SkillsBench without skills and 43.82 vs. 39.33 on Terminal-Bench 2.1.
Table R1: Training on the Skill-Dep. and Verifier-Passed subsets under matched budgets (500 GLM-5.2 trajectories each).
Each Skill-Dep. trajectory is also worth more: the Skill-Dep. student is higher on SkillsBench with skills (42.39 vs. 32.10) and, with only 500 trajectories, comes close to the 45.50 obtained with all 1,769 GLM-5.2 trajectories (Table 5). Verifier-Passed trajectories still help (+8.76 with skills and +4.61 without on SkillsBench, over the base), which is why we keep them as fallback environments.
The 39.2% reflects how demanding the criterion is, not a lack of useful tasks. The criterion is a construction-time device for making tasks as hard as possible: it targets tasks that a strong reference agent (GPT-5.4 under Codex) cannot complete on its own without the skill, yet completes with it. The remaining tasks are not simply easier; some could not be completed by the reference agent with or without the skill. As stated in Section 4.1, they are still built from their source skills and pass verification.
Changes to the paper: Table R1 and the matched-budget comparison above have been added as a new Appendix E.2.