acceptodds
Under review as a conference paper at ICLR 2027

SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving

Abstract

Human-written agent skills encode rich workflows for real-world problem solving, but are typically used as external inference-time instructions rather than internalized as reusable model capabilities. We introduce SkillGym, a framework that transforms these skills into executable, verifiable training environments for large language model agents. Its skill-to-task pipeline instantiates concrete tasks, verifies outcomes with code-based checkers, and assesses empirical skill dependence through contrastive executions. We construct and release 2,756 environments across 12 categories and collect 8,364 successful trajectories from multiple models and harnesses, averaging 49 tool calls and over 60k logged text tokens. These resources support supervised fine-tuning on verified workflows and reinforcement learning with outcome-based rewards. Under Claude Code, supervised fine-tuning improves Qwen3.5-35B-A3B by 199 Elo on GDPval-AA v2, 19.10 percentage points on Terminal-Bench 2.1, and 28.13 and 14.68 points on SkillsBench v1.1 with and without skills, respectively. Our 35B SkillGym-Agent reaches 51.47% on skill-assisted SkillsBench, exceeding reported scores for Claude Sonnet 4.6, GPT-5.4 Mini, and DeepSeek V4 Pro. Without skills, it also surpasses skill-assisted base models under Codex and Claude Code, suggesting reusable procedural competence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Skills, Generalist Agents, Tool Use, Agentic Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F9a2OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SkillGym, a framework for converting human-written agent skills into executable training environments. The pipeline combines manually designed category-specific templates, automated environment construction and refinement, code-based outcome verification, and contrastive executions with and without skills. The resulting resource contains 2,756 environments and 8,364 successful trajectories collected from multiple teacher models and agent harnesses. Supervised fine-tuning of Qwen3.5-35B-A3B on these trajectories improves performance on GDPval-AA v2, Terminal-Bench 2.1, and SkillsBench v1.1.

Critical strengths
  1. Substantial resource-construction effort. The dataset covers diverse task categories and packages executable environments, task-specific verifiers, and long-horizon interaction trajectories. This infrastructure could be useful for subsequent agent-training research.

  2. Consistent improvements over the original model. Under Claude Code, the trained model improves by 19.10 percentage points on Terminal-Bench and by 28.13 and 14.68 points on SkillsBench with and without skills, respectively. These results show consistent performance gains across the evaluated settings.

Critical weaknesses
  1. The central intuition is straightforward: human-written skills contain useful procedures, tasks can be generated to exercise those procedures, and successful executions can provide training supervision. The paper implements this idea through substantial engineering, but offers limited additional insight into when or why this approach is particularly effective. The experiments do not sufficiently substantiate its central claim of internalizing human skills. The improvements without inference-time skill access provide partial support for this claim. However, they do not rule out the alternative explanation that the gains arise from general capability improvements through SFT on high-quality agent trajectories.

  2. The contribution of skill-dependent supervision remains unclear. Only 39.2% of the constructed environments satisfy the paper's empirical skill-dependency criterion, while the remaining environments are retained as verifier-passed fallback tasks. This suggests that grounding human-written skills into tasks that demonstrably benefit from those skills is nontrivial. Moreover, only 36.8% of successful trajectories explicitly invoke their corresponding target skills. These statistics leave unclear whether skill-dependent supervision is the main source of the reported improvements.

Questions for authors

The most significant questions are already listed under weaknesses. In addition, I would be interested to get a clarification on the following point:

  1. Does training on the Skill-Dep. subset produce greater skill-free transfer than training on the Verifier-Passed subset under matched budgets?

  2. The notion of skill internalization is insufficiently defined. How do you define skill internalization, and what evidence distinguishes it from general improvements in agent capability through SFT?

3 comments

We thank the reviewer for the thoughtful review. We appreciate the recognition of the substantial engineering behind SkillGym and the observation that the improvements without inference-time skill access provide partial support for the internalization claim. The new experiments below strengthen this evidence.

1. About the Contribution of Skill-Dependent Supervision (W2, Q1)

Under a matched budget, training on Skill-Dep. trajectories gives greater skill-free transfer than training on Verifier-Passed trajectories. We randomly sampled 500 of the 935 Skill-Dep. trajectories and 500 of the 834 Verifier-Passed trajectories, all from the same teacher and harness (GLM-5.2 under Claude Code; Table 2), and fine-tuned the same Qwen3.5-35B-A3B checkpoint on each with the same settings. On both evaluations that provide no skills, the Skill-Dep. student is higher: 18.87 vs. 16.74 on SkillsBench without skills and 43.82 vs. 39.33 on Terminal-Bench 2.1.

Table R1: Training on the Skill-Dep. and Verifier-Passed subsets under matched budgets (500 GLM-5.2 trajectories each).

Training dataTerminal-Bench 2.1SkillsBench v1.1SkillsBench v1.1 (w/o)
None (Base)39.3323.3412.13
SkillGym Verifier-Passed39.3332.1016.74
SkillGym Skill-Dep.43.8242.3918.87

Each Skill-Dep. trajectory is also worth more: the Skill-Dep. student is higher on SkillsBench with skills (42.39 vs. 32.10) and, with only 500 trajectories, comes close to the 45.50 obtained with all 1,769 GLM-5.2 trajectories (Table 5). Verifier-Passed trajectories still help (+8.76 with skills and +4.61 without on SkillsBench, over the base), which is why we keep them as fallback environments.

The 39.2% reflects how demanding the criterion is, not a lack of useful tasks. The criterion is a construction-time device for making tasks as hard as possible: it targets tasks that a strong reference agent (GPT-5.4 under Codex) cannot complete on its own without the skill, yet completes with it. The remaining tasks are not simply easier; some could not be completed by the reference agent with or without the skill. As stated in Section 4.1, they are still built from their source skills and pass verification.

Changes to the paper: Table R1 and the matched-budget comparison above have been added as a new Appendix E.2.

2. About the Definition of Skill Internalization and Its Evidence (W1, Q2)

We define skill internalization as the acquisition or strengthening of reusable procedural patterns from human-written skills in model parameters through training, enabling the model to apply them to new tasks without explicitly retrieving or invoking the corresponding skills at inference time. These patterns include task-dependent decisions, execution constraints, verification steps, and recovery strategies. Prior work uses a similar notion: SKILL0 (Lu et al., 2026) also internalizes agent skills into model parameters so that no skill retrieval is needed at inference.

The approach works because it moves the high-quality human knowledge in skills into the model (W1). Human-written skills hold high-quality procedural knowledge, but at inference time they act only as external context, so their benefit depends on retrieval, context length, and faithful instruction following (Section 1). SkillGym instead turns each skill into verified tasks and trains on trajectories that carry the procedure out to completion, including the decisions, checks, and recoveries that the skill text leaves implicit. The same pipeline turns any skill into tasks, so the training data grows with the skill library and is not limited by what fits in the context. Without skills, the trained model already exceeds the base model that has the skills in context: on SkillsBench, 26.81% vs. 23.34% under Claude Code and 21.08% vs. 5.33% under Codex (Table 4).

The approach is most effective on tasks just beyond what a strong agent can do on its own (W1). Under a matched budget, trajectories from tasks that a strong agent cannot complete without the skill but completes with it (Skill-Dep.) train a stronger model than the same number of Verifier-Passed trajectories (Table R1).

Two results show that the gains come from learning the skills' procedures, not from general capability improvements through SFT (W1, Q2).

  • The student carries out source-skill procedures with no skill available (Table R2). Internalization is not a mechanism separate from SFT; SFT is how the procedures enter the parameters, and what distinguishes internalization is what is learned. On Terminal-Bench 2.1, where no skill is available and the student neither reads a skill file nor calls the Skill tool, behaviors prescribed by the source skills become far more frequent than in the base: stating a root cause before editing rises from 7.4% to 58.4% of tasks, and checking a requirements list before finishing from 6.2% to 32.6%. On fix-code-vulnerability, for example, the student located the removed validation through Git history, reproduced the failure, and restored both checks together, whereas the base patched one location at a time. A lexical search finds no verbatim reuse of skill scripts or commands, so what transfers is the procedure, not the text.

Table R2: Procedural behaviors on Terminal-Bench 2.1, extracted from the raw trajectories with heuristic rules (tasks showing the behavior / tasks with saved sessions).

BehaviorRelated source skillSkillGym-AgentBase
States a hypothesis or root cause before editingsystematic-debugging52/89 (58.4%)6/81 (7.4%)
Explicitly checks exit codesverification-before-completion44/89 (49.4%)2/81 (2.5%)
Checks a requirements list before finishingverification-before-completion29/89 (32.6%)5/81 (6.2%)
  • SFT on high-quality agent trajectories alone does not reproduce the gains (Table R3). We fine-tuned the same Qwen3.5-35B-A3B checkpoint on 1,869 SWE-Gym (Pan et al., 2025) trajectories from the public SWE-Hero release, a high-quality agent-trajectory corpus not derived from skills, with the same training settings as our single-teacher SkillGym students (1,722 and 1,769 trajectories; Table 5). Both SkillGym students outperform the SWE-Gym student by a wide margin on all three metrics.

Table R3: Same-backbone comparison of skill-derived and non-skill-derived training data under comparable budgets.

Training dataTerminal-Bench 2.1SkillsBench v1.1SkillsBench v1.1 (w/o)
None (Base)39.3323.3412.13
SWE-Gym (non-skill-derived)11.246.974.95
SkillGym (DeepSeek V4 Pro subset)43.8228.8119.02
SkillGym (GLM-5.2 subset)55.0645.5025.10

Changes to the paper: In Section 1, we have added the definition of skill internalization, citing SKILL0 (Lu et al., 2026), and the explanation of why and when the approach is effective. We have also added the trajectory analysis with Table R2 as a new Appendix B.5, and Table R3 with its analysis as a new Appendix E.1.

We hope the definition and the new comparisons clarify the role of skill-derived supervision.

Dear Reviewer F9a2,

Thank you again for the time and effort you put into reviewing our work!

We understand that this is a busy period and wanted to briefly follow up on our rebuttal. We have added the requested matched-budget comparison of the Skill-Dep. and Verifier-Passed subsets, clarified our definition of skill internalization, and provided additional trajectory-level analyses and a SWE-Gym comparison.

Whenever you have a chance, we would be grateful for your feedback on whether these additions address your concerns about the contribution of skill-dependent supervision and the evidence for skill internalization. We would be happy to clarify any remaining questions.

Thank you for your time and consideration!

Best, The Authors of Submission 26445

Reviewer_NbZ4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SkillGym, a framework that transforms human-written agent skills into executable, verifiable training environments. It combines structured skill cards with manually designed task templates to construct tasks with code-based verifiers, and uses contrastive executions with and without skills to assess empirical skill dependence under a reference configuration. The resulting resource comprises 2756 environments across 12 categories and 8364 successful trajectories collected from multiple models and agent harnesses. Supervised fine-tuning of Qwen3.5-35B-A3B on these trajectories improves performance on professional work, terminal execution, and skill-based tasks, including task solving without inference-time skills. Teacher-mixture ablations further examine the benefits and trade-offs of combining different trajectory sources.

Critical strengths
  1. Converting skill documents into verifiable training resources is a meaningful systems and data contribution. SkillGym integrates manual templates, executable environments, outcome verification, and refinement while distinguishing executability from empirical skill dependence. Its 2756 environments and 8364 successful trajectories provide a substantial resource for learning workflows across domains. The checks of update propagation, recovery after restart, and persistence in Appendix C further illustrate how the approach can cover stateful tasks.
  2. Results with the same backbone and system prompt support training utility. Under Claude Code, GDPval improves from 974 to 1173 Elo and Terminal-Bench from 39.33% to 58.43% while retaining the same backbone and system prompt. These gains across task types provide positive evidence for the overall training recipe without depending on direct rankings against published scores for frontier models.
Critical weaknesses
  1. The evidence connecting aggregate gains to internalization of transferable skills remains limited. Table 4 supports the utility of the training recipe, including gains without inference-time skills, but does not directly show which procedures encoded in the source skills are learned and reused in new task instances. The results are also consistent with learning from stronger teachers or improving general tool use. Table 5 varies teacher and data composition without distinguishing these explanations. The paper therefore demonstrates the utility of workflow trajectories more directly than its distinctive scientific claim about internalizing and transferring human procedural knowledge. This limits what can be concluded about the contribution of deriving training tasks from skills.
  2. The substantive validity of the generated tasks and verifiers lacks representative assessment. The resource's value depends on preserving meaningful decisions and constraints from source skills and detecting substantive failures to complete the resulting tasks. The feasibility checks and skill contrasts reported in Section 3.2 offer limited independent evidence of this correspondence, while Section 4 mainly reports coverage, pass rates, and trajectory statistics. Section 3.1 limits the scope of checks for content tasks, and the stronger runtime checks in Appendix C illustrate one environment without establishing quality across the corpus. This leaves uncertainty about whether verifier-selected demonstrations consistently represent correct workflows and whether the environment suite provides reliable training feedback.
Questions for authors
  1. What evidence directly supports learning and transfer of procedures from the source skills? Beyond aggregate score gains, can you identify specific decisions, constraints, or recovery strategies that the student acquires and reuses in new task instances? An analysis of existing results at the skill level, or a narrowly scoped matched comparison, could clarify the connection to skill-derived training and its relationship to gains in general tool use.
  2. How has the substantive validity of the generated tasks and verifiers been assessed? For representative existing tasks, can you show how important requirements from the source skills are preserved and whether outputs omitting those requirements are rejected? Existing independent assessments, examples of critical failures, or analyses of verifier errors would support the resource-quality claim more directly than additional coverage or trajectory-length statistics.
3 comments

Thank you for the constructive review and for noting that "Table 4 supports the utility of the training recipe, including gains without inference-time skills." We address the two concerns below. 1. About Evidence for Internalization of Skill Procedures (W1, Q1)

We give two kinds of evidence: what the student does on evaluation tasks where no skill is available (Table R1), and a matched comparison with high-quality agent trajectories from a strong model that are not derived from skills (Table R2). Together they show that a stronger teacher or general tool use alone does not explain the gains.

The student reuses source-skill procedures on new tasks. We compared student and base rollouts on the same Terminal-Bench 2.1 tasks under Claude Code. No skill is available in these environments, and the student neither read a skill file nor invoked the Skill tool. Both models pass the two tasks below, so the difference lies in how they reach the solution.

  • The student finds the root cause before fixing (systematic-debugging). On fix-code-vulnerability, the student used Git history and diffs to locate the removed validation in _hkey and _hval, reproduced the failure, restored both checks together, and ran regression tests plus targeted malformed-header checks. The base used no Git command and patched one location at a time after further failures.
  • The student verifies before finishing (verification-before-completion). On merge-diff-arc-agi-task, the student tested before committing, checked a ten-item requirements list, and reran verification before finishing, although the task does not prescribe this sequence. The base committed before its first test and finished without a final rerun or checklist.

These cases are not isolated: across Terminal-Bench 2.1, the behaviors prescribed by these two skills appear in far more student trajectories than base trajectories (Table R1).

Table R1: Procedural behaviors on Terminal-Bench 2.1, extracted from the raw trajectories with heuristic rules (tasks showing the behavior / tasks with saved sessions).

BehaviorRelated source skillSkillGym-AgentBase
States a hypothesis or root cause before editingsystematic-debugging52/89 (58.4%)6/81 (7.4%)
Explicitly checks exit codesverification-before-completion44/89 (49.4%)2/81 (2.5%)
Checks a requirements list before finishingverification-before-completion29/89 (32.6%)5/81 (6.2%)

A strong teacher and high-quality agent trajectories alone do not reproduce the gains. We fine-tuned the same Qwen3.5-35B-A3B checkpoint with identical settings on 1,869 trajectories from SWE-Gym (Pan et al., 2025), a high-quality agent-trajectory corpus produced by a strong model on tasks that are not derived from skills, comparable in size to our single-teacher subsets (1,722 and 1,769 trajectories). Both SkillGym students outperform the SWE-Gym student on all three metrics by a wide margin (Table R2).

Table R2: Same-backbone comparison of skill-derived and non-skill-derived training data under comparable budgets.

Training dataTerminal-Bench 2.1SkillsBench v1.1SkillsBench v1.1 (w/o)
None (Base)39.3323.3412.13
SWE-Gym (non-skill-derived)11.246.974.95
SkillGym (DeepSeek V4 Pro subset)43.8228.8119.02
SkillGym (GLM-5.2 subset)55.0645.5025.10

Changes to the paper: We have added the trajectory analysis (the two cases and Table R1) as a new Appendix B.5, and the SWE-Gym comparison (Table R2, with the analysis above) as a new Appendix E.1.

Reference: Pan, J., Wang, X., Neubig, G., Jaitly, N., et al. (2025). Training Software Engineering Agents and Verifiers with SWE-Gym. In ICML 2025.

2. About the Validity of Generated Tasks and Verifiers (W2, Q2)

We ensure and check in four ways that each task and its verifier stay tied to the requirements of the source skill; Table R3 gives three representative examples.

  1. The construction prompts and rules tie every verifier to the source skill. Verification requirements are hand-written in each of the 63 sub-category templates (Section 3.1). Before writing a task, the construction agent must read the full skill directory (SKILL.md and the scripts and resources it references), identify the skill's key capabilities, and record in plan.json the required outputs, what the verifier must check, and why the task should be easier with the skill. The task's difficulty must come from what the skill supports, not from environment setup or sheer coding. In every revision, an automated review separate from task authoring enforces the verifier rules: main tests assert only outcomes stated in the instruction; separate anti-shortcut tests target input tampering, answer copying, bypassed requirements, and fabricated outputs (Section 3.2); the untouched initial environment must fail; and neither the verifier nor the reference solution may read the skill files. Every released task ships with its plan (Appendix C.2), so this link can be inspected task by task.
  2. Every revision is executed with and without the skill, and the verifiers also accept independently produced solutions. The reference solution must first pass the verifier in freshly built containers; the reference agent (GPT-5.4 under Codex) then attempts two otherwise identical copies of the task, with and without the skill. If the with-skill run passes and the verifier rejects the without-skill run (runs with no valid verifier result do not count), the task is labeled Skill-Dep.; otherwise it is revised and rerun within the preset limit, about ten rounds on average for accepted tasks. In total, 1,081 verifiers accepted the with-skill attempt and rejected the without-skill one. Each task with a passing teacher run is passed by about three of the four model–harness configurations on average (Table 2); three of these four configurations use models that took no part in construction.
  3. Training on verifier-selected trajectories improves downstream results, especially on SkillsBench. We keep only executions accepted by the verifier, anti-shortcut checks included. The trained model improves every reported metric on GDPval-AA v2, Terminal-Bench 2.1, and SkillsBench v1.1 under both harnesses (Table 4), with large gains on SkillsBench, whose tasks are built around skills: under Claude Code, from 23.34% to 51.47% with skills and from 12.13% to 26.81% without. This indicates that the selected trajectories carry the procedures of their source skills and that the model internalizes them; Table R1 above shows this at the level of individual procedures.
  4. Table R3 shows, for three representative tasks, how a requirement from the source skill is carried into the task and enforced by the verifier.

Table R3: Examples of requirements carried into tasks and how verifiers enforce them.

TaskRequirement carried into the taskHow the verifier checks itWhat is rejected
Invoice organizationKeep the original documents intact and organize copies (from the source skill)Input-integrity checks and per-copy SHA-256 comparisonsMissing copies, altered originals, mismatched copy contents
Data analysisFollow the exclusion rules and ranking limits given in the inputsChanges those rules and limits, reruns the submitted analysis entrypoint, and compares the outputOutputs that do not reflect the changed inputs, such as hard-coded results
Deployment (Appendix C.2)Reach a synchronized, restart-safe deployment with persistent storage and correct readiness (from the source skill electric-deployment)A trusted runtime checks initial synchronization, injected updates, restart recovery, exact HTTP-200 readiness, and persisted-cache reuseDeployments that do not propagate updates, lose state on restart, or report readiness too early

Changes to the paper: In Section 3.2, we have stated that each revision receives one with-skill and one without-skill execution, with a pointer to a new Appendix C.4. Appendix C.4 describes the constraints in the construction prompts, what plan.json records, the verifier rules (outcome-only main tests, anti-shortcut tests, a failing initial environment, no access to skill files), the validation run in every revision (reference solution, with- and without-skill pair, revision and rerun within the preset limit), and includes Table R3.

We thank the reviewer again and are glad to clarify any remaining questions.

Dear Reviewer NbZ4,

Thank you again for the time and effort you put into reviewing our work!

We understand that this is a busy period. We wanted to briefly let you know that we have posted detailed responses to your questions and concerns, including additional trajectory-level analyses and experimental comparisons, and updated the manuscript accordingly.

Whenever you have a chance, we would be grateful for your feedback on whether our responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best, The Authors of Submission 26445

Reviewer_ERC8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SkillGym provides a useful framework for turning human-written skills into executable training tasks. The dataset is substantial, and the improvements across several agent benchmarks are strong, especially without inference-time skills. I still have some concerns about how much of the gain comes from skill-based task construction, the reliability of skill-dependency tests, and differences in evaluation settings. However, I lean toward acceptance given the scale of the dataset and the consistent improvements over the base model.

Critical strengths
  1. Useful training resource. SkillGym provides 2,756 executable environments across 12 categories and 8,364 successful trajectories validated by code-based checkers. The successful trajectories average 49 tool calls, making this a useful resource for training agents on extended, multi-step workflows.

  2. Strong performance gains. SkillGym-Agent achieves substantial improvements over its base model across multiple benchmarks. The gains without inference-time skills are particularly interesting and are consistent with improved reusable procedural competence. These results support the value of the collected workflow trajectories for agent training.

Critical weaknesses
  1. Limited attribution of gains to skill-derived task construction. The experiments demonstrate the usefulness of the resulting training corpus, but do not isolate the specific contribution of deriving tasks from human-written skills. Table 5 mainly varies teacher composition and harness coverage. A baseline trained on verified trajectories from non-skill-derived tasks under a comparable training budget is not reported. This makes it difficult to distinguish the contribution of skill-derived task construction from the benefits of additional high-quality agent supervision.

  2. Limited validation of the skill-dependency criterion. The dataset contains 1,081 environments satisfying the empirical skill-dependency criterion and 1,675 verifier-passed fallback environments. This distinction is clearly disclosed. However, the paper does not specify the number of repeated with- and without-skill executions or assess the stability of these construction-time labels. It also does not report a controlled training comparison between the Skill-Dep. and fallback subsets, leaving the contribution of the skill-dependent subset unclear.

  3. Evaluation comparability. As acknowledged in Appendix D.2, the Codex base model and SkillGym-Agent use different system prompts, so the Codex gains cannot be attributed solely to fine-tuning. Public model comparisons also use different harnesses and evaluation settings, as noted in Appendix D.1, which limits the interpretation of cross-model rankings.

Questions for authors
  1. Could the authors compare against a baseline trained on verified trajectories from non-skill-derived tasks, using the same backbone, a comparable supervised-token budget, and matched training settings? Matching teacher sources and task coverage where feasible would further strengthen this comparison.

  2. How many with- and without-skill executions were used per task for the construction-time skill-dependency assessment, and how was label stability assessed? Could the authors compare training on the Skill-Dep. and fallback subsets under matched training budgets?

  3. Could the authors evaluate the Codex base model and SkillGym-Agent with the same system prompt? Separately, were overlap or near-duplicate checks performed between the training tasks/source skills and the evaluation benchmarks, particularly SkillsBench? If so, please report the procedure and findings.

3 comments

We thank the reviewer for the careful reading and for noting that "the experiments demonstrate the usefulness of the resulting training corpus" and that the distinction between Skill-Dep. and fallback environments "is clearly disclosed." We respond to each concern below.

1. About Attribution of Gains to Skill-Derived Task Construction (W1, Q1)

We added the requested baseline. We fine-tuned the same Qwen3.5-35B-A3B checkpoint on the SWE-Gym (Pan et al., 2025) subset of the public SWE-Hero release, a high-quality agent-trajectory corpus that is not derived from skills, using the same training settings (epochs, learning rate, context length) as our single-teacher students in Table 5. The budget is comparable: 1,869 SWE-Gym trajectories versus 1,722 (DeepSeek V4 Pro) and 1,769 (GLM-5.2) SkillGym trajectories, with similar per-trajectory token length.

Table R1: Same-backbone comparison of skill-derived and non-skill-derived training data under comparable budgets.

Training dataTerminal-Bench 2.1SkillsBench v1.1SkillsBench v1.1 (w/o)
None (Base)39.3323.3412.13
SWE-Gym (non-skill-derived)11.246.974.95
SkillGym (DeepSeek V4 Pro subset)43.8228.8119.02
SkillGym (GLM-5.2 subset)55.0645.5025.10

Both SkillGym students outperform the SWE-Gym student on all three metrics by a wide margin. Even the weaker SkillGym student evaluated without skills (19.02) exceeds the SWE-Gym student evaluated with skills (6.97). This supports the soundness of our framework and the high quality of the sampled trajectories.

Changes to the paper: We have added this baseline, with its setup, Table R1, and the analysis above, as a new Appendix E.1.

Reference: Pan, J., Wang, X., Neubig, G., Jaitly, N., et al. (2025). Training Software Engineering Agents and Verifiers with SWE-Gym. In ICML 2025.

2. About the Skill-Dependency Criterion (W2, Q2)

Each revision is run once with and once without the skill. Construction has a preset limit of 10 hours and 20 revision rounds per task. A task is labeled Skill-Dep. as soon as one revision passes with the skill and fails without it; a verified task that reaches the limit without such a revision is labeled Verifier-Passed. The criterion is meant to make tasks as hard as possible: it targets tasks that a strong reference agent (GPT-5.4 under Codex) cannot complete on its own without the skill, yet completes with it. As Section 3.2 states, the label is evidence of dependency under the reference configuration, not proof that every agent needs the skill.

We improve label stability through multi-round iteration. When a revision does not meet the criterion, we revise the task and its environment and rerun the whole sequence within the limit. Each round first requires a separate review, static checks, and a passing reference solution; the with- and without-skill runs then use two freshly built copies of the task that differ only in whether the skill is installed. Runs with no valid verifier result never count.

Under a matched budget, training on Skill-Dep. tasks outperforms training on Verifier-Passed tasks on all three metrics. We randomly sampled 500 of the 935 Skill-Dep. trajectories and 500 of the 834 Verifier-Passed trajectories, all from the same teacher and harness (GLM-5.2 under Claude Code; Table 2), and fine-tuned the same Qwen3.5-35B-A3B checkpoint on each with the same settings.

Table R2: Training on the Skill-Dep. and Verifier-Passed subsets under matched budgets (500 GLM-5.2 trajectories each).

Training dataTerminal-Bench 2.1SkillsBench v1.1SkillsBench v1.1 (w/o)
None (Base)39.3323.3412.13
SkillGym Verifier-Passed39.3332.1016.74
SkillGym Skill-Dep.43.8242.3918.87

The largest gap is on SkillsBench with skills (42.39 vs. 32.10), where 500 Skill-Dep. trajectories come close to the 45.50 obtained with all 1,769 GLM-5.2 trajectories (Table R1). Verifier-Passed trajectories still improve SkillsBench over the base (+8.76 with skills, +4.61 without), which supports keeping them as fallback environments.

Changes to the paper: In Section 3.2, we have stated that each revision receives one with-skill and one without-skill execution, and added the full procedure as a new Appendix C.4 (including the 10-hour, 20-round limit and the rule that runs without a valid verifier result do not count). Table R2 and this analysis have been added as a new Appendix E.2.

3. About Evaluation Comparability and Train–Test Overlap (W3, Q3)

The Codex comparison already uses the same system prompt. Under Codex (Tables 4 and 13), the base model and SkillGym-Agent both use the no-applypatch prompt, which differs from the standard Codex system prompt only in removing the description of the apply_patch tool; this tool is not compatible with non-GPT models in Codex. Appendix D wrongly states that the base model used the standard prompt. We apologize for this reporting error and have corrected it. The Codex gains in Table 4 therefore reflect fine-tuning.

Skill selection and task construction are independent of the evaluation benchmarks. Tasks are built from publicly available human-written skills and manually designed templates, and no task instructions, input assets, reference solutions, or task-specific verifiers from any evaluation benchmark, including SkillsBench, were used. Before training, we also checked our source skills against the skill packages shipped by SkillsBench (names, descriptions, SKILL.md contents, and accompanying scripts) and removed the matching skills: trajectories from tasks whose source skill matched a SkillsBench skill in content were excluded from training. A few other skills share only a name with a SkillsBench skill (e.g., pdf), and their contents differ. As a result, no skill underlying the training trajectories is identical or near-duplicate to a SkillsBench skill.

Changes to the paper: We have corrected the Codex setup in Appendix D.1 (both models use no-applypatch) and removed the sentence in Appendix D.2 stating that the Codex comparison also changes the system prompt. In Section 5.1, we have stated that no instructions, input assets, reference solutions, or verifiers from any evaluation benchmark were used to build training tasks, and that trajectories from tasks whose source skill matches a SkillsBench skill in content were excluded before training; Appendix D.1 now gives the details of this comparison.

We hope these additions and corrections address the reviewer's concerns.

Dear Reviewer ERC8,

Thank you again for the time and effort you put into reviewing our work!

We understand that this is a busy period. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns, including additional experimental results and clarifications, and updated the manuscript accordingly.

Whenever you have a chance, we would be grateful for your feedback on whether our responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best, The Authors of Submission 26445

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.