acceptodds
Under review as a conference paper at ICLR 2027

Skill2Task: Learning from Skills through Graph-Guided Task Synthesis for Agentic RL

Abstract

Large language model (LLM) agents can reuse past experience during reinforcement learning (RL) through natural-language skills distilled from their trajectories on training tasks. However, we uncover a bottleneck in SkillRL: generalization can stall in later training stages. Skills distilled from individual task trajectories may also contain overly broad, conflicting, or incorrect recommendations. These challenges motivate broadening the agent’s training experience to support further policy improvement and provide opportunities to test potentially unreliable recommendations through execution feedback. We therefore introduce a framework that uses skills to specify and construct training tasks with varied goals and conditions. To this end, we organize skills into a graph that captures their recommendations and applicability conditions, together with relations that describe how skills can be combined and when their recommendations conflict. The graph guides task synthesis in existing environments, with verifiable success criteria and checks for task relevance and solvability. Starting from a SkillRL checkpoint, we incorporate the resulting tasks into continued RL, using environment feedback to train the agent to combine skills and decide when to follow, adapt, or reject their recommendations. On ALFWorld, Qwen3-4B-Instruct trained with tasks synthesized by our framework achieves higher average success rates than SkillRL on both seen and unseen tasks, with gains of 8.1 percentage points on unseen Pick&Place tasks and 11.1 points on unseen Examine tasks.

Primary area:
reinforcement learning
Keywords:
Skill-Guided Task Synthesis, Agentic RL, Skill Graphs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_jXQnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies two limitations of SkillRL: continued training can plateau, and automatically distilled skills can become overfitted, conflicting, or incorrect. It proposes Skill2Task, which organizes skills into a typed relation graph and uses skill properties and relations to synthesize new tasks for continued RL. Experiments with Qwen3-4B and Qwen3-8B on ALFWorld and search-augmented QA report improvements over SkillRL on both seen and unseen evaluations.

Critical strengths
  1. The paper identifies an important and underexplored failure mode of skill-augmented agents: automatically accumulated skills can encode overgeneralized, conflicting, or invalid guidance. Experiments also show that such skills can directly impair task execution, offering a clear motivation for improving how learned skills are evaluated and reused.
Critical weaknesses
  1. Skill2Task appears to rely heavily on environment-specific grounding, including permitted edits, legal goals, transition rules, source evidence, and a planner or blind solver for solvability verification. This assumption is well matched to ALFWorld and QA but may substantially limit applicability to open-ended environments where valid task modifications and automatic success criteria are unavailable. Moreover, the task-synthesis teacher receives rich, potentially privileged environment information, but the paper does not evaluate whether directly providing the same information to the acting agent would already yield comparable performance. Without this baseline, it is difficult to separate the benefit of task synthesis from that of introducing stronger environment-specific prior knowledge.

  2. The motivation for grounding skill revision in executable tasks is reasonable, since environment feedback can provide evidence beyond teacher self-assessment. However, the paper does not compare against a direct skill-revision baseline in which the same teacher revises flagged skills without synthesizing and executing additional tasks. It is therefore unclear whether task synthesis is necessary, or whether most gains could be obtained through direct skill-bank refinement at substantially lower cost.

  3. Graph construction relies on skill-text similarity and co-retrieval on the original training tasks to propose candidate skill pairs, but this crucial retrieval stage is neither specified nor evaluated. This may miss semantically dissimilar skills that are not retrieved together.

Questions for authors
  1. Are the baselines matched for total compute? Please report the additional RL training cost and the Teacher/solver cost for graph construction, task generation, and validation.
2 comments

We thank the reviewer for the thoughtful questions on environment grounding, direct skill revision, and retrieval coverage. We clarify the shared synthesis framework and the complementary contributions of policy learning and library refinement below.

Q1. Does grounding restrict applicability or introduce privileged information?

Thank you for the question. Skill2Task synthesizes training tasks in both ALFWorld and Web/Search, yielding 1,017 and 738 accepted tasks, respectively. ALFWorld instantiates executable goals in existing scenes; Web/Search constructs questions grounded in source material and validates answers through independent retrieval. These settings cover both environment interaction and information retrieval. The shared framework turns skill conditions and interactions into task requirements, which each environment instantiates and verifies. Its operation therefore extends beyond a simulator with a planner.

We would also like to clarify the role of grounding information. Permitted edits and legal-goal constraints define valid training instances; plans, reference answers, and source evidence support offline verification. These are task-construction and checking inputs, rather than additional solution instructions supplied to the acting policy. The policy learns through its own interactions and task rewards, and uses the standard observations and tools at evaluation. Giving an actor a description of the construction constraints does not supply the execution experience generated by practicing those tasks. This distinction explains why grounding is part of the synthesis interface, while executable practice drives policy learning.

Q2. Is task synthesis necessary, or would direct skill revision be enough?

Thank you for this suggestion. Figure 5(b) separates policy learning from library refinement through checkpoint/library swaps. With the original library fixed, our policy improves unseen performance over the SkillRL policy on both ALFWorld and Search. With our policy fixed, the final library further improves performance. Conversely, pairing the SkillRL policy with our final library still falls below the full Skill2Task agent. These comparisons show that both components contribute: supplying better guidance alone does not recover the gains from their joint evolution.

For a direct skill-editing comparison, we use the same strong teacher, GLM-5.2, to revise the SkillRL library by correcting recommendations and adding conditions or exceptions. The SkillRL policy checkpoint is held fixed; no synthetic tasks or additional RL are introduced. This isolates the effect of revising the library itself.

ALFWorld Qwen3-4B configurationOverall
SkillRL policy + original library80.5 ± 2.3
Same SkillRL policy + teacher-revised library81.4 ± 1.3
Full Skill2Task83.8 ± 1.8

Direct library revision improves overall success from 80.5% to 81.4%, while full Skill2Task reaches 83.8%. Together with Figure 5(b), this comparison supports the complementary contribution of policy learning: Skill2Task updates the policy through executable practice and refines the library from the resulting experience.

Q3. Can pair retrieval miss useful skill relations?

Thank you for this question. Candidate pairs are proposed through both skill-text neighborhoods and task-level co-retrieval. The latter can introduce skills with different descriptions that are relevant to a shared task, so candidate construction is not restricted to semantically similar pairs. The teacher subsequently evaluates their conditions, behaviors, and effects to identify the relation. This combines semantic and task-contextual relevance to identify useful skill interactions for task synthesis.

Q4. Are total compute budgets matched?

Thank you for asking about total compute. The SkillRL-continuation and Skill2Task-continuation comparisons match the policy-training setup: the same starting checkpoint, eight GPUs, learning rate 1e-6, rollout counts, episode limits, and optimizer steps. Skill2Task additionally incurs offline teacher and solver work. Across ALFWorld and Search, the measured totals are 2,457 teacher API calls and 1,948 solver API calls. We report this additional API usage separately from policy RL; the matched budget refers to policy training, rather than total end-to-end compute.

Dear reviewer, the discussion period is about to close. Please read the authors' response and let us know briefly which of your concerns remain. If you have any further questions for the authors, please post them soon so the authors have time to respond.

Reviewer_6dCHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Skill2Task, a framework that generates new training tasks from previously learned agent skills. It constructs a skill relation graph that captures skill applicability, reliability, and interactions, including support, prerequisite, and conflict relations. The graph guides task synthesis, followed by relevance and solvability verification. The synthesized tasks are then used for continued reinforcement learning. Experiments on ALFWorld and search-augmented question answering show improvements over SkillRL and other baselines using Qwen3-4B and Qwen3-8B.

Critical strengths
  1. Well-motivated approach to skill-guided learning. The paper identifies meaningful limitations of existing skill libraries, including conflicting, overfitted, and incorrect recommendations. The analysis in Section 3 provides concrete examples and demonstrates that unreliable skills can negatively affect agent performance. Using these limitations to guide task synthesis is an interesting and intuitive idea.
  2. Encouraging experimental results across different settings. The method consistently improves overall performance over SkillRL on ALFWorld and search benchmarks at both model scales. In particular, the improvements on unseen Pick&Place and Examine tasks are promising. The ablation studies also provide evidence that both node-conditioned and relation-conditioned task synthesis contribute to performance.
Critical weaknesses
  1. The specific contribution of graph structure remains insufficiently established. Although Table 3 shows improvements from relation-conditioned synthesis, the gains over node-only synthesis are relatively modest. The paper lacks stronger controls, such as graph-free task generation or randomized skill relations with matched task quantities and difficulty. Consequently, it remains unclear how much of the improvement comes from explicitly modeling skill relations versus simply introducing more diverse synthetic training tasks.
  2. Limited evidence for the effectiveness and efficiency of the synthesis process. The main experiments rely on continued RL initialized from a strong SkillRL checkpoint, making training-budget comparisons important. While the paper includes original-task continuation analyses, fully matched comparisons across benchmarks would strengthen the conclusions. Moreover, the search benchmark improvements are relatively small, and the paper provides limited quantitative analysis of task acceptance rates, synthesis quality, and teacher-model costs. These omissions make it difficult to assess the scalability and cost-effectiveness of the framework.
Questions for authors
  1. Can the authors compare graph-guided synthesis with simpler task-generation strategies, such as graph-free synthesis or randomized skill relations, under matched task counts and training budgets?
  2. Can the authors provide more details about the number of generated and accepted tasks, validation rejection rates, and the computational cost of graph construction and task synthesis?
  3. Can the authors report multi-seed results for the ablation experiments and clarify whether the gains over matched-budget SkillRL continuation are statistically significant?
2 comments

We thank the reviewer for the positive assessment of the motivation, stress-task diagnosis, and improvements on unseen ALFWorld tasks. We address the requested controls and accounting below.

Q1. How much does the graph structure contribute beyond adding diverse synthetic tasks?

Thank you for this important question. Table 3 already contains the node-only result (w/o relation tasks, 82.9% seen, 80.6% unseen, 81.8% overall). To directly isolate the contribution of relation labels, we add one matched pair control and report the existing edge-only branch alongside it.

The two edge-only conditions use the same co-retrieved skill pairs, seed tasks, accepted-task count, validation checks, RL steps, and random seeds. Edge-only (typed relation) is the existing w/o node tasks branch: it synthesizes tasks from skill pairs with their Support/Prerequisite/Conflict labels, but contains no node-conditioned tasks. Edge-only (no relation label) gives the teacher the same co-retrieved pairs but removes the relation label; it also contains no node-conditioned tasks. Thus, the only changed variable between the two edge-only conditions is the typed relation annotation, and no arbitrary skill matching is introduced.

ALFWorld Qwen3-4B variantSeenUnseenOverall
Full typed Skill2Task (node + edge)84.5 ± 1.183.1 ± 2.683.8 ± 1.8
Edge-only (typed relation; w/o node tasks)80.7 ± 0.881.3 ± 1.381.0 ± 0.7
Edge-only (no relation label; co-retrieved pair)78.8 ± 2.381.1 ± 1.179.9 ± 1.7

The untyped control tests whether the teacher can infer the latent interaction from the two skills’ conditions, behaviors, and effects without an explicit label. The observed gap is consistent with Support, Prerequisite, and Conflict reflecting relationships already implicit in the skill texts. Explicit labels specify the intended interaction, while the measured ablations show that the full graph combines relation-conditioned and node-conditioned practice and reaches 83.8%, above typed edge-only (81.0%) and node-only (81.8%).

Q2. How many tasks are generated and accepted, and what are the costs?

Thank you for asking for the complete synthesis accounting. The two-stage validation pipeline is explicit: a teacher checks target relevance, then a planner or blind solver checks solvability, followed by deduplication. In ALFWorld, candidates must correspond to legal goals and a valid plan. In Search, the solver does not see the reference answer and must produce an evidence-supported answer that matches it.

EnvironmentGeneratedRelevance rejectedSolvability rejectedDuplicatesAcceptedAcceptance rate
ALFWorld1,18793 (7.8%)70 (6.4% of 1,094)71,01785.7%
Search1,063189 (17.8%)134 (15.3% of 874)273869.4%

The lower Search acceptance rate is expected because exact-answer agreement and evidence support are stricter than the ALFWorld goal checker.

The policy-training budget is matched between SkillRL continuation and Skill2Task continuation: both start from the same best SkillRL checkpoint and use eight A800 GPUs, learning rate 1e-6, the same rollout configuration, episode limits, optimizer steps, and evaluation protocol. Skill2Task uses a 1:1 mixture of original and validated synthetic tasks, whereas standard SkillRL continuation uses the original tasks. Across ALFWorld and Search, the offline pipeline additionally used 2,457 teacher API calls and 1,948 solver API calls. These are measured API-call totals, reported separately from the matched policy-training budget.

Separately, Tables 1 and 2 report consistent overall improvements over SkillRL: ALFWorld improves 80.5→83.8 at 4B and 83.6→86.7 at 8B; Search improves 44.9→46.2 and 46.2→47.9. These are the main-table comparisons; the continuation analysis is shown in Figures 5 and 7.

Q3. Can you report multi-seed ablations?

Thank you for asking about run-to-run variation. We report the results across three random seeds below:

VariantSeenUnseenOverall
Full Skill2Task84.5 ± 1.183.1 ± 2.683.8 ± 1.8
From initial model79.3 ± 3.278.4 ± 2.178.8 ± 2.4
w/o node tasks80.7 ± 0.881.3 ± 1.381.0 ± 0.7
w/o relation tasks82.9 ± 2.680.6 ± 0.981.8 ± 1.4

Full Skill2Task improves over node-only synthesis by 2.0 points overall and 2.5 points on unseen tasks, and over removing node tasks by 2.8 and 1.8 points, respectively.

Dear reviewer, the discussion period is about to close. Please read the authors' response and let us know briefly which of your concerns remain. If you have any further questions for the authors, please post them soon so the authors have time to respond.

Comments on the paper

5 comments

We thank the reviewer for highlighting the distinction between task diversity and typed relations, and for asking how the policy handles misleading skills while they remain retrievable.

Q1. How does synthesis from co-retrieved pairs without relation labels perform?

Thank you for this precise control question. The following comparison keeps the same pair pool, seed tasks, validation, accepted-task count, RL steps, and random seeds. Neither edge-only condition includes node-conditioned tasks.

ALFWorld Qwen3-4B variantSeenUnseenOverall
Full typed Skill2Task (node + edge)84.5 ± 1.183.1 ± 2.683.8 ± 1.8
Edge-only (typed relation; w/o node tasks)80.7 ± 0.881.3 ± 1.381.0 ± 0.7
Edge-only (no relation label; co-retrieved pair)78.8 ± 2.381.1 ± 1.179.9 ± 1.7

The two edge-only conditions use exactly the same co-retrieved pairs; the only change is whether Support/Prerequisite/Conflict is supplied to the teacher. The observed gap is consistent with the teacher inferring the latent interaction from the two skills’ conditions, behaviors, and effects. The measured ablations show that the full graph combines relation-conditioned and node-conditioned practice and achieves the highest mean success, reaching 83.8% overall versus 81.0% for typed edge-only and 81.8% for node-only synthesis.

Q2. Does the final policy learn to reject or adapt to misleading recommendations when the flagged skills remain available?

Thank you for highlighting this mechanism-level question. Section 3 shows that unreliable guidance changes the outcome on the same SkillRL checkpoint: removing the flagged skills improves performance across Conflict, Overfitted, and Incorrect stress tasks. Figure 5(b) provides the corresponding policy/library swap analysis.

The most informative comparison is Final policy + Original skill bank versus SkillRL policy + Original skill bank. With the original bank held fixed, our policy improves unseen performance on both ALFWorld and Search while potentially unreliable recommendations remain available for retrieval. This supports improved policy behavior beyond library refinement alone. The comparison Final policy + Final skill bank versus Final policy + Original skill bank shows the additional benefit of library refinement. Together, the swaps show that Skill2Task improves through joint evolution of policy behavior and the skill bank, rather than through removing problematic skills alone. Figure 5(a) provides consistent execution evidence: Skill2Task produces fewer environment rejections and fewer repeated failures than standard SkillRL continuation.

Q3. How are the synthesized tasks controlled and validated?

Thank you for asking about the synthesis procedure. The accepted-task counts are 1,017 for ALFWorld and 738 for Search. Every candidate first receives target-relevance review, then solvability verification, and finally deduplication. The same validation pipeline is used for node-conditioned, typed edge-only, and untyped edge-only tasks. Policy comparisons start from the same SkillRL checkpoint and use the same RL steps, rollout settings, and 1:1 original/synthetic mixture, so the ablations differ in task conditioning rather than optimization budget.

The openreview system alerted me to a potential issue:

A figure's embedded file says its values were not measured

- Figure 1 (learning curves; “SkillRL plateaus”). The embedded figure file says: "Schematic SkillRL/Ours curves, not measured trajectories; Static Skill horizontal means from ALFWorld table"

The abstract states this figure's finding.

The note is stored in the figure's own file inside the PDF and is not visible on the page. Most PDF tools show it in the properties of the embedded figure.

I checked and confirmed that this issue is real. Could the authors please explain?

Thank you for pointing this out. Figure 1 was intended as a schematic illustration to give readers an intuitive view of the trends of the compared methods. It was based on our actual experimental results, with an AI assistant helping with plotting, smoothing, and interpolation between the available data points. The underlying experimental values are real, but the curves were not intended to represent point-by-point measured training trajectories.

We appreciate that this purpose should have been stated more clearly in the visible caption. We will clarify the figure’s schematic nature and plotting procedure, and revise the abstract to ensure its claims accurately reflect the measured evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.