acceptodds
Under review as a conference paper at ICLR 2027

CoSkill: Joint Reinforcement Learning of Reasoning and Meta-Skill Agents for Skill–Policy Co-Evolution

Abstract

Skill libraries improve the sample efficiency of agentic reinforcement learning (RL) by enabling large language model (LLM) agents to reuse procedural knowledge. Yet existing paradigms exhibit structural shortcomings: they either decouple skill evolution from policy optimization or instantiate meta-skills as fixed workflows. Both treat skills as passive objects to be managed, limiting the flexible evolution of skills and their co-adaptation with the reasoning agent. To address these limitations, we propose CoSkill, a unified multi-agent RL framework that recasts the static meta-skill workflow as a learnable Meta-Skill Agent and jointly trains it with a Reasoning Agent over a hierarchical skill library. By modeling the Reasoning and Meta-Skill Agents as a cooperative team sharing a single backbone, CoSkill enables end-to-end co-adaptation: the Reasoning Agent conditions its actions on a retrieved task skill and step skills selected from its child set, while its task performance guides the Meta-Skill Agent in refining those step skills. Experiments on ALFWorld and WebShop show that CoSkill substantially outperforms prior skill-based and RL baselines, achieving success rates of 98.4% and 90.6%, respectively (+3.5 and +6.2 pp). As shown in Figure 1, CoSkill achieves superior early-stage sample efficiency, asymptotic performance, and wall-clock efficiency. Ablation studies and quantitative analyses further demonstrate that treating meta-skill evolution as an RL-trainable agent substantially accelerates learning while yielding more selective, reliable, and structurally stable skill evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Multi-agent system, Skill optimization, Co-evolving

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_23JXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CoSkill, a framework that jointly trains a Reasoning Agent and a Meta-Skill Agent using a shared language model and a hierarchical skill library. The Reasoning Agent retrieves task-level and step-level guidance, while the Meta-Skill Agent edits step skills based on execution feedback. Proposed edits are evaluated by re-executing the same task, with the change in return providing feedback for the editor. Both roles are trained through GiGPO. Experiments on ALFWorld and WebShop examine task performance, learning efficiency, skill evolution, and generalization.

Critical strengths
  1. CoSkill jointly trains task execution and skill editing using a shared language model. The hierarchical skill library provides both task-level and step-level guidance, allowing the Meta-Skill Agent to modify individual procedures based on execution feedback while the reasoning policy is updated through reinforcement learning.

  2. The authors retain the editing and verification procedures when removing the Meta-Skill Agent’s RL objective, helping assess the contribution of learning to edit. Experiments with a frozen Reasoning Agent also show improvements, providing evidence that skill-editing training is useful even without updating the reasoning model.

  3. The paper includes experiments on the skill hierarchy, joint-update frequency, verification attempts, and unseen task types. The case studies provide examples of how learned edits preserve task constraints and retain reusable procedures, offering additional evidence about the behavior of the Meta-Skill Agent.

Critical weaknesses
  1. The initial skill library is generated by an external model, but the authors do not specify which model is used. Given the importance of this initialization, an analysis of skill quality, library size, and task coverage would help clarify how strongly the reported results depend on the initial library.

  2. Figure 5 shows that, as training progresses, UPDATE accounts for 97–99% of the Meta-Skill Agent’s decisions, while the other operations are rarely selected. This makes it unclear whether the reported gains come from learning to choose appropriate editing operations or mainly from improving existing skill content. The current analysis does not sufficiently distinguish these two explanations.

  3. CoSkill relies on a pre-built library to start online learning. Although INSERT can add new step skills, it requires an existing parent task skill. The authors do not evaluate CoSkill with an empty library or with initial skills generated by its own backbone. Its ability to start skill learning without externally prepared knowledge therefore remains unclear.

  4. The experiments are limited to two relatively constrained benchmarks. Additional evaluation on more challenging tasks, such as coding, would strengthen the empirical evidence.

Questions for authors
  1. The generalization experiment considers held-out task types within the same environment. Would CoSkill remain effective under larger distribution shifts, such as tasks from a different environment or domain? How well would the learned skills and editing policy generalize to these settings?
  2. How sensitive is CoSkill to the quality of the initial skill library? If the initial skills contain errors or omit important procedures, can joint training effectively correct them, and how would this affect learning efficiency and final performance? A case study illustrating how CoSkill handles incorrect or incomplete initial skills would be helpful.
  3. Could the authors clarify whether the task-level skill content is also updated when a revised bundle is promoted? Appendix D describes step-level editing operations, whereas Figure 4 refers to an “Edited Task Skill.”
2 comments

Thank you for the rebuttal and additional experiments. However, my concern regarding W4 remains. The new experiments do not sufficiently demonstrate generalization to more complex scenarios. Therefore, I will maintain my original rating of 2.

AuthorsOpenReview ↗

Thanks for your comment.

Thank you for your comment. To address W4, we have added results on two additional benchmarks: TriviaQA (single-hop) and HotpotQA (multi-hop). These results demonstrate that Coskill yields consistent improvements across four diverse benchmarks: ALFWorld, WebShop, TriviaQA, and HotpotQA.

With the rebuttal deadline approaching, we unfortunately do not have sufficient time to evaluate additional benchmarks. We sincerely appreciate your suggestions, which have helped strengthen our work. Moving forward, we plan to extend Coskill to real-world applications, and we appreciate your continued interest in our work.

Reviewer_WqJSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes CoSkill, a reinforcement learning framework that jointly learns to execute tasks and refine the skills used to solve them. A Reasoning Agent and a Meta-Skill Agent share a Qwen2.5-7B-Instruct backbone and operate over a hierarchical library of task-level guidance and step-level procedures. The Meta-Skill Agent proposes edits based on execution feedback; these edits are evaluated by rerunning the task, with the average return improvement over the original attempt serving as the editing reward. Both roles are jointly optimized using GiGPO, while verified skill revisions are selectively retained in the library.

The paper reports success rates of 98.4% on ALFWorld and 90.6% on WebShop, compared with 94.9% for RetroAgent and 84.4% for D2Skill (O3), respectively, in its main comparison. These correspond to gains of 3.5 and 6.2 percentage points. Ablations examine the contributions of learned skill editing, hierarchical organization, and joint optimization. Additional experiments with a frozen reasoner provide evidence that training the editor can improve task performance independently of updates to the reasoning policy.

Critical strengths
  1. The method connects skill-content optimization to an explicit execution-based learning signal. Sections 3.2–3.4, especially Equation 13 and Algorithm 2, distinguish the baseline trajectory, staged revisions, verification trajectories, and role-specific credit assignment. This avoids crediting a proposed revision solely from the transition that motivated it. The contribution is consequential because it makes the editing policy itself trainable while retaining a shared backbone and a structured skill library, rather than relying entirely on a fixed external editing workflow.

  2. The paper provides substantive evidence that training the editor matters. The no-Meta-Skill-RL ablation retains editing and verification, so its comparison controls for the mere presence of these mechanisms: full CoSkill reaches 70.31% versus 57.81% at step 20 and 95.31% versus 92.19% at step 100 (Table 2). Appendix H goes further by using a separately parameterized, frozen reasoner: the Qwen experiment reaches a historical-best 42.19% versus 15.63% at step 40 (Table 4). These results support useful learning in the editor rather than attributing every gain to improvements in the reasoning weights. Figure 5 also connects the intervention to fewer empty candidate bundles and less cumulative promotion, although those diagnostics alone do not prove semantic edit quality.

Critical weaknesses
  1. The cross-method accuracy advantage needs a clearer common evaluation protocol. Appendix J transparently distinguishes reproduced baselines from results imported under their original protocols. However, the headline gains of 3.5 points on ALFWorld and 6.2 points on WebShop are over RetroAgent and D2Skill (O3), respectively, both imported results. The paper does not establish that these comparisons use identical evaluation task sets, aggregation rules, and checkpoint-selection procedures. For CoSkill's main ALFWorld result, Appendix K reports historical-best validation performance; its three-seed mean of 97.40% with a standard deviation of 1.81 points is useful evidence of run-to-run stability, but does not establish performance on tasks independent of checkpoint selection. The reproduced baselines and controlled ablations remain informative. My concern is specifically that the reported margins over the strongest prior methods are not yet supported by a clearly comparable evaluation.

  2. The budget analysis does not establish equal total interaction or compute costs. Appendix G matches rollout-group sizes, but Algorithm 2 and Table 7 specify that each CoSkill baseline rollout is followed by two full verification attempts, whose trajectories also train the reasoner. Matching the group-size parameter therefore need not match the amount of environment experience or training data. D2Skill also uses paired rollouts, so the relative overhead cannot be inferred from CoSkill's attempt count alone. Appendix F usefully measures the effect of verification count, and Figure 1 reports wall-clock comparisons, but neither resolves the cross-method accounting without a breakdown of actual interactions and comparable hardware/timing settings. This limits the evidence for the claimed sample- and wall-clock-efficiency advantage, while leaving the controlled evidence for learning the editor intact.

Questions for authors
  1. Do CoSkill, the reproduced baselines, and the imported RetroAgent/D2Skill (O3) results use the same evaluation task sets and success definitions, including the WebShop product/goal split? Please distinguish total evaluation-set size from validation batch size in the 128/64 descriptions. Also, Appendix J describes a macro average, but Skill1's six Table 1 category scores average approximately 91.1%, versus the reported 93.7%. Is the aggregate instead computed over individual episodes, and is the same aggregation used across methods?

  2. Were the same ALFWorld tasks used both to select the historical-best checkpoint and to report the headline score? If so, can the selected checkpoints for CoSkill and a strong reproduced baseline be evaluated on tasks not used for selection, with uncertainty reported across available runs or evaluation tasks? This would clarify the robustness of the performance advantage beyond the three-seed historical-best statistics already provided.

  3. For the comparisons in Appendix G and Figure 1, what are the cumulative environment interactions and compute costs, including verification and editing, at comparable success levels? Please specify the GPU models and timing boundaries, including whether offline library construction is counted. Appendix K reports approximately 66 hours for the default M=2 run, whereas Appendix F reports 157.23 hours for a separate M=2 run; which hardware or workload differences explain the gap, and which settings underlie Figure 1?

Reviewer_5yffOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoSkill trains a Reasoning Agent and a Meta-Skill Agent that share one Qwen2.5-7B backbone over a hierarchical skill library (task skills with child step skills). After each step, the Meta-Skill Agent proposes INSERT, UPDATE, DELETE, or KEEP on the active step skill. Edits are staged, the task is re-run M=2M=2 times, and the edit is rewarded by Δiskill=1M∑mRi(m)−Ri(0)\Delta^{skill}_i = \frac{1}{M}\sum_{m} R^{(m)}_i - R^{(0)}_i. Both roles are trained jointly with GiGPO. The method reports 98.4% on ALFWorld and 90.6% on WebShop, with ablations, a frozen-reasoner study, and analysis of edit behavior.

Critical strengths
  1. The frozen-reasoner study (Appendix H) shows that training only the skill editor raises success from 15.63% to 42.19%. This directly supports the main claim that the editor should be a learned policy.
  2. Figure 5 and the case studies explain the mechanism well. Without RL the editor deletes more skills, grows the library faster, and leaves 12.4% of bundles empty, while the RL editor makes targeted updates.
Critical weaknesses
  1. All main numbers use the best validation score over many checkpoints on what appears to be a 64-task set (98.44% = 63/64). One task is 1.56 points, and key ablation gaps (3.12 and 1.56 points) are within the seed range of the full model (3.13 points). Table 1 also mixes reproduced and copied baseline numbers.
  2. Cost and external help are not fully accounted for. Verification triples the episodes per task, and the budget study matches group size, not environment steps. Appendix K says the M=2M=2 run took 66 hours, but Appendix F says 157 hours, which affects Figure 1(c). The initial library is built by an unnamed external LLM.
Questions for authors
  1. What is the validation set size, and what are the final-checkpoint results for CoSkill and the reproduced baselines?
  2. Which wall-clock number for M=2M=2 is correct, and which run is used in Figure 1(c)?
  3. Which external LLM builds the initial library, and how does GiGPO perform with the same library but no editing?
  4. Can you compare methods under matched environment steps?
  5. Are the evaluation tasks disjoint from the tasks used to build the offline library?

Suggestions for future work:

  1. Evaluate on a less saturated benchmark and a second backbone family.
1 comment

Thank you for the detailed response and the new experiments. The final-checkpoint results, the Qwen2.5-7B-initialized library, and the fixed-library GiGPO control address my main concerns, and I have raised my rating to 3.

One point remains unclear. The response states that the main experiments use M=1 and that M=2 in Appendix K was a typo. However, Table 7 and Appendix F also state that M=2 is the default, and Appendix F reports that the M=1 run peaks at 95.31% and ends at 93.75%, while the M=2 run reaches 98.44%. The headline 98.4% and the new final-checkpoint 98.43% match the M=2 run. Could you clarify which M the Table 1 results use, and whether Figure 1(c) and the budget comparison in Appendix G use the same run?

I also note that the evaluation tasks overlap with the tasks used to build the offline library. Since the reproduced baselines follow the same setup, the comparison remains fair, but I suggest stating this clearly in the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.