acceptodds
Under review as a conference paper at ICLR 2027

A³-Skill: Learning Autonomous Management of Executable Tool Skills via Pure Reinforcement Learning

Abstract

Large language models (LLMs) increasingly rely on external tools to solve complex tasks. Tool skills enable LLMs to reuse procedural knowledge from past tool interactions. Existing approaches, however, often rely on fragmented skill learning, skill-operation supervision, or natural-language skill representations. We study autonomous tool-skill management: jointly acquiring, applying, and adapting executable tool skills within a single policy. We propose A-Skill, a two-stage reinforcement learning framework without supervised skill operations. In the first stage, we adopt a progressive curriculum that gradually transitions from reference-guided skill construction to autonomous skill acquisition and refinement. In the second stage, a leave-one-scenario-out strategy trains the policy to retain, revise, or rebuild imperfectly matched transferred skills through successive task interactions. We evaluate A-Skill on AppWorld across independent transfer, sequential adaptation, and cold-start acquisition, and additionally assess cross-benchmark generalization on BFCL-v3. On AppWorld, A-Skill improves TGC over the strongest competing baseline by 13.63% under sequential adaptation and 15.78% under cold-start acquisition. It also achieves 32.00% accuracy on BFCL-v3, compared with 28.00% for the strongest baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
tool skill

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3AHeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

A3A^3-Skill trains a single policy with RL to build, execute, update and keep executable tool skills, without supervised skill operations. Training has two stages: a progressive curriculum, followed by leave-one-scenario-out adaptation. The method is evaluated on AppWorld and BFCL-v3.

Critical strengths
  1. The problem is well motivated, and the two-stage design is coherent.
  2. The analyses are informative: textual vs. executable skills, skill-management decisions, and execution cost.
  3. The case study clearly illustrates the skill lifecycle.
Critical weaknesses
  1. Small absolute gains. The headline relative improvements amount to about 6 tasks on AppWorld and 6 problems on BFCL. SAGE is better in Independent Transfer.
  2. Unclear attribution. No variant disables skills at test time, and the tight turn budget may favor skills simply because they compress calls.
  3. Incremental contribution. RL-trained executable skills (SAGE) and RL-based skill evolution (SkillForge, WER, Skill-α\alpha) already exist, and the closely related methods are not compared experimentally.
Questions for authors
  1. Where do the Stage I reference trajectories come from?
  2. How does performance change with skills disabled at test time, or with a larger turn budget?
1 comment

Thank you for the response. I have read the rebuttal; the new ablations address my attribution concern (W2) and Q1. My assessment of the contribution and effect size is unchanged, so I maintain my rating.

Reviewer_huapOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A³-Skill trains a single policy to build, execute, update, and retain executable tool skills. Stage I progressively replaces successful reference trajectories with the model’s own interactions. Stage II excludes target-scenario skills during training to encourage adaptation of imperfectly matched skills. GRPO rewards combine task completion with downstream skill reuse and workflow-graph alignment.

On AppWorld with Qwen3-4B-Instruct-2507, A³-Skill improves sequential and cold-start TGC over the strongest competing baseline by approximately 3.57 percentage points in each setting, while independent-transfer TGC is slightly lower. The paper also reports component ablations, reduced code generation, and 32% versus 28% accuracy on a held-out BFCL-v3 subset with target-domain training permitted.

Critical strengths
  • A promising training design for acquiring and adapting executable skills.
  • Evaluation separates relevant capabilities and demonstrates generation savings.
Critical weaknesses
  1. The paper does not convincingly establish its incremental contribution over the closest prior work (SAGE, , https://arxiv.org/html/2512.17102v2), already learns executable skill generation, use, and update within one acting model, using sequential task chains and credit from later successful reuse. Although its main configuration uses SFT, SAGE also evaluate initializations without SFT, including RL warm-up. Consequently, the strongest distinctions are the curriculum, cross-scenario training, and workflow reward—not unified skill management or RL-only training by themselves. The paper acknowledges adapted baselines, but Appendix B.2 does not establish whether its SAGE implementation retains the defining sequential rollout and skill-integrated rewards. Without that comparison being specified, the reported gains do not convincingly demonstrate an advance over the nearest existing approach.

  2. The central workflow reward may measure the occurrence of a skill rather than its useful contribution. Equations (4)–(5) compare a skill with the best-matching subgraph of the observed task workflow. If this workflow includes the skill’s own expanded execution, a straight-line skill can match itself and receive maximal alignment even when it performs irrelevant reads before other tools complete the task. A stronger interpretation of “executed and used” could prevent this, but the paper does not define it operationally. The reward ablation shows that the signal helps, but does not establish whether graph matching adds information beyond rewarding successful skill invocation, particularly if removing alignment also changes the bonus magnitude. This matters because verifying reusable procedural structure is a central justification for the proposed reward. The concern is conditional on the implementation, it is not evidence that reward exploitation actually occurred.

Questions for authors
  1. Does the SAGE baseline retain its original sequential rollout and skill-integrated reward? Can you establish the benefit of the proposed curriculum and cross-scenario training against its RL-warm-up variant under comparable training conditions, and clarify the substantive distinction from SkillMaster’s learned lifecycle management?

  2. Does the target workflow graph include the skill’s own execution, and what prevents an executed but unnecessary skill from receiving high utility? Please provide a concrete scored example or diagnostic, and clarify whether the reward ablation isolates graph information from a change in the strength of the reuse bonus.

Reviewer_FT5xOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes A³-Skill, a two-stage reinforcement learning framework that trains a single policy to acquire, apply, and refine reusable executable tool skills. It combines progressive skill acquisition with cross-scenario adaptation. Experiments on AppWorld and BFCL-v3 show improvements over task-only RL, supervised skill learning, and prompting-based skill-use baselines.

Critical strengths
  1. The paper addresses an interesting problem: training agents to accumulate and improve reusable skills through interaction. Learning these capabilities within a unified policy is a promising direction.

  2. The curriculum and reward components are supported by multiple ablations. Results on a 4B model demonstrate benefits over both RL-only and SFT+RL baselines, as well as prompting-based skill reuse.

Critical weaknesses
  1. The training configurations of important baselines, particularly ToolRL and SAGE, are insufficiently specified. Training steps and total interaction budgets are not separately reported, and SAGE’s teacher-generated supervision is underdescribed. These omissions make it difficult to assess whether the improvements reflect the proposed method rather than differences in training information or budget.

  2. All controlled experiments use a single 4B model. It remains unclear whether the benefits persist with stronger models, limiting the demonstrated scope of the approach.

Questions for authors
  1. Please specify ToolRL’s and SAGE’s training steps, batch/rollout settings, and total training interaction budgets. How were these matched to A³-Skill?

  2. For SAGE, which teacher model generated the supervision, how many trajectories were used, how were they constructed and filtered, and what was the SFT schedule? Did the baselines receive comparable access to successful reference trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.