acceptodds
Under review as a conference paper at ICLR 2027

Meta-Skill Learning for Tool-Using Environment Adaptation in Large Language Model Agents

Abstract

Large language model (LLM) agents can often generalize to new tool-using environments from pre-prepared tool definitions and environment descriptions; however, such specifications rarely capture the implicit operational rules that govern success in non-standard environments. Interaction trajectories instead provide rich environment-specific knowledge, exposing structured patterns of success, failure, and recovery, and suggesting that such experience can be systematically distilled into reusable skills for accomplishing new tasks within the same environment. In this paper, we study training an LLM agent to rapidly adapt to new environments by learning a meta-skill, defined as the ability to explore a new environment, summarize environment-specific operational knowledge from limited interactions, and apply the resulting skill to subsequent tasks. We formulate this problem as meta-reinforcement learning over a distribution of tool-using environments and propose a meta-skill learning framework for environment adaptation. Experiments on multiple benchmarks show that environment-specific adaptation consistently improves performance on new tasks, supporting that tasks within the same environment share reusable skills. Moreover, the proposed framework outperforms strong baselines on both in-distribution and out-of-distribution environments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large langauage model, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YXTROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MetaSkill, a meta-reinforcement learning framework that trains LLM agents to adapt to unseen tool-using environments through limited interactions. The method consists of three stages involving environment exploration, extraction of reusable environment-specific skills, and skill-guided task execution. The authors introduce a stage-wise optimization strategy to train these components using downstream task rewards. Experiments on BFCL, τ-bench, τ²-bench, and ACEBench show improvements over the evaluated baselines, including on unseen environments and cross-benchmark transfer.

Critical strengths
  1. The paper identifies a concrete limitation of existing agents in adapting to implicit tool conventions and operational rules in unseen environments.

  2. The proposed stage-wise optimization offers a practical approach to credit assignment across exploration, skill extraction, and exploitation.

  3. The evaluation covers multiple model sizes, unseen environments, and cross-benchmark transfer, with ablation studies supporting the effectiveness of the proposed training design.

Critical weaknesses

Although I like the work, several concerns remain, and I would appreciate the authors’ responses.

  1. My main concern is the claim of generalizability. Without adaptation, MetaSkill shows limited additional gains on τ-bench, τ²-bench, and ACEBench and often performs similarly to RL and Experience-guided RL. Therefore, what does meta-skill training contribute beyond training-free approaches such as FAMA [1], which already use failure analysis to improve agent decisions? Please provide a comparison that establishes this benefit.

  2. My second concern is reliability. In real service environments, a failed task can frustrate customers or affect the system, and repair may not be possible. Although MetaSkill is trained, its adaptation still uses interactions that may contain failures. How does it address settings where test-time trial and error is unacceptable? Does training improve first-attempt reliability, or mainly improve subsequent tasks after feedback?

  3. I remain unconvinced that the results establish broad generalization learned during training. The largest improvements occur on BFCL, while two-shot gains over the strongest trained baseline on the transfer benchmarks are only 1.2 to 2.7 percentage points. Could the BFCL gains reflect shared tool-use logic between training and evaluation environments, while transfer improvements mainly depend on target-environment experience? Moreover, these small gaps may reflect evaluation variability. Please evaluate more than five runs, report confidence intervals, and compare decoding temperatures with equal tuning budgets across methods to establish whether the gains persist.

  4. Several skill-based RL methods already exist. Therefore, the authors should compare MetaSkill with representative methods from this literature and clearly identify its algorithmic contribution. Without these comparisons, the paper does not sufficiently establish novelty at the algorithmic level.

  5. Please specify which 17 BFCL environments were used for training and which three were held out, including task IDs and counts. Also clarify whether all training tasks come from public BFCL data or include additional constructed data. Since tasks can involve multiple tool classes, how are environments defined and separated? Without these details, the unseen-environment evaluation cannot be independently verified.

  6. Please identify the user-simulator models and decoding settings for τ-bench, τ²-bench, and ACEBench. Although Appendix C.3 reports three evaluation runs, please clarify what varies between runs given the agent’s greedy decoding. For ACEBench’s agent portion, please also specify the evaluated subsets and scoring protocol.


Reference:

[1] FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments. https://arxiv.org/abs/2604.25135

Questions for authors

Please read the weaknesses.

Reviewer_wH8iOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MetaSkill trains a shared LLM to collect support-task trajectories, extract textual operational guidance, and use that guidance on distinct query tasks. All three stages receive downstream query-success rewards through separate GRPO groups. At deployment, the model adapts through generated guidance without parameter updates. Experiments with Qwen3-4B and Qwen3-8B report improvements on held-out BFCL tasks and environments, and transfer to τ-bench, τ²-Bench, and ACEBench. Ablations examine the stage-specific updates and compare against end-to-end GRPO.

Critical strengths
  1. Training exploration and skill extraction using success on a distinct query task directly rewards information that transfers beyond the support task. This is a substantive distinction from same-task retry methods such as LaMer. Jointly training the three roles in one model gives a practical formulation of this objective.

  2. All methods reportedly use the same inference pipeline and budget within each adaptation setting, making this useful evidence that training improves how the model uses support experience.

Critical weaknesses
  • The evaluation does not yet establish what environmental knowledge is genuinely held out. The official BFCL data contains 20 API-class combinations built from eight APIs, each appearing in at least four combinations. If these are the paper's 20 environments, holding out three combinations still leaves every API represented in training. This tests new combinations, but does not by itself establish adaptation to unseen API implementations or operational rules. The paper does not define its grouping precisely enough to resolve this distinction.

  • Task-family isolation is equally consequential for the cross-task claim: in the inspected official snapshot, 185 of 200 corresponding base/long-context records have identical complete user-question sequences. Different record IDs therefore do not establish independent tasks. The authors need to explain how related variants are separated across training/evaluation and support/query tasks. This does not demonstrate leakage, and the cross-benchmark results remain encouraging, but the ambiguity directly limits the interpretation of the central BFCL evidence.

  • The benefit of stage-wise optimization is not separated from additional training samples. Algorithm 1 evaluates 3G query trajectories across its three groups, while the end-to-end GRPO ablation is described as a single group; the paper does not establish matched total training budgets. The reported 50.2% versus 47.3% unseen/three-shot comparison consequently does not tell us whether the gain comes from stage-wise conditioning or additional query evaluations and training signal. This matters because the stage-wise algorithm is a central methodological contribution. The theoretical analysis does not resolve this confound: it establishes properties of reward variation and conditional baselines, rather than a variance guarantee for the implemented GRPO estimator. A controlled training comparison would be more consequential than further theoretical elaboration.

Questions for authors
  • What defines an environment, and which three are held out? Please show how underlying task families—including variants across BFCL categories—are partitioned between training/evaluation and adaptation support/query sets. If the holdout is by API combination, which operational knowledge is intended to be unseen?

  • How do the total training interactions and query evaluations compare between MetaSkill and end-to-end GRPO? Does MetaSkill retain its advantage when training budgets and stage-loss weighting are controlled?

Reviewer_TJcGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a MAML-style algorithm to optimize LLM skills. The inner loop, for each samples environment, proposes skills (explore, extract, then exploit), and the outer loop optimizes LLM weights to improve expected ability to adapt to new environment.

Critical strengths

This paper presents a meaningful meta-learning formulation for tool-using agents. The paper moves beyond optimizing individual task success and explicitly optimizes performance after environment-specific adaptation.

Critical weaknesses

Insufficient disentanglement of where the improvements originate.

Because exploration, skill extraction, and exploitation share the same trainable LLM parameters, the reported gains do not clearly establish which capability is primarily responsible for the improvements. In particular, better post-adaptation performance does not necessarily imply that the model learns to extract higher-quality skills; it could also result from better skill utilization or changes in the underlying task-solving policy. The stage-wise training ablations are useful but do not fully isolate these factors.

Questions for authors

Can the authors disentangle improvements in exploration, skill extraction, and skill utilization?

How robust are the improvements when support and query tasks differ substantially in their workflows?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion and Manuscript Revisions

Dear Area Chair and Reviewers,

We thank Reviewers TJcG, wH8i, and YXTR for their careful reviews. We summarize our responses and additional evidence below. Manuscript revisions are marked in blue.

1. Core Contribution:

MetaSkill jointly trains exploration, skill extraction, and utilization through performance on new query tasks after learning from support tasks in the same environment. Its contribution is their coordinated ability to use environment experience for cross-task adaptation. We assess this ability through post-adaptation accuracy under the same support budget and inference pipeline, without test-time parameter updates.

2. Key Evidence and Additional Evaluations:

On BFCL unseen environments, Qwen3-8B MetaSkill, RL, and Experience-guided RL start at similar accuracy (42.9%, 42.8%, and 43.1%) but reach 50.2%, 44.2%, and 43.6% after three-shot adaptation (Table 1). Shared tool familiarity alone therefore does not explain MetaSkill's advantage.

We added five-run cross-benchmark evaluations with independently sampled support tasks and 90% confidence intervals. Without further parameter training, MetaSkill retains the highest mean post-adaptation accuracy across all transfer domains, both model sizes, and both adaptation budgets (Section 3.2, Tables 2–5). Representative Qwen3-8B two-shot results are:

Transfer settingBest trained baseline (%)MetaSkill (%)
τ-bench Retail47.750.1
τ²-Bench Retail33.234.4
ACEBench English Agent34.037.2

The baseline is the better of RL and Experience-guided RL. Full tables report confidence intervals; Appendix C.3 specifies the evaluation protocol, including fixed agent and user-simulator temperatures of 0.0.

3. Point-by-Point Summary:

Reviewer TJcG

  • W1 / Q1: Which of the three capabilities improves? MetaSkill learns their coordination. Removing any stage-specific training objective reduces performance, supporting the contribution of all three to the complete adaptation process (Appendix D.1).
  • Q2: Can experience help tasks with different workflows? Tasks can reuse environment rules without sharing the same workflow. Our case study demonstrates this transfer on a successfully completed query (Appendix D.3, Case 2).

Reviewer wH8i

  • W1 / Q1: What is unseen at test time? BFCL tests new tasks in held-out API combinations; the other benchmarks additionally test new API implementations and operational rules. The partition is explicit in Appendix C.2 (transfer results: Section 3.2).
  • W2 / Q1: Could related task variants cause leakage? Related variants stay on the same side of the training/held-out environment split. Support/query pairing also stays within the same category, preventing the cross-category pairing raised by the reviewer (Appendices C.2–C.3).
  • W3 / Q2: Does the training comparison account for different budgets? We compare performance after both methods reach a plateau: 50.2% for MetaSkill versus 47.3% for end-to-end GRPO under identical test-time conditions. We clarified this comparison and disclosed training rollout counts and loss weights (Table 7; Appendix C.3).

Reviewer YXTR

  • W1: What does training add beyond zero-shot performance or training-free adaptation? MetaSkill learns to use environment experience more effectively, as shown by higher post-adaptation accuracy under the same inference conditions. Learning this capability complements training-free adaptation design (Tables 1–5).
  • W2: Must the evaluated task fail before adaptation helps? No. Experience comes from other tasks in the same environment and guides the new query from its first attempt. Our response explains how historical or staging interactions could supply this experience.
  • W3: Are gains explained by shared tools or evaluation variability? Baselines trained on the same BFCL tools benefit less from adaptation. Added five-run transfer evaluations with 90% confidence intervals retain MetaSkill's highest mean accuracy across all tested domains and both model sizes (Tables 2–5).
  • W4: What distinguishes MetaSkill from prior skill-based RL? Its objective explicitly trains experience acquired on one set of tasks for usefulness on different tasks in the same environment. Our response explains this distinction from same-task retries and skill-library methods (Appendix A).
  • W5: Are the data sources and partitions clear? All tasks come from public BFCL data. We provided the complete environment partition and task counts in our response and documented the construction in Appendix C.2.
  • W6: Are evaluation settings reproducible? We specified the user simulator, decoding settings, support sampling, evaluation repetitions, and ACEBench subset and scoring protocol (Appendices C.2–C.3).

Thank you for your consideration. We welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.