- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MetaSkill, a two-layer framework for LLM agents equipped with reusable skills. The Method layer maintains a library of 26 manually authored problem-solving methods and each method specifies entry conditions, exclusions, role slots, state variables, policy rules, invariants, review checks, repair rules, and termination criteria. At task time, the agent selects a method and binds its abstract roles to task objects. The Execution layer uses the instantiated method to determine intermediate results, select and coordinate native skills, and check outcomes. On failure, it either repairs execution locally or reselects the method.
Experiments cover 89 SkillsBench tasks and 995 ToolHop examples, with two agent frameworks and two LLMs. MetaSkill outperforms BoT and AgentSkillOS in all six configurations, by 3.51–13.48 pp over the strongest baseline. Ablations on SkillsBench indicate that method-guided orchestration outperforms injecting method definitions as context, and that method reselection contributes 4.50 pp.
Critical strengths
- The paper targets a relevant gap. Reasoning-template approaches offer reusable strategies that are not tied to concrete skill execution. Skill-composition approaches organize execution without method-level guidance on what to check or when to revise.
Critical weaknesses
-
The central claim is not isolated. Direct injection removes both the method library and the explicit orchestration loop, and there is no "orchestration without methods" control, so the gains cannot be attributed to methodological knowledge. The existing numbers point the other way: injecting all 26 methods (55.06) scores below BoT (60.67), and MetaSkill without reselection exactly matches BoT, which leaves a simple check-and-retry loop as an equally plausible explanation.
-
The ToolHop error analysis contradicts the proposed mechanism. MetaSkill's advantage comes from eliminating incomplete executions (69 -> 0), while completed-but-incorrect answers rise (284 -> 329). Among completed executions, MetaSkill is no more accurate than BoT (64.7% vs. 65.8%), so the gain reflects finishing more reliably rather than better dependency reasoning.
-
The SkillsBench evidence is statistically weak. Margins over the strongest baseline amount to only 4-12 of 89 tasks, from a single run with no variance estimates or significance tests. All ablation claims, including the 4.50 pp (4-task) reselection effect, rest solely on this benchmark.
Questions for authors
- How does the full orchestration loop perform without any method library, in the Table 5 configuration? This control would show whether the gains come from methodological knowledge or from orchestration alone.
- On ToolHop, what is each method's accuracy among completed executions, and how are incomplete executions distributed across chain lengths? Figure 2 suggests the advantage comes mainly from completing execution rather than from more accurate answers.
- Are the SkillsBench results from a single run, and can you report variance and paired significance tests across multiple runs? The margins correspond to only 4–12 of 89 tasks.
- Which tasks were the 26 methods derived from, and do any overlap with the SkillsBench evaluation tasks?
2 comments
Thanks for the detailed reply. The new Q1 controls are useful, but since the method's contribution remains confounded with the much higher token budget and the Q2 breakdown is still missing, I will keep my original score.
Response to Reviewer 4TWe
Q1: Isolating the contribution of method knowledge
We added two controls on the same 89 SkillsBench tasks using PI Agent with DeepSeek-V4.1-Flash. The gap is full MetaSkill task accuracy minus the accuracy of each configuration.
The no-library control retains planning, online skill selection, execution, checking, and bounded repair. Its gap to full MetaSkill measures the contribution of adding curated methods together with their role bindings and decision rules. Generating a method for each task under the same budget also yields lower accuracy, supporting the value of reusing knowledge across tasks. Direct injection supplies the curated definitions as text. Its lower accuracy shows the additional benefit of applying those definitions to guide execution.
Q2: Completed accuracy and incomplete executions
All methods in the ToolHop experiment share a maximum of nine execution steps. An execution is classified as incomplete if it has not produced a final answer file meeting the submission requirements before the budget expires. This differs from submitting an incorrect answer. Incompleteness therefore reflects the ability to organize useful calls and deliver an answer within a finite budget, rather than an external failure independent of the solving strategy.
Some baseline executions exhaust steps on indirect, redundant, or erroneous tool calls without obtaining information needed downstream. MetaSkill binds roles to the entities being resolved, uses method rules to constrain call dependencies and required results, and reuses available results. These constraints direct limited steps toward useful progress. Reducing incomplete executions is therefore a benefit of method-guided tool orchestration under the shared step budget, supporting the proposed mechanism rather than contradicting it.
Q3: Repeated runs and statistical tests
The SkillsBench results are from single runs. We have not completed independent reruns or paired significance tests, and do not report cross-run variance or statistical significance.
Improvements across all six configurations show a consistent direction across models, agents, and benchmarks; the added controls examine the source of the gain. Neither measures run-to-run variability. The results support improvements in the evaluated settings, with stability across independent runs still to be quantified. We will conduct independent repeat runs to assess the stability of these results.
Q4: Method sources and evaluation-task overlap
Using only public task descriptions and native Skills as source material, we built the library through LLM-assisted analysis and human synthesis. No execution trajectories, failure outcomes, or scoring feedback were used in construction. We first organized seven families by the main work needed to complete tasks, using task goals and Skill-based solution approaches to scope cross-task comparisons. Within each family, we compared applicability conditions, key dependencies, and decision principles to extract reusable logic.
Task objects became bindable roles; shared relations guiding progress, checking, and adjustment were retained, while implementations stayed in native Skills. LLMs assisted analysis and candidate drafting; humans compared candidates, checked sources, revised, and selected 26 structured MetaSkills across the seven families. Section 4.1.1 and Appendix A clarify the process and library in the revision.
For example, separate-nuisance-estimate shares reference checks across economic detrending and video analysis while leaving estimators in native Skills. Methods encode no reference answers, target outputs, or complete solutions.
Evaluation fixes definitions; runtime adaptations do not update them. ToolHop is not a construction source. Applying the library there improves accuracy by 3.51-4.72% over the strongest baseline, supporting reuse beyond SkillsBench source procedures.