- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the MAS-Skill paradigm, which extends skill extraction from extracting skills of individual agents to extracting skills from the collaboration of multiple agents. It selects better-performing configurations from 15 basic configurations, packages them as base skills, and then combines and transforms these stored base skills through genetic algorithms and other mechanisms to form new skills.
Critical strengths
-
The experiments are relatively complete, and ablation studies are also included.
-
The problem itself is important and practical: how to construct collaboration topologies is indeed a problem.
-
Cross-model experiments are conducted, demonstrating generalizability.
-
The paper considers not only accuracy, but also token consumption.
Critical weaknesses
-
The claimed contribution of extending skill extraction from single agents to multi-agent topologies has limited novelty, since there have already been a large number of works on skills for multi-agent topologies.
-
The paper evaluates 15 base configurations, selects better-performing configurations and summarizes them into skills, and then updates them through methods such as genetic algorithms. This seems more like searching for the best skills first and then searching for better solutions within a neighborhood around their topologies, which may make the comparison with other algorithms unfair. The performance improvement may also come from more extensive offline search rather than from the skill itself.
-
The performance improvement seems relatively small. The main advantage of the paper is also unclear, as the method seems to mainly combine existing techniques according to its own pipeline. It is not clear what new insights MAS-Skill provides beyond this integration.
Questions for authors
-
The paper claims that its contribution is to extend skill extraction from single agents to multi-agent topologies? However, there have already been a large number of works on skills for multi-agent topologies.
-
The paper uses 15 base configurations to calculate accuracy, selects better-performing configurations and summarizes them into skills, and then updates them through methods such as genetic algorithms. I feel that this is more like searching for the best skills first, and then searching for the optimal solution within a neighborhood around their topologies. Is this unfair when comparing with other algorithms? Is it possible that the performance improvement comes from more extensive offline search rather than from the skill itself?
-
The performance improvement seems relatively small. I do not really understand what the biggest advantage of this paper is. It seems to simply combine existing methods according to its own pipeline. Could the authors explain the advantages in more detail? What new insights can MAS-Skill bring?
3 comments
Q2: Are comparisons fair with 15 configurations and genetic search? Do gains reflect extra offline search or skills?
A2: Thank you for raising the possibility that the improvement is due to greater offline search. We address this in two steps: first account for the preparation investment, then compare ways of using exactly the same learned library. This distinguishes the cost of acquiring collaboration knowledge from the contribution of selecting it at inference.
The 15 initial configurations are the combinations of five topologies and three agent counts (Sec. 3.2, Eq. 3). They provide empirically evaluated starting points rather than defining the complete search space. Fusion can change role assignments, add or remove agents, rewire dependencies and revise synthesis instructions using successful and failed trajectories. Offspring are selected on development data (Eq. 8), and the resulting records are retained for later queries. We acknowledge that this process requires computation.
Following your suggestion, we report offline and online token usage for AFlow[6], MaAS[7] and MAS-Skill on all six benchmarks. Offline usage includes exploration, experience extraction, fusion and validation; online usage includes planning, gating, execution and synthesis. Values are total input plus output tokens in millions, with the same evaluation set across methods within each benchmark.
Table L2a. Offline construction cost (million tokens)
| Method | GSM8K | MATH | MultiArith | MBPP | HumanEval | HotpotQA |
|---|---|---|---|---|---|---|
| AFlow[6] | 52.26 | 52.88 | 42.06 | 2.70 | 2.48 | 3.29 |
| MaAS[7] | 36.37 | 22.45 | 27.16 | 1.88 | 1.82 | 1.96 |
| MAS-Skill | 40.90 | 56.68 | 44.62 | 7.54 | 3.99 | 2.39 |
MAS-Skill has a substantial preparation cost, exceeding MaAS on all benchmarks and AFlow on four. This makes the investment explicit: the method spends computation to obtain a library of useful collaborations and the evidence needed to reuse them.
Table L2b. Online evaluation cost (million tokens)
| Method | GSM8K | MATH | MultiArith | MBPP | HumanEval | HotpotQA |
|---|---|---|---|---|---|---|
| AFlow[6] | 6.37 | 4.66 | 5.10 | 2.69 | 1.66 | 1.33 |
| MaAS[7] | 9.25 | 7.23 | 5.37 | 2.33 | 1.50 | 1.26 |
| MAS-Skill | 5.09 | 3.97 | 4.70 | 2.28 | 1.35 | 1.26 |
The learned library is then used at a lower online cost than AFlow on all benchmarks and at a lower or equal cost than MaAS. Together with the accuracy results in the original Table 1 and Table L1 above, this shows the intended benefit of the preparation stage: stronger task performance and more economical inference after learning. The full deployment cost should still include the initial investment, amortized over subsequent queries.
Cost accounting alone does not isolate the skill-selection contribution. We therefore also compare query-dependent retrieval with the best development-selected skill per type from the same base-plus-fused library. Both retain complete executable records and share models, tools and generation limits; both HotpotQA conditions use provided subtypes.
Table L3. Selection with the learned library held fixed
Accuracy / pass@1 (%):
| Method | GSM8K | MATH | MultiArith | MBPP | HumanEval | HotpotQA | Avg. |
|---|---|---|---|---|---|---|---|
| Best fixed skill/type | 93.08 | 51.65 | 98.50 | 72.43 | 88.55 | 54.50 | 76.45 |
| MAS-Skill | 93.18 | 53.09 | 99.00 | 73.02 | 88.55 | 54.75 | 76.93 |
Query-dependent selection improves five point estimates and ties on HumanEval. Because even the fixed comparator benefits from the learned library, this comparison focuses on how the knowledge is used rather than whether it was acquired.
Mean online tokens per query:
| Method | GSM8K | MATH | MultiArith | MBPP | HumanEval | HotpotQA |
|---|---|---|---|---|---|---|
| Best fixed skill/type | 13,330 | 13,076 | 11,097 | 8,744 | 10,928 | 9,729 |
| MAS-Skill | 13,157 | 10,960 | 9,404 | 5,980 | 8,798 | 9,645 |
Retrieval reduces usage across all six benchmarks while maintaining or improving performance. This difference cannot be attributed to additional offline search, since the library is identical. The two experiments therefore answer different parts of your concern: learning the library has a real preparation cost, and selecting an appropriate learned collaboration provides an additional execution benefit. We will present these results together so that the efficiency and mechanism claims are supported by the relevant controls.
Q3: Given the relatively small performance gains and the combination of existing techniques, what are the main advantages and new insights of MAS-Skill beyond integrating these components?
A3: Thank you for asking us to explain the broader insight and the practical importance of the gains. The central idea is that a main agent can reuse knowledge of how a team should collaborate, rather than reconstructing that knowledge independently for every query. The useful resource is not only an answer pattern or a role prompt, but a complete protocol connecting responsibilities, communication, tools and synthesis with evidence about its applicability.
The added experiments help make this insight concrete. Table L1 compares the approach with recent workflow, topology and skill-based methods, including Skill-MAS[2], and shows the strongest average performance. Table L3 holds the learned library fixed and shows that query-dependent use improves the accuracy–cost trade-off. Tables L2a–L2b account for the preparation required to obtain those records and the lower online usage once they are available. These results connect the proposed representation to measurable consequences, rather than relying on the novelty of the individual search or retrieval components.
The specialist-collaboration experiment follows Sec. 4.1 and Appendices F.2–F.3 of the original paper. It contains 90 scenarios: 30 each in travel, mortgage and software, with pools of 10, 6 and 8 available specialists, respectively. These are the available pool sizes, not a requirement to invoke every specialist for each query. All methods use GPT-4o-mini as the execution backbone and share the scenario context and evaluation protocol. Each domain uses three folds: 20 scenarios support skill learning and the other 10 are held out for evaluation, rotating so that each scenario contributes exactly one test result.
Scoring is performed by a GPT-4o-mini evaluator. It receives the predefined user-facing and system-facing assertions, the final response and the recorded tool actions, and judges whether each requirement is satisfied. A scenario receives an Overall GSR of 1 only if all required assertions pass, and 0 otherwise. The reported value is the mean success rate over the 90 scenarios. This evaluates completion of the specified goals rather than asking the model for an unconstrained overall quality rating.
Table L4. Specialist collaboration: goal-success rate
| Method | Expert collaboration (GSR) |
|---|---|
| AFlow[6] | 0.233 |
| MaAS[7] | 0.300 |
| MAS-Skill | 0.444 |
MAS-Skill satisfies all required assertions in 40 of 90 scenarios, compared with 27 for MaAS and 21 for AFlow. In this setting, the improvement concerns whether a coordinated task succeeds, rather than only a small difference on a general question-answering benchmark. The original Sec. 5.3 and Fig. 4 also report lower mean online token usage for these collaborations, providing a complementary efficiency perspective.
The broader lesson is that collaboration experience can be retained at the team level and made available for conditional reuse. The learning stage provides evaluated alternatives and their execution lessons; the inference stage selects and instantiates an appropriate alternative. We will emphasize this connection, supported by the new comparisons and the specialist results, as the main contribution beyond integrating familiar components.
Thanks again for your careful and valuable comments, hope our response addresses your concerns. If our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating.
References
[1] Jiayu Wang et al. SkillOrchestra: Learning to Route Agents via Skill Transfer. arXiv:2602.19672, 2026.
[2] Hehai Lin, Qi Yang, and Chengwei Qin Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems. Findings of EMNLP 2026, pp. 33685–33718.
[3] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig Agent Workflow Memory. ICML 2025, pp. 63897–63911.
[4] Guibin Zhang et al. G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks. ICML 2025, pp. 76678–76692.
[5] Rui Ye et al. MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems. ICML 2025.
[6] Jiayi Zhang et al. AFlow: Automating Agentic Workflow Generation. ICLR 2025.
[7] Guibin Zhang et al. Multi-agent Architecture Search via Agentic Supernet. ICML 2025.
Response to Reviewer 1o65: Novelty of Reusable Collaboration Skills, Fairness of Offline Search, and Empirical Advantages
Thank you for your careful review and constructive suggestions. Your comments helped us identify where the motivation, comparisons and evaluation details needed a clearer explanation. Below, we clarify the relevant parts of the original paper and present the additional experiments addressing your concerns.
Q1: What is novel about extending skill extraction to multi-agent topologies, given the existing work on skills for multi-agent systems?
A1: Thank you for asking us to clarify the novelty relative to prior work on multi-agent skills and topology design. We agree that the contribution should be explained more precisely than simply extending skills from one agent to multiple agents. Our contribution is to learn complete collaboration protocols as reusable, query-selectable resources for the main agent. The motivation follows the two paradigms discussed in Sec. 1 and Fig. 1 of the original paper.
Rule-based systems prescribe roles and communication patterns in advance. They incur little topology-selection overhead, but the same pattern may not suit queries with different dependencies or difficulty. Search-based systems can discover better configurations, but evaluating candidates requires computation during design or inference. Offline search avoids repeatedly optimizing every test query, yet still leaves the question of how the resulting experience should be organized and used for different later queries. This is the problem addressed by MAS-Skill: retaining a compact set of effective collaborations together with evidence for choosing among them.
A MAS-Skill record links execution, experience and applicability. The configuration in Eq. 1 specifies topology, agent count, agent definitions and synthesis. Equation 2 adds suitable-query descriptions, successful and failed examples, distilled lessons and a quality score. These fields connect learning to use: Pareto selection preserves empirically useful accuracy–cost choices; genetic fusion changes structures and synthesis using execution experience; retrieval and gating then determine which record should serve the current query (Secs. 3.2–3.3). The accepted record directly instantiates the collaboration, while a weak match triggers one-pass planning.
This also distinguishes MAS-Skill from recent skill-oriented work. SkillOrchestra[1] supports agent routing, and Skill-MAS[2] evolves orchestration principles. MAS-Skill instead makes the reusable object a complete executable team protocol, with record-specific evidence about when to use or reject it. The advantage is a direct connection between an evaluated collaboration and its subsequent execution: the main agent can reuse the topology, roles, tools and synthesis together, rather than reconstructing all of them from general guidance.
Following your suggestion, we added AWM[3], G-Designer[4], MAS-GPT[5], SkillOrchestra[1] and Skill-MAS[2] on the six main benchmarks. The evaluations use GPT-4o-mini for execution and share test manifests and scorers. MAS-GPT retains its released generator; AWM is adapted from web workflows. Baselines appear in order of first public release. Scores are accuracy (%) or code pass@1 (%), and Avg. is the unweighted mean.
Table L1. Comparison with additional related methods (%)
MAS-Skill achieves the strongest average result, including a 5.99-point advantage over Skill-MAS and improvements over it on all six benchmarks. The results support the usefulness of retrieving complete collaboration protocols, especially for code generation and multi-hop question answering. A2/A5 further examines whether the benefit can be explained by greater offline search. We will revise the contribution statement around this concrete representation, learning-to-execution connection and supporting evidence, rather than claiming that multi-agent skills themselves are new.