acceptodds
Under review as a conference paper at ICLR 2027

MAS-Skill: Skill-Guided Multi-Agent Systems via Collaborative Topology Transfer

Abstract

Large language model (LLM)-based multi-agent systems (MAS) benefit from structured collaboration, yet existing approaches either use a fixed topology for all queries or perform costly per-query topology search without reusing successful interaction patterns. We propose MAS-Skill, a framework that models multi-agent collaboration as a reusable skill. During offline learning, MAS-Skill conducts controlled A/B tests over topology and agent-count configurations, selects accuracy-cost Pareto-optimal configurations as base skills, and evolves them through genetic recombination and mutation guided by successful and failed trajectories. Each skill records a collaboration topology, agent roles, tool assignments, synthesis strategy, applicability conditions, and execution experience. At inference time, a main agent retrieves a type-aware, high-confidence skill and instantiates its collaboration plan; when no reliable match exists, it constructs a new plan autonomously, thereby avoiding per-query topology search. Across six benchmarks covering mathematical reasoning, code generation, and multi-hop question answering, MAS-Skill achieves 92.8% on GSM8K, 52.9% on MATH, 99.0% on MultiArith, 84.2% on MBPP, 94.7% on HumanEval, and 87.5% on HotpotQA, yielding an average score of 85.17% and outperforming the strongest compared baseline by 1.58 percentage points. On a multi-agent collaboration benchmark, MAS-Skill achieves an average Overall Goal Success Rate of 0.444 while using fewer online inference tokens than AFlow and MaAS. These results demonstrate that reusable collaboration skills provide an effective alternative to repeated runtime topology search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1o65OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the MAS-Skill paradigm, which extends skill extraction from extracting skills of individual agents to extracting skills from the collaboration of multiple agents. It selects better-performing configurations from 15 basic configurations, packages them as base skills, and then combines and transforms these stored base skills through genetic algorithms and other mechanisms to form new skills.

Critical strengths
  1. The experiments are relatively complete, and ablation studies are also included.

  2. The problem itself is important and practical: how to construct collaboration topologies is indeed a problem.

  3. Cross-model experiments are conducted, demonstrating generalizability.

  4. The paper considers not only accuracy, but also token consumption.

Critical weaknesses
  1. The claimed contribution of extending skill extraction from single agents to multi-agent topologies has limited novelty, since there have already been a large number of works on skills for multi-agent topologies.

  2. The paper evaluates 15 base configurations, selects better-performing configurations and summarizes them into skills, and then updates them through methods such as genetic algorithms. This seems more like searching for the best skills first and then searching for better solutions within a neighborhood around their topologies, which may make the comparison with other algorithms unfair. The performance improvement may also come from more extensive offline search rather than from the skill itself.

  3. The performance improvement seems relatively small. The main advantage of the paper is also unclear, as the method seems to mainly combine existing techniques according to its own pipeline. It is not clear what new insights MAS-Skill provides beyond this integration.

Questions for authors
  1. The paper claims that its contribution is to extend skill extraction from single agents to multi-agent topologies? However, there have already been a large number of works on skills for multi-agent topologies.

  2. The paper uses 15 base configurations to calculate accuracy, selects better-performing configurations and summarizes them into skills, and then updates them through methods such as genetic algorithms. I feel that this is more like searching for the best skills first, and then searching for the optimal solution within a neighborhood around their topologies. Is this unfair when comparing with other algorithms? Is it possible that the performance improvement comes from more extensive offline search rather than from the skill itself?

  3. The performance improvement seems relatively small. I do not really understand what the biggest advantage of this paper is. It seems to simply combine existing methods according to its own pipeline. Could the authors explain the advantages in more detail? What new insights can MAS-Skill bring?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1o65: Novelty of Reusable Collaboration Skills, Fairness of Offline Search, and Empirical Advantages

Thank you for your careful review and constructive suggestions. Your comments helped us identify where the motivation, comparisons and evaluation details needed a clearer explanation. Below, we clarify the relevant parts of the original paper and present the additional experiments addressing your concerns.

Q1: What is novel about extending skill extraction to multi-agent topologies, given the existing work on skills for multi-agent systems?

A1: Thank you for asking us to clarify the novelty relative to prior work on multi-agent skills and topology design. We agree that the contribution should be explained more precisely than simply extending skills from one agent to multiple agents. Our contribution is to learn complete collaboration protocols as reusable, query-selectable resources for the main agent. The motivation follows the two paradigms discussed in Sec. 1 and Fig. 1 of the original paper.

Rule-based systems prescribe roles and communication patterns in advance. They incur little topology-selection overhead, but the same pattern may not suit queries with different dependencies or difficulty. Search-based systems can discover better configurations, but evaluating candidates requires computation during design or inference. Offline search avoids repeatedly optimizing every test query, yet still leaves the question of how the resulting experience should be organized and used for different later queries. This is the problem addressed by MAS-Skill: retaining a compact set of effective collaborations together with evidence for choosing among them.

A MAS-Skill record links execution, experience and applicability. The configuration in Eq. 1 specifies topology, agent count, agent definitions and synthesis. Equation 2 adds suitable-query descriptions, successful and failed examples, distilled lessons and a quality score. These fields connect learning to use: Pareto selection preserves empirically useful accuracy–cost choices; genetic fusion changes structures and synthesis using execution experience; retrieval and gating then determine which record should serve the current query (Secs. 3.2–3.3). The accepted record directly instantiates the collaboration, while a weak match triggers one-pass planning.

This also distinguishes MAS-Skill from recent skill-oriented work. SkillOrchestra[1] supports agent routing, and Skill-MAS[2] evolves orchestration principles. MAS-Skill instead makes the reusable object a complete executable team protocol, with record-specific evidence about when to use or reject it. The advantage is a direct connection between an evaluated collaboration and its subsequent execution: the main agent can reuse the topology, roles, tools and synthesis together, rather than reconstructing all of them from general guidance.

Following your suggestion, we added AWM[3], G-Designer[4], MAS-GPT[5], SkillOrchestra[1] and Skill-MAS[2] on the six main benchmarks. The evaluations use GPT-4o-mini for execution and share test manifests and scorers. MAS-GPT retains its released generator; AWM is adapted from web workflows. Baselines appear in order of first public release. Scores are accuracy (%) or code pass@1 (%), and Avg. is the unweighted mean.

Table L1. Comparison with additional related methods (%)

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQAAvg.
AWM[3]86.9247.7498.3370.3887.7971.7077.14
G-Designer[4]92.3041.3699.0068.0483.9768.8875.59
MAS-GPT[5]86.2649.1890.6768.3390.0866.8075.22
SkillOrchestra[1]93.4653.2997.6772.1491.6077.9081.01
Skill-MAS[2]90.2550.5698.3371.3589.7574.8479.18
MAS-Skill92.8052.8899.0084.2094.6687.5085.17

MAS-Skill achieves the strongest average result, including a 5.99-point advantage over Skill-MAS and improvements over it on all six benchmarks. The results support the usefulness of retrieving complete collaboration protocols, especially for code generation and multi-hop question answering. A2/A5 further examines whether the benefit can be explained by greater offline search. We will revise the contribution statement around this concrete representation, learning-to-execution connection and supporting evidence, rather than claiming that multi-agent skills themselves are new.

Q2: Are comparisons fair with 15 configurations and genetic search? Do gains reflect extra offline search or skills?

A2: Thank you for raising the possibility that the improvement is due to greater offline search. We address this in two steps: first account for the preparation investment, then compare ways of using exactly the same learned library. This distinguishes the cost of acquiring collaboration knowledge from the contribution of selecting it at inference.

The 15 initial configurations are the combinations of five topologies and three agent counts (Sec. 3.2, Eq. 3). They provide empirically evaluated starting points rather than defining the complete search space. Fusion can change role assignments, add or remove agents, rewire dependencies and revise synthesis instructions using successful and failed trajectories. Offspring are selected on development data (Eq. 8), and the resulting records are retained for later queries. We acknowledge that this process requires computation.

Following your suggestion, we report offline and online token usage for AFlow[6], MaAS[7] and MAS-Skill on all six benchmarks. Offline usage includes exploration, experience extraction, fusion and validation; online usage includes planning, gating, execution and synthesis. Values are total input plus output tokens in millions, with the same evaluation set across methods within each benchmark.

Table L2a. Offline construction cost (million tokens)

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQA
AFlow[6]52.2652.8842.062.702.483.29
MaAS[7]36.3722.4527.161.881.821.96
MAS-Skill40.9056.6844.627.543.992.39

MAS-Skill has a substantial preparation cost, exceeding MaAS on all benchmarks and AFlow on four. This makes the investment explicit: the method spends computation to obtain a library of useful collaborations and the evidence needed to reuse them.

Table L2b. Online evaluation cost (million tokens)

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQA
AFlow[6]6.374.665.102.691.661.33
MaAS[7]9.257.235.372.331.501.26
MAS-Skill5.093.974.702.281.351.26

The learned library is then used at a lower online cost than AFlow on all benchmarks and at a lower or equal cost than MaAS. Together with the accuracy results in the original Table 1 and Table L1 above, this shows the intended benefit of the preparation stage: stronger task performance and more economical inference after learning. The full deployment cost should still include the initial investment, amortized over subsequent queries.

Cost accounting alone does not isolate the skill-selection contribution. We therefore also compare query-dependent retrieval with the best development-selected skill per type from the same base-plus-fused library. Both retain complete executable records and share models, tools and generation limits; both HotpotQA conditions use provided subtypes.

Table L3. Selection with the learned library held fixed

Accuracy / pass@1 (%):

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQAAvg.
Best fixed skill/type93.0851.6598.5072.4388.5554.5076.45
MAS-Skill93.1853.0999.0073.0288.5554.7576.93

Query-dependent selection improves five point estimates and ties on HumanEval. Because even the fixed comparator benefits from the learned library, this comparison focuses on how the knowledge is used rather than whether it was acquired.

Mean online tokens per query:

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQA
Best fixed skill/type13,33013,07611,0978,74410,9289,729
MAS-Skill13,15710,9609,4045,9808,7989,645

Retrieval reduces usage across all six benchmarks while maintaining or improving performance. This difference cannot be attributed to additional offline search, since the library is identical. The two experiments therefore answer different parts of your concern: learning the library has a real preparation cost, and selecting an appropriate learned collaboration provides an additional execution benefit. We will present these results together so that the efficiency and mechanism claims are supported by the relevant controls.

Q3: Given the relatively small performance gains and the combination of existing techniques, what are the main advantages and new insights of MAS-Skill beyond integrating these components?

A3: Thank you for asking us to explain the broader insight and the practical importance of the gains. The central idea is that a main agent can reuse knowledge of how a team should collaborate, rather than reconstructing that knowledge independently for every query. The useful resource is not only an answer pattern or a role prompt, but a complete protocol connecting responsibilities, communication, tools and synthesis with evidence about its applicability.

The added experiments help make this insight concrete. Table L1 compares the approach with recent workflow, topology and skill-based methods, including Skill-MAS[2], and shows the strongest average performance. Table L3 holds the learned library fixed and shows that query-dependent use improves the accuracy–cost trade-off. Tables L2a–L2b account for the preparation required to obtain those records and the lower online usage once they are available. These results connect the proposed representation to measurable consequences, rather than relying on the novelty of the individual search or retrieval components.

The specialist-collaboration experiment follows Sec. 4.1 and Appendices F.2–F.3 of the original paper. It contains 90 scenarios: 30 each in travel, mortgage and software, with pools of 10, 6 and 8 available specialists, respectively. These are the available pool sizes, not a requirement to invoke every specialist for each query. All methods use GPT-4o-mini as the execution backbone and share the scenario context and evaluation protocol. Each domain uses three folds: 20 scenarios support skill learning and the other 10 are held out for evaluation, rotating so that each scenario contributes exactly one test result.

Scoring is performed by a GPT-4o-mini evaluator. It receives the predefined user-facing and system-facing assertions, the final response and the recorded tool actions, and judges whether each requirement is satisfied. A scenario receives an Overall GSR of 1 only if all required assertions pass, and 0 otherwise. The reported value is the mean success rate over the 90 scenarios. This evaluates completion of the specified goals rather than asking the model for an unconstrained overall quality rating.

Table L4. Specialist collaboration: goal-success rate

MethodExpert collaboration (GSR)
AFlow[6]0.233
MaAS[7]0.300
MAS-Skill0.444

MAS-Skill satisfies all required assertions in 40 of 90 scenarios, compared with 27 for MaAS and 21 for AFlow. In this setting, the improvement concerns whether a coordinated task succeeds, rather than only a small difference on a general question-answering benchmark. The original Sec. 5.3 and Fig. 4 also report lower mean online token usage for these collaborations, providing a complementary efficiency perspective.

The broader lesson is that collaboration experience can be retained at the team level and made available for conditional reuse. The learning stage provides evaluated alternatives and their execution lessons; the inference stage selects and instantiates an appropriate alternative. We will emphasize this connection, supported by the new comparisons and the specialist results, as the main contribution beyond integrating familiar components.

Thanks again for your careful and valuable comments, hope our response addresses your concerns. If our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating.

References

[1] Jiayu Wang et al. SkillOrchestra: Learning to Route Agents via Skill Transfer. arXiv:2602.19672, 2026.

[2] Hehai Lin, Qi Yang, and Chengwei Qin Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems. Findings of EMNLP 2026, pp. 33685–33718.

[3] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig Agent Workflow Memory. ICML 2025, pp. 63897–63911.

[4] Guibin Zhang et al. G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks. ICML 2025, pp. 76678–76692.

[5] Rui Ye et al. MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems. ICML 2025.

[6] Jiayi Zhang et al. AFlow: Automating Agentic Workflow Generation. ICLR 2025.

[7] Guibin Zhang et al. Multi-agent Architecture Search via Agentic Supernet. ICML 2025.

Reviewer_FsjyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MAS-Skill represents a reusable multi-agent collaboration skill as an executable configuration comprising a topology, agent roles, tool bindings, and synthesis instructions, together with applicability information and execution experience. It constructs base skills through matched configuration comparisons and accuracy–cost selection, and further develops fused skills through genetic optimization. At inference time, the system retrieves a skill using query similarity, negative examples, and quality information, checks applicability, and falls back to collaboration planning when necessary.

The paper evaluates general reasoning and code-generation tasks, specialist collaboration, component ablations, and transfer across models and datasets. It reports a macro-average score of 85.17% across six general benchmarks, versus 83.59% for MaAS, and a specialist-collaboration goal success rate of 0.444, versus 0.300 for MaAS. The general-benchmark comparison includes previously published baseline results, whereas the specialist comparison uses a shared evaluation protocol.

Critical strengths
  1. A concrete and reusable representation of collaboration. The skill includes executable structure, role and tool assignments, synthesis instructions, and applicability evidence. The combination of configuration comparison, accuracy–cost selection, and validated fusion forms a reasonably complete system. The appendix also clarifies that the configuration comparisons evaluate complete configurations rather than identify a causal effect of topology alone.

  2. Useful empirical coverage and relevant ablations. The specialist benchmark reports an improvement from 0.300 to 0.444 in goal success rate over MaAS while using approximately 5.5% fewer online tokens. Tables 3 and 9 already examine removing the skill pool, genetic fusion, and configuration comparison. For example, removing fusion decreases performance by 0.67 percentage points on GSM8K and 2.29 points on HumanEval. The authors also explicitly acknowledge limitations of heterogeneous-model transfer and exploratory topology judgments.

Critical weaknesses
  1. The distinctive value of the skill mechanism is insufficiently isolated. The representation of a complete transferable collaboration protocol is a meaningful distinction from capability-routing approaches such as the cited SkillOrchestra. However, much of the pipeline combines established ideas: offline workflow search, accuracy–cost selection, experience reuse, evolutionary optimization, and similarity-based retrieval. The paper needs stronger evidence that its particular skill representation and query-dependent selection add substantial value.

    The existing component ablations are useful, but do not compare retrieval with a fixed development-selected skill per task type from the same library, under comparable execution budgets. This would help distinguish the value of query-specific selection from that of constructing and reusing a good workflow. Likewise, Appendix H.2 defines the out-of-domain “without skill” baseline as direct inference, so those gains do not isolate skill transfer from additional collaboration and tool use. In cross-model transfer, supporting roles retain GPT-4o-mini, as the authors explicitly acknowledge; a comparison with the same model pool and tools but without learned skill selection is still needed. The current evidence therefore supports the integrated system more strongly than the independent contribution of its central skill mechanism.

  2. Evaluation details and numerical inconsistencies limit confidence in the reported comparisons. The HumanEval skill in Appendix G automatically tests sub-agent code and returns failure feedback for revision. The paper should identify the source of these internal assertions and explain their separation from held-out scoring tests. Appendix F.3 states that BigCodeBench public instructions are separate from executable gold tests, but this does not fully specify which assertions the internal verifier can access. Comparable test feedback and revision allowances for baselines also need clarification.

    There are concrete reporting issues as well. Table 3 reports a GSM8K cost of 4.53 for the full method, while Table 9 reports 3.31; the other ablation entries agree. Relative to the no-skill-pool cost of 3.52, this changes whether the full method increases or decreases cost. In addition, the reported MultiArith accuracy of 99.00% is not an integer success fraction of the stated 180 evaluation items, and the BigCodeBench baseline of 46.3% is not obtainable by rounding an integer success fraction of 150 items to one decimal place. These values may reflect aggregation or rounding conventions that are not documented. Raw success counts, run counts, and aggregation rules are needed to reconcile them. Finally, because several general-benchmark baselines are inherited from published results, their small score differences should be interpreted with appropriate caution rather than as fully controlled comparisons.

Questions for authors
  1. With the same skill library, model pool, tools, and comparable token budgets, how does query-dependent retrieval compare with always using the best development-selected skill for each task type? Please also report how often retrieval is accepted versus falling back to planning, so that the source of the gains can be assessed.

  2. Which test assertions are available to the internal code verifier in Appendix G: prompt-visible examples, independently generated tests, or benchmark scoring tests? Please specify their separation from final evaluation tests and whether compared systems receive equivalent testing and revision access.

  3. Please reconcile the full-system GSM8K costs of 4.53 and 3.31 in Tables 3 and 9, and provide success counts and aggregation rules explaining the MultiArith and BigCodeBench percentages relative to the evaluation sizes in Table 6.

10 comments
AuthorsOpenReview ↗

Response to Reviewer Fsjy: Isolating the Skill Mechanism’s Contribution, Ensuring Evaluation Fairness, and Clarifying Test Access and Reported Results

Thank you for your careful review and constructive suggestions. Your comments helped us identify where the motivation, comparisons and evaluation details needed a clearer explanation. Below, we clarify the relevant parts of the original paper and present the additional experiments addressing your concerns.

Q1: What evidence isolates the value of the skill representation and query-dependent selection beyond established workflow search, experience reuse and retrieval methods?

A1: Thank you for recognizing the value of a transferable collaboration protocol and for asking us to establish what the skill mechanism adds. We agree that the contribution needs to be supported at two levels: why a collaboration record is a useful representation, and whether selecting and reusing such records improves actual execution.

The motivation is to make collaboration experience available to the main agent as an executable resource. As explained in Sec. 1 of the original paper, fixed rules provide inexpensive but relatively rigid coordination, while search offers flexibility at the cost of evaluating candidate configurations during preparation or inference. Merely finding a good configuration does not specify how its successful and failed executions should inform a later query. MAS-Skill connects these stages: it extracts empirically supported configurations, fuses their structures and experience, and retains both the resulting protocol and evidence of its applicability.

This connection is explicit in Eq. 2. The executable part specifies the team, tools, dependencies and synthesis procedure; the accompanying fields describe suitable queries, successful and failed cases, and distilled lessons. The latter are used by retrieval and acceptance, rather than serving only as documentation. Compared with SkillOrchestra[1], which informs agent selection, MAS-Skill supplies a complete collaboration. Compared with Skill-MAS[2], which evolves orchestration guidance, it retrieves a previously evaluated executable record. The practical advantage is that a reliable match already specifies how the team should work, while an unsuitable match can be rejected before execution.

Following your suggestion, we added AWM[3], G-Designer[4], MAS-GPT[5], SkillOrchestra[1] and Skill-MAS[2] to the six main benchmarks. They cover workflow reuse, topology generation, system generation and skill-based orchestration. Execution uses GPT-4o-mini with common test manifests and scorers; MAS-GPT retains its released generator, and AWM is adapted from web workflows. Baselines are ordered by first public release. The scores are accuracy (%) or code pass@1 (%), with an unweighted average.

Table F1. Comparison with additional related methods (%)

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQAAvg.
AWM[3]86.9247.7498.3370.3887.7971.7077.14
G-Designer[4]92.3041.3699.0068.0483.9768.8875.59
MAS-GPT[5]86.2649.1890.6768.3390.0866.8075.22
SkillOrchestra[1]93.4653.2997.6772.1491.6077.9081.01
Skill-MAS[2]90.2550.5698.3371.3589.7574.8479.18
MAS-Skill92.8052.8899.0084.2094.6687.5085.17

MAS-Skill achieves the highest average performance and exceeds Skill-MAS on all six benchmarks. This provides evidence that the proposed way of retaining and using collaboration knowledge is competitive with recent alternatives, rather than comparing only with older fixed workflows.

To test whether the improvement merely comes from running multiple agents, we also retain the execution models and tools but remove the skill library. The no-skill system constructs its collaboration through one-shot planning, rather than using single-agent direct inference.

Table F2. Learned skills versus one-shot multi-agent planning (%)

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQAAvg.
Without skills91.2848.9789.0873.3186.2677.9077.80
MAS-Skill92.8053.0998.5084.2094.6687.5085.13

The skill condition improves every benchmark, with a 7.33-point average gain. The benefit therefore remains when the alternative can still organize and execute a multi-agent team. This supports the value of retaining learned collaboration knowledge. A2 then uses a shared-library control to separate query-dependent selection from simply possessing a useful workflow, addressing the second part of your concern directly. We will organize the revised discussion around this sequence of evidence.

Q2: How does retrieval compare with the best development-selected skill from the same library under matched budgets? Do out-of-domain gains persist with collaboration and tools matched, and do cross-model gains persist with the same model pool, including GPT-4o-mini support, but without learned skill selection?

A2: Thank you for laying out the controls needed to distinguish the skill mechanism from additional computation and assistance. Following your suggestions, we added three comparisons, each retaining the resources relevant to the alternative explanation you identified.

First, we hold the learned library fixed and change only how a skill is selected. The fixed baseline chooses the best skill per type on development data, retaining all roles, tools, instructions and experience. Both conditions use the same model pool, tools and generation limits. HotpotQA uses provided subtypes in both conditions.

Table F3. Retrieval versus the best fixed skill per type

Accuracy / pass@1 (%):

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQAAvg.
Best fixed skill/type93.0851.6598.5072.4388.5554.5076.45
MAS-Skill93.1853.0999.0073.0288.5554.7576.93

Retrieval has higher point estimates on five benchmarks and ties on HumanEval. The fixed comparator already contains strong learned workflows, so the modest remaining accuracy difference is specifically informative about selection.

Mean online tokens per query:

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQA
Best fixed skill/type13,33013,07611,0978,74410,9289,729
MAS-Skill13,15710,9609,4045,9808,7989,645

The retrieval condition reduces usage on every benchmark while maintaining or improving performance. This demonstrates an execution benefit after the library is already available. Since its construction is shared, the difference cannot be attributed to extra offline search. Actual token use is reported rather than forced to equal the common generation limit.

Second, we separate backbone transfer from GPT-4o-mini assistance on MBPP. In the homogeneous setting, the named backbone handles every online call, including planning and gating. In the heterogeneous setting, supporting roles may use GPT-4o-mini. Each has a no-skill multi-agent planning control with access to the same model pool and tools.

Table F4a. Homogeneous execution on MBPP (%)

MethodQwen2.5-3BQwen3-8BMistral-7BLlama-3.1-8BAvg.
Without skills55.1368.6231.0955.1352.49
MAS-Skill54.8466.8637.8356.8954.11

Mistral and Llama benefit from transferred skills without GPT-4o-mini assistance, whereas the Qwen models do not improve. This shows both that transfer can work without the stronger helper and that its effectiveness depends on how well the execution backbone follows the transferred protocol.

Table F4b. Heterogeneous execution on MBPP (%)

MethodQwen2.5-3BQwen3-8BMistral-7BLlama-3.1-8BAvg.
Without skills54.8468.3330.7954.8452.20
MAS-Skill68.3373.0254.8468.3366.13

With the same supporting model pool available to both conditions, skill-guided execution improves all four backbones. The gains therefore cannot be explained simply by giving only the skill condition access to GPT-4o-mini. This evaluates the complete skill-guided execution strategy, including its role assignments; the available pool is matched, while realized model calls can differ.

Third, we replace the original direct-inference transfer baseline with a matched multi-agent control. The original Figure 7 tests zero-shot use of frozen source-domain skills, but its direct-inference baseline does not isolate the effect of extra collaboration. The new comparison freezes the library and matches the models, tool access, generation limits and team size for each question.

Table F5. Out-of-domain transfer with matched team sizes (%)

MethodDAPO-MathBigCodeBenchPopQAAvg.
Without skills29.3346.0025.3333.55
MAS-Skill30.6749.3324.6734.89

Skills improve the point estimates on DAPO-Math and BigCodeBench, while PopQA remains close. The paired intervals include zero, so this is evidence of feasible cross-domain reuse rather than a resolved gain on every target. Taken together, these controls answer the three confounds separately: selection can help with the library fixed, transfer is not explained solely by helper-model access, and the out-of-domain comparison now retains multi-agent execution in both conditions.

Reviewer_gEPhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

MAS-Skill learns multi-agent collaboration configurations as reusable skills during offline skill learning and retrieves them at inference time. During offline skill learning, the method evaluates five topologies under three different agent-count settings, retains at most two base skills for each task type, and generates additional fused skills using a genetic algorithm. Each skill also stores information about when it should be applied and under what conditions it may fail, enabling more informed skill selection at inference time. During inference, the method selects skills using TF-IDF retrieval and a confidence gate. When no suitable skill is found, it falls back to a one-pass planner.

Critical strengths

Storing skill applicability conditions and failure cases, and retrieving relevant skills using TF-IDF, is a practical idea. The method is evaluated on diverse benchmarks, including transfer experiments.

Critical weaknesses
  1. The motivation and problem formulation are not sufficiently justified.

-The paper argues that rule-based methods lack adaptability, search-based methods incur high inference-time costs, and methods that optimize global workflows require extensive design-time search. Among these paradigms, design-time workflow optimization is the most directly relevant comparison. However, the motivation is framed as if the proposed method addresses the limitations of all three paradigms, which seems somewhat unnatural.

-The proposed method itself also relies on design-time optimization, yet an analysis of its design-time computational cost appears to be missing.

-MaAS, one of the baselines, also optimizes an agentic supernet at design time and samples query-dependent architectures at inference time. Therefore, the distinction between MAS-Skill and MaAS is not sufficiently clear from the motivation.

-Prior studies that learn to construct query-specific multi-agent systems through offline training, or retrieve workflows from past trajectories, are largely overlooked. A clearer discussion and comparative analysis of related methods, such as G-Designer, MAS-GPT, and Agent Workflow Memory[1,2,3], would be necessary.

  1. The contribution of query-specific skill retrieval is unclear.

-The method retains at most two base skills per task type, in addition to generating fused skills. Given this relatively small number of base skills, it is unclear how much benefit query-specific skill retrieval provides compared to simply using a fixed skill for each task type.

-The paper states that task-type metadata is used when available, while a coarse task type is predicted otherwise. However, it is unclear which setting is used for each benchmark. If ground-truth task-type labels are provided only to MAS-Skill, this may introduce additional information unavailable to the baselines, and the contribution of this information is unclear.

-Although the paper reports aggregate inference-time costs and component ablations, these experiments do not isolate the efficiency benefits of query-specific skill retrieval and the confidence gate compared to fixed per-task-type configurations. In addition, a comparison of design-time computational costs is missing.

Relevant References

[1] MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems (Ye et al., ICML 2025)

[2] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks (Zhang et al., ICML 2025)

[3] Agent Workflow Memory (Wang et al., ICML 2025)

Questions for authors

1Were ground-truth task-type labels used during inference on HotpotQA and MATH? If so, how does performance change when the model must predict the task type instead of using the ground-truth label?

2.Does the proposed method outperform a baseline that uses a single best-performing configuration for each task type? This comparison would help clarify the contribution of query-specific skill retrieval.

3.What is the computational cost of offline skill learning during the design phase? Please also provide a comparison with MaAS in terms of design-time computational cost.

10 comments
AuthorsOpenReview ↗

Response to Reviewer gEPh: Distinctions from Prior Work, Benefits of Query-Specific Skill Retrieval, and Offline Learning Costs

Thank you for your careful review and constructive suggestions. Your comments helped us identify where the motivation, comparisons and evaluation details needed a clearer explanation. Below, we clarify the relevant parts of the original paper and present the additional experiments addressing your concerns.

Q1: Why is the motivation framed around limitations of rule-based methods, inference-time search and design-time workflow optimization, when design-time optimization is the closest comparison?

A1: Thank you for pointing out that the motivation was not sufficiently clear. We would first like to clarify that the original paper distinguishes two existing paradigms, rule-based and search-based construction; design-time workflow optimization is a setting within the second paradigm, not a third independent category (Sec. 1 and Fig. 1). We agree that offline workflow optimization is the closest comparison and will make that relationship clearer.

The underlying problem is that the collaboration best suited to one query may be inefficient or ineffective for another. As discussed in Sec. 1 of the original paper, a problem that can be decomposed into dependent subproblems may benefit from a specialist chain, whereas one that requires exploring alternative solutions may benefit from parallel brainstorming. Rule-based systems make collaboration inexpensive to select, but a prescribed topology and role assignment cannot easily accommodate this variation. Search-based systems offer greater flexibility by evaluating candidate structures. When performed for individual queries, this evaluation adds online overhead; when performed offline, the resulting workflow or architecture distribution still requires substantial preparation.

MAS-Skill addresses how the experience obtained during construction can be retained and reused for an appropriate later query. It keeps a compact library of complete collaborations rather than requiring every query to use one fixed configuration. Each record contains both the executable configuration and evidence about when it works or fails (Sec. 3.1, Eq. 2). Offline Pareto selection preserves useful accuracy–cost trade-offs, and fusion combines structures with lessons from their execution trajectories (Sec. 3.2). At inference, the main agent retrieves and instantiates a suitable record; only an unreliable match triggers one-pass planning (Sec. 3.3).

This explains the connection between the two limitations and our design: a library provides alternatives to a fixed template, while direct reuse avoids repeating candidate evaluation for familiar queries. It also makes the learned collaboration available for transfer to another execution backbone. We do not rely on the claim that offline learning is inexpensive.

Q2: What is the computational cost of the design-time optimization used by MAS-Skill?

A2: Thank you for highlighting the missing design-time accounting. MAS-Skill does require offline computation, and our original inference-cost comparison did not capture that investment. Following your suggestion, we separately report offline construction and online evaluation for MAS-Skill, AFlow[1] and MaAS[2] on all six benchmarks. Offline usage includes configuration exploration, experience extraction, fusion and validation, including recorded unsuccessful candidates. Online usage includes planning, applicability checks, agent execution and synthesis. Both tables count input plus output tokens in millions, with the same evaluation set used by each method within a benchmark.

Table G1a. Offline construction cost (million tokens)

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQA
AFlow[1]52.2652.8842.062.702.483.29
MaAS[2]36.3722.4527.161.881.821.96
MAS-Skill40.9056.6844.627.543.992.39

The results show that the skill library requires a substantial initial investment. MAS-Skill spends more offline tokens than MaAS and, on four benchmarks, more than AFlow. This preparation produces reusable collaborations and their execution evidence; it should be considered together with the cost of subsequently using those records.

Table G1b. Online evaluation cost (million tokens)

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQA
AFlow[1]6.374.665.102.691.661.33
MaAS[2]9.257.235.372.331.501.26
MAS-Skill5.093.974.702.281.351.26

After construction, MAS-Skill has lower online usage than AFlow on every benchmark and lower or equal usage than MaAS. Since evaluation sizes are matched within each benchmark, the same conclusion holds for mean usage per query. Across the six evaluation sets, the total online reductions are 14.49% relative to AFlow and 30.77% relative to MaAS. Together with the accuracy evidence in the original Table 1 and the added comparison in A4, these results support the intended trade-off: investing in reusable collaboration knowledge during preparation to improve the accuracy and efficiency of later execution.

Q3: How does MAS-Skill differ from MaAS, which also optimizes an agentic supernet offline and samples query-dependent architectures at inference time?

A3: Thank you for asking for a more precise comparison with MaAS. You are right that MaAS[2] already supports query-dependent architectures. The distinction is what is learned and reused, rather than whether the architecture depends on the query. MaAS learns an agentic supernet and samples a query-dependent system from that distribution. MAS-Skill learns a library of explicit collaboration records and selects a record together with evidence of its applicability.

The distinction can be seen in Eqs. 1–2 of the original paper. A configuration specifies topology, agent count, agent definitions and synthesis instructions. A skill additionally stores suitable-query descriptions, successful and failed examples, distilled experience and a quality score. These fields are used operationally: Eq. 10 combines query–skill similarity with a failed-example penalty and a quality adjustment, and the confidence gate determines whether the selected collaboration should be instantiated. Thus, the experience is not only used during optimization; it remains available when deciding how to handle a later query.

Once accepted, the record specifies the roles, tools and communication structure that the main agent executes. This gives MAS-Skill a direct route from past collaboration experience to a new team configuration, and allows the same record to be tested with another backbone. We will clarify this process in the motivation and related work. The cost comparison in A2 and the fixed-library comparison in A9 then test its practical consequences, rather than treating query dependence or transferability alone as a unique property.

Q4: How does MAS-Skill compare with related methods that learn query-specific multi-agent systems offline or retrieve workflows from past trajectories, such as G-Designer, MAS-GPT and Agent Workflow Memory?

A4: Thank you for pointing us to these closely related methods. We agree that a convincing comparison should consider both how collaboration knowledge is retained and how it is used at inference. G-Designer[3] learns task-aware communication graphs, MAS-GPT[4] learns to generate executable systems, and AWM[5] retains reusable workflows. SkillOrchestra[6] uses learned skill information to select agents. These are relevant alternatives to the reuse mechanism in MAS-Skill, and we have expanded both the discussion and the experiments accordingly.

Skill-MAS[7] is also closely related: it evolves strategy-level instructions for an orchestrator. Our distinction is to store complete executable collaborations, with their own applicability conditions and execution evidence, and select among them for individual queries. In MAS-Skill, a successful retrieval supplies not just advice for generating a team, but the topology, roles, tool assignments and synthesis procedure needed to instantiate one (Sec. 3.1). Its failure evidence also informs selection and rejection (Sec. 3.3). This makes the link between a previously evaluated collaboration and its later use explicit.

Following your suggestion, we added AWM[5], G-Designer[3], MAS-GPT[4], SkillOrchestra[6] and Skill-MAS[7] to the six-benchmark comparison. The execution backbone is GPT-4o-mini, with common evaluation manifests and scorers. MAS-GPT retains its released system generator; AWM is adapted from web workflows to the benchmark tasks. The table reports our task-adapted evaluations, rather than copying results from the methods’ original benchmarks. Baselines are ordered by first public release. Values are accuracy (%) for reasoning and question answering, and pass@1 (%) for code generation; Avg. is their unweighted mean.

Table G2. Comparison with additional related methods (%)

MethodGSM8KMATHMultiArithMBPPHumanEvalHotpotQAAvg.
AWM[5]86.9247.7498.3370.3887.7971.7077.14
G-Designer[3]92.3041.3699.0068.0483.9768.8875.59
MAS-GPT[4]86.2649.1890.6768.3390.0866.8075.22
SkillOrchestra[6]93.4653.2997.6772.1491.6077.9081.01
Skill-MAS[7]90.2550.5698.3371.3589.7574.8479.18
MAS-Skill92.8052.8899.0084.2094.6687.5085.17

MAS-Skill achieves the highest average score, outperforming the strongest added baseline by 4.16 percentage points and Skill-MAS by 5.99 points. The advantage over Skill-MAS holds across all six benchmarks, with particularly clear gains on code generation and multi-hop question answering. This supports the practical value of retrieving executable collaboration knowledge in addition to retaining general orchestration guidance. The fixed-library experiments in A5/A9 provide a complementary control for query-specific selection. We will include these methods and results in the revision so that the contribution is positioned against the most relevant alternatives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.