acceptodds
Under review as a conference paper at ICLR 2027

MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?

Abstract

Multi-agent systems (MAS) offer a scalable path forward for agentic AI, comprising multiple LLM-based agents, each assigned a system prompt and a position within a workflow that governs inter-agent coordination and output aggregation. System prompts thus form a critical and accessible optimization surface: they specify agents' roles and behaviors, enabling system-level improvements without model finetuning. Although prompt optimization has shown substantial potential for single LLMs, extending it to MAS poses distinct challenges, notably an exponentially growing search space. It remains unclear whether, when, and by how much prompt optimization improves MAS performance, and how sensitive such gains are to system configuration. In this work, we systematically study system-prompt optimization across a broad range of MAS setups varying in task, workflow, communication protocol, and team size. We benchmark both natural multi-agent extensions of state-of-the-art single-agent prompt optimizers and recent prompt optimizers designed specifically for MAS. The results reveal its potential to unlock significant gains while exposing open challenges, characterizing when and how much prompt optimization helps across diverse MAS settings.

Primary area:
datasets and benchmarks
Keywords:
Multi-Agent LLM Systems, Agentic AI, Prompt Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sqGROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MAS-PromptBench, a benchmark for evaluating prompt optimization in multi-agent LLM systems. It systematically studies eight optimizers across different tasks, workflow topologies, communication protocols, and team sizes, revealing when prompt optimization is effective and where it faces challenges.

Critical strengths
  1. Comprehensive benchmark. The paper evaluates diverse MAS configurations and prompt optimizers, providing a useful benchmark for future research.
  2. Practical insights. The empirical findings offer useful guidance on when prompt optimization benefits multi-agent systems.
Critical weaknesses
  1. Limited generalizability across LLMs. In Section 4.5, Sequential achieves the largest gain (+8.6) with Qwen3.5-9B, while Independent (+13.6) and Centralized (+12.8) perform best with Llama-3.1-8B-Instruct. These substantially different results raise concerns about the generalizability of the paper’s conclusions across different LLMs.
Questions for authors

Why does free-form communication consistently yield larger prompt optimization gains than structured communication, and does it also lead to better final performance?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer sqGR: Part One

We sincerely thank the reviewer for the support of our work. It is gratifying to know that the reviewer finds the benchmark comprehensive and its empirical findings useful guidance on when prompt optimization benefits MAS.

W1: Limited generalizability across LLMs (Sec. 4.5)

Thank you for the careful reading of our task-model ablation and for raising this important concern.

  • In our experiments, Qwen3.5-9B and Llama-3.1-8B-Instruct lead to generally consistent findings, although the exact gains differ. Under both task models, the gain depends strongly on the topology (it varies across the five topologies by 7.5 points with Qwen3.5-9B and by 13.1 points with Llama-3.1-8B-Instruct), no optimizer is best on every topology, and the two models agree on the best optimizer on Independent, Sequential and Centralized. The most improved topology does change, as the paper reports; our takeaway, that multi-agent systems need topology-aware prompt optimizers, holds under both.
  • Different LLMs often behave very differently, so generalization across all LLMs is generally not achievable for most systems and tasks. For example, a good ordering of few-shot examples for one model is not transferable to another [1], the performance of a prompt format correlates only weakly between models [2], and different instruction templates change both the absolute and the relative performance of LLMs [3]. Model providers also publish model-specific prompting guidance and ask developers to re-tune their prompts when moving to a new model [4, 5].
  • We use Qwen3.5-9B as the main task model and Llama-3.1-8B-Instruct as the second one, because they are two of the most capable open models. Both are widely used open-weight models from different model families, which makes the findings on them meaningful in practice.

[1] Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022, May). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8086-8098).

[2] Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024, May). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations (Vol. 2024, pp. 25055-25083).

[3] Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024). State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12, 933-949.

[4] OpenAI. (n.d.). Using GPT-5.5. OpenAI Developers. Retrieved November 15, 2026, from https://developers.openai.com/api/docs/guides/latest-model/gpt-5.5

[5] Anthropic. (n.d.). Prompting best practices. Claude Docs. Retrieved November 15, 2026, from https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices

AuthorsOpenReview ↗

Reply to Reviewer sqGR: Part Two

Q1:

  • Why does freeform communication yield larger gains than structured communication?
  • Does it lead to better final performance?

Thank you so much for raising this question, since it concerns a key finding of this work.

Freeform communication leaves agent messages open to revision, which gives prompt optimization more room to improve. Under Freeform, the initial system prompts specify no message format, so the optimizer revises both the content and the form of what each agent passes to the next agent. Under Semi-structured and Structured, the initial system prompts already contain the protocol's message format (slots for status, evidence, confidence and next step), so less is left for the optimizer to add.

The optimized prompts show this on BFCL, where the larger gain appears (+12.5 points with Freeform, against +5.2 with Semi-structured and +5.1 with Structured). With the Sequential topology under Freeform, the optimized Analyzer prompt adds an explicit message format that the initial prompt does not contain: "Your output must be a JSON block containing a list of function-call dictionaries in this exact format". The larger gain of Freeform also appears on three additional tasks (see our response to Reviewer 8cHW, W2).

The final performance after optimization is similar across the three protocols, with Freeform slightly higher. The test scores of the optimized prompts are 71.9 with Freeform, against 70.9 with Semi-structured and 71.4 with Structured, averaged over the three tasks, the four multi-agent topologies and MAS-GEPA, MAS-MIPRO and MASPO. This is consistent with our takeaway, which concerns the prompt-optimization gain and not the final performance: Freeform communication gives prompt optimization more room to improve, and the paper does not claim that it yields the best final performance.

Dear Reviewer sqGR,

We sincerely thank you for taking the time to review our work and for your positive score. Your valuable suggestions have greatly contributed to refining our paper, and we sincerely appreciate your constructive feedback. As the ICLR author–reviewer discussion period concludes on Nov 18 (AoE), we would greatly appreciate your feedback on our rebuttal responses to ensure that all your concerns have been adequately addressed. Moreover, we would be happy to engage in further discussions and address any additional questions or suggestions you may have.

Best regards,
Authors

Reviewer_8cHWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MAS-PromptBench to evaluate system-prompt optimization within fixed multi-agent system configurations. The benchmark covers nine existing task datasets, five workflow configurations including a single-agent baseline, and variations in communication protocol and team size. The main experiments compare four optimizers, with additional studies involving four further optimizers and an alternative task model. These factors are investigated through different experimental subsets rather than a complete factorial evaluation. The primary metric is the task-score improvement over the initial prompts. The results show positive, negligible, and negative gains, with optimizer performance and topology rankings depending on the task and task model. The authors further argue that freeform communication offers greater optimization opportunities and that larger teams make prompt optimization more difficult.

Critical strengths
  1. A potentially reusable evaluation framework: The paper brings tasks, workflows, communication protocols, team sizes, and optimizers into a common evaluation framework, addressing the difficulty of comparing MAS prompt-optimization methods across heterogeneous setups. Measuring end-task improvements within fixed system configurations is practically meaningful. If the implementations and configurations are reliably reusable, this resource could reduce the cost of evaluating future methods.
  2. Useful evidence of failures and model dependence: The paper reports negligible and negative gains and explicitly demonstrates changes in topology rankings across task models. These observations help calibrate expectations about prompt optimization and show why an optimizer should be validated within the intended deployment configuration.
Critical weaknesses
  1. The effect of initial prompts is unclear: Initial prompts are generated separately for each topology, but the paper does not report whether the results hold with different starting prompts. Larger gains could partly come from weaker or easier-to-improve prompts. It is therefore unclear how much of the difference comes from the topology itself.
  2. Some main findings depend on the task mix: The protocol and team-size studies mainly use three tasks. Recalculating Tables 4–5 without BFCL removes both the average advantage of freeform communication and the decline in gains as teams grow. This does not indicate a problem with BFCL, but it limits how broadly these findings can be applied.
Questions for authors
  1. Do optimizers start from the same prompts? For the same MAS setup, do the initial prompts change across optimizers or random seeds? In Table 4, why are the freeform Independent HotpotQA baseline scores 39.3 for GEPA and 53.1 for MIPRO?
  2. Do the findings hold across starting prompts and tasks? Have the main findings been checked with different reasonable initial prompts? Please also provide a per-task analysis showing whether the protocol and team-size findings are mainly driven by one task.
  3. How is the budget allocated as teams grow? As team size grows, how is the optimization budget allocated, and how many prompt updates does each agent receive? Are validation calls included? This would help clarify whether larger teams receive too few opportunities for optimization.
7 comments
AuthorsOpenReview ↗

Reply to Reviewer 8cHW: Part One

We thank the reviewer for the thorough and constructive feedback. It is gratifying to know that the reviewer finds our common evaluation framework practically meaningful and the evidence of failures and model dependence useful. We address each point below.

W1: Unclear effect of initial prompts on the gains

Thank you very much for raising this important point, which helps us clarify how the initial prompts are constructed. The larger gains do not come from weaker or easier-to-improve initial prompts:

  • The initial prompts are generated by a powerful LLM, Qwen3.5-122B-A10B, which is stronger than the task model, Qwen3.5-9B. The agents therefore start from prompts written by a stronger model than their own, not from weak prompts. The initial prompts are also generated systematically: the prompts of all topologies come from the same prompt template (the meta prompt), which is filled for each agent with the task domain, the topology description, the available tools and the agent's role. For a given task, the task domain and the tools are the same in every topology, so the initial prompts of two topologies differ only in the topology description and the agents' roles. These two parts have to differ, because an agent's system prompt specifies its role in the workflow, and the roles are different in each topology (e.g., a coordinator and sub-agents in Centralized, and the stages of a chain in Sequential).
  • For the same MAS setup, the initial prompts are the same for every optimizer and every optimizer seed. They are generated once and shared by all eight optimizers, and the three optimizer seeds of an optimizer are three independent optimization runs from the same initial prompts. The gain of each run is computed against the baseline evaluated in the same run.
  • As the reviewer suggested, we additionally ran the topology study with a second set of initial prompts, and the topology finding still holds. We randomly generated a second set of initial prompts from the same prompt template as the original set, and repeated the topology study with the four optimizers on HotpotQA, LiveCodeBench and BFCL. Under both sets, the topologies rank in the same order by average gain: Sequential (+8.6 with the original set and +8.7 with the second set), Centralized (+4.8 and +3.7), Independent (+1.4 and +1.6) and Decentralized (+1.1 and +0.9). The average gain of each topology differs between the two sets by at most 1.1 points. The two sets start from different baselines (for example, 69.2 against 65.3 on average for Centralized), yet give the same pattern of gains. The gain therefore depends on the topology and not on the particular initial prompts. The table below reports the full results with the second set, where each cell gives baseline / optimized (gain ± standard deviation across three optimizer seeds); the results with the original set are the Freeform columns of Table 4 of the paper. We have added these results to the revised paper (Sec. 4.5 and Appendix C.2), and Appendix E.1 now states how the initial prompts are generated and shared.
MethodDatasetIndependentSequentialCentralizedDecentralized
MAS-GEPAHotpotQA47.3 / 56.0 (+8.7 ± 1.5)52.3 / 52.0 (−0.3 ± 0.6)48.0 / 47.7 (−0.3 ± 0.6)57.3 / 56.3 (−1.0 ± 1.0)
LiveCodeBench92.0 / 92.0 (0.0 ± 0.0)94.0 / 96.7 (+2.7 ± 3.1)84.0 / 82.7 (−1.3 ± 1.2)86.7 / 88.7 (+2.0 ± 3.5)
BFCL50.3 / 51.7 (+1.3 ± 1.5)46.0 / 78.3 (+32.3 ± 11.6)67.3 / 65.0 (−2.3 ± 1.5)82.7 / 82.7 (0.0 ± 3.6)
MAS-MIPROHotpotQA47.0 / 47.0 (0.0 ± 0.0)51.7 / 50.7 (−1.0 ± 0.0)45.7 / 46.3 (+0.7 ± 1.2)58.0 / 58.3 (+0.3 ± 2.1)
LiveCodeBench94.7 / 95.3 (+0.7 ± 1.2)90.7 / 90.7 (0.0 ± 0.0)77.3 / 77.3 (0.0 ± 0.0)86.7 / 94.7 (+8.0 ± 2.0)
BFCL50.7 / 50.7 (0.0 ± 0.0)45.0 / 87.3 (+42.3 ± 7.0)67.0 / 83.7 (+16.7 ± 6.8)81.7 / 80.7 (−1.0 ± 1.0)
MAPROHotpotQA54.3 / 54.3 (0.0 ± 0.0)52.3 / 50.7 (−1.7 ± 1.5)45.3 / 45.0 (−0.3 ± 0.6)57.7 / 58.3 (+0.7 ± 1.2)
LiveCodeBench94.0 / 94.0 (0.0 ± 0.0)90.0 / 90.0 (0.0 ± 0.0)82.7 / 82.7 (0.0 ± 0.0)84.0 / 84.0 (0.0 ± 0.0)
BFCL50.3 / 50.3 (0.0 ± 0.0)47.0 / 47.3 (+0.3 ± 4.5)66.3 / 82.0 (+15.7 ± 1.2)83.0 / 87.3 (+4.3 ± 1.2)
MASPOHotpotQA45.7 / 51.0 (+5.3 ± 3.5)51.7 / 53.7 (+2.0 ± 1.0)45.0 / 44.3 (−0.7 ± 0.6)58.3 / 58.0 (−0.3 ± 4.0)
LiveCodeBench94.7 / 96.7 (+2.0 ± 2.0)88.0 / 88.0 (0.0 ± 0.0)88.7 / 88.7 (0.0 ± 0.0)86.0 / 86.0 (0.0 ± 0.0)
BFCL51.3 / 53.0 (+1.7 ± 4.9)46.0 / 73.3 (+27.3 ± 4.2)66.3 / 82.3 (+16.0 ± 6.0)81.7 / 79.3 (−2.3 ± 3.1)
Average Δ+1.6 ± 0.6+8.7 ± 1.3+3.7 ± 0.8+0.9 ± 0.7
AuthorsOpenReview ↗

Reply to Reviewer 8cHW: Part Two

W2: Protocol and team-size findings depend on the task mix (BFCL)

We sincerely thank the reviewer for the careful recalculation of Tables 4–5, which we confirm. To test how broadly the two findings apply, we add new experiments on three additional tasks, one per domain (MATH, SWE-Bench and API-Bank), and the results are consistent with both findings. We ran both studies on them with MAS-GEPA, MAS-MIPRO and MASPO on the four multi-agent topologies.

The two tables below give the per-task analysis on the paper's three tasks and the three additional ones (in bold), for the communication protocols (n=4n=4) and the team sizes (Freeform). Each value is the gain averaged over the three optimizers and the four topologies. The row Average Δ is the paper's average over its three tasks, and Average Δ without BFCL is the reviewer's recalculation. All averages are computed from unrounded values.

DatasetFreeformSemi-structuredStructured
HotpotQA+1.8+2.0+3.2
LiveCodeBench+0.40.0+0.8
BFCL+12.5+5.2+5.1
Average Δ+4.9+2.4+3.0
Average Δ without BFCL+1.1+1.0+2.0
MATH+2.8+1.6+2.3
SWE-Bench+2.5+1.7+1.3
API-Bank+3.1+2.6+2.7
Average Δ of the three additional tasks+2.8+2.0+2.1
Datasetn=4n=4n=8n=8n=10n=10
HotpotQA+1.8+1.0+2.5
LiveCodeBench+0.4+1.3+1.0
BFCL+12.5+5.7+4.2
Average Δ+4.9+2.6+2.6
Average Δ without BFCL+1.1+1.1+1.8
MATH+2.8+3.4+1.6
SWE-Bench+2.5+1.7+1.7
API-Bank+3.1+2.2+3.2
Average Δ of the three additional tasks+2.8+2.4+2.1

From the tables, we make the following observations:

  • On BFCL, both findings of the paper hold: Freeform communication gives prompt optimization more room to improve, and a larger team size increases the challenge of prompt optimization. With Freeform communication, the gain is +12.5 points, more than twice the gain with Semi-structured (+5.2) or Structured (+5.1) communication. As the team grows from n=4n=4 to n=8n=8 and n=10n=10, the gain decreases from +12.5 to +5.7 and +4.2. This is consistent with our analysis in Sec. 4.1: BFCL is a tool-calling task with explicit function names and output formats that the system prompts can directly shape. Prompt optimization therefore yields a large gain on BFCL, and this gain is sensitive to the communication protocol and the team size.
  • On HotpotQA and LiveCodeBench, prompt optimization yields only a small gain under every communication protocol and team size. The gain stays between 0.0 and +3.2 points in every setting, so neither the protocol nor the team size changes it much: averaged over the two tasks, the three protocols differ by at most 1.0 point, and the three team sizes by at most 0.7 points. For HotpotQA, a reasoning task, local prompt improvements are often lost before they reach the final answer, as analyzed in Sec. 4.1. For LiveCodeBench, the initial prompts already reach 86.0–94.0% with Freeform communication and n=4n=4 (Table 4 of the paper), which leaves little room for improvement.
  • Freeform yields the largest gain on each of the three additional tasks (MATH, SWE-Bench and API-Bank). On the three tasks together, the gain is +2.8 with Freeform, against +2.0 with Semi-structured and +2.1 with Structured.
  • The gain decreases with the team size on the three additional tasks together, from +2.8 at n=4n=4 to +2.4 at n=8n=8 and +2.1 at n=10n=10. It is lower at n=10n=10 than at n=4n=4 on MATH and SWE-Bench.

We have added these results to the revised paper (Sec. 4.3, Sec. 4.4 and Appendix C.3). The full results for the three additional tasks are on our project page and in Tables 12 and 13 of the revised paper.

AuthorsOpenReview ↗

Reply to Reviewer 8cHW: Part Three

Q1:

  • Do all optimizers and random seeds start from the same initial prompts?
  • Why do the freeform Independent HotpotQA baselines differ (39.3 for GEPA vs. 53.1 for MIPRO)?

Thank you for the careful reading of Table 4 and for these detailed questions, which we are very glad to answer.

Yes, for the same MAS setup, the initial prompts are the same for every optimizer and every optimizer seed. Please see our response to W1 for the details.

Both values are evaluations of the same initial prompts on the same HotpotQA test items in separate identical evaluation runs: Each optimization run evaluates the initial prompts itself. The gain of each run is computed against the baseline evaluated in the same run, so the difference between the two baselines does not enter the gain of either optimizer. The two results for MAS-GEPA and MAS-MIPRO give different baselines due to:

  • The outputs of the task model vary between runs. The task model is sampled at temperature 0.2 and served by a pool of vLLM servers, which batch each request with the other concurrent requests. The same request is therefore batched differently in each run, and its output can differ even with a fixed decoding seed [1].
  • This configuration amplifies the variation. On HotpotQA, each agent searches Wikipedia in several steps, so a different query in an early step changes the evidence in all later steps. The four agents work independently, and the final answer is chosen by an exact-match majority vote. When the vote is split (e.g., two against two), one changed answer changes the final answer. Across the 12 Freeform configurations (three tasks × four topologies), the baselines of the optimizers differ by 13.8 points only in this one. In the other eleven, they differ by at most 6.0 points, and on HotpotQA by at most 2.7.

To quantify this variation, we re-evaluated the same initial prompts on the same test items five times in this configuration:

Evaluation12345Average ± SD
Baseline EM (%)46.052.043.054.056.050.2 ± 5.5

The five evaluations range from 43.0 to 56.0, a spread of 13.0 points, which is comparable to the 13.8-point difference between 39.3 and 53.1. The difference between the two baselines is therefore consistent with the run-to-run variation of this configuration.

[1] vLLM Team. (n.d.). Frequently asked questions: Can the output of a prompt vary across runs in vLLM? vLLM Documentation. Retrieved November 15, 2026, from https://docs.vllm.ai/en/latest/usage/faq

Q2:

  • Do the findings hold across starting prompts?
  • Do the findings hold across tasks? Provide a per-task analysis: are the protocol and team-size findings driven by one task?

Thank you for these important questions, which we address with additional experiments.

Yes, the topology finding holds with a second set of initial prompts. Please see our response to W1 for the results.

Both findings also hold on tasks other than BFCL. Please see our response to W2 for the per-task analysis and the full results.

Reviewer_oRR7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work provides a benchmark for evaluating system prompt optimizers for MAS. It evaluates across three domains, five workflow topologies, communication protocols, and team sizes over eight prompt optimizers, including four representative optimizers: MAS-GEPA, MAS-MIPRO, MAPRO, and MASPO. The experimental results show that prompt optimization is effective but sensitive to certain configurations. Overall, the analysis gives insight into when prompt optimization works for MAS, such that the tasks need to be explicit and controllable to benefit from prompt optimization approaches.

Critical strengths
  1. It provides a very practical framework for evaluating the gains of prompt optimization to the MAS system, showing that MAS needs topology-aware prompt optimizers.

  2. The findings also reveal that freeform communication is currently the best form of communication protocol among agents. The benchmark can elicit useful information like this.

  3. This work includes good coverage of the MAS prompt optimizer and tasks.

Critical weaknesses
  1. The evaluation focuses exclusively on the instructions. However, it is not rational to exclude the skills or context that are also considered a critical part of the prompt and can be optimized via prompt optimization.

  2. It includes a collection of existing benchmarks rather than introducing new tasks, and there are no new proposals for optimizers as well.

  3. The topology set studied in this work has a limited set of forms. Other forms of multi-agent workflows that are really practical in real-world applications have not been covered.

Questions for authors
  1. Have you considered other forms of topology?

  2. In a decentralized MAS, how do agents communicate with each other?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer oRR7: Part One

We sincerely thank the reviewer for the careful review and constructive feedback. It is rewarding to know that the reviewer finds MAS-PromptBench a very practical framework for evaluating the gains of prompt optimization for MAS, with good coverage of prompt optimizers and tasks. In what follows, we provide our response to the reviewer's comments.

W1: Why optimize only the instructions, excluding skills and context?

Thank you for this question. The reviewer is correct that, in practice, each agent in a MAS receives more than its system prompt: its input may also include task-specific skills, relevant memory, retrieved context and available tools. As discussed in Remark 1 of the paper, this work focuses on the system prompt as a first step. We start with the system prompt because it is the core of the instruction set. It sits at the top of the instruction hierarchy and defines each agent's role and workflow (Remark 1), which determine how the agents in a MAS work together. And our formulation extends naturally to optimizing any subset of these components. Extending MAS-PromptBench to skills and retrieved context is an interesting direction for future work, and we will certainly pursue it.

W2: Contribution without new tasks or new optimizers

We appreciate this comment and are very glad to clarify the contribution of this work. MAS-PromptBench is a benchmark-and-analysis paper: its contribution is a controlled evaluation framework and the findings it produces. Before proposing new optimizers, we believe it is more important to first establish the value and potential of prompt optimization for MAS, and then to design new optimizers based on where this potential lies. Without this understanding, it is unclear whether new optimizers are needed, and for which MAS configurations. We therefore take a step back and ask a fundamental question that is still open:

How much can prompt optimization help in MAS, and how does its effect vary across configurations?

MAS-PromptBench answers this question by measuring the gains of eight representative optimizers across a diverse set of fixed MAS configurations. The result is a capability map of existing MAS prompt optimizers, which serves as a foundation for designing new ones. The map shows where prompt optimization still has substantial room for improvement, and where its sensitivity to the MAS configuration calls for more principled algorithm design.

Moreover, we want to highlight that benchmark-and-analysis studies, which evaluate existing methods on existing tasks under one controlled protocol, are an important and well-established type of contribution in LLM-based MAS research [1–3]. Using existing tasks and optimizers is a deliberate design choice: a difference in gain can then be attributed to the MAS configuration and the optimizer, not to a new task or algorithm.

Concretely, MAS-PromptBench makes two main contributions:

  • A comprehensive benchmark for MAS prompt optimization. MAS-PromptBench integrates six MAS-specific prompt optimizers, which cover almost all existing ones, and MAS extensions of two widely used single-agent optimizers, GEPA and MIPRO. It evaluates them on nine tasks across reasoning, coding and tool calling, under a wide range of MAS configurations: five topologies, three communication protocols, four team sizes and two task models, all built in one framework. A new optimizer can be plugged in and evaluated under all of these configurations with the same protocol.
  • A capability map of existing optimizers, with insights into where prompt optimization has more potential. The gain of existing optimizers depends strongly on the MAS configuration: averaged over the nine tasks and the optimizers, it ranges from +1.0 to +4.8 points across the five topologies, and no optimizer is best on every topology, so MAS need topology-aware prompt optimizers. Prompt optimization also shows more potential on tasks with explicit and verifiable agent-local behaviors, such as coding and tool calling, than on reasoning tasks. It has more room to improve under Freeform communication than under the Semi-structured and Structured protocols, and in smaller teams than in larger ones.

[1] Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., ... & Stoica, I. (2026). Why do multi-agent llm systems fail?. Advances in Neural Information Processing Systems, 38.

[2] Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. D., & Pretorius, A. (2023). Should we be going mad? a look at multi-agent debate strategies for llms. arXiv preprint arXiv:2311.17371.

[3] Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., ... & Liu, X. (2025). Towards a science of scaling agent systems. arXiv preprint arXiv:2512.08296.

AuthorsOpenReview ↗

Reply to Reviewer oRR7: Part Two

W3 & Q1:

  • Limited topology forms; practical real-world workflows not covered
  • Have you considered other forms of topology?

Thank you for this comment and the question on other topologies. The five topologies in MAS-PromptBench are already among the most common agent workflows in practice. Each one is a standard pattern in widely used agent frameworks and design guides:

TopologyNames in the sourcesSources
SingleSingle-agent systemOpenAI [4], Google Cloud [6]
IndependentParallelization (voting); concurrent agentsAnthropic [1], OpenAI Agents SDK [5], Google Cloud [6], Azure [9], AutoGen [10]
SequentialPrompt chaining; sequential pipelineAnthropic [1], OpenAI Agents SDK [5], Google Cloud [6], Google ADK [7], Azure [9], AutoGen [10], CrewAI [11]
CentralizedOrchestrator–workers; manager with agents as tools; supervisorAnthropic [1], Claude Code subagents [3], OpenAI [4, 5], Google Cloud [6], Google AI co-scientist [8], Azure [9]
DecentralizedMulti-agent debate; group chat; swarmGoogle Cloud [6], Azure [9], AutoGen [10]

Four of them (Single, Independent, Centralized and Decentralized) are also among the agent architectures studied by Kim et al. [12].

To answer the reviewer's question, we add two additional widely used real-world topologies to MAS-PromptBench and evaluate them with the four optimizers on all nine tasks: Fan-out/Fan-in, as in Anthropic's multi-agent Research system [2] and Google ADK [7], and Iterative Refinement, as in the evaluator-optimizer workflow [1], Google ADK [7] and Claude Code [13]. Fan-out/Fan-in differs from Centralized in its workers: in Centralized, the coordinator dispatches subtasks to sub-agents with different specialized roles, whereas in Fan-out/Fan-in the coordinator decomposes the task once, three workers with the same role solve the subtasks in parallel, and the coordinator synthesizes their results. In Iterative Refinement, a generator produces a candidate, two critics review it, and a verifier either accepts it or returns it with feedback to the generator, for at most three rounds. The diagrams below show their structure.

Fan-out/Fan-in (one coordinator and three workers)

                         Input task
                             │
                             ▼
                  ┌─────────────────────┐
                  │     Coordinator:    │
                  │  Decompose the task │
                  └──────────┬──────────┘
          ┌──────────────────┼──────────────────┐
          ▼                  ▼                  ▼
   ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
   │  A1: Solve  │    │  A2: Solve  │    │  A3: Solve  │
   │  subtask 1  │    │  subtask 2  │    │  subtask 3  │
   └──────┬──────┘    └──────┬──────┘    └──────┬──────┘
          └──────────────────┼──────────────────┘
                             ▼
                  ┌─────────────────────┐
                  │     Coordinator:    │
                  │  Synthesize results │
                  └──────────┬──────────┘
                             ▼
                        Final answer

Iterative Refinement (one generator G, two critics C1 and C2, and one verifier V; at most three rounds)

                        Input task
                             │
                             ▼
                  ┌─────────────────────┐
                  │  G: Generate/revise │◄────────────────────┐
                  │    the candidate    │                     │
                  └──────────┬──────────┘                     │
                  ┌──────────┴──────────┐                     │
                  ▼                     ▼                     │
          ┌───────────────┐     ┌───────────────┐             │
          │   C1: Review  │     │   C2: Review  │             │
          │ the candidate │     │ the candidate │             │
          └───────┬───────┘     └───────┬───────┘             │
                  └──────────┬──────────┘                     │
                             ▼                                │
                  ┌─────────────────────┐                     │
                  │ V: Verify candidate │                     │
                  │  and give feedback  │                     │
                  └──────────┬──────────┘                     │
                             │                                │
                             ├── FAIL, round < 3 ─────────────┘
                             │
                             └── PASS, or round = 3 ──► Final answer
AuthorsOpenReview ↗

Reply to Reviewer oRR7: Part Three

W3 & Q1 (continued)

These results extend MAS-PromptBench from five to seven topologies, and the gain still depends on the topology. Averaged over the four optimizers, it is +2.4 points on Iterative Refinement and +1.5 on Fan-out/Fan-in, within the range of the five original topologies (+1.0 to +4.8). This supports the paper's takeaway that MAS need topology-aware prompt optimizers. The table below reports the average gain over the nine tasks for all seven topologies, with the standard deviation across three optimizer seeds; the two new topologies are in bold, and "—" marks inapplicable settings. The per-task results are on our project page and in Table 9 of the revised paper. We have added these results to the revised paper (Sec. 4.2 and Appendix C.1).

TopologyMAS-GEPAMAS-MIPROMAPROMASPO
Single+1.6 ± 1.0+3.0 ± 0.7——
Independent+2.5 ± 1.0+1.2 ± 0.6−0.1 ± 0.4+1.1 ± 0.8
Sequential+6.0 ± 2.2+7.5 ± 1.1+1.2 ± 0.9+4.7 ± 0.7
Centralized+0.2 ± 1.8+3.3 ± 1.1+1.9 ± 0.6+1.4 ± 0.7
Decentralized+0.9 ± 1.0+1.2 ± 0.8−0.4 ± 0.9+2.4 ± 0.8
Fan-out/Fan-in+0.6 ± 1.2+2.6 ± 1.4+2.1 ± 0.9+0.6 ± 1.0
Iterative Refinement+2.2 ± 2.0+3.4 ± 1.2+0.3 ± 0.8+3.9 ± 1.0

[1] Anthropic. (2024, December 19). Building effective agents. https://www.anthropic.com/engineering/building-effective-agents

[2] Anthropic. (2025, June 13). How we built our multi-agent research system. https://www.anthropic.com/engineering/multi-agent-research-system

[3] Anthropic. (n.d.). Create custom subagents. Claude Code Docs. Retrieved November 15, 2026, from https://code.claude.com/docs/en/sub-agents

[4] OpenAI. (2025, April). A practical guide to building agents. https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf

[5] OpenAI. (n.d.). Agent orchestration. OpenAI Agents SDK. Retrieved November 15, 2026, from https://openai.github.io/openai-agents-js/guides/multi-agent

[6] Google Cloud. (2026, May 28). Choose a design pattern for your agentic AI system. Cloud Architecture Center. https://docs.cloud.google.com/architecture/choose-design-pattern-agentic-ai-system

[7] Google. (2025, December 16). Developer's guide to multi-agent patterns in ADK. Google Developers Blog. https://developers.googleblog.com/en/developers-guide-to-multi-agent-patterns-in-adk

[8] Google Research. (2025, February 19). Accelerating scientific breakthroughs with an AI co-scientist. https://research.google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist

[9] Microsoft. (2026, February 12). AI agent orchestration patterns. Azure Architecture Center. https://learn.microsoft.com/azure/architecture/ai-ml/guide/ai-agent-design-patterns

[10] Microsoft. (n.d.). Multi-agent design patterns. AutoGen. Retrieved November 15, 2026, from https://microsoft.github.io/autogen/stable/user-guide/core-user-guide/design-patterns/intro.html

[11] CrewAI. (n.d.). Processes. CrewAI Documentation. Retrieved November 15, 2026, from https://docs.crewai.com/en/concepts/processes

[12] Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., ... & Liu, X. (2025). Towards a science of scaling agent systems. arXiv preprint arXiv:2512.08296.

[13] Anthropic. (2025, September 29). Building agents with the Claude Agent SDK. https://www.anthropic.com/engineering/building-agents-with-the-claude-agent-sdk

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their constructive feedback. We are glad that the reviewers find MAS-PromptBench a very practical framework with good coverage of prompt optimizers and tasks (Reviewer oRR7), a common evaluation framework that is practically meaningful (Reviewer 8cHW), and a comprehensive benchmark whose findings offer useful guidance on when prompt optimization benefits MAS (Reviewer sqGR). Following the reviewers' comments, we ran three sets of additional experiments, and the results are consistent with the paper's findings:

  • Two new topologies (Reviewer oRR7, W3 & Q1). We added Fan-out/Fan-in and Iterative Refinement to MAS-PromptBench and evaluated them with the four optimizers on all nine tasks. Their average gains are +1.5 and +2.4 points, within the range of the five original topologies (+1.0 to +4.8), and the gain still depends on the topology.
  • A second set of initial prompts (Reviewer 8cHW, W1 and Q2). We repeated the topology study with the four optimizers on HotpotQA, LiveCodeBench and BFCL. Under both sets, the topologies rank in the same order by average gain, and the average gain of each topology differs between the two sets by at most 1.1 points.
  • Three additional tasks for the protocol and team-size studies (Reviewer 8cHW, W2 and Q2). We repeated both studies on MATH, SWE-Bench and API-Bank with MAS-GEPA, MAS-MIPRO and MASPO on the four multi-agent topologies. Freeform communication yields the largest gain on each task (+2.8 points on average, against +2.0 and +2.1), and the gain decreases with the team size on the three tasks together (+2.8, +2.4 and +2.1 at nn = 4, 8 and 10).

We have also updated the paper, the code and the project page:

  • Revised paper. The revision includes the results above (Sec. 4.2–4.5 and Appendix C), with all changes highlighted in teal.
  • Code and project page. Our anonymous code repository [link] and project page [link] now include the two new topologies and the second set of initial prompts.

We respond to each reviewer's comments in the individual responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.