- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MAS-PromptBench, a benchmark for evaluating prompt optimization in multi-agent LLM systems. It systematically studies eight optimizers across different tasks, workflow topologies, communication protocols, and team sizes, revealing when prompt optimization is effective and where it faces challenges.
Critical strengths
- Comprehensive benchmark. The paper evaluates diverse MAS configurations and prompt optimizers, providing a useful benchmark for future research.
- Practical insights. The empirical findings offer useful guidance on when prompt optimization benefits multi-agent systems.
Critical weaknesses
- Limited generalizability across LLMs. In Section 4.5, Sequential achieves the largest gain (+8.6) with Qwen3.5-9B, while Independent (+13.6) and Centralized (+12.8) perform best with Llama-3.1-8B-Instruct. These substantially different results raise concerns about the generalizability of the paper’s conclusions across different LLMs.
Questions for authors
Why does free-form communication consistently yield larger prompt optimization gains than structured communication, and does it also lead to better final performance?
5 comments
Reply to Reviewer sqGR: Part Two
Q1:
- Why does freeform communication yield larger gains than structured communication?
- Does it lead to better final performance?
Thank you so much for raising this question, since it concerns a key finding of this work.
Freeform communication leaves agent messages open to revision, which gives prompt optimization more room to improve. Under Freeform, the initial system prompts specify no message format, so the optimizer revises both the content and the form of what each agent passes to the next agent. Under Semi-structured and Structured, the initial system prompts already contain the protocol's message format (slots for status, evidence, confidence and next step), so less is left for the optimizer to add.
The optimized prompts show this on BFCL, where the larger gain appears (+12.5 points with Freeform, against +5.2 with Semi-structured and +5.1 with Structured). With the Sequential topology under Freeform, the optimized Analyzer prompt adds an explicit message format that the initial prompt does not contain: "Your output must be a JSON block containing a list of function-call dictionaries in this exact format". The larger gain of Freeform also appears on three additional tasks (see our response to Reviewer 8cHW, W2).
The final performance after optimization is similar across the three protocols, with Freeform slightly higher. The test scores of the optimized prompts are 71.9 with Freeform, against 70.9 with Semi-structured and 71.4 with Structured, averaged over the three tasks, the four multi-agent topologies and MAS-GEPA, MAS-MIPRO and MASPO. This is consistent with our takeaway, which concerns the prompt-optimization gain and not the final performance: Freeform communication gives prompt optimization more room to improve, and the paper does not claim that it yields the best final performance.
Dear Reviewer sqGR,
We sincerely thank you for taking the time to review our work and for your positive score. Your valuable suggestions have greatly contributed to refining our paper, and we sincerely appreciate your constructive feedback. As the ICLR author–reviewer discussion period concludes on Nov 18 (AoE), we would greatly appreciate your feedback on our rebuttal responses to ensure that all your concerns have been adequately addressed. Moreover, we would be happy to engage in further discussions and address any additional questions or suggestions you may have.
Best regards,
Authors
Reply to Reviewer sqGR: Part One
We sincerely thank the reviewer for the support of our work. It is gratifying to know that the reviewer finds the benchmark comprehensive and its empirical findings useful guidance on when prompt optimization benefits MAS.
Thank you for the careful reading of our task-model ablation and for raising this important concern.