- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper targets a limitation of MAS: existing approaches typically specify only the roles agents should play (who to act as), without providing concrete guidance on how to carry out their work (how to do the work), which limits overall system performance. To address this, the paper uses a strong model to construct role and skill libraries based on the validation set, which are then provided to an orchestrator. The orchestrator decomposes each input task into subtasks, assigns an appropriate role and skill to each subtask, and establishes the dependencies among subtasks, thereby organizing the execution of the entire MAS. To further improve orchestration, the paper self-evolves two natural-language experience modules, Environment Knowledge and Team Strategy, on the validation set. In each round, both modules are updated through contrastive reflection on multiple executions of the same task; after a fixed number of rounds, the version that performs best on the validation set is frozen for testing. Experiments show that the method improves MAS performance across multiple benchmarks, suggesting that both Role–Skill Composition and Self-Evolving Orchestration contribute to better task performance.
Critical strengths
-
The framework constructs role and skill libraries in advance, and the orchestrator dynamically selects appropriate role–skill combinations for each task. This addresses a limitation of existing MAS, which typically assign roles to agents without providing specific guidance on how to carry out their work.
-
The self-evolving Environment Knowledge and Team Strategy modules iteratively distill task characteristics and orchestration experience, providing additional guidance to the orchestrator and thereby enhancing its orchestration capabilities.
Critical weaknesses
-
Both the role and skill libraries are generated by GPT-5.6-sol, a model stronger than the three backbone models used in the MAS framework. Consequently, it is difficult to determine whether the observed performance gains stem from the Role–Skill Composition and Self-Evolving Orchestration mechanisms themselves, or simply from the additional knowledge provided by a more powerful model. It would be helpful to conduct supplementary experiments where these libraries are generated by weaker models, or directly by the backbone models themselves.
-
Both Environment Knowledge and Team Strategy are provided to the orchestrator in their entirety as natural-language documents, without any filtering. Since some of this information may be irrelevant to the current task, the method could benefit from an appropriate retrieval mechanism that filters out such extraneous content.
Questions for authors
-
Could you report the performance of MASCraft when the role and skill libraries are generated by weaker models?
-
As shown in Figure 3, the improvements from self-evolution vary considerably in stability across benchmarks. What do you consider to be the primary reason? Furthermore, the validation curves for DRB and BCP do not appear to have converged. Beyond running a fixed number of rounds and selecting the best-performing one on the validation set, are there more principled stopping criteria?
3 comments
Follow-up on our response
Dear Reviewer,
Thank you again for taking the time to review our paper and share your feedback. Is there anything we could clarify in our response or any remaining concerns we could address?
Response to the Authors
Thank you for your response. The additional experiments and clarifications have addressed most of my concerns. However, I believe my current score is already appropriate, so I will keep it unchanged.
Response to Reviewer GPFV
Thank you for the positive assessment and helpful suggestions. We address library construction, policy relevance, and evolution stability below.
R1. Dependence on the Library-Construction Model (W1, Q1)
Using the execution backbone to construct the libraries led to modest score decreases across the three benchmarks relative to GPT-5.6-sol. We evaluated four library-construction models while keeping GLM-5.2-Fast-Preview fixed as the execution backbone.
These results suggest that a sufficiently capable construction model is important, but a stronger external model is not essential for comparable performance in these settings.
R2. Task Relevance of Policy Context (W2)
We include both policy modules in full for two reasons:
We will explore task-based retrieval and filtering to address potentially irrelevant guidance as the policies grow.
R3. Evolution Stability and Stopping Criteria (Q2)
(1) Stability across benchmarks. Performance may fluctuate across rounds because textual policy updates offer limited control over update direction and magnitude, which gradients and step sizes help specify in parameter optimization. We observed similar fluctuations in our reproductions of Skill-MAS and other baselines, consistent with reports in OPRO (p. 4, §2.3) [1], SIPDO (p. 6, Table 1) [2], and STEVE (p. 2, Figure 1(b)) [3]. Our comparison of successive policies suggests that overly aggressive updates may have contributed to some of the observed declines (see our response to Reviewer LWES, R2). DRB’s continuous, multidimensional scoring may help explain its smoother trajectory by capturing incremental improvements in coverage and organization.
(2) Additional rounds and stopping criteria. We extended evolution to 20 rounds on DRB and BCP but observed no substantial further gains on the held-out test sets. We will explore validation-based early stopping with a predefined improvement threshold and patience as an alternative to a fixed round limit.
Thank you again for your constructive feedback. We will incorporate the additional results and clarifications into the revised paper and welcome any further questions or concerns.
References
[1] Large Language Models as Optimizers. ICLR 2024.
[2] SIPDO: Closed-Loop Prompt Optimization via Synthetic Data Feedback. ICLR 2026.
[3] STEVE: Stabilizing Textual Gradient-Based Prompt Optimization via Error-Driven Refinement and Regularized Verification. Findings of IJCNLP–AACL 2026.