- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduce an MAS evolution method, which treats the overall system as a DAG and tries to modify each edge based on the feedback during test time. Testing on two models and 4 benchmarks, they demonstrate the effectiveness of their method. Besides, the authors also include some analysis to better explain their method.
Critical strengths
- Good formulation, which treat the overall process as a modification of a DAG.
- Solid experiments, including several benchmarks and models.
- I think the method is well designed.
Critical weaknesses
- The huge cost for the evolution process.
- We may need to design a specific framework for each dataset, which limits its usage significantly.
- I think the proposed method need a powerful critic and it cannot generalize to open-ended scenarios.
Questions for authors
- I think the first two figures look like AI-generated product, especially for figure 2. Would it possible to draw them on your own?
- I still worry about the effectiveness on real world problems. But at least, it is a decent research product. Could the authors explain more about how to use it in real world scenarios?
2 comments
Thanks for your response, I will keep my positive score.
Thank you for your positive comments on our problem formulation, experiments, and method design. We respond to each point below.
Q1: Figures 1 and 2
Thank you for the suggestion. We drew Figures 1 and 2 ourselves, with assistance from AI tools as disclosed in our AI Use Statement. We used a simple schematic style so that readers can quickly understand the workflow of EMAS. In the revised paper, we have revised Figure 2 to make it more concise and easier to read.
Q2: Use in real-world scenarios
Large-scale studies find that real user requests concentrate in a small number of categories: nearly 80% of ChatGPT conversations fall into practical guidance, seeking information, and writing (Chatterji et al., 2025, NBER Working Paper 34255), and LMSYS-Chat-1M clusters the user prompts of 100K randomly sampled English conversations into 20 topic clusters (Zheng et al., ICLR 2024).
Deployment therefore works by category. Tasks are grouped by category, one EMAS system is maintained for each category, and new tasks are routed to the corresponding category. No manual design is needed for a new category: the Initial MAS for each category is designed fully automatically by an LLM, and the thresholds and acceptance rules of EMAS are identical across the 4 benchmarks and 2 backbones. As tasks accumulate, the system can continue to evolve, and each revision must pass paired Validation before it is deployed. This approach is suitable for real-world scenarios:
We have added this motivation to the introduction (Section 1) of the revised paper.
Summary
Thank you again for your constructive feedback. We hope this response answers your questions.