- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MAGIC, which progressively generates multi-agent collaboration structures based on the input question, selecting each node's role, whether to use a single agent or a predefined group, and how they are connected. The authors use reinforcement learning to train the graph construction policy and provide intermediate rewards based on performance on other training questions, structural complexity, and role repetition. The paper evaluates the method on eight benchmarks covering knowledge reasoning, mathematics, coding, and tabular tasks, and includes component comparisons and comparisons of training costs and inference token consumption.
Critical strengths
-
The proposed method achieves good results on multiple tasks. The main experiments cover eight benchmarks and include three complete training and evaluation runs, with mean scores higher than those of the compared methods. The efficiency experiments on four benchmarks also show that performance improvements do not generally lead to higher inference token consumption, which adds to the method's practical value.
-
The paper compares the effects of using single agents or agent groups at different stages and finds that not all stages require the same scale of collaboration. On the tasks tested, selectively using groups provides a better performance–cost trade-off than using single agents throughout or groups throughout. This provides useful experimental guidance on how to allocate collaboration resources.
Critical weaknesses
-
The method's added contributions are not clear enough
Previous work such as MaAS already includes single-agent operators such as CoT and multi-agent modules such as Debate and Ensemble in the same operator library, selecting and combining them based on the question. This paper provides both single-agent and group implementations for the same role, but does not discuss how this design differs specifically from MaAS or what additional benefits these differences bring.
For reward design, the authors use four other training questions to evaluate the structure being constructed, but how much this feedback helps construct the graph for the current question has not been evaluated separately. The existing comparisons remove this feedback together with the terms for structural complexity and role repetition, so they do not show how much benefit comes from the four-question evaluation itself. More detailed ablations are needed to determine how much additional value the new reward provides over simpler alternatives.
-
The stability of the key experimental results needs to be verified
The main experiments were run three times, but Q2 and Q3 were each run only once. In Q2, the new reward improves MMLU-Pro accuracy from 75% to 76%, which is only one more correct answer out of 100 test questions; the improvement on TAT-QA is only 0.5 F1 points. Q3 compares performance at the same training cost, but also lacks repeated runs. It is not yet clear whether these gains are stable.
Questions for authors
-
The paper already reports some training costs and inference token consumption. What are the total training token consumption, training wall-clock time, and test wall-clock time for this method and the main comparison methods? What are the corresponding model-call concurrency settings? Do the performance gains justify the cost changes introduced by the new components?
-
Q2 and Q3 have each been run only once. If training is repeated independently, do these gains remain stable? How much do the results vary?
-
Keeping the final task reward, how do the results compare with the full method if only structural information is added, or if only feedback from four other training questions is added? How does the training cost change in each case? How much independent benefit comes from running these four additional questions?
-
Previous work such as MaAS already includes single-agent operators and multi-agent modules in the same operator library. How does this paper's mixed-granularity approach differ in substance from those earlier approaches?