- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FRAME, an algorithm that progressively selects agent roles and generates a communication structure based on the question and the agents' intermediate outputs, ultimately building a complete multi-agent structure. The method uses RFF-UCB to select roles and a lightweight neural network to decide which edges to connect. When the constructed multi-agent structure cannot complete the task, it uses interventional verification to identify the problematic role, modifies its prompt, and reconstructs the graph. Effective role repairs obtained during training are saved and reused. The authors evaluate the method on eight benchmarks, including code generation and mathematics.
Critical strengths
-
Adjusting subsequent roles and communication between agents based on actual problem-solving results is a meaningful direction. Earlier outputs can help determine which agents to select next and how the agents should collaborate. The ablation results in Table 3 support using intermediate outputs to guide graph construction.
-
The experiments cover multiple tasks, and the main experiments use one open-source model and one closed-source model. The proposed algorithm also achieves noticeable improvements on MMLU-Pro and LiveCodeBench.
Critical weaknesses
-
Insufficient reporting of the training procedure and full training costs
In the proposed algorithm, each time a new node is added, the current partial graph needs to be executed again on the current training question. According to the paper, each training question undergoes three graph-construction runs. At each construction step, three edge configurations are sampled, and each configuration is executed three times. If failure repair is triggered, additional correction and graph reconstruction are required. This training procedure involves a large number of repeated model calls. However, the paper does not report the token consumption and wall-clock time of the full training process on each benchmark, nor does it compare training costs with other baselines. The authors need to report these results and specify the number of concurrent API requests so that it is possible to assess whether the performance gains from the additions to the algorithm are worth the corresponding training costs.
In addition, although the paper provides the procedure and sampling counts for a single question during training, it does not report the total number of epochs, the total number of parameter updates, or when the overall training procedure ends. These details also need to be provided.
-
Issues with the experimental statistics; the sources of the HumanEval training and test sets also need to be explained
The average number of nodes and the proportion of single-agent graphs in Table 11 do not match. For GSM8K, the single-agent proportion is 88.3%, and every remaining graph contains at least two nodes, so the minimum average number of nodes should be 1.117, rather than the reported 1.07. The same issue occurs for HumanEval: its single-agent proportion is 89.0%, and the maximum number of nodes is five, so the average number of nodes can be at most 1.44, below the reported 1.48. If these numbers come from the same set of final topologies, they cannot hold simultaneously. The authors need to explain this issue.
In addition, Table 8 reports 40 training questions and 164 test questions for HumanEval, but the standard HumanEval dataset contains only 164 questions in total. The source of the 40 training questions therefore needs to be explained. If these 40 questions were sampled from the full set of 164 questions and the same 164 questions were then used for testing, the training and test sets would overlap. The authors need to clarify this issue and explain where the training data come from, as this directly affects the validity of the experimental results.
Questions for authors
-
What are the total training token consumption and wall-clock time for the proposed algorithm and the main comparison baselines on each benchmark? Do these statistics include all calls for candidate graph execution, interventional verification, role modification, and graph reconstruction? What is the number of concurrent requests when calling the models?
-
What are the complete training details for each benchmark? How many epochs are used in total, how many parameter updates are performed in total, and what is the condition for ending training?
-
How are the average number of nodes and the proportion of single-agent graphs in Table 11 calculated? Do the two columns come from the final topologies of the same set of test instances? Based on the single-agent proportions in Table 11 and the node-count limits in the paper, the average number of nodes should be at least 1.117 for GSM8K and at most 1.44 for HumanEval, which do not match the reported 1.07 and 1.48. How should these discrepancies be explained?
-
Where do the 40 training questions for HumanEval come from? Are the 164 questions used for testing the full standard HumanEval dataset?