- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper mainly contributes evaluation methods and experimental practices for LLM multi-agent systems. It distinguishes three factors: structural prompts, the number of candidates and their selection rule specified in advance, and actual resource consumption. The authors propose an auditable criterion for checking that requests remain unchanged when generating 1, 3, or 5 candidates. The cooperative cooking experiment uses a fixed voting rule to select actions, while HumanEval measures whether a candidate pool contains a correct program, without voting.
Critical strengths
-
The paper has a clear research question: Fixing the number of candidates and matching actual token consumption answer different questions. Selecting runs based on costs produced after treatment can change what is being estimated.
-
The paper is honest and cautious interpretation: The authors do not interpret non-significant structural effects as “no effect,” or generalize the decline under voting to “more inference is always harmful.” Scoring corrections and their impact are also disclosed separately.
Critical weaknesses
-
Repeated sampling, degradation under voting, and the risks of adjusting for post-treatment variables have already been studied. The main additions are an auditable criterion, preregistered factorial experiments, and fixed margins for practical importance. The paper needs to explain more clearly what new insights this combination provides beyond existing evaluations.
-
The voting experiment tests only one fixed rule, without comparing other candidate-selection methods. The negative results therefore apply only to the combined sampling-and-voting policy. The observed changes also go against both preregistered mechanism predictions, leaving the reason for the performance decline unresolved.
-
After scoring corrections, the single-candidate pass rate on HumanEval reaches 92.7%, leaving limited room for improvement. These results therefore cannot establish that structural prompts are unhelpful on harder tasks, nor should they be treated as a replication of the cooking results.
Questions for authors
See weakness