- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper explores how the performance of LLM-based multi-agent systems changes as the number of agents increases, and why scaling eventually saturates.
The authors adopt Steiner's task taxonomy to classify different multi-agent tasks based on how team members' efforts combine. They focus in particular on two types: disjunctive tasks and compensatory tasks. Through both theoretical analysis and large-scale experiments, they find that these tasks exhibit specific scaling limitations.
Overall, I think this paper presents an empirical study with relatively comprehensive experiments and a systematic analysis. However, the conclusions regarding scaling are largely reflected in prior work; the authors are using an existing task taxonomy to systematically validate and frame them. Because of that, I feel the novelty is a bit lacking.
Critical strengths
-
the authors provide a very clear conceptual framework, using Steiner's task taxonomy to offer a systematic way to understand different types of multi-agent collaboration. Although the taxonomy itself isn't new, applying it to LM systems feels quite novel.
-
The experiments are quite comprehensive. They cover 13 open-weight models, scale team sizes up to 30, and compare different tasks, rounds, and team compositions. Compared to previous work, this approach is much more systematic, and the exploration of these scaling patterns is consequently much more convincing.
Critical weaknesses
-
the novelty of this paper is somewhat limited, and there is a lack of new insights. The main theoretical findings can largely be derived directly from classical statistical principles, and they align closely with existing papers on multi-agent scaling. For instance, with a disjunctive task, pass@N increasing with the number of samples and agents is basically a straightforward probability result, and plurality voting converging to the model answer is just a direct corollary of the law of large numbers. As for compensatory tasks, the model has highly relevant priors that are already discussed in the literature.
-
regarding the taxonomy, although applying it to LLM-based multi-agent collaboration is novel, as a theoretical framework, it only provides a clear classification without yielding new conclusions that go beyond existing analytical frameworks. In other words, the taxonomy doesn't offer much beyond organizing existing theoretical results, and it lacks useful new theoretical predictions.
Questions for authors
-
what does the author feel is the contribution of this work compared to existing papers on multi-agent scaling?
-
how does the Stenner's task taxonomy provide new insights or predictions that would not directly follow from analyzing the aggregation mechanisms using existing statistical tools?
5 comments
Thank you for the response. I still have a few concerns.
- Regarding the existing literature, I feel the authors' comments on the contributions of prior work are not entirely fair. For instance, Kim et al. (2026) also analyzed the scaling behavior of different task structures in a very comprehensive study. Their conclusion isn't simply that coordination helps on parallelizable tasks but hurts on sequential ones. It feels to me that the insights in this work are similar, just using a different text task taxonomy.
- Regarding the selection rule that the authors additionally derived based on this taxonomy during the rebuttal phase: I haven't been able to spend a lot of time going through all this newly added content in detail. However, I did take a close look at it. To me, it seems to rely on the posterior performance of different methods on tasks to re-select the corresponding approach. Because of that, I don't find this selection rule to be a particularly informative approach.
Of course, please correct me if I've misunderstood anything.
We thank the reviewer for the follow-up and attention to detail. In our opinion there is a misunderstanding.
Re 1
We agree that our one-sentence summaries of the most related papers compressed both studies. We can extend the textual analysis and add in annexes a comparison table such as the one below:
| Chen et al. | Kim et al. | This paper | |
|---|---|---|---|
| Prediction | Accuracy vs. no of calls , fitted per query | 20-param linear regression; inputs include coordination metrics from multi-agent runs | Closed-form limits (Prop. 2.1) from single-agent samples |
| Task type | language | not specified (some seem to be divisible as per Steiner) | Unitary Stainer task types |
| Benchmarks | 5 (multiple-choice questions, + a synthetic dataset) | 6: web browsing, finacial analysis, etc | 5 disjunctive, 4 compensatory |
| Aggregation | Vote, Filter-Vote | plurality/majority vote | aggregation rule: vote for disjunctive, average for compensatory, union for additive, etc. |
| MAS topology | independent non debating zero shot answers | 4 topologies, zero shot + debate, tool use | single full mesh, debate, zero shot, self-correction |
| MAS size | 4 | 1-9 | 1-30 |
| MAS model composition | homo | homo + hetero | homo + hetero |
| Model type | proprietary (GPT-3.5) | 9 (OpenAI, Google and Anthropic) | 11-13 open weight/recipe (3-72 bn params) |
| Main conclusion | “easy” queries/less calls, “hard” queries/more calls, and non-monotone behavior can emerge when a task contains both types of queries | identify an empirical capability-saturation threshold beyond which additional agents are unlikely to improve performance | theory-guided framework in which the task type determines the rule that combines agent contributions and this rule determines the large-team limit |
Chen et al. Our disjunctive limit is the share of queries their Lemma 1 calls easy, and we do not claim it as new. They study voting on tasks with few possible answers. Continuous estimates, the choice between voting and averaging, revision, and mixed-model pools are where our two task types part ways.
Kim et al. They study how agent workflows are structured on agentic tasks and set static benchmarks aside. We study how contributions combine within a single subtask, which is one node in their architectures (Fig 2), on static benchmarks. Where we overlap, we agree: gains saturate, and their preliminary mixed-model teams did not get past capability saturation either, which matches our disjunctive pools. The compensatory reversal is outside their design. They also note that it is still open how findings about human teams carry over to AI agents. Steiner's taxonomy is one such finding, and we test it.
Re 2
"the selection rule that the authors additionally derived based on this taxonomy during the rebuttal phase".
The task selection and answer aggregation rules (first and last cols of Tab 1) is what Sec 2 describes and Sec 3 evaluates and was there at submission time. It says: if task = disjunctive do voting aggregation, esle if task = compensatory do averaging aggregation, if task = additive, do union agregation, etc. The new Fig 1 proves that 1) it works on dicretionary and compensatory benchmarks across up to 13 open weight and open recipe models 3-72 bn parameters and 2) we can pre-estimate the limits with Eq. 1 and 2 of the revised paper (Eq. 3,4 of original).
"it seems to rely on the posterior performance of different methods on tasks to re-select the corresponding"
When a task comes in, it is assigned to 1 of 5 Steiner task types (task selection rule). Based on that, the correct aggregation rule is assigned enabling the use of statistics tools to predict the limit. There's no re-selecting and no posterior in this sense. Fig 1 validates the aggregation rules so perhaps that's why the reviwer may think it's posterior because it uses experiments from Sec 3 to validate.
"I don't find this selection rule to be a particularly informative approach"
To make a simple analogy, probably not a perfect one, this statement can be understood as "Knowing whether a machine learning task is regression, classification, clustering or other is not informative". It's like saying that when solving a classification task it's not informative to know that precition, recall and F1 are probably appropriate metrics to evaluate the performance. Same as by knowing a ML task is of type regression (compensatory analogy), we know that the metric is say MSE (averaging analogy) and we also know that bagging usually helps (more agents help or not analogy). Example analogy below:
| ML analogue | Known result analogue | ‖ | Steiner type | Predicted and Measured here |
|---|---|---|---|---|
| Bagging regressors | error → bias, variance | ‖ | compensatory | error predicted by Eq. 2 within 0.6%; gain capped by bias share β = 0.87–0.97 |
| Voting classifiers | accuracy → share of inputs with correct modal label | ‖ | disjunctive | gain capped by Eq. 1, limit within 0.5 points |
Crystallizing the novelty of the framework and the new findings it enables
Thank you for the insightfull questions, please see also General Response above.
what does the author feel is the contribution of this work compared to existing papers on multi-agent scaling?
To date, two works directly study the link between task properties and multi-agent group performance a) Chen et al. (2024) formalize query difficulty and show how its variation explains voting's scaling behavior and b) Kim et al (2026) estimate task properties from data and find that coordination helps on parallelizable tasks but can hurt on sequential ones. In this work, with the proposed framework, the approach is the opposite. Starting from the task (or benchmark), it is mapped to the taxonomy, that informs the aggregation rules selection, which then enables deriving the estimated scaling. In the newly revised Abstract, Section 1 and new Figures 1 and 2 we try to make this clear.
how does the Stenner's task taxonomy provide new insights or predictions that would not directly follow from analyzing the aggregation mechanisms using existing statistical tools?
We thank the reviewer for this question. We agree that the large-N limits in Proposition 2.1 (Section 2.3) follow from standard statistical tools, and we do not claim new limit theorems. Rather, statistics answers what a given aggregation rule converges to, whereas Steiner’s taxonomy answers the system-design questions around that rule: which rule the task requires a priori, what the team could achieve in principle, and which non-aggregation levers matter (Sections 1 and 2, Figure 2). This yields three tested predictions that do not follow from aggregation statistics alone:
Statistics analyzes a chosen operator while the taxonomy assigns the rule before seeing data (Table 1; Sections 2.1–2.2; Appendix A) and supplies potential productivity (pass@N) to measure process loss (Sections 2.1, 3.1, and 3.2). By testing all five Steiner rules on the same teams it can be seen that the assigned rule achieves the largest gain across all nine benchmarks, while swapping disjunctive and compensatory rules costs 4.2 disjunctive points and 75% of compensatory MSE reduction (Section 1, Figure 1a, b; Appendix D, Figure 6). The large N=30 process loss (4.8–20.1 disjunctive points in Section 3.2; 55.1% pass@30 vs. 33.5% realized on RealFP in Section 3.1 and Table 4) shows that remaining headroom lies in selection or bias reduction, not larger N (Section 4).
Aggregation statistics are agnostic to revision, which shifts the answer distribution before aggregation. Paired with demonstrability (Laughlin & Ellis, 1986, in Appendix B.2.1; contrasted with Appendix B.2.2), the taxonomy predicts revision helps disjunctive tasks (verifiable steps) but not compensatory estimates.
Empirically, revision boosts disjunctive accuracy by +27.9 points on GSM8K and +15.5 on GSM-Hard (Section 3.1; Appendix D, Table 7; driven by a second attempt in self-revision controls in Section 3.1 and Appendix D, Table 6), whereas on all four compensatory benchmarks it cuts within-item variance by 16–24% but changes item-level bias (β=87-97% of MSE; Section 3.3) by <3%, leaving the floor unchanged (Figure 1d; Appendix E.1, Table 18)—and observing peers increases error relative to self-revision on three of four benchmarks (Section 3.3; Appendix E.1, Tables 16 and 17).
Steiner ties disjunctive potential to the strongest member, whereas compensatory teams beat all members only when opposite-sign biases bracket the truth (Section 1; Section 3.3). Accordingly, our best 5-model disjunctive pool trails its strongest member in 10/10 splits (by 3.4–4.3 points; Section 3.2, Table 2), while the 7B–8B pool beats its best member by 16.7% and 17.2% on RealFP and SciOly, yet fails when biases share a sign (+54% error on Guesstimate, where all 5 models overestimate 14/15 items, and on ForecastBench; Section 3.3, Table 3; Appendix E.1, Table 13).