acceptodds
Under review as a conference paper at ICLR 2027

Multi-agent Scaling Across Disjunctive and Compensatory Tasks

18 views

Abstract

Multi-agent LLM systems are often expected to improve as team size increases, yet the scaling behavior may depend on task structure. Our central contribution is to introduce Steiner’s taxonomy of group tasks as a framework for analyzing multi-agent LLM scaling and focusing the analysis on disjunctive and compensatory tasks. We model independently sampled agents as conditionally independent given the item, which yields their large-team limits: plurality voting converges to the model’s modal answer, and averaging converges to the model’s item-level bias. Across selected representative benchmarks, 13 open-weight models, and teams of up to 30 agents, we find qualitatively different scaling behavior. On disjunctive tasks, the probability that at least one agent is correct grows by 5–20 points with team size, but plurality voting over agents that answer directly realises almost none of this potential, as the model predicts to within 0.5 points on average. Multi-round revision raises accuracy considerably, yet the gain is nearly the same with one peer as with 29. In contrast, scaling provides little benefit on Fermi estimation, despite its natural suitability for aggregation: item-level biases shared across the samples of a model account for about 87% of the squared error, so averaging reduces error by only about 6%. Combining model families helps on Fermi estimation but does not surpass the strongest member on disjunctive tasks. These results show that task structure, together with the mechanism combining member outputs, is a fundamental determinant of team scaling.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
multi-agent scaling, open weight model, open recipe model, Steiner taxonomy, disjunctive task, compensatory tasl

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MPiLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores how the performance of LLM-based multi-agent systems changes as the number of agents increases, and why scaling eventually saturates.

The authors adopt Steiner's task taxonomy to classify different multi-agent tasks based on how team members' efforts combine. They focus in particular on two types: disjunctive tasks and compensatory tasks. Through both theoretical analysis and large-scale experiments, they find that these tasks exhibit specific scaling limitations.

Overall, I think this paper presents an empirical study with relatively comprehensive experiments and a systematic analysis. However, the conclusions regarding scaling are largely reflected in prior work; the authors are using an existing task taxonomy to systematically validate and frame them. Because of that, I feel the novelty is a bit lacking.

Critical strengths
  1. the authors provide a very clear conceptual framework, using Steiner's task taxonomy to offer a systematic way to understand different types of multi-agent collaboration. Although the taxonomy itself isn't new, applying it to LM systems feels quite novel.

  2. The experiments are quite comprehensive. They cover 13 open-weight models, scale team sizes up to 30, and compare different tasks, rounds, and team compositions. Compared to previous work, this approach is much more systematic, and the exploration of these scaling patterns is consequently much more convincing.

Critical weaknesses
  1. the novelty of this paper is somewhat limited, and there is a lack of new insights. The main theoretical findings can largely be derived directly from classical statistical principles, and they align closely with existing papers on multi-agent scaling. For instance, with a disjunctive task, pass@N increasing with the number of samples and agents is basically a straightforward probability result, and plurality voting converging to the model answer is just a direct corollary of the law of large numbers. As for compensatory tasks, the model has highly relevant priors that are already discussed in the literature.

  2. regarding the taxonomy, although applying it to LLM-based multi-agent collaboration is novel, as a theoretical framework, it only provides a clear classification without yielding new conclusions that go beyond existing analytical frameworks. In other words, the taxonomy doesn't offer much beyond organizing existing theoretical results, and it lacks useful new theoretical predictions.

Questions for authors
  1. what does the author feel is the contribution of this work compared to existing papers on multi-agent scaling?

  2. how does the Stenner's task taxonomy provide new insights or predictions that would not directly follow from analyzing the aggregation mechanisms using existing statistical tools?

5 comments
AuthorsOpenReview ↗

Crystallizing the novelty of the framework and the new findings it enables

Thank you for the insightfull questions, please see also General Response above.

what does the author feel is the contribution of this work compared to existing papers on multi-agent scaling?

To date, two works directly study the link between task properties and multi-agent group performance a) Chen et al. (2024) formalize query difficulty and show how its variation explains voting's scaling behavior and b) Kim et al (2026) estimate task properties from data and find that coordination helps on parallelizable tasks but can hurt on sequential ones. In this work, with the proposed framework, the approach is the opposite. Starting from the task (or benchmark), it is mapped to the taxonomy, that informs the aggregation rules selection, which then enables deriving the estimated scaling. In the newly revised Abstract, Section 1 and new Figures 1 and 2 we try to make this clear.

how does the Stenner's task taxonomy provide new insights or predictions that would not directly follow from analyzing the aggregation mechanisms using existing statistical tools?

We thank the reviewer for this question. We agree that the large-NN limits in Proposition 2.1 (Section 2.3) follow from standard statistical tools, and we do not claim new limit theorems. Rather, statistics answers what a given aggregation rule converges to, whereas Steiner’s taxonomy answers the system-design questions around that rule: which rule the task requires a priori, what the team could achieve in principle, and which non-aggregation levers matter (Sections 1 and 2, Figure 2). This yields three tested predictions that do not follow from aggregation statistics alone:

  1. A priori rule selection and process loss (Sections 1, 2, and 3.1–3.2; Appendix D):

Statistics analyzes a chosen operator while the taxonomy assigns the rule before seeing data (Table 1; Sections 2.1–2.2; Appendix A) and supplies potential productivity (pass@N\text{pass@}N) to measure process loss (Sections 2.1, 3.1, and 3.2). By testing all five Steiner rules on the same teams it can be seen that the assigned rule achieves the largest gain across all nine benchmarks, while swapping disjunctive and compensatory rules costs 4.2 disjunctive points and 75% of compensatory MSE reduction (Section 1, Figure 1a, b; Appendix D, Figure 6). The large N=30N=30 process loss (4.8–20.1 disjunctive points in Section 3.2; 55.1% pass@30\text{pass@}30 vs. 33.5% realized on RealFP in Section 3.1 and Table 4) shows that remaining headroom lies in selection or bias reduction, not larger NN (Section 4).

  1. When deliberation helps (Sections 3.1–3.3; Appendices B.2, D, and E.1):

Aggregation statistics are agnostic to revision, which shifts the answer distribution before aggregation. Paired with demonstrability (Laughlin & Ellis, 1986, in Appendix B.2.1; contrasted with Appendix B.2.2), the taxonomy predicts revision helps disjunctive tasks (verifiable steps) but not compensatory estimates.

Empirically, revision boosts disjunctive accuracy by +27.9 points on GSM8K and +15.5 on GSM-Hard (Section 3.1; Appendix D, Table 7; driven by a second attempt in self-revision controls in Section 3.1 and Appendix D, Table 6), whereas on all four compensatory benchmarks it cuts within-item variance by 16–24% but changes item-level bias (β=87-97%\beta = 87\text{-}97\% of MSE; Section 3.3) by <3%, leaving the floor unchanged (Figure 1d; Appendix E.1, Table 18)—and observing peers increases error relative to self-revision on three of four benchmarks (Section 3.3; Appendix E.1, Tables 16 and 17).

  1. Heterogeneous teams vs. best member (Sections 3.2 and 3.3; Appendix E.1):

Steiner ties disjunctive potential to the strongest member, whereas compensatory teams beat all members only when opposite-sign biases bracket the truth (Section 1; Section 3.3). Accordingly, our best 5-model disjunctive pool trails its strongest member in 10/10 splits (by 3.4–4.3 points; Section 3.2, Table 2), while the 7B–8B pool beats its best member by 16.7% and 17.2% on RealFP and SciOly, yet fails when biases share a sign (+54% error on Guesstimate, where all 5 models overestimate 14/15 items, and on ForecastBench; Section 3.3, Table 3; Appendix E.1, Table 13).

Thank you for the response. I still have a few concerns.

  1. Regarding the existing literature, I feel the authors' comments on the contributions of prior work are not entirely fair. For instance, Kim et al. (2026) also analyzed the scaling behavior of different task structures in a very comprehensive study. Their conclusion isn't simply that coordination helps on parallelizable tasks but hurts on sequential ones. It feels to me that the insights in this work are similar, just using a different text task taxonomy.
  2. Regarding the selection rule that the authors additionally derived based on this taxonomy during the rebuttal phase: I haven't been able to spend a lot of time going through all this newly added content in detail. However, I did take a close look at it. To me, it seems to rely on the posterior performance of different methods on tasks to re-select the corresponding approach. Because of that, I don't find this selection rule to be a particularly informative approach.

Of course, please correct me if I've misunderstood anything.

We thank the reviewer for the follow-up and attention to detail. In our opinion there is a misunderstanding.

Re 1

We agree that our one-sentence summaries of the most related papers compressed both studies. We can extend the textual analysis and add in annexes a comparison table such as the one below:

Chen et al.Kim et al.This paper
PredictionAccuracy vs. no of calls KK, fitted per query20-param linear regression; inputs include coordination metrics from multi-agent runsClosed-form limits (Prop. 2.1) from single-agent samples
Task typelanguagenot specified (some seem to be divisible as per Steiner)Unitary Stainer task types
Benchmarks5 (multiple-choice questions, + a synthetic dataset)6: web browsing, finacial analysis, etc5 disjunctive, 4 compensatory
AggregationVote, Filter-Voteplurality/majority voteaggregation rule: vote for disjunctive, average for compensatory, union for additive, etc.
MAS topologyindependent non debating zero shot answers4 topologies, zero shot + debate, tool usesingle full mesh, debate, zero shot, self-correction
MAS size41-91-30
MAS model compositionhomohomo + heterohomo + hetero
Model typeproprietary (GPT-3.5)9 (OpenAI, Google and Anthropic)11-13 open weight/recipe (3-72 bn params)
Main conclusion“easy” queries/less calls, “hard” queries/more calls, and non-monotone behavior can emerge when a task contains both types of queriesidentify an empirical capability-saturation threshold beyond which additional agents are unlikely to improve performancetheory-guided framework in which the task type determines the rule that combines agent contributions and this rule determines the large-team limit

Chen et al. Our disjunctive limit π\pi is the share of queries their Lemma 1 calls easy, and we do not claim it as new. They study voting on tasks with few possible answers. Continuous estimates, the choice between voting and averaging, revision, and mixed-model pools are where our two task types part ways.

Kim et al. They study how agent workflows are structured on agentic tasks and set static benchmarks aside. We study how contributions combine within a single subtask, which is one node in their architectures (Fig 2), on static benchmarks. Where we overlap, we agree: gains saturate, and their preliminary mixed-model teams did not get past capability saturation either, which matches our disjunctive pools. The compensatory reversal is outside their design. They also note that it is still open how findings about human teams carry over to AI agents. Steiner's taxonomy is one such finding, and we test it.

Re 2

"the selection rule that the authors additionally derived based on this taxonomy during the rebuttal phase".

The task selection and answer aggregation rules (first and last cols of Tab 1) is what Sec 2 describes and Sec 3 evaluates and was there at submission time. It says: if task = disjunctive do voting aggregation, esle if task = compensatory do averaging aggregation, if task = additive, do union agregation, etc. The new Fig 1 proves that 1) it works on dicretionary and compensatory benchmarks across up to 13 open weight and open recipe models 3-72 bn parameters and 2) we can pre-estimate the limits with Eq. 1 and 2 of the revised paper (Eq. 3,4 of original).

"it seems to rely on the posterior performance of different methods on tasks to re-select the corresponding"

When a task comes in, it is assigned to 1 of 5 Steiner task types (task selection rule). Based on that, the correct aggregation rule is assigned enabling the use of statistics tools to predict the limit. There's no re-selecting and no posterior in this sense. Fig 1 validates the aggregation rules so perhaps that's why the reviwer may think it's posterior because it uses experiments from Sec 3 to validate.

"I don't find this selection rule to be a particularly informative approach"

To make a simple analogy, probably not a perfect one, this statement can be understood as "Knowing whether a machine learning task is regression, classification, clustering or other is not informative". It's like saying that when solving a classification task it's not informative to know that precition, recall and F1 are probably appropriate metrics to evaluate the performance. Same as by knowing a ML task is of type regression (compensatory analogy), we know that the metric is say MSE (averaging analogy) and we also know that bagging usually helps (more agents help or not analogy). Example analogy below:

ML analogueKnown result analogue‖Steiner typePredicted and Measured here
Bagging regressorserror → bias, variance‖compensatoryerror predicted by Eq. 2 within 0.6%; gain capped by bias share β = 0.87–0.97
Voting classifiersaccuracy → share of inputs with correct modal label‖disjunctivegain capped by Eq. 1, limit within 0.5 points
Reviewer_tgAyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Steiner's taxonomy of group tasks as a framework for analyzing multi-agent LLM scaling in disjunctive and compensatory tasks. It suggests that scaling is more effective in disjunctive tasks than compensatory tasks.

Critical strengths
  • The framework is a useful way for characterizing different tasks.
Critical weaknesses
  • As agent workflows become more complex, it seems that most tasks will fall in the heterogeneous group. It may be useful to think about the atomic unit of tasks.
  • The formal framework is somewhat disconnected.
Questions for authors

Minor:

  • some citation formats are not correct.
3 comments
AuthorsOpenReview ↗

Atomic unit of tasks and formal framework connection

Thank you for the insightfull questions, please see also General Response above.

As agent workflows become more complex, it seems that most tasks will fall in the heterogeneous group. It may be useful to think about the atomic unit of tasks.

We agree that complex workflows may combine multiple task types. Steiner distinguishes divisible tasks from unitary tasks; we therefore treat a workflow as a composition of unitary subtasks, each with an identifiable rule for combining contributions. The workflow need not have one global task label. Figure 2 illustrates this decomposition and its design implications; Section 2 connects each subtask’s rule to its aggregation model and scaling limit. Automatic task decomposition and empirical evaluation of the other three task types remain open work.

The formal framework is somewhat disconnected.

To connect the formal framework to the experiments, the revision adds Figure 1: (a, b) applies all five combination rules to the same Round-1 teams and (c, d) plot all three rounds. All panels overlay the predictions of Proposition 2.1. Figure 3 evaluates the predicted curves on the task’s assigned rule across team sizes and revision rounds. The Introduction, Section 2 lead-in, and Discussion now state the same task structure → combination rule → scaling limit link.

We will double check the citation formatting and references in the revised manuscript.

I have read the response. I still think this paper does not really address the interesting case that is prevalent in current agentic tasks.

We thank the reviewer for the follow-up. We take the point to be that current agentic tasks are multi-step workflows, whereas our experiments use single subtasks on static benchmarks. We agree with the description and explain why the framework nevertheless bears on the agentic case.

The reviewer asks for the atomic unit of agentic tasks. We provide it in Steiner's terms as the unitary subtask: a step at which several agents address the same question and their outputs are combined by one rule. Atomicity is always relative to something, and in this framework is relative to the aggregation rule. The same holds in other frameworks: in hierarchical task networks (Erol, Hendler & Nau, 1994) a task is primitive relative to the available operators, in hierarchical task analysis (Annett & Duncan, 1967) relative to a stopping rule, and in crowdsourcing (Kittur et al., 2011) relative to the microtask a platform accepts. We do not dispute that other decompositions exist and will note them in Section 2.

The paper treats each step as a unitary Steiner task and shows that knowing its type has predictive value. Kim et al. (2026) report an empirical capability-saturation threshold that holds also for 13 preliminary mixed-model teams on BrowseComp-Plus. They present the threshold as an observation. In our framework it is a consequence of the combination rule.

The agent architectures in Kim et al. contain such steps, and the limit of a workflow follows from the limits of its steps: a workflow in which every step must succeed cannot exceed its weakest step, a voting step cannot exceed its π however many agents feed it, and an averaging step cannot reduce the error below the bias its agents share. Single-agent runs therefore suffice to identify the limiting step of a complex workflow and the measure that raises its limit. Table 1 and Appendix A the assign the agentic benchmarks, including SWE-bench, OSWorld, TheAgentCompany and BrowseComp-Plus from Kim 2026, to the conjunctive type, whose limit is the weakest step. As a conjunctive task can be composed of several disjunctive/conjunctive/compensatory/discretionary aggregated by the deduplicated set union, etc, using Proposition 2 we can estimate the weakest step from single-agent samples.

To show that the step limits compose, we "simulate" conjunctive workflows from the existing Round-1 experimental data we have: a workflow succeeds only if every step succeeds; each step is an item of a different benchmark, solved by the same team of N agents with the rule of its type; items are paired at random (13 models, N = 1–30). For the chain ARC → MMLU-Hard → GSM8K → GSM-Hard, the chain accuracy predicted from single-agent samples alone (the product of the step predictions of Proposition 2.1) matches the observed chain accuracy to within 0.2 points at every N; the predicted chain limit (the product of the step limits π_j) is reached at N = 30 to within 0.2 points and lies below the weakest-step bound min_j π_j in all 13 models. The weakest step identified from single-agent samples is the weakest observed step in 13 of 13 models. With a compensatory step (ARC → GSM8K → RealFP, geometric mean, success within one order of magnitude), the agreement is within 1.5 points and the weakest step is identified in 12 of 13 models. Since the chain limit is a product, the step whose upgrade raises it most is the one with the largest ratio of oracle limit to plurality limit; this step, predicted from single-agent samples, is the best observed target in 8 of 13 (first chain) and 12 of 13 (second chain) models. Steps are independent by construction, so the analysis tests the step predictions and the identification of the limiting step, not dependence between steps.

We can add to the Limitations section: "The experiments evaluate unitary subtasks on static benchmarks. Predicting the performance of a complete agentic workflow requires decomposing it into typed subtasks and evaluating the conjunctive, additive and discretionary types; we leave both to future work".

Reviewer_Jsq1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when adding more LLM agents improves performance. The authors use Steiner’s taxonomy to analyze voting and averaging, with experiments on independent responses and multi-round discussions. They show how popular wrong answers and shared estimation biases can limit the gains from larger teams.

Critical strengths
  • The distinction between having a correct answer in the team and successfully selecting it is useful and clearly explained.

  • The experiments cover a good range of models, including analysis of temperature and teams made up of different models.

Critical weaknesses
  • I am not quite sure what additional insight comes from Steiner’s taxonomy. The paper discusses related work on voting and scaling, including Chen et al. (2024), but what can this taxonomy explain or predict that the previous analysis cannot? This needs to be clearer.

  • The answer-first prompt makes the discussion results difficult to interpret. Most models answer before giving their reasoning, while later rounds have access to reasoning from earlier responses. Some improvement could therefore come from another chance to think through the question. The paper acknowledges this, but does not include a baseline that separates this effect from discussion with peers.

  • Section 2.4 compares wrong-answer probability with estimation-error size when discussing exponential versus polynomial scaling. These are different quantities. Appendix B.1 also notes that the probability of a large estimation error can decrease exponentially, so the distinction in the main text needs more care.

Questions for authors
  • Could you test reasoning-before-answering prompts and add a single agent that revises its own answer without seeing other responses? Would the conclusions about team size still hold?

  • For the “label-clean” RealFP subset, how do you establish that the reference is wrong when most models disagree with it? Could these instead be mistakes shared by the models?

2 comments
AuthorsOpenReview ↗

New insights from the taxonomy and some additional results

Thank you for the insightful questions, please see also General Response above.

I am not quite sure what additional insight comes from Steiner’s taxonomy. The paper discusses [...], including Chen et al. (2024), but what can this taxonomy explain or predict that the previous analysis cannot? This needs to be clearer.

We have updated Section 1 and Figure 1 to make the explanatory and predictive power of Steiner’s taxonomy explicit. Whereas Chen et al. (2024) and Kim et al. (2026) either analyze a single aggregation mechanism or fit task properties post-hoc from empirical runs, Steiner’s taxonomy provides an a priori structural framework (Task Type -> Combination Rule -> Asymptotic Ceiling, Proposition 2.1; Figure 2) that explains and predicts three phenomena prior analyses cannot.

  1. It unifies discrete selection and continuous error cancellation: Chen et al. (2024) formalize item difficulty strictly for discrete voting, which represents only one category in Steiner’s taxonomy (disjunctive tasks, generalized in our Proposition 2.1) and cannot explain continuous estimation (compensatory tasks), where scaling relies on error cancellation rather than modal selection. By separating these regimes, we derive the compensatory MSE floor and show that systematic item-level bias accounts for 87–97% of squared error across 48 model-benchmark pairs, explaining why homogeneous averaging and debate cap out at ~10% MSE reduction regardless of team size (Figure 1b, d).

  2. It predicts cross-rule mismatch penalties: rather than estimating parallel versus sequential properties from data like Kim et al. (2026), Steiner’s taxonomy prescribes the optimal aggregation rule directly from task structure. Evaluating all five Steiner rules on the exact same agent outputs (new Figure 1a, b; Appendix Figure 6) confirms that the taxonomy-assigned rule consistently dominates, whereas swapping the disjunctive and compensatory rules lowers disjunctive accuracy.

  3. It predicts opposite multi-model dynamics: separating modal selection ceilings (π\pi) from bias floors (Ek[bk2]\mathbb{E}_k[b_k^2]) explains why heterogeneous model pools underperform their single best expert on disjunctive tasks (Table 2) due to vote dillution, yet outperform their best individual model on compensatory Fermi tasks (Table 3, Table 13) where opposite-sign biases cancel out.

Could you test reasoning-before-answering prompts and add a single agent that revises its own answer without seeing other responses? [...]

We added a self-revision control (Tables 6 and 17), we are running additional experiments and update. On two disjunctive benchmarks and two models, Round-3 accuracy with and without peer responses differs by at most 0.6 points; in two-agent teams, 29–40% of the gain comes from corrections neither agent had produced earlier. On the four compensatory benchmarks, peer revision increases error relative to self-revision on three benchmarks and has no significant effect on Guesstimate. After self-revision, averaging 30 agents still reduces error by only 5.2–10.4%, with item-level bias accounting for 80–95% of MSE. These controls support the conclusion that team-size effects remain limited in the tested settings, while the narrow disjunctive control bounds how broadly we can generalize.

We are running reasoning-first prompts and will add. Two models that naturally reason before their formatted answer have lower correctness correlation and larger voting gains, but their gains also largely saturate by N=5N=5. This is observational evidence, not a controlled prompt comparison; the revised paper states this as a limitation. However, there are a number of prior works that show this, including Kim et al (2026), Chen et al. (2024), Wen er at. (2025).

Exponential probability versus MSE.

We revised the theory discussion to compare like quantities. Under ideal assumptions, both voting failure probability and the probability of exceeding a fixed estimation-error threshold can decay exponentially, while expected squared error decreases as 1/N1/N. The important difference in our setting is the limit under item difficulty, bias, and correlation: plurality approaches π\pi, while averaging approaches the shared-bias floor (Section 2; Appendix C).

For the “label-clean” RealFP subset, how do you establish that the reference is wrong when most models disagree with it? Could these instead be mistakes shared by the models?

We do not establish that every excluded reference is wrong. RealFP is known to have issues, we manually checked some facts across encyclopedias and the labels seem wrong. E.g. How many human ancestors do I have? Label 2x10^90, several reputable sources say the number is closer to 10^8. Table 9 adds examples. The trends are similar with or without those references and can be seen on additional compensatory tasks (updated Figure 3), including another Fermi benchmark (ScioLy) in the revised version.

I would like to thank the authors for their responses. I have no further questions and will keep my positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful feedback and recognizing the novelty and usefulness of the proposed framework.

The feedback and questions revealed that our original contribution and findings were not sufficially crystallized in the initial version of the paper. Our contribution is a formal framework that can estimate LLM multi-agent team scaling per task type. The framework assumes unitary tasks and connects Steiner's task taxonomy to aggregation statistics to predict how task structure and its combination rule determine scaling limits and may inform multi-agent LLM design choices.

The revision makes this contribution explicit in the updated Abstract, Introduction and Section 2, adds a same-team comparison of all five combination rules (Figure 1), connects complex workflows to unitary subtasks (Figure 2), and evaluates the predicted scaling curves across rounds (Figure 3). The revised paper also adds three more benchmarks for the compensatory tasks, self-revision controls and clarifies the limits of the RealFP sensitivity subset.

All changes are marked with blue in the paper while reviwers questions are answered below.

The most important textual changes with the crystallized contribution and findings are also pasted here for convenience:

This work. We develop a formal framework that can estimate LLM multi-agent team scaling per task type. The framework assumes the five unitary tasks from Steiner's taxonomy and proposes the opposite approach to the current state of the art. Rather than starting from empirical observation, we propose starting from the task (or benchmark), then map the task to the taxonomy (Table 1). The mapping informs the aggregation rule selection (last column of the table), which then enables deriving the estimated scaling as in Proposition 2.1. The verified scaling (Figure 1) may inform multi-agent LLM system design choices (Figure 2).

Findings. (1) By applying all five rules to the same teams, the rule assigned by the correct task type gives the largest improvement in our tested settings; the other task's rule costs 4.2 accuracy points on disjunctive tasks and forgoes three quarters of compensatory squared-error reduction as in Figure 1(a,b). This finding indicates that studying and developing multi-agent LLM systems may be Steiner unitary task type sensitive. (2) Across 65 disjunctive model-task pairs, plurality approaches the correct-modal share within 0.5 points initially and 1.1 after revision Figure 1(c); self-revision controls separate a second attempt from peer effects. (3) Across 48 compensatory model-benchmark pairs, item-level bias accounts for 87-97% of squared error, limiting averaging gains to 2-7% of the absolute error Figure 1(d); revision leaves this floor nearly unchanged. (4) Mixed-model teams exceed their strongest member only on the two Fermi benchmarks, where member biases differ in sign, indicating directions along which to improve multi-agent LLM performance on Fermi tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.