acceptodds
Under review as a conference paper at ICLR 2027

Understanding LLM Multi-Agent Scaling: Agent Composition and Evidence Coverage

Abstract

Collective intelligence depends not only on how many agents participate, but also on what each contributes. We examine how model and persona composition shapes the returns from scaling LLM-based multi-agent systems. An evidence-coverage framework shows that equal individual coverage need not yield equal collective recovery: observations can overlap in what they miss. Across seven benchmarks and groups of two to sixteen agents, combining model and persona diversity achieves the highest seven-task macro-average accuracy among the tested configurations at every evaluated agent count in both voting and debate. Persona diversity provides larger average gains in mixed-model than in single-model groups across these settings, although benefits vary across tasks. We further analyze multi-round response trajectories using semantic effective rank to characterize how output diversity varies across agent configurations. Together, these findings highlight agent composition as a key dimension of LLM multi-agent scaling, motivating a focus on the complementary contributions of additional agents rather than their number alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large language models, Multi-agent systems, Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_PgViOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how model and persona composition affect scaling in LLM multi-agent systems. It introduces an evidence-coverage model, where observations recover potentially overlapping evidence components and relates collective information recovery to joint non-coverage rather than individual marginal coverage alone. Empirically, the paper compares combinations of model diversity and persona diversity across seven benchmarks, two aggregation workflows, and multiple agent counts. It further analyzes output trajectories using semantic effective rank and examines the relationship between output diversity and accuracy.

Critical strengths

The empirical results are broad and systematically factorized. The paper jointly varies agent count, model diversity, persona diversity, task, and workflow rather than drawing conclusions from a single MAS configuration.

Critical weaknesses
  1. The theoretical contribution appears loosely connected to the empirical measurements. First, the theoretical framework assumes a noise-free reveal-or-erasure observation model (Assumption 2.3) where revealed evidence components are unconditionally correct, explicitly ignoring hallucinations, invalid reasoning steps, persuasive misinformation, sycophancy, and aggregation bottlenecks. Second, the authors acknowledge that their empirical benchmarks do not measure, track, or operationalize the latent evidence variables or coverage masks. Consequently, the theoretical model functions as an abstract mathematical analogy rather than an explanatory or predictive tool for the observed scaling curves.

  2. Limited incremental explanatory value of the proposed semantic-diversity diagnostic. While K^\hat{K} is positively associated with accuracy in aggregate, the paper's own regression sensitivity analysis shows that its incremental explanatory value becomes very small after controlling for task, workflow, and agent count ΔR2\Delta R^2=0.0014 in the strongest reported specification). This suggests that much of the aggregate association between semantic diversity and accuracy is accounted for by differences across tasks, workflows, configurations, and agent counts. Consequently, the current evidence does not convincingly establish that K^\hat{K} provides a generally useful diagnostic signal beyond these readily observable factors.

Questions for authors
  1. How do you control for the confounding factor of prompt quality in L2 and L4? If agents are given NN variations of a generic system prompt with neutral rephrasings rather than expert role personas, how much of the "persona diversity" gain remains?

  2. Can the authors directly operationalize the theoretical notion of joint non-coverage using agent outputs and test whether it predicts the observed gains from heterogeneous composition?

  3. Can the authors report paired uncertainty/significance estimates for the key L4-vs-L3 and L4-vs-best-single-backbone comparisons?

  4. Do the conclusions persist under matched token/inference budgets?

2 comments

We thank the reviewer for the detailed feedback on prompt controls, evidence coverage, uncertainty, and evaluation budgets.

Q1. Prompt quality.

At fixed task, workflow, and N, L3 and L4 use the same backbone slots. L1 and L2 compare prompt assignments on each backbone before averaging. These comparisons estimate the effect of our particular role assignments. In Formal Logic at N = 2, Persona replaces one formal-logician instruction with a commonsense-reasoner instruction. The design measures the effect of these specific role assignments and their interaction with the backbone. Neutral paraphrase controls and rotated role assignments would further separate role content from diversity. These controls are outside the current evaluation.

Q2. Joint omission.

The evidence-coverage framework models correctly revealed clues and separates marginal coverage from joint omission. Proposition 3.6 compares shared-propensity dependence with independent coverage at matched marginal rates. The benchmarks record final-answer accuracy. They do not record the theoretical coverage quantities at the agent level. A direct empirical test would require annotated clues, per-agent recovery records, joint omission counts, and separate tracking of false claims. Semantic effective rank describes output geometry; evidence coverage requires per-agent clue annotations.

Q3. Uncertainty and single-backbone comparison.

At N = 16, L4 exceeds L3 by 5.74 percentage points in Vote and 5.44 in Debate on the seven-task average. Gains are positive on all seven Vote tasks and on six Debate tasks, with one tie. Removing each task in turn gives gains of 5.09 to 6.51 percentage points in Vote and 4.73 to 6.34 in Debate. These ranges show that no single task determines the direction of the macro-average difference. They quantify sensitivity to benchmark composition; confidence intervals would quantify sampling uncertainty.

Against the best observed persona-diverse single backbone on Formal Logic, L4 wins at one of five sizes in Vote and four in Debate. On ARC, it wins at one Vote size and none in Debate. These comparators were selected by test accuracy. The resulting comparisons are descriptive and use a test-set oracle; deployment-time superiority would require validation-based selection.

Q4. Matched budgets.

At fixed N within each workflow, agent counts and call schedules are matched. Token usage, inference cost, and latency can still differ. Table 2 also compares different values of N and therefore does not match call counts. Our results characterize accuracy at the specified team sizes and workflows. Equal-budget and equal-cost performance remains unmeasured.

Overall, the empirical finding is that a role assignment can have different effects across backbone compositions and workflows. The evidence-coverage framework provides a theoretical account of why complementary observations may matter, while the experiments measure accuracy under the tested configurations.

Thank you again for your detailed review. Could you let us know whether our rebuttal addressed your questions and concerns, or whether any points still need clarification?

Reviewer_VWeDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how multi-agent systems and different personas can improve coverage under two approaches, voting and debate. However, the experimental results are relatively superficial and lack sufficient experiments to provide practical insights for real-world applications.

Critical strengths

The paper's experiments consider multiple model architectures and datasets, with a relatively extensive set of experiments. Meanwhile, under relatively strong assumptions, the paper provides some theoretical results to guide the experiments.

Critical weaknesses

The paper lacks practical guidance for real-world multi-agent systems. For example, how many agents should we choose, what personas should these agents have, and how should we determine their personas? The paper demonstrates a very interesting phenomenon, but fails to show how this phenomenon can help us improve the capabilities of multi-agent systems.

Questions for authors

Regarding the independence assumptions in Assumptions 2.3 and 3.4, it is difficult to believe that different agents are independent if they share the same base model. The authors should consider adding empirical tests of these assumptions on real-world datasets to examine whether the extent to which these assumptions are violated in practice is negligible.

For aggregating outputs from different LLMs, beyond voting and debate, there are some recent methods that account for differences in model capabilities, such as Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information, which would be worth exploring.

2 comments

We thank the reviewer for the constructive suggestions on practical guidance, independence assumptions, and aggregation.

Response to Practical guidance.

We recommend testing role assignments on the intended backbone combination and workflow, rather than transferring results from a single-backbone team. A practical procedure is to compare shared and distinct roles at fixed N on held-out target-task examples, then choose N based on accuracy and cost. This procedure gives task-specific guidance for selecting roles and team sizes.

WinoGrande illustrates why roles should be tested in the intended team. Under Debate at N = 16, the role change reduces the single-backbone mean by 12.16 percentage points but improves the mixed team by 6.35. L4 still trails L1 by 8.99 percentage points. The example also shows why role-change gains should be considered alongside each team's overall accuracy.

Response to Independence.

Assumption 2.3 requires independent latent evidence components and a coverage mask independent of their values. It does not require independent agent misses. Proposition 3.1 allows dependent coverage. Assumption 3.4 introduces independent coverage across channels for a component, yielding Theorem 3.5. Proposition 3.6 compares this case with shared-propensity dependence at matched marginal rates.

Two channels with 50% marginal coverage recover a clue 75% of the time under independent misses, compared with 50% under perfectly shared misses. This comparison applies to the specified dependence models. Other dependence structures may behave differently, and model heterogeneity alone gives no guarantee of independence. Independence is therefore a condition used for Theorem 3.5, not a measured property of real backbone pairs.

Response to Aggregation.

Ai et al., Beyond Majority Voting, consider aggregation methods that use model quality and dependence. We vary team composition while keeping Vote or Debate fixed. Reaggregating the same response panel would help separate composition effects from aggregation effects. Our results characterize team composition under Vote and Debate.

Thank you for clarifying your concern about the connection between the coverage model and the experiments. Our analysis allows agents using the same backbone to miss the same evidence. Proposition 3.1 expresses residual uncertainty in terms of the probability that all channels miss a given component, allowing dependence across channels. Proposition 3.6 then shows that, even at the same individual coverage rates, a shared coverage propensity can increase joint misses relative to the independent case in Assumption 3.4. This establishes how dependence can limit evidence recovery in the model. Please let us know if this helps resolve your concern.

Reviewer_uDyeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how model diversity and persona diversity affect the performance of LLM-based multi-agent systems. The authors compare four agent configurations: homogeneous teams in which agents share the same model and persona, teams that vary only in persona, teams that vary only in model, and teams that vary in both. They evaluate these configurations across seven tasks, two collaboration strategies, voting and debate, and team sizes ranging from 2 to 16 agents. The results show that teams with both model and persona diversity achieve the highest macro-average accuracy among the tested configurations, although the effect of diversity varies substantially across tasks and settings. To explain why heterogeneous teams may help, the paper introduces an evidence-coverage framework. It models each agent as having a chance to reveal useful evidence and analyzes how overlap in agents’ evidence coverage affects the amount of evidence collectively recovered by a team.

Critical strengths
  1. The paper studies agent composition across a broad set of conditions, including seven tasks, two collaboration strategies, and multiple team sizes. Its comparison of model diversity and persona diversity helps clarify how these two sources of heterogeneity relate to multi-agent performance.
  2. The paper also reports task-level variation, rather than relying only on aggregate results, which makes the findings more informative.
Critical weaknesses
  1. The empirical contribution is not clearly distinguished from prior work. The main finding, that heterogeneous agent teams can outperform homogeneous teams, seems intuitive. The paper should more clearly explain what its systematic comparison reveals beyond this general observation and how that insight differs from prior studies of diversity in multi-agent systems.
  2. The experiments do not directly validate the evidence-coverage framework. The framework offers a possible explanation for why diverse agents may help, but the experiments do not measure which evidence each agent discovers or whether teams with better evidence coverage perform better. It therefore remains unclear whether the framework explains the reported results or serves mainly as a post hoc interpretation.
  3. The experiments do not isolate the effect of diversity itself. Changing the models or personas may also change agents’ underlying capabilities, prompts, and response styles. As a result, performance differences between team configurations cannot be attributed to diversity alone. More controlled comparisons would help clarify which factors drive the observed gains.
  4. The benefits of diversity are inconsistent across tasks and settings. Although teams with both model and persona diversity achieve the highest macro-average accuracy, the gains vary substantially across tasks, and some configurations perform worse than homogeneous teams. The paper could better analyze the conditions under which diversity helps or hurts, rather than relying primarily on aggregate results.
  5. The scope of the evaluation limits the generality of the conclusions. The study covers several tasks, workflows, and team sizes, but it remains unclear whether the findings extend to other models, tasks, or multi-agent interaction designs. Evaluation on a broader range of models and settings would help establish how widely the conclusions apply.
  6. The comparison with the “best single-backbone” baseline may be optimistically biased. For ARC and Formal Logic, the paper states that it selects the best single-backbone result using test accuracy. Selecting the strongest baseline on the same test set used for comparison can favor that baseline by chance and makes the comparison difficult to interpret. Please use a validation set to select the baseline, or report uncertainty that accounts for this selection.
  7. The reported associations between semantic diversity and accuracy may be overstated by the statistical analysis. The permutation test shuffles diversity values across all 220 observations, which pool results from multiple tasks, configurations, and workflows. These observations may not be independent: several share the same task or experimental setting. The reported p<0.001p<0.001 may therefore reflect the pooled structure as well as the diversity–accuracy association. A task- or setting-aware analysis would make the statistical evidence easier to assess.
  8. The authors should repeat runs within each task, workflow, and agent-count setting while varying the agent composition. They should report whether changes in semantic diversity are consistently associated with changes in accuracy across runs.
Questions for authors
  1. In Formal Logic, the shared-prompt configuration achieves only 0.79%–6.35% accuracy, while the persona-diverse configuration reaches 34.92%–54.76%, a gain of approximately 40–41 percentage points. Could the authors provide the prompts and explain the answer-extraction and scoring procedures used to obtain these results? Have they replicated this large gain across independent runs, and could output formatting or answer parsing affect it?
  2. Could the authors quantify uncertainty in the main results? Several test sets contain only 126 questions, and some comparisons show small accuracy differences. For example, in Table 2, the macro-average for L4 with 2 agents is less than one percentage point higher than that for L1 with 16 agents. Did the authors conduct paired tests or bootstrap analyses, and how much variation arises from random tie-breaking in Debate or model-call variability? Which main comparisons remain robust when uncertainty is taken into account?
  3. Does the permutation test account for dependence among observations? Table 6 reports �0��0� after unrestricted permutation across 220 pooled observations. Since multiple observations come from the same task, configuration, or workflow, they may not be independent. Do the reported correlations and significance levels remain stable under task-aware permutations or clustered bootstrap analyses?
  4. How should the authors interpret the weak incremental association between semantic diversity and accuracy after controlling for task and workflow? The reported increase in �1��1� from adding �2��2� drops from 0.143 in the basic regression to 0.0062 after controlling for task and workflow, and to 0.0014 when agent count is treated categorically. Does this indicate that semantic diversity provides little additional explanatory information beyond the experimental setting? Could the authors clarify which conclusion about the relationship between diversity and accuracy is supported by these results?
6 comments

We thank the reviewer for the careful reading and detailed questions. They help clarify both the contribution of the comparison and the interpretation of our results.

Contribution. Prior work has shown that heterogeneous teams can improve performance. Our study asks a more specific question: how does agent composition affect the returns from scaling? We develop an evidence-coverage framework showing how joint omissions limit collective recovery even at matched individual coverage rates. Our systematic 2×2 evaluation examines model and persona diversity across seven tasks, two workflows, and five team sizes. Beyond L4's consistent macro-average lead, we find that persona gains depend on backbone composition and can even reverse direction. We further analyze response trajectories using semantic effective rank, examining its relationship with accuracy under different controls.

Q1. Formal Logic prompts and scoring.

The large Formal Logic gain deserves particular scrutiny because changing a role prompt can affect both reasoning and whether a usable answer is produced. We provide the five retained role templates in the accompanying disclosure. At N = 2, Base uses two formal logicians, while Persona pairs a formal logician with a commonsense reasoner. Both use the same answer format and scoring rule. The scorer extracts the final brace-delimited choice, applies majority voting over valid answers, and resolves ties uniformly. Questions without valid votes are scored incorrect. The historical run used 126 questions with seed 42.

Our audit identified an important factor in interpreting the result. In a retained 50-question fragment of the gpt-5-mini Persona run, the formal-logician agent produced no text, while the commonsense agent responded to 24 questions. The scorer correctly reproduced the recorded extractions. This shows that answer availability must be considered alongside reasoning accuracy when interpreting the large Base–Persona gap. The fragment cannot determine the contribution of each factor without corresponding Base outputs. We will clarify this distinction in Appendix D.1. The closed-source experiments are separate from our main seven-task evaluation.

Q2. Uncertainty and backbone selection.

The key question is whether L4's advantage reflects a consistent composition effect or a few favorable benchmarks. The fixed-size comparisons provide useful evidence here. At N = 16, L4 exceeds L3 by 5.74 percentage points in Vote and 5.44 in Debate. These gains occur across all seven Vote tasks and six Debate tasks, with one tie. They also remain positive when any single benchmark is removed. This pattern suggests that the macro-average advantage is distributed across tasks.

The two-agent comparison offers a complementary perspective on how team composition affects performance at different agent counts. L4 at N = 2 reaches the L1 sixteen-agent macro-average in both workflows, although its smaller margins are sensitive to benchmark selection. We interpret this as motivation to evaluate smaller heterogeneous teams, while recognizing that task-omission analysis does not replace paired confidence intervals or independent-run estimates, which the retained aggregates cannot provide.

For ARC and Formal Logic, the best single backbone was selected using test accuracy. We will explicitly identify this as a test-set oracle so that its role as a retrospective comparator is clear.

Q3. Dependence in the permutation test.

We conducted an additional sensitivity analysis on the same 220 observations, centering rank and accuracy within each task, workflow, and agent count, followed by task-aware configuration-profile permutations.

The positive association persists within matched settings. Total effective rank yields r=0.432r=0.432, p=0.0262p=0.0262, while correct-answer rank and the correct/incorrect rank ratio yield r=0.517r=0.517, p=0.0009p=0.0009, and r=0.416r=0.416, p=0.0019p=0.0019, respectively, using 9,999 permutations. These results show that the rank–accuracy association extends beyond differences between tasks and workflows. The inference remains conditional on configuration-profile exchangeability.

Q4. Incremental value of effective rank.

We agree that effective rank provides limited additional explanatory power once the experimental setting is known. Its incremental R2R^2 decreases from 0.143 in the basic regression to 0.0062 with task and workflow controls, and to 0.0014 with categorical agent count.

However, effective rank serves a different analytical purpose. Accuracy measures whether a team succeeds, while effective rank characterizes the semantic variation among its responses. For example, on ARC, L2 exhibits higher effective rank than L3 but lower accuracy. This distinction helps reveal how agent composition changes response diversity without assuming that greater diversity necessarily improves performance. We therefore use effective rank to analyze output behavior, with its predictive limitations made explicit.

Thank you for taking the time to respond to my review and for providing the additional analyses. I appreciate the clarifications. However, the rebuttal does not systematically address each of the critical weaknesses raised in my review. While it responds to the four numbered questions, several concerns remain unanswered or are addressed only at a general level, including the empirical validation of the evidence-coverage framework, the confounding between diversity and differences in model capability or persona content, and the repeated runs. I therefore consider the rebuttal incomplete and stick to my original score.

  1. Empirical contribution and novelty — partially addressed. The response clarifies the paper’s focus on how agent composition affects scaling and highlights the interaction between model and persona diversity. However, it does not clearly distinguish these findings from prior work on heterogeneous multi-agent systems. Restating the contributions can not showcase the novelty.

  2. Empirical validation of the evidence-coverage framework — not addressed. The response explains the framework’s intended role, but the experiments still do not measure evidence coverage or test whether it predicts the observed performance differences.

  3. Attribution of the effect to diversity — not addressed. The response does not provide controlled comparisons that separate diversity from differences in model capability, persona content, or response style.

  4. Repeated runs —not addressed. The task-omission analysis does not show whether the results are stable across independent runs. The authors acknowledge that the retained aggregate results cannot provide confidence intervals or estimates of run-to-run variation, but report no new repeated runs. Concern remains unaddressed.

Reviewer_h7inOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how agent composition affects multi-agent scaling. Its theory shows that correlated blind spots reduce collective evidence recovery even at matched individual coverage rates. Experiments cross model and persona diversity across seven benchmarks, two workflows, and 2–16 agents. Combining both diversity sources leads the macro-average at every tested agent count, improving over the homogeneous baseline by 7.8 points in voting and 9.9 in debate at 16 agents.

Critical strengths

The 2×2 design directly addresses the research question, with matched call counts within each workflow. The finding that two diverse agents match or exceed the mean homogeneous 16-agent baseline is practically interesting, although it does not establish equivalent token or latency savings.

The paper is appropriately candid about its limitations. It explicitly acknowledges that the experiments do not measure the theoretical coverage quantities and reports that semantic diversity’s incremental association with accuracy largely disappears after controlling for task, workflow, and agent count.

Critical weaknesses

The main concern is statistical reliability. Results generally lack uncertainty estimates or clear evidence of repeated independent runs. With benchmark subsets containing only a few hundred examples, it is difficult to assess which reported differences are robust.

The theory also remains disconnected from the experiments. Neither coverage nor joint omission is estimated, so the proposed explanation is not directly tested. The results demonstrate configuration effects but do not establish evidence complementarity as their mechanism.

Questions for authors

Can the authors provide paired confidence intervals and repeated-run results? Can they directly measure evidence coverage and joint omissions on a controlled task to test the proposed mechanism?

2 comments

We thank the reviewer for the careful assessment, especially the questions about statistical reliability and the relationship between the theory and experiments.

Response toStatistical reliability.

We examined whether any single benchmark drives the macro-average results. At each workflow and agent count, L4 ranks first among the four configurations on the full seven-task average. It remains first in 69 of 70 comparisons when one task is removed. The exception is Debate at N = 2 without HellaSwag, where L3 reaches 66.96% accuracy and L4 reaches 66.72%.

The comparison between two and sixteen agents is more sensitive:

L4 (N = 2) minus L1 (N = 16)7-task averageRange after removing one task
Vote+0.48 %-1.59 to +1.88 %
Debate+1.58 %-0.35 to +5.73 %

These checks show that L4's fixed-size ranking is stable to removing one benchmark, while the two-agent threshold is sensitive to task composition. We therefore describe the latter as reaching the L1 (N = 16) accuracy on this seven-task suite, rather than as evidence that two agents generally replace sixteen. The omission ranges assess benchmark sensitivity, not statistical uncertainty. The retained results are aggregate accuracies, so they do not provide paired confidence intervals or run-to-run variance estimates.

Response to Theory and evidence.

Proposition 3.6 compares independent coverage with a specific shared-propensity model at matched marginal rates. If two channels each recover a clue half the time, independent misses yield 75% team coverage, while perfectly shared misses yield 50%. This is a theoretical result about evidence recovery. Our experiments measure final-answer accuracy, not per-agent evidence recovery. We therefore present evidence coverage as a theoretical account that motivates the composition study, rather than as an empirically verified mediator of the observed accuracy gains.

Thank you again for your time and thoughtful review. We just wanted to check in and see if our response addressed your questions on benchmark sensitivity and the scope of our theoretical model. Please let us know if you have any further questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.