- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how model and persona composition affect scaling in LLM multi-agent systems. It introduces an evidence-coverage model, where observations recover potentially overlapping evidence components and relates collective information recovery to joint non-coverage rather than individual marginal coverage alone. Empirically, the paper compares combinations of model diversity and persona diversity across seven benchmarks, two aggregation workflows, and multiple agent counts. It further analyzes output trajectories using semantic effective rank and examines the relationship between output diversity and accuracy.
Critical strengths
The empirical results are broad and systematically factorized. The paper jointly varies agent count, model diversity, persona diversity, task, and workflow rather than drawing conclusions from a single MAS configuration.
Critical weaknesses
-
The theoretical contribution appears loosely connected to the empirical measurements. First, the theoretical framework assumes a noise-free reveal-or-erasure observation model (Assumption 2.3) where revealed evidence components are unconditionally correct, explicitly ignoring hallucinations, invalid reasoning steps, persuasive misinformation, sycophancy, and aggregation bottlenecks. Second, the authors acknowledge that their empirical benchmarks do not measure, track, or operationalize the latent evidence variables or coverage masks. Consequently, the theoretical model functions as an abstract mathematical analogy rather than an explanatory or predictive tool for the observed scaling curves.
-
Limited incremental explanatory value of the proposed semantic-diversity diagnostic. While is positively associated with accuracy in aggregate, the paper's own regression sensitivity analysis shows that its incremental explanatory value becomes very small after controlling for task, workflow, and agent count =0.0014 in the strongest reported specification). This suggests that much of the aggregate association between semantic diversity and accuracy is accounted for by differences across tasks, workflows, configurations, and agent counts. Consequently, the current evidence does not convincingly establish that provides a generally useful diagnostic signal beyond these readily observable factors.
Questions for authors
-
How do you control for the confounding factor of prompt quality in L2 and L4? If agents are given variations of a generic system prompt with neutral rephrasings rather than expert role personas, how much of the "persona diversity" gain remains?
-
Can the authors directly operationalize the theoretical notion of joint non-coverage using agent outputs and test whether it predicts the observed gains from heterogeneous composition?
-
Can the authors report paired uncertainty/significance estimates for the key L4-vs-L3 and L4-vs-best-single-backbone comparisons?
-
Do the conclusions persist under matched token/inference budgets?
2 comments
Thank you again for your detailed review. Could you let us know whether our rebuttal addressed your questions and concerns, or whether any points still need clarification?
We thank the reviewer for the detailed feedback on prompt controls, evidence coverage, uncertainty, and evaluation budgets.
Q1. Prompt quality.
At fixed task, workflow, and N, L3 and L4 use the same backbone slots. L1 and L2 compare prompt assignments on each backbone before averaging. These comparisons estimate the effect of our particular role assignments. In Formal Logic at N = 2, Persona replaces one formal-logician instruction with a commonsense-reasoner instruction. The design measures the effect of these specific role assignments and their interaction with the backbone. Neutral paraphrase controls and rotated role assignments would further separate role content from diversity. These controls are outside the current evaluation.
Q2. Joint omission.
The evidence-coverage framework models correctly revealed clues and separates marginal coverage from joint omission. Proposition 3.6 compares shared-propensity dependence with independent coverage at matched marginal rates. The benchmarks record final-answer accuracy. They do not record the theoretical coverage quantities at the agent level. A direct empirical test would require annotated clues, per-agent recovery records, joint omission counts, and separate tracking of false claims. Semantic effective rank describes output geometry; evidence coverage requires per-agent clue annotations.
Q3. Uncertainty and single-backbone comparison.
At N = 16, L4 exceeds L3 by 5.74 percentage points in Vote and 5.44 in Debate on the seven-task average. Gains are positive on all seven Vote tasks and on six Debate tasks, with one tie. Removing each task in turn gives gains of 5.09 to 6.51 percentage points in Vote and 4.73 to 6.34 in Debate. These ranges show that no single task determines the direction of the macro-average difference. They quantify sensitivity to benchmark composition; confidence intervals would quantify sampling uncertainty.
Against the best observed persona-diverse single backbone on Formal Logic, L4 wins at one of five sizes in Vote and four in Debate. On ARC, it wins at one Vote size and none in Debate. These comparators were selected by test accuracy. The resulting comparisons are descriptive and use a test-set oracle; deployment-time superiority would require validation-based selection.
Q4. Matched budgets.
At fixed N within each workflow, agent counts and call schedules are matched. Token usage, inference cost, and latency can still differ. Table 2 also compares different values of N and therefore does not match call counts. Our results characterize accuracy at the specified team sizes and workflows. Equal-budget and equal-cost performance remains unmeasured.
Overall, the empirical finding is that a role assignment can have different effects across backbone compositions and workflows. The evidence-coverage framework provides a theoretical account of why complementary observations may matter, while the experiments measure accuracy under the tested configurations.