acceptodds
Under review as a conference paper at ICLR 2027

Beyond Symmetric Agents: Cognitive Diversity and Multiagent Debate in Small Language Models

Abstract

Multi-agent debate (MAD) reportedly improves reasoning and factuality over single-model inference, but prior work treats agents as symmetric peers, leaving open what drives the gains. We test the hypothesis that cognitive diversity among agents is the driver, in the setting where the question is still measurable: small open-weight models with benchmark headroom. Across 23 models from nine vendor families, five tasks, and 5,500+ debate and control runs, we vary diversity along three axes (personas, sampling temperature, and model identity) pairing every debate configuration with a generation-budget-matched majority-vote control. The hypothesis is rejected on every axis. Debate beats single-agent inference (3–7 points where tasks have headroom) but at matched budget conditions it ties or even loses to self-consistency sampling at 1.6 the wall-clock and 3.4 the token cost. Persona prompting reduces accuracy and a dose-response experiment over each model's full combinatorial persona space shows the cost is a persona tax, not a diversity tax: redundant personas hurt most, while maximally-diverse teams recover part of the loss. Furthermore, mixed-model teams lose to majority votes over their own rosters, with accuracy tracking member capability rather than heterogeneity, and nearly all of debate's benefit comes from the first exchange of answers. We further identify a pervasive measurement hazard in which debate transcripts silently overflow serving context windows, whose correction alone moves our debate-versus-sampling comparison from points to parity. Our results recast reported MAD gains as an ensemble-sampling effect and provide the budget-matched, contamination-checked baseline bar that future debate mechanisms should be required to clear.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent debate, multi-agent systems, LLM agents, self-consistency, test-time compute, ensemble methods, majority voting, persona prompting, cognitive diversity, small language models, open-weight models, LLM evaluation, reasoning, benchmark evaluation, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fUUMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks the question of why mult-agent debate (MAD) sometimes beats a single model and if agent diversity is key to making it work. The authors tested on 23 open weight models and 5 tasks, varying the diversity of the agents in three ways -- assigned persons, temperature, tempuerture and different models in a team. They compare each case to a control and find 1. MAD tends to beat single models but ties or loses to the majority voting and costs more 2. Reduce debt personas reduce accuracy. 3. Temperature diversity has no effect 4. Mixed models are slightly better than homgenousl model s and 6. almost all the impoirvement comes from the first dexchange in the debate.

Critical strengths

The authors do a rigorous job matching the debate arm with the control arm -- this makes the results solid

The persona dose-response ladder is clever and helps better disentagle "personas hurt" vs "diversity hurts"

Critical weaknesses

The definition of "cognitive diversity" is very narrow and as a result the key claim of the paper supports is more narrow. All the persona teams came from the extreme tier, which often include task irrelevant characters.

Null results are reported as equivalence but there are no statistics to demonstrate this.

Questions for authors

Can you report results with the moderate persona tier, or with task-relevant personas such as different expert roles or reasoning strategies?

Are the Qwen3.5-27B anomalies in Table 17 explained by context overflow or another serving issue?

1 comment
AuthorsOpenReview ↗

Response to Reviewer fUUM (rating 2, constructive)

We thank the reviewer for a review that improved the paper twice over; both critiques led to valuable additions to our paper.

W1/Q1. "Cognitive diversity" is narrow: all persona teams came from the extreme tier, often task-irrelevant. Can you report moderate-tier or task-relevant personas (expert roles, reasoning strategies)?

Yes; this was the main experiment we ran during the discussion period. Compute limitations meant delays we were not anticipating but we did run a new arm: a 50-persona task-relevant pool, half expert roles (research mathematician, biographer, archivist, fact-checker, examiner…), half reasoning strategies (verification specialist, decomposition strategist, back-substitution checker, process-of-elimination reasoner…), selected into teams with the identical embedding → MaxDet machinery, run on 12 tier-stratified models × 5 tasks × n ∈ {3,5,7} against the paper's existing budget-matched debate and sampling controls.

The short answer to the reviewer's question is yes, persona content matters, and we were able to quantify by how much: task-relevant teams beat extreme teams by +1.48pp (paired, 95% CI [+0.52, +2.49]); the extreme pool's significant costs decrease notably (GSM −1.72pp → −0.44pp n.s.; CSQA −2.53pp → −0.08pp n.s.) and math turns significantly positive (+3.67pp [+1.31, +6.44]). On the judged factuality task the picture reverses: task-relevant personas REDUCE claim precision among verifiable claims (−6.8pp [−9.2, −4.5] vs no-persona; −4.9pp vs the extreme pool, 36 matched cells, two-judge panel) and even biographer/fact-checker personas make biographies less precise. The persona tax is therefore partly a property of the extreme pool on scored tasks, task-dependent in sign, and we reviewed/rephrased the paper's claim accordingly in the uploaded revision. The core finding is unchanged at its strongest test: even the most task-relevant pool gives no significant net benefit over no-persona debate (+0.71pp [−0.41, +1.89]), and its round-1 gain (+1.91pp [+0.24, +3.78]) is eroded by subsequent debate, which we consider to be further evidence that the debate stage does ensemble-aggregation work rather than deliberative refinement. Perhaps more decisively for the paper's headline: task-relevant-persona debate remains statistically indistinguishable from budget-matched sampling (vs uniform SC −0.57pp [−2.16, +1.13]; vs diverse SC +0.51pp [−1.43, +2.64]) and even the best persona pool does not make debate beat the same budget spent on sampling.

W2. Null results are reported as equivalence without statistics.

The reviewer is correct, and we adjusted/fixed our results: every "≈" claim now carries a two-one-sided-tests (TOST) result with ±1pp equivalence bounds on the paired-by-problem deltas (pooled over 320 model × task × budget cells, overflow-bound cells excluded, cluster bootstrap over cells). We find that:

  • Debate vs budget-matched temperature-diverse sampling: pooled −0.42pp, 90% CI [−0.87, +0.04] → formally equivalent at ±1pp. This is the paper's headline null hypothesis, and it is now a proper statistical statement.
  • Debate vs uniform sampling at matched budget: −0.65pp, 95% CI [−1.11, −0.20] not equivalent; debate is statistically worse than even uniform sampling here, which strengthens rather than weakens the paper's conclusion.
  • Mixed-model debate vs own-roster ensemble on MMLU: −0.01pp, 95% CI [−0.90, +0.85] → equivalent at ±1pp (the other three tasks show significant losses, as previously reported).

The claims whose CIs do not fit the bounds are restated as "not distinguishable" rather than "equivalent."

Q2. Are the Qwen3.5-27B anomalies in Table 17 explained by context overflow or another serving issue?

The reviewer's suspicions were spot on. It is context overflow, and we can show it explicitly. At the model's 16K serving window, its 7-agent biography/GSM cells end with up to 100% empty final-round responses (the wrapper returns empty on overflow rather than crashing); accuracy collapses accordingly. When we re-ran the same experiments at the 131K window, the same cells recovered; math 0.638 → 1.000, final-round-empty 43% → 0%. Table 17's anomalous cells are exactly the window-bound ones and the revision marks them and promotes the per-window curve into the main text (Figure 3, Section 9). Independently, our replication-from-public-release reproduced the same cells overflowing and the anomaly is a deterministic property of the configuration and not a serving accident.

We believe both of your substantive requests are now delivered: the task-relevant persona experiment and formal equivalence statistics for every null claim. If these resolve W1 and W2, we would be grateful if you would consider raising your score to acceptance.

Reviewer_q9QCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the role of cognitive diversity in MAD, considering three dimensions: persona, sampling temperature, and model identity. They evaluate 23 open-weight models across five tasks. The results suggest that MAD does not consistently outperform independent sampling, and that persona prompting may reduce performance.

Critical strengths
  • The paper studies a relevant question about the role of agent diversity in multi-agent debate.

  • The authors conduct experiments across multiple models and settings.

Critical weaknesses
  • The main concern is that several key findings have already been extensively studied. Prior work [1, 2, 3] has shown that MAD often fails to outperform self-consistency or majority voting under comparable inference budgets. Furthermore, the roles of personas, model heterogeneity, and diversity in MAD have also been investigated [1, 3, 4]. The paper does not clearly establish sufficiently novel insights beyond these existing studies.

  • Each automatically scored task uses only 100 questions (which is not representative enough; for example, GSM8K has more than 8k tasks, but the authors only use 100), and the biography task contains only 40 subjects. The arithmetic task is also insufficiently described, and several tasks exhibit high baseline accuracy. These settings limit the strength of the paper's broad conclusions about MAD and cognitive diversity.

[1] Should We Be Going MAD? A Look at Multi-Agent Debate Strategies for LLMs. ICML 2024.

[2] Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? NeurIPS 2025.

[3] Stop Overvaluing Multi-Agent Debate — We Must Rethink Evaluation and Embrace Model Heterogeneity.

[4] Demystifying Multi-Agent Debate: The Role of Confidence and Diversity. ACL 2026.

Questions for authors

I leave the other three weaknesses here:

  • The paper treats persona prompting, sampling temperature, and model identity as three dimensions of cognitive diversity, but provides limited justification for why these factors meaningfully capture differences in reasoning processes. In particular, sampling temperature primarily controls decoding randomness rather than necessarily inducing different cognitive strategies.

  • The persona experiments rely on highly unconventional roles, such as a pre-lingual infant or a Zen master communicating through koans, which may be inherently unsuitable for the evaluated reasoning tasks. The observed performance degradation is therefore not particularly surprising and may not generalize to more meaningful, task-relevant role specialization.

  • The context-window overflow analysis identifies an engineering implementation issue, but ensuring sufficient context capacity is primarily an experimental validity concern rather than a novel scientific insight into MAD.

2 comments
AuthorsOpenReview ↗

Response to Reviewer q9QC (rating 1, novelty) - 1/2

We thank the reviewer for being direct and for mentioning the four references, all of which we engage below. We respectfully believe the novelty assessment conflates our starting point (which is indeed established, and which we cite) with our contributions (which no cited work contains or resolves, and which in one case contradict a cited conclusion).

"Prior work [1,2,3] has shown that MAD often fails to outperform self-consistency or majority voting under comparable inference budgets."

Yes, agreed, and we say it so in the paper. That is our premise and not our claim. But "comparable budgets" does more work in this sentence than the cited papers support:

  • Smit et al. [1] benchmark protocol families on GPT-3.5 (API) across seven multiple-choice sets with accuracy-vs-dollar scatter without generation-matched control per debate configuration. Their own "Limitations" section call for exactly what we did in our study: "future works could extend this line of work using open-source models and in-house infrastructure."

  • Choi et al. [2] compare majority voting over N round-0 answers with debate at N(T+1) generations, showing that the cheaper baseline ties with debate. Our self-consistency controls match debate's budget upward (k = N × rounds samples, uniform and temperature-diverse variants, per cell). Only the matched-up design supports our actual claim, that the same budget spent on sampling does the same work, and only it exposes the task dependence (debate +4pp on math/GSM, −3pp on MMLU/CSQA at matched budget), which no cited work reports.

  • Zhang et al. [3] is a position paper (4 models, two proprietary, ~6-call convention) whose footnote expressly declines technical depth.

On model heterogeneity, our result contradicts [3]. Their "heterogeneity is a universal antidote" conclusion compares mixed-model debate against homogeneous debate only. We ran the control they omitted, mixed team's own generation-matched majority vote, across 230 designed, tier-stratified teams: mixed debate loses to its own roster's vote, −1.04pp, 95% CI [−1.50, −0.61] (team-level bootstrap, 10k resamples), while winning only 83/320 paired cells. We contend that a controlled result that reverses a cited paper's conclusion cannot be redundant with it.

"The roles of personas … and diversity in MAD have also been investigated [1,3,4]."

  • In [1], "Multi-Persona" is a fixed two-agent angel/devil protocol, not a diversity axis that we implemented.
  • In [2] (not [1]/[3]), personas appear in exactly one table: one model, two tasks, hand-picked roles, described by the authors as "suggesting the potential benefit." We consider this a mere "hint", and not a proper and meticulous investigation.
  • In [4], "diversity" is answer-pool diversity, measured by the number of distinct initial answers, and the authors explicitly set persona prompting aside as not directly increasing it. Their "Limitations" section states their experiments "focus on homogeneous agents" and that heterogeneous models "may introduce additional dynamics not captured in this work," and call for open-ended tasks. Both taskas are sections of our paper.

We can confidently say that no cited work contains a persona pool with embedding-geometry team construction; a dose-response ladder separating the persona tax (cost of persona text, −3.9pp at round 1) from a diversity tax (none: diverse teams recover, redundant teams do not); 23 open models across 9 vendor families and 0.6B–109B; a judged open-ended factuality task with a vendor-disjoint multi-judge panel and a judge-coupling ablation; a contamination check (GSM-Symbolic: +0.78pp, no signal) and a headroom measurement (MMLU-Pro: −15.1pp below MMLU for 23/23 models).

AuthorsOpenReview ↗

Response to Reviewer q9QC (rating 1, novelty) - 2/2

"Sampling temperature primarily controls decoding randomness rather than inducing different cognitive strategies."

We agree and our paper actually treats temperature diversity as the standard practice to be audited, not as a cognitive mechanism: temperature-diverse teams are the default in widely used MAD implementations, so a study of what drives debate gains must include them as a control arm. Our null on this axis (pooled −0.30pp after overflow exclusion) is the check of exactly what practitioners do.

"Unconventional personas … degradation is unsurprising and may not generalize to task-relevant role specialization."

This is the one point where new data was needed and we ran additional experiments during the discussion period (global comment, point 1): a 50-persona task-relevant pool (expert roles + reasoning strategies), identical selection protocol, 12 models × 5 tasks × 3 team sizes, against the existing budget-matched controls. The reviewer's intuition is partly confirmed and now quantified: task-relevant personas outperform the extreme pool by +1.48pp (paired, 95% CI [+0.52, +2.49]) and remove its significant per-task costs (GSM −1.72pp → −0.44pp n.s.; CSQA −2.53pp → −0.08pp n.s.; math turns significantly positive, +3.67pp). However, the paper's substantive conclusion survives this most-favorable test: even task-relevant personas give no significant net improvement over no-persona debate (final +0.71pp [−0.41, +1.89]), their round-1 benefit (+1.91pp) is decreased rather than amplified by debate, and no persona configuration closes the debate-vs-matched-sampling gap. On the judged factuality task the reviewer's generalization doubt is in fact answered in the other direction: even biographer/fact-checker personas reduce claim precision (−6.8pp [−9.2, −4.5] vs no-persona, 36 matched cells) — persona effects are content- and task-dependent, and in no configuration a net benefit. The revision restates the persona claim as content-bounded accordingly.

"Each task uses only 100 questions … settings limit the strength of the broad conclusions."

We can address this concern with four responses. (i) Scale sits in the number of controlled cells and not in the per-cell question count: 5,500+ runs, every cell carrying a 95% bootstrap CI, and a second-seed slice replicating the homogeneous grid at r=0.978. (ii) The headline comparisons are paired by problem, so CI widths, which are reported for every headline, already deal with the 100-question draw. (iii) The saturation concern cuts against the cited priors, and not our paper: [2]/[3] evaluate models at ≥0.90 on GSM8K; our pool was chosen to allow for headroom and our MMLU-Pro slice quantifies it. Biography uses 40 subjects because that is the complete Du et al. (2023) dataset, kept for comparability with the work we re-examine. (iv) Assuming a reasonable level of uniformity across all datasets/benchmarks, answering 100 or 500 problems shouldn't change the expected accuracy much, but only the uncertainty around it, which we measure and compare across models with the bootstrapping experiments.

"The context-window overflow analysis … is primarily an experimental validity concern rather than a novel scientific insight."

The overflow finding is indeed a measurement hazard in exactly this genre of study, and its correction is large enough to flip conclusions: our debate-vs-sampling comparison moves from −1.8pp to parity when the artifact is removed. The cited papers run 5-round debates on 7–8B models ([2]) and multi-call protocols ([3]) without reporting window audits. If we found that the hazard flips signs in our data, prior comparisons inherit the burden of showing they were not affected. We consider establishing, measuring, and correcting a conclusion-inverting artifact a contribution of the same kind the field now expects for data contamination, and our replication-from-release reproduced the artifact and its correction end-to-end.

We genuinely value your critique and believe it has contributed to our work. We also naturally recognize that the burden of demonstrating novelty rests on us. We believe the budget-matched controls absent from [1–4], the reversal of [3]’s heterogeneity conclusion under the control it did not even include, the measured overflow correction, and the new task-relevant persona experiment run during this discussion together show that the paper’s contributions are not contained in the cited work, two of which explicitly call for this study in their own limitations. If this response changes your assessment, we would be grateful if you would consider revising your score to acceptance.

Reviewer_dPwtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures how diversity can affect the quality of the results in multi-agent debate systems. In short, a task (mathematical reasoning, general and commonsense knowledge, factual generation) can be accomplished by resorting to multiple agents interacting and confronting among each other. The authors instantiate a protocol where agents exhibit diversity, in terms of personas (i.e., system prompts), constraints on the output probabilities, and underlying model. The authors implement an experimental protocol and compare the result-s with two baselines: 1) homogeneous teams; 2) ensembles of single-unit teams. The results are essentially negative: diversity does not seem to provide a competitive advantage.

Critical strengths

The experimental protocol is extremely curated and several different configurations are compared against baselines and control experiments.

Critical weaknesses
  1. Readability is low in several key aspects. Too many concepts seem overly compressed, thus harming the readability. I’ll give some examples in the next section
  2. The results hold for small language models. Since context window can play a significant role, the significance of the results could be strengthened by measuring how different context capacities influence the results.
Questions for authors
  1. Model diversity and diversity metric are not very well explained. For model diversity, the composition of the experiments is not clear. For persona diversity, my understanding is that it depends on the embedding of the system prompts. The mathematical details are not clearly expressed and are distributed all over the paper.
  2. What does figure 1 represent on the y axis? The text mentions an unspecified “cost accuracy” that is never defined nor explained. (I would guess it represents the loss/gain in accuracy).
  3. Tables 4 and 5 are not readable in the format they are. Again, I was able to grasp the general idea but the details are lost in the very dense text containing concepts and parameters that are not explained. Please explain all the components in these tables.
  4. What is r in the experiments?
  5. Maximal diversity in figure 2 does not seem to show a clear tendency towards a reduction of the gap (at least, not in all the tasks). What are the confidence intervals here?
  6. The concept of capacity tax is not really explained.
  7. As far as I understand, the number of rounds is fixed to 3. How does changing this parameter affects the results?
1 comment
AuthorsOpenReview ↗

Response to Reviewer dPwt (rating 2, presentation-focused)

We thank the reviewer for the detailed review for calling our protocol "extremely curated." All of your seven questions have been addressed below. We concede that five were presentation defects on our side (fixed), and the other two have been answered by experiments already in the paper's appendix that we failed to mention in the main 9-page body of the article. Objectively, point by point:

W1/Q1. Model diversity composition unclear; persona diversity mathematics distributed all over the paper.

Fixed in revision §3.3 (Diversity Metric, blue text): one subsection now states the full pipeline; each persona is embedded by the participant model itself (mean-pooled final-layer hidden state of "You are {persona}.") with pairwise cosine distances form D, and teams are exact brute-force maximizers of det(S) (MaxDet, volume = spread) or min pairwise distance (MaxMin) over all C(50, n) subsets. The "mixed-model" team composition (tier stratification, class constraints, balanced marginals) is now stated as a fixed four-step protocol in §3.2 (blue text).

Q2. Figure 1's "cost accuracy" is never defined.

Our mistake. The y-axis is accuracy and the x-axis is generation cost (total completions \times mean completion length); the caption now defines both concepts. "Cost-accuracy" as a compound term has been removed from the text.

Q3. Tables 4 and 5 are not readable.

We rebuilt them and split them into per-axis panels, with parameters moved to the caption, abbreviations expanded, and each column keyed to the section that is related to it.

Q4. What is r?

Rounds. The term has now been defined at first use and in every caption that uses it.

Q5. Figure 2: does maximal diversity reduce the gap? Confidence intervals?

The reviewer is correct in that the original per-cell intervals were too wide to support the trend visually. The current revision replaces Figure 2 with the statistically correct plot. Since both arms answer identical seeded problems, we plot the paired per-problem delta (persona rung vs no-persona debate), pooled across the 19 models per rung, with cluster bootstrap CIs (10,000 resamples). The trend is now testable and it is statistically significant: the persona cost shrinks by about +0.17pp per diversity decile, 95% CI [+0.06, +0.27] (cluster bootstrap over models), from −4.3pp [−6.6, −2.2] at the most-redundant rung to −1.7pp [−3.1, −0.4] at the campaign MaxDet team, meaning that diversity recovers part of the cost, but every rung stays significantly below the no-persona baseline. (MMLU cells enter unpaired because its question draws differ across fetches which is a documented data property; all other tasks are paired per problem.) The new task-relevant persona arm (global comment, point 1) answers whether this residual cost is a property of personas per se or of the extreme pool. We find it is content-dependent; task-relevant personas erase the cost (+1.48pp over extreme, paired CI [+0.52, +2.49]) without conferring a significant net benefit (+0.71pp [−0.41, +1.89]).

Q6. "Capacity tax" not explained.

We renamed the concept and defined it properly: the accuracy cost of adding personas, measured at round 1 (before any debate): −3.9pp pooled at round 1, recovering to −2.4pp after debate (GSM: −5.6pp → −0.3pp). The revision touches upon this once, with the per-persona decomposition (worst: "cosmic" −11.3pp; best: "kantian" +4.4pp).

W2/Q7. Rounds are fixed at 3; how do results change with rounds? And context capacities should be varied.

Both experiments existed in our data and are now presented in the revision: (a) a rounds ablation over rounds ∈ {1, 2, 4, 5} on 10 tier-stratified mixed teams (40 cells per setting) showed mean accuracy of 84.6% → 87.2% → 87.6% → 87.0%. The first exchange pretty much carries all of the benefit (+2.6pp from r=1 to r=2). Beyond r=3 reasoning tasks plateau (math 99.2% at both r=4 and r=5) while knowledge tasks decline somewhat (CSQA 81.0% at r=2 → 78.6% at r=5) which is consistent with the paper's first-exchange mechanism finding; (b) a context-window grid re-running the window-sensitive models at 16K, 32K, 131K and model-maximum windows: this grid is the basis of the paper's overflow correction, and the revision promotes the per-window curve into the main text (Figure 3, Section 9). At 16K, 7-agent debates of long-form models produce up to 100% empty final rounds; at 131K the same configurations recover fully (e.g. Qwen3.5-27B math 0.638 → 1.000). This mainly happens with Qwen-family models.

All your questions were actionable, and each is now either fixed in the revision (definitions of nn, rr, capacity tax; Figure 1 axis; selection protocol in §3.3; rebuilt table captions) or answered with data (rounds ablation; context-capacity curve, now Figure 3). If you find these resolve your concerns about presentation and the role of context capacity, we would be grateful if you would consider updating your score for acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A brief summary of added experiments, new stats and overall revision

We thank all 3 reviewers for their careful read and attention to our manuscript. All 3 reviews do converge on three similar concerns and actionable points, which will be highlighted below:

  1. "Are the persona results an artifact of extreme personas?" (q9QC, fUUM) --- We added a new experiment during the discussion period: a pool of 50 "task-relevant personas" with 25 expert roles such as 'research mathematician', 'biographer', 'fact-checker', 'examiner', and 25 reasoning strategies such as 'verification specialist', 'decomposition strategist', 'process-of-elimination reasoner' and so on, to try and address the reviewers' concerns about persona selection. These were selected into teams by the identical embedding, with the MaxDet protocol as per the paper's experiments, and were run across 12 tier-stratified models which spanned all three capability tiers + all 5 tasks, with n ∈ {3,5,7}, and against the paper's existing budget-matched controls. The new/added results (scored tasks, paired per problem against the same no-persona baselines) indicate that the persona cost is indeed "content-dependent" with task-relevant personas beating extreme ones by +1.48pp [95% CI +0.52, +2.49] and erasing the extreme pool's significant per-task costs. That said, even this most-favorable pool confers no significant final benefit over 'no-persona debate' (+0.71pp [−0.41, +1.89]). Further, its round-1 gain of +1.91pp [+0.24, +3.78] loses to debate. The paper's conclusion is, therefore, marginally bounded but not truly changed or overturned: persona content sets the sign of a small effect, and no persona pool closes the debate-vs-sampling gap as task-relevant-persona debate remains statistically indistinguishable from budget-matched sampling (vs uniform SC −0.57pp [−2.16, +1.13]; vs temperature-diverse SC +0.51pp [−1.43, +2.64]), with no detectable advantage. On the judged factuality task, even task-relevant personas reduce claim precision (−6.8pp [−9.2, −4.5] vs no-persona over 36 matched cells), meaning that in no configuration do personas confer a net benefit.

  2. "Null results need equivalence statistics." (fUUM) We added two one-sided tests (TOST) with equivalence bounds of ±1pp on the paired-by-problem deltas. Debate vs budget-matched temperature-diverse sampling (320 cells, overflow cells were excluded): pooled −0.42pp, 90% CI [−0.87, +0.04], which are formally equivalent at ±1pp. Debate vs uniform sampling: −0.65pp, 95% CI [−1.11, −0.20] are not equivalent with debate being statistically worse than even uniform sampling at matched budget.

  3. "Presentation." (dPwt) The revision (uploaded with this response) defines every term the review flagged, consolidates the diversity-metric math into a single subsection, and rebuilds Tables 4–5. All changes are in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.