- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks the question of why mult-agent debate (MAD) sometimes beats a single model and if agent diversity is key to making it work. The authors tested on 23 open weight models and 5 tasks, varying the diversity of the agents in three ways -- assigned persons, temperature, tempuerture and different models in a team. They compare each case to a control and find 1. MAD tends to beat single models but ties or loses to the majority voting and costs more 2. Reduce debt personas reduce accuracy. 3. Temperature diversity has no effect 4. Mixed models are slightly better than homgenousl model s and 6. almost all the impoirvement comes from the first dexchange in the debate.
Critical strengths
The authors do a rigorous job matching the debate arm with the control arm -- this makes the results solid
The persona dose-response ladder is clever and helps better disentagle "personas hurt" vs "diversity hurts"
Critical weaknesses
The definition of "cognitive diversity" is very narrow and as a result the key claim of the paper supports is more narrow. All the persona teams came from the extreme tier, which often include task irrelevant characters.
Null results are reported as equivalence but there are no statistics to demonstrate this.
Questions for authors
Can you report results with the moderate persona tier, or with task-relevant personas such as different expert roles or reasoning strategies?
Are the Qwen3.5-27B anomalies in Table 17 explained by context overflow or another serving issue?
Response to Reviewer fUUM (rating 2, constructive)
We thank the reviewer for a review that improved the paper twice over; both critiques led to valuable additions to our paper.
Yes; this was the main experiment we ran during the discussion period. Compute limitations meant delays we were not anticipating but we did run a new arm: a 50-persona task-relevant pool, half expert roles (research mathematician, biographer, archivist, fact-checker, examiner…), half reasoning strategies (verification specialist, decomposition strategist, back-substitution checker, process-of-elimination reasoner…), selected into teams with the identical embedding → MaxDet machinery, run on 12 tier-stratified models × 5 tasks × n ∈ {3,5,7} against the paper's existing budget-matched debate and sampling controls.
The short answer to the reviewer's question is yes, persona content matters, and we were able to quantify by how much: task-relevant teams beat extreme teams by +1.48pp (paired, 95% CI [+0.52, +2.49]); the extreme pool's significant costs decrease notably (GSM −1.72pp → −0.44pp n.s.; CSQA −2.53pp → −0.08pp n.s.) and math turns significantly positive (+3.67pp [+1.31, +6.44]). On the judged factuality task the picture reverses: task-relevant personas REDUCE claim precision among verifiable claims (−6.8pp [−9.2, −4.5] vs no-persona; −4.9pp vs the extreme pool, 36 matched cells, two-judge panel) and even biographer/fact-checker personas make biographies less precise. The persona tax is therefore partly a property of the extreme pool on scored tasks, task-dependent in sign, and we reviewed/rephrased the paper's claim accordingly in the uploaded revision. The core finding is unchanged at its strongest test: even the most task-relevant pool gives no significant net benefit over no-persona debate (+0.71pp [−0.41, +1.89]), and its round-1 gain (+1.91pp [+0.24, +3.78]) is eroded by subsequent debate, which we consider to be further evidence that the debate stage does ensemble-aggregation work rather than deliberative refinement. Perhaps more decisively for the paper's headline: task-relevant-persona debate remains statistically indistinguishable from budget-matched sampling (vs uniform SC −0.57pp [−2.16, +1.13]; vs diverse SC +0.51pp [−1.43, +2.64]) and even the best persona pool does not make debate beat the same budget spent on sampling.
The reviewer is correct, and we adjusted/fixed our results: every "≈" claim now carries a two-one-sided-tests (TOST) result with ±1pp equivalence bounds on the paired-by-problem deltas (pooled over 320 model × task × budget cells, overflow-bound cells excluded, cluster bootstrap over cells). We find that:
The claims whose CIs do not fit the bounds are restated as "not distinguishable" rather than "equivalent."
The reviewer's suspicions were spot on. It is context overflow, and we can show it explicitly. At the model's 16K serving window, its 7-agent biography/GSM cells end with up to 100% empty final-round responses (the wrapper returns empty on overflow rather than crashing); accuracy collapses accordingly. When we re-ran the same experiments at the 131K window, the same cells recovered; math 0.638 → 1.000, final-round-empty 43% → 0%. Table 17's anomalous cells are exactly the window-bound ones and the revision marks them and promotes the per-window curve into the main text (Figure 3, Section 9). Independently, our replication-from-public-release reproduced the same cells overflowing and the anomaly is a deterministic property of the configuration and not a serving accident.
We believe both of your substantive requests are now delivered: the task-relevant persona experiment and formal equivalence statistics for every null claim. If these resolve W1 and W2, we would be grateful if you would consider raising your score to acceptance.