acceptodds
Under review as a conference paper at ICLR 2027

Multi-Agent Debate Amplifies but Rarely Creates: A Calibrated Dissection at the Frontier of Model Capability

Abstract

Multi-agent debate is widely deployed in the belief that interaction lets large language models (LLMs) reason beyond their individual capabilities, yet many benchmark–model combinations used to justify it are now saturated, obscuring how much of the reported gain comes from interaction itself. We rebuild the evidence base on a pre-registered difficulty window that retains only items on which agents genuinely disagree, and evaluate homogeneous multi-instance debate, a heterogeneous five-model panel, Asch-style confederate manipulations, and equal-budget replications of three role- and aggregation-based designs. Here debate converges quickly but improves slowly, and its consensus is often confidently wrong: agreement forms by the second round, yet up to 72.7% of a model's contested items end in a unanimously wrong consensus. What drives convergence depends on where an item sits in the panel's capability distribution: where someone can solve an item, correct answers propagate; where no one can, wrong answers are adopted 108 times against 4, and the strongest model becomes the largest source of adopted errors. Exchanging reasoning traces from the first round doubles adoption of the observed majority and halves minority survival, so the channel that repairs weaker members also accelerates consensus collapse on unsolvable items. Neither scripted saboteurs nor dedicated critics shifted these outcomes, and no replicated design improved significantly over plain heterogeneous debate. Debate therefore acts more as a selector over answers its members can already produce than as a creator of new ones. Progress may depend less on richer protocols than on panel coverage, staged exposure of reasoning, and knowing when consensus deserves trust.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Debate, Large Language Models, Consensus Formation, Social Conformity, Error Propagation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2DD3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper examines whether multi-agent debate propagates existing correct answers or generates new ones. It compares three instances of the same model with a panel of five different models. Both configurations exchange answers alone or answers with reasoning over three debate rounds. The five-model experiments share 45 contested questions. A separate probe uses 40 questions that all models answered incorrectly during calibration. The findings suggest that debate mainly propagates answers already available within the group. On the contested pool, reasoning-based debate increases verdict accuracy from 80.0% to 84.4%, but reduces the share of questions with any correct member answer from 93.3% to 86.7%.

Critical strengths

The analysis distinguishes agreement from correctness and answer availability. The paper tracks individual accuracy, group verdicts, and whether any member holds a correct answer. Table A3 shows that the five-model panel can improve its verdict accuracy while losing correct answers held by individual members. This is useful evidence that stronger consensus does not necessarily preserve the group’s useful information. It provides insight that final accuracy alone would miss.

The scoring audit addresses errors that directly affect the experimental setup. Appendix A.1 reports that 40 of the 50 code errors initially attributed to one model were actually answer-extraction errors. Correcting these errors changes the model’s measured performance and which questions enter the contested pools. This matters because the study depends on identifying questions that models genuinely fail. The audit strengthens the basis for that selection.

Critical weaknesses

The experiments do not establish whether newly correct answers come from interaction or additional sampling. Section 3.1 reports four creation events in the homogeneous reasoning condition. It interprets them as individual discoveries that debate merely spreads. However, there is no matched no-peer control to separate these explanations. An answer missing from the initial samples is also not necessarily beyond a model’s capability. The creation counts need reconciliation too: the introduction says the all-wrong probe never produces a correct answer, while Appendix A.6 reports one generated item after revised scoring. These issues matter because the paper’s main claim concerns what debate can create. The conclusion should stay tied to the observed answers unless stronger controls support it.

The protocol comparisons do not fully separate the method effect from the message format. Table 4 compares methods that exchange reasoning with an answer-only baseline. Section 3.3 attributes MADC’s entire three-item gain to this format change. However, Table A3 reports 37 correct verdicts for answer-only debate and 38 for reasoning-based debate. MADC reaches 40 in Table 4. The shared successes cited in the text do not fully explain these different totals. A paired comparison with the reasoning baseline is needed to separate gains and losses. This does not establish that MADC is better, but it weakens the claim that reasoning alone explains its gain.

Questions for authors

What are the final creation counts after the scoring corrections, including the generated item reported in Appendix A.6? Is there a matched no-peer comparison that supports attributing newly correct answers to additional sampling rather than interaction?

Can you report the item-level wins and losses between MADC and plain debate with reasoning? How do these results explain the difference between their 40 and 38 correct verdicts?

4 comments

We thank reviewer 2DD3 for their time and constructive feedback. We address the main concerns below.

W1: The experiments do not establish whether newly correct answers come from interaction or additional sampling ... The conclusion should stay tied to the observed answers unless stronger controls support it.

Q1: What are the final creation counts after the scoring corrections, including the generated item reported in Appendix A.6? Is there a matched no-peer comparison that supports attributing newly correct answers to additional sampling rather than interaction?

Regarding the reviewer's concern about whether debate truly possesses the capacity to create, we respond as follows.

First, we clarify the precise meaning of the capability boundary. We agree that the absence of a correct answer in initial samples is not an absolute proof of impossibility. To make this operational boundary more rigorous, we redesigned Round 0 around self-consistency: each agent generates three independent responses without peer visibility and enters the debate with its majority-voted answer. We have incorporated the details of our self-consistency design into Section 3. Thus, an all-wrong Round-0 indicates that no agent surfaced a valid solution across three independent draws per model, rather than a single-pass omission. Accordingly, we revise the statement in the Introduction from "never" to "rarely". Furthermore, we explicitly scope our conclusions to frontier tasks operating at the models' capability boundary, noting that on easier tasks well within capability limits, models may still generate correct answers.

To ensure greater statistical reliability, we substantially expanded our evaluation dataset, increasing the contested subset from 45 to 340 instances and the all-wrong subset from 40 to 220 instances.

Second, we provide a no-peer control that separates "interaction" from "sampling.":

  • The homogeneous condition uses three instances of the same model; because their weights are identical, their exchanges carry no information beyond what a single instance's sampling distribution could produce, so homogeneous debate is informationally equivalent to repeated sampling from a single distribution — precisely the "additional sampling" control the reviewer requests.
  • The heterogeneous condition, in contrast, uses five distinct models, so peer messages carry genuinely new information and constitute the "interaction". In the heterogeneous all-wrong condition, creation is 0 across all items.

The two settings also exhibit broadly consistent model-level rates of produced new correct answers and flips. Together, new correct answers are therefore driven by additional sampling, and interaction is responsible only for amplification, not creation.

Third, we correct the counting of creation. R3 reports the updated operational creation counts (items where all agents fail at Round 0 but reach a correct verdict by Round 3) on the expanded dataset after revised scoring. In homogeneous panels evaluated on their respective contested pools, creation counts decrease gradually from M1 to M5 (4 / 3.3 / 3.3 / 3 / 2 under answer-only; 7.3 / 6 / 5.7 / 2 / 2 under with-reasoning). The heterogeneous panel combines complementary model strengths led by M1, raising contested creation above the best single-model baseline from 4 to 5.7 (answer-only) and from 7.3 to 10.7 (with-reasoning). Even on the harder all-wrong pool, the heterogeneous panel integrates cross-model capabilities to achieve 4 creations under answer-only sharing and 8 under with-reasoning sharing.

Table R3. Operational creation counts after scoring corrections.

ConfigurationMessage formatTotal runsOperational creation (Avg)
Homogeneous (contested)answer-only80/80/80/100/1004 / 3.3 / 3.3 / 3 / 2
Homogeneous (contested)with reasoning80/80/80/100/1007.3 / 6 / 5.7 / 2 / 2
Heterogeneous contestedanswer-only3405.7
Heterogeneous contestedwith reasoning34010.7
Heterogeneous all-wronganswer-only3404
Heterogeneous all-wrongwith reasoning3408

The single "generated item" previously noted in Appendix A.6 (item omni-60) was a classification artifact: under the corrected scoring, at least one agent already produced a valid answer at Round 0. Consequently, omni-60 represents an instance of amplification rather than creation, and we have reclassified it into the contested pool and updated Appendix A.6 accordingly.

W2: The protocol comparisons do not fully separate the method effect from the message format. Table 4 compares methods that exchange reasoning with an answer-only baseline. Section 3.3 attributes MADC’s entire three-item gain to this format change. However, Table A3 reports 37 correct verdicts for answer-only debate and 38 for reasoning-based debate. MADC reaches 40 in Table 4. The shared successes cited in the text do not fully explain these different totals. A paired comparison with the reasoning baseline is needed to separate gains and losses. This does not establish that MADC is better, but it weakens the claim that reasoning alone explains its gain.

Q2: Can you report the item-level wins and losses between MADC and plain debate with reasoning? How do these results explain the difference between their 40 and 38 correct verdicts?

Following the reviewer's requirements, we supplement a paired comparison against the with-reasoning baseline in original manuscript, see Table R4.

Table R4. Item-level wins and losses between MADC and plain debate in original manuscript.

ComparisonItems wonItems lostNet gainp
With reasoning vs Answer-onlyomni-147, omni-50, omni-97omni-120, omni-60+11.00
MADC vs Answer-onlyomni-147, omni-50, omni-97None+30.25
MADC vs With reasoningomni-120, omni-60None+20.50

Relative to answer-only, with-reasoning wins omni-147/50/97 and loses omni-120/60 (net +1, p = 1.00); relative to answer-only, MADC wins three items with no losses (net +3, p = 0.25); and relative to with-reasoning, MADC wins omni-120/60 with no losses (net +2, p = 0.50). This precisely explains the gap between 40 and 38: MADC's 40 = 35 (solved by all three) + 3 (solved only by with-reasoning) + 2 (solved only by answer-only). The five additional items MADC solves are those on which the two plain-debate formats complement each other, and the remaining five items defeat all three. In short, MADC recovers the union of items already solvable by the two plain debate formats rather than extrapolating beyond the capability boundary of plain debate.

Regarding whether these results establish that MADC is superior, none of the observed net gains are statistically significant (p = 0.25 and p = 0.50). More fundamentally, our round-level diagnostics reveal that while sharing reasoning traces increases convergence, member accuracy, and final verdict accuracy, it simultaneously lowers oracle accuracy: across multi-round exchanges, a solitary agent that initially answered correctly is frequently swayed by the majority and flips to an incorrect answer, causing the panel to lose its only valid solution. This confirms that debate acts primarily as an amplification mechanism rather than a capability enhancer, and can actively erode accuracy via true-to-false flips.

Finally, regarding finite-sample variance, expanding our contested and all-wrong pools, as well as three independent seeds (0/1/2), we find that model responses fluctuate within their capability range, and the original 45-item sample indeed contains an element of coincidence. Under our expanded evaluation, our primary finding holds: MADC yields no statistically significant improvement over plain debate baselines.

Thank you for the detailed response and additional experiments. I have no major concerns, and I will increase the soundness score.

I am unsure it reached the highest overall score (4 - clear acceptance) though. I maintain my original positive score as 3.

Thanks!

Thank you very much for your thoughtful engagement throughout the discussion, and for raising the soundness score. We would like to add one remark on the broader motivation of this work, which we hope may help in your overall assessment.

Multi-agent debate has become a highly active area in LLM applications, with a steady stream of papers on improving and evaluating MAD systems — even at ICLR 2027, a keyword search for "multi-agent" returns many related studies. However, after carefully analyzing this large body of work, we were left with a set of persistent questions that the literature does not answer: once we set aside the widely used (and largely saturated) benchmarks and small-scale dummy models, how accurate is MAD in realistic settings, and can more rounds and more agents actually solve genuinely hard problems? These basic questions are, to our knowledge, missing from the literature. Although the headline outcome may seem predictable — debate helps surface more correct answers — no complete study had directly established it under controlled, difficulty-calibrated conditions.

Beyond these anticipated findings, our more fine-grained evaluation surfaced phenomena that may hinder the practical adoption of MAD: at the capability boundary, models rapidly converge (within ≤2 rounds) onto a plausible answer and abandon diverse exploration; in this regime, exchanging more context is counterproductive, and adding more rounds or agents only increases computational waste. Existing methods have various limitations and neither identify nor adequately address this failure mode.

We therefore believe this study offers concrete value to the MAD community: it provides the missing controlled measurements, a calibrated evaluation protocol, and actionable evidence about when debate helps, when it merely consumes budget, and why. We hope these contributions merit consideration for a higher overall rating.

Thank you once again for your time and careful consideration.

Reviewer_ius2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of multi-agent debate, investigating whether debate enables models to discover new solutions or primarily amplifies existing correct answers. The authors construct difficulty-calibrated evaluation sets and analyze answer transitions, consensus formation, and model interactions across homogeneous and heterogeneous debate settings. The results suggest that debate mainly propagates existing answers, while increased interaction can accelerate incorrect consensus.

Critical strengths
  • The paper provides a systematic investigation of MAD beyond final accuracy. The experimental design covers multiple model configurations and intervention strategies, offering useful observations about multi-agent interaction.

  • The paper introduces a difficulty-calibrated evaluation protocol to avoid benchmark saturation and better expose interaction effects, which strengthens the credibility and interpretability of its empirical observations.

Critical weaknesses
  • My main concern is that the authors should provide a clearer position of this work in the MAD evaluation works. Several observations have already been to some extent discussed in prior work. For example, [1] and [2] have questioned the effectiveness of MAD compared with simpler inference strategies, while [3] has investigated conformity and answer transitions. Although this paper provides additional measurements, its novel contributions and insights beyond these studies are not sufficiently clear.

  • While understanding debate dynamics is valuable, the paper does not clearly establish how its additional process-level analyses translate into more reliable final decisions or actionable improvements to MAD systems. Given that the primary objective of MAD is to improve solution quality, the practical significance of these observations could be better justified.

  • Several key experiments are conducted on only 40–45 tasks with single-seed runs, which raises concerns about reliability and generalizability.

  • The contested-item selection criterion favors tasks where correct answers are already available among the agents, potentially influencing the conclusion that debate primarily amplifies rather than creates solutions. This would weaken the evidence supporting the paper's broader conclusions.

[1] Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?

[2] Stop Overvaluing Multi-Agent Debate — We Must Rethink Evaluation and Embrace Model Heterogeneity

[3] Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

Questions for authors

Please refer to weakness sections.

6 comments

We thank reviewer ius2 for their time and constructive feedback. We address the main concerns below.

W1: My main concern is that the authors should provide a clearer position of this work in the MAD evaluation works. Several observations have already been to some extent discussed in prior work. For example, [1] and [2] have questioned the effectiveness of MAD compared with simpler inference strategies, while [3] has investigated conformity and answer transitions. Although this paper provides additional measurements, its novel contributions and insights beyond these studies are not sufficiently clear.

[1] Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?

[2] Stop Overvaluing Multi-Agent Debate — We Must Rethink Evaluation and Embrace Model Heterogeneity

[3] Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

We agree that [1], [2], and [3] have made contributions by questioning the gains of MAD over simpler inference strategies, highlighting model heterogeneity, and investigating conformity and answer transitions. While our work shares thematic similarities with these studies, it differs in its target measurements, experimental setups, and the boundaries of its conclusions.

Specifically:

  • [1] decomposes MAD into voting and debate using 7B/32B models on standard benchmarks such as GSM8K and MMLU, showing that gains stem primarily from majority voting.
  • [2] evaluates four medium-scale models across multiple benchmarks, such as GSM8K and MMLU-Pro, to argue that MAD is overvalued and advocates heterogeneity as a general remedy.
  • [3] evaluates five homogeneous agents on benchmarks including MMLU-Pro and GPQA under single-round peer exposure, separating answer flips into spontaneous instability, strict conformity, and reasoning-induced adoption, showing that peer adoption is not merely conformity.

In contrast, our work differs from [1–3] in three key dimensions:

  1. Benchmark: Having diagnosed that conventional benchmarks are largely saturated for frontier models, we pre-registered a 30% to 80% difficulty window and curated two frontier evaluation pools (contested and all-wrong) from over 10k problems, thereby making interaction effects genuinely observable.
  2. Model configuration: Rather than relying on small-to-medium models as in [1] and [2] (e.g., Qwen2.5-7B and Llama-3.1-8B-Instruct from 2024) or a single homogeneous model family as in [3] (e.g., GPT-4o from 2024), we deliberately construct a five-model Pareto frontier of newer, larger models spanning 30B to 2.4T parameters to reflect a realistic scenario (newly added descriptions; detailed in Section 3).
  3. Experimental design: We systematically cover both homogeneous and heterogeneous panels ([3] considers only homogeneous models, while [1] and [2] focus primarily on homogeneous settings) and introduce causal interventions absent from all three prior works, namely distractors and critics in Experiment 2, advancing the analysis from answer flips to whether debate actually creates or destroys correct solutions.

Grounded in our experiments across expanded model pools, the most fundamental distinction between our work and above three papers lies in the implications of our findings. Whereas prior works offer skepticism or general prescriptions on conventional tasks, we demonstrate that at the model capability boundary, when agents lack initial consensus, adding more context, or interaction rounds accelerates premature convergence, discourages deep verification, and ultimately lead to a collapse into an incorrect consensus.

Thus, our contribution is not merely reiterating that MAD is overvalued, but to identify a more actionable research direction: how to preserve diverse exploration rather than encourage blind convergence when models operate at the limits of their capabilities. This perspective points to two promising avenues for future work: controlled context exposure, as well as mechanisms governing model entry and exit from the deliberation process.

We add the analysis of these three papers to the revised Related Work section.

W2: While understanding debate dynamics is valuable, the paper does not clearly establish how its additional process-level analyses translate into more reliable final decisions or actionable improvements to MAD systems. Given that the primary objective of MAD is to improve solution quality, the practical significance of these observations could be better justified.

We agree that process-level analyses must ultimately translate into more reliable final decisions or actionable system improvements rather than remaining purely observational. In fact, our findings point directly toward that goal and reveal critical operational boundaries for deploying MAD systems.

First, our findings yield actionable design principles. Across three controlled experiments (homogeneous versus heterogeneous panels crossed with answer-only versus reasoning-sharing protocols, causal interventions with saboteurs and critics, and budget-matched replications of MADC, PEAR, and Free-MAD-N), we establish a fundamental asymmetry: correct answers already present at Round 0 are far more likely to survive into the final decision than initially absent answers are to be newly created. Thus, the practical value of MAD lies in selection and amplification rather than creation. This directly motivates three recommendations: (1) deploy heterogeneity to maximize initial solution coverage at Round 0; (2) treat reasoning traces as a controlled dose by enforcing independent initial generation or selective sharing; and (3) interpret consensus jointly with task difficulty, since rapid agreement on frontier and difficult tasks often signals collective collapse rather than verification.

Second, our results define clear operational boundaries. At the capability boundary, when agents lack initial consensus, scaling context length, interaction rounds, or debater count does not improve stability or accuracy, as roughly 90% of debates converge by Round 3 with virtually no novel discovery thereafter. Therefore, our work redirects future MAD engineering from simple scaling toward preventing premature convergence under high disagreement, specifically motivating controlled context exposure and dynamic agent entry and exit mechanisms.

W3: Several key experiments are conducted on only 40–45 tasks with single-seed runs, which raises concerns about reliability and generalizability.

To address this concern regarding sample size and single-seed variance, we have substantially strengthened our evaluation along three lines.

First, we expanded our evaluation pools: the contested set grew from 45 to 340 items and the all-wrong set from 40 to 220 items, all selected using the same pre-registered filtering criteria from over 10k raw questions. Using these frontier benchmarks is essential rather than incidental: conventional suites such as GSM8K, GPQA, and MMLU-Pro are largely saturated for the models we study (90% average accuracy in our pilots), which would render interaction effects unobservable. The corresponding description has been updated in Section 2.2 (Task Pools).

Second, we introduced multi-seed replication by re-running all five models (M1–M5) across three independent seeds (0, 1, 2). Experiments indicate that accuracy remains highly stable across seeds, and our core conclusions hold: solution creation remains absent, with accuracy fluctuating within low single digits. Details can be found in Table R1 of our response to Reviewer 6LHo’s W1/Q1.

Third, we mitigated single-pass sampling noise at Round 0 by redesigning the initialization around self-consistency. Specifically, each agent now samples three independent responses and enters the debate with its majority-voted answer, ensuring that any Round-0 disagreement reflects genuine differences among models rather than sampling stochasticity. We have incorporated the details of our self-consistency design into Section 3.

Because these curated items sit right at the models' capability boundary, the outcome distribution stabilizes reliably, and further scaling of the dataset is unlikely to alter our conclusions. A detailed breakdown is provided in Appendix 4 of the revised manuscript.

W4: The contested-item selection criterion favors tasks where correct answers are already available among the agents, potentially influencing the conclusion that debate primarily amplifies rather than creates solutions. This would weaken the evidence supporting the paper's broader conclusions.

Regarding the reviewer's concerns about the selection criterion for the contested pool, we would like to clarify that amplification and creation are measured on two disjoint, complementary evaluation pools, meaning the contested filtering criteria cannot contaminate our findings on solution creation.

First, on the contested pool, we intentionally retained items where at least two models already held the correct answer (300 out of 340 items). This design specifically isolates the direction of amplification: when a valid solution is already present, does group interaction amplify the truth or derail models toward an incorrect majority? In our expanded heterogeneous experiments, shifts toward the majority are dominated by error-to-correct repairs: models adopted correct answers 234 times versus incorrect answers 129 times under answer-only sharing (and 363 versus 295 times under reasoning sharing). Consequently, final accuracy improved from 84.3% to 86.5% (answer-only) and 81.5% to 85.9% (with reasoning), while the false-consensus rate remained as low as 1.8% under answer-only sharing (rising to 10.2% with reasoning traces). Thus, the contested pool evaluates how existing truths are amplified, not whether new ones are invented.

Second, our conclusion that MAD rarely creates new solutions is established independently on the all-wrong pool, where every model fails at Round 0. Even in our expanded runs, genuine creation remains exceedingly rare: correct verdicts emerged on only 4 of 220 items under answer-only sharing and 8 of 220 with reasoning traces, and trace inspection indicates that these isolated cases reflect standard inference-time sampling variance from extended generation rather than inter-model synthesis. This aligns directly with our homogeneous multi-instance controls in Experiment 1, where the rate of newly emerging correct answers closely matches each model's baseline independent flip rate, confirming that occasional breakthroughs stem from single-model sampling distributions rather than debate mechanics.

Therefore, our core finding that MAD primarily amplifies existing solutions rather than creating novel ones is not an artifact of the contested selection criteria; rather, it is independently established by the all-wrong pool and cross-validated across both experimental regimes.

Reviewer_6LHoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether multi-agent debate enables LLMs to solve problems beyond their individual capabilities or primarily amplifies existing solutions through information exchange. To avoid benchmark saturation, the authors introduce a calibrated evaluation framework that selects challenging items based on single-shot model accuracy. The study examines homogeneous and heterogeneous multi-agent debate, the impact of sharing reasoning traces, adversarial and critical agent interventions, and three existing debate strategies under constrained computational budgets.

Critical strengths
  1. The paper addresses a fundamental question regarding the effectiveness of multi-agent debate: whether improvements arise from genuinely new reasoning capabilities or from selecting and propagating existing answers. This distinction is valuable for understanding the limitations of multi-agent reasoning systems.
  2. Rather than relying solely on final accuracy, the paper examines answer transitions, convergence, minority survival, and incorrect consensus across multiple debate rounds. These analyses provide useful insights into how information propagates among agents.
  3. The authors investigate homogeneous and heterogeneous agent configurations, answer-only versus reasoning-sharing communication, adversarial interventions, and several existing debate strategies. The calibration-based selection of challenging problems also helps address benchmark saturation.
Critical weaknesses
  1. The main heterogeneous experiments use only 45 questions, while the all-wrong evaluation contains 40 questions. Furthermore, experiments use a single random seed. These relatively small sample sizes limit the statistical power and robustness of the findings, particularly the conclusions about the effectiveness of alternative debate strategies.
  2. Although the paper evaluates several multi-agent debate strategies, it lacks a sufficiently comprehensive comparison with independent sampling, self-consistency, or single-model inference using equivalent computational budgets. Such baselines are important for determining whether multi-agent interaction provides additional benefits beyond repeated sampling and answer aggregation.
  3. The paper primarily defines creation based on whether a correct answer is absent from the initial agent responses. However, absence from a small number of initial samples does not imply that an answer lies outside the models' underlying capabilities or sampling distributions. The all-wrong experiments therefore do not fully establish a capability boundary. Additional sampling-based controls would help distinguish genuinely novel reasoning from solutions accessible through repeated independent inference.
  4. The comparisons between debate strategies are not fully controlled. In Experiment 3, the replicated methods exchange reasoning traces while the baseline uses answer-only messages, making it difficult to isolate the effects of the proposed structural modifications. Additionally, changes in reasoning effort during heterogeneous experiments may introduce further variation.
Questions for authors
  1. Can the authors provide additional experimental runs with different random seeds and larger evaluation pools to assess the statistical robustness of the main findings?
  2. How would the conclusions about solution creation change when compared against a compute-matched independent sampling baseline? Could some apparently novel solutions simply arise from additional inference attempts rather than inter-agent interaction?
  3. Can the authors compare the replicated debate strategies against a reasoning-sharing baseline instead of an answer-only baseline to better isolate their architectural contributions?
  4. How sensitive are the findings to the number of participating agents, the number of debate rounds, and the selected difficulty window?
4 comments

We thank reviewer 6LHo for their time and constructive feedback. We address the main concerns below.

W1: The main heterogeneous experiments use only 45 questions, while the all-wrong evaluation contains 40 questions. Furthermore, experiments use a single random seed. These relatively small sample sizes limit the statistical power and robustness of the findings, particularly the conclusions about the effectiveness of alternative debate strategies.

Q1: Can the authors provide additional experimental runs with different random seeds and larger evaluation pools to assess the statistical robustness of the main findings?

Following the reviewer's suggestions, we have substantially expanded our evaluation datasets, conducted multi-seed stability experiments, and redesigned the Round-0 initialization via self-consistency to eliminate single-pass stochasticity. Notably, scaling up the datasets revealed that the small original pools had inflated accuracy, as re-evaluation resulted in substantial accuracy drops across all five homogeneous pools (see Table R1).

First, to strengthen statistical power, we expanded the contested evaluation pool from 45 to 340 items and the all-wrong pool from 40 to 220 items, curated from over 10k raw questions to retain genuinely challenging, high-contention cases. Specifically, we drew these new items under the exact same filtering criteria from Humanity's Last Exam (HLE) , Omni-Math and LiveCodeBench. Because our study evaluates capable, modern models to reflect real-world deployment rather than artificially weak baselines, standard benchmarks frequently used in prior work (e.g., GSM8K, GPQA, MMLU-Pro) proved largely saturated in our pilot runs (90% average accuracy), making frontier benchmarks essential for meaningful multi-agent debate. The corresponding description has been updated in Section 2.2 (Task Pools).

Second, following the reviewer's recommendation, we evaluated all five models (M1–M5) across three independent random seeds (0/1/2) on the expanded dataset. As shown in Table R1, (1) in the homogeneous setting, each model works on its own pool of items, so accuracy fluctuates across models; notably, M4's post-debate accuracy approaches or even exceeds that of M1 and M2 (under both answer-only and with-reasoning), which we attribute to M4's stronger self-correction ability on its own pool; (2) in the heterogeneous setting, accuracy fluctuates little because all five models share a single pool: for every contested item, at least one of the five models holds the correct answer, whereas a homogeneous model's pool consists only of items that the model itself cannot solve. Hence, the heterogeneous panel maintains higher accuracy.

Table R1. Multi-seed performance consistency (accuracy, %) on the expanded pool in Round 3.

SettingMessage formatseed 0seed 1seed 2
Homogeneous contested (M1/M2/M3/M4/M5)Answer-only18.8 / 16.2 / 7.5 / 17.0 / 11.017.5 / 16.2 / 10.0 / 13.0 / 13.017.5 / 18.8 / 8.8 / 16.0 / 12.0
Homogeneous contested (M1/M2/M3/M4/M5)with reasoning26.2 / 21.2 / 22.5 / 27.0 / 16.026.2 / 22.5 / 16.2 / 31.0 / 13.023.8 / 25. 0 / 16.2 / 32.0 / 14.0
Heterogeneous contestedAnswer-only80.380.982.3
Heterogeneous contestedwith reasoning83.283.284.7
Heterogeneous all-wrong poolAnswer-only3.24.58.2
Heterogeneous all-wrong poolwith reasoning7.75.58.1

Across these three independent runs, our empirical conclusions remain consistent: we observe no creation behavioral shifts, and accuracy fluctuations remain strictly within the low single digits. Because these curated problems operate right at the models' capability boundary, the performance distribution stabilizes reliably, and further scaling of the dataset is unlikely to alter our core conclusions. Detailed experimental breakdowns are provided in Appendix 4 of the revised manuscript.

Finally, to guarantee robustness against single-pass sampling noise, we standardized the Round-0 initialization across all experiments using self-consistency. Specifically, instead of relying on a single initial generation, each agent independently produces three responses and commits to the majority-voted answer before entering the subsequent discussion. We have incorporated the details of the self-consistency design into Section 3.

W2: Although the paper evaluates several multi-agent debate strategies, it lacks a sufficiently comprehensive comparison with independent sampling, self-consistency, or single-model inference using equivalent computational budgets. Such baselines are important for determining whether multi-agent interaction provides additional benefits beyond repeated sampling and answer aggregation.

Q2: How would the conclusions about solution creation change when compared against a compute-matched independent sampling baseline? Could some apparently novel solutions simply arise from additional inference attempts rather than inter-agent interaction?

Regarding compute-matched baselines, we set a 64k-token context limit across all experiments to accommodate the difficulty of our curated problems. Even under this ceiling, model spillover can still be observed. For example, M2 and M5 exhibit context overflow rates of 9.46% and 15.97%, respectively, with single-pass inference exceeding 30 minutes. Inspecting their traces, we found these failures stem from models falling into repetitive reasoning loops (scored as incorrect when no valid answer can be extracted), meaning that extending the token budget further would still fail to address them. When a model systematically hits its reasoning or context ceiling on a problem, simply allocating an equivalent compute budget to independent repeated sampling or standard self-consistency predominantly yields homogeneous failure modes rather than breakthrough solutions.

To disentangle genuine inter-agent interaction from mere additional inference attempts, we rely on two controlled designs. First, under the homogeneous multi-instance condition in Experiment 1, the two settings we originally designed, "answer only" and "with reasoning," essentially implement the "independent sampling" you pointed out. Second, in our original Experiment 2, we conducted two sub-experiments: one added two saboteurs, and the other added a critic (M1), so that a persistent external voice changes only the interaction while holding capability and computational budget fixed. Third, our redesigned Round-0 initialization applies independent multi-sample voting (self-consistency) to each agent before debate begins. Together, these controls isolate the causal effects of interaction from the gains achievable through repeated sampling alone.

W3: The paper primarily defines creation based on whether a correct answer is absent from the initial agent responses. However, absence from a small number of initial samples does not imply that an answer lies outside the models' underlying capabilities or sampling distributions. The all-wrong experiments therefore do not fully establish a capability boundary. Additional sampling-based controls would help distinguish genuinely novel reasoning from solutions accessible through repeated independent inference.

To address this concern directly, we organize our response into two parts: first, how we rigorously establish the empirical capability boundary, and second, how our sampling controls confirm that genuinely novel creation is rare.

Regarding the capability boundary, items in the all-wrong pool are frontier problems filtered through repeated filtering to verify that models consistently fail on them. To eliminate single-pass sampling stochasticity, we introduced three safeguards in our revised evaluation:

  1. substantially expanding the evaluation pools;
  2. replicating all five models across three random seeds (0/1/2);
  3. redesigning Round 0 around self-consistency, where each agent independently samples three responses and enters the debate with its majority-voted answer.

Thus, the absence of a correct answer at Round 0 reflects that each agent's three-sample majority vote missed the correct answer, rather than an accidental single-sample omission. Furthermore, we intentionally constructed a five-model Pareto frontier spanning 30B to 2.4T parameters to reflect realistic deployment (newly added descriptions; detailed in Section 3), and evaluated it through three experiment types: the contested pool tests self-correction under information sharing, the all-wrong pool tests whether agents can synthesize a valid consensus from mutually erroneous starts, supplemented by the saboteur/critic interventions (Experiment 2) and three role-based designs (Experiment 3) as controls.

Second, regarding the conclusion that MAD rarely creates new solutions, scaling up the dataset confirms that accuracy does not improve on boundary tasks. When models cannot solve a problem independently, they tend to conform to existing peer responses regardless of correctness and abandon deeper verification; adding more rounds or debaters rarely generates novel solutions. Crucially, the few instances of newly emerging correct answers occurred in homogeneous three-instance panels under the with-reasoning condition. Trace analysis shows that these isolated breakthroughs stem from extra independent sampling draws of the single model itself rather than inter-agent synthesis. This demonstrates that such rare events are accessible through repeated independent inference, thereby supporting our core conclusion that MAD primarily amplifies existing solutions rather than creating new ones.

W4: The comparisons between debate strategies are not fully controlled. In Experiment 3, the replicated methods exchange reasoning traces while the baseline uses answer-only messages, making it difficult to isolate the effects of the proposed structural modifications. Additionally, changes in reasoning effort during heterogeneous experiments may introduce further variation.

Q3: Can the authors compare the replicated debate strategies against a reasoning-sharing baseline instead of an answer-only baseline to better isolate their architectural contributions?

Thank you for pointing this out. Rather than a missing experiment, this was purely a reporting omission: all three designs were run with “with reasoning”. Table R2 presents the complete comparison using the reasoning-sharing panel as the direct reference:

Table R2. Comparison of debate strategies.

MethodAxisCalls per itemFinal acc. (r0-r3)Net flipsp
heterogeneousanswer-only (baseline)1581.2 (80.3-82.3)——
heterogeneouswith reasoning (baseline)1583.7 (83.2-84.7)——
MADCpositional ordering1587.9 (87.1-88.8)+6.00.12
PEARconfidence, routing, critique1579.0 (78.2-79.7)-7.71.00
Free-MAD-Nanti-conformity prompting1081.2 (80.6-82.1)0.01.00

As shown in Table R2, modifying the debate protocols yields negligible gains over the reasoning-sharing baseline. Specifically, when models lack initial consensus, exposing them to verbose reasoning traces accelerates premature convergence: they quickly conform to peer opinions instead of engaging in deep reasoning to verify whether the answer is correct, ultimately collapsing into a shared wrong consensus. Consequently, scaling architectural modifications, interaction rounds, model amounts, or computational budgets rarely overcomes this bottleneck. In future work, we will further explore how to preserve reasoning diversity when models operate at their capability boundary.

Q4: How sensitive are the findings to the number of participating agents, the number of debate rounds, and the selected difficulty window?

With respect to the reviewer's questions concerning these three dimensions, we respond to them one by one below.

  • Number of agents. We deliberately selected five models spanning strong-to-weak capabilities to form a Pareto frontier where no single model dominates across all domains. This allows us to test both whether noise from stronger models interferes with weaker models and whether a correct answer can trigger a broadcasting effect across the group.
  • Number of Debate Rounds. In our experiments, answers largely converged after three rounds, with additional rounds yielding limited gains. Extending interaction beyond this threshold incurs diminishing returns at a prohibitive computational cost: our setup already requires 30 API calls per item (5 agents × 3 parallel sampling draws in Round 0, followed by 3 sequential rounds in Rounds 1–3), averaging 5 minutes per round, or over 40 minutes in extreme cases, for a minimum runtime exceeding 10 minutes per item. Moreover, longer accumulated contexts across extra rounds exacerbate conformity among weaker agents toward the majority opinion. Thus, adding more rounds is neither effective nor computationally practical.
  • Selected Difficulty Window. Our filtered evaluation items are intentionally challenging and require extended reasoning chains. We selected 30%~80% difficulty window specifically to avoid task saturation and preserve genuine inter-agent disagreement, as only contested scenarios allow us to cleanly observe and measure solution creation and amplification during debate.

Based on our findings, exposing agents to excessive context degrades reasoning efficiency and accelerates premature conformity .

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

We sincerely thank all reviewers for their time and constructive feedback. We believe that their comments and questions have greatly improved our manuscript, and we have revised it accordingly.

Main differentiation and innovation:

  1. We find a pervasive benchmark-saturation problem for frontier models and, through large-scale filtering, curate a high-quality, pre-registered evaluation pool (contested and all-wrong) on which debate effects are genuinely observable.
  2. We deliberately construct a five-model Pareto frontier spanning 30B–2.4T parameters to make capability differences surface during debate, enabling us to examine whether models are more often misled or corrected. This distinguishes our work from prior studies relying on small-scale or single-family models.
  3. We design three systematic experiments that progressively isolate the mechanisms of multi-agent debate. Experiment 1 examines amplification versus creation by crossing homogeneous and heterogeneous panels with answer-only and reasoning-sharing message formats; Experiment 2 applies causal interventions through scripted saboteurs and a dedicated critic; Experiment 3 replicates three role- and aggregation-based designs (MADC, PEAR, Free-MAD-N) under matched budgets. Together, these three experiments disentangle what debate creates from what it merely amplifies, and demonstrate that neither external pressure nor engineered protocols extends the panel's capability boundary.
  4. Our core finding and recommendation: at a model’s capability boundary, when agents have not yet reached consensus, excessive context accelerates premature convergence, causing them to abandon deep reasoning and collapse into a consensus whose correctness is uncertain. Additional design choices, interaction rounds, debaters, or computational budgets rarely mitigate this problem. Future work will explore how to preserve diverse exploration rather than encourage blind convergence when models operate at their capability limits.

The main concerns of the reviewers:

  • Statistical robustness: limited sample size and single-seed runs. (6LHo; ius2)
  • Baseline design: missing compute-matched controls separating interaction from additional sampling. (6LHo; 2DD3)
  • Experimental control: message format confounded with method effect. (6LHo; 2DD3)
  • Measurement validity: the definition and measurement of "creation" and the capability boundary. (6LHo; ius2; 2DD3)

We believe that the revised paper fully addresses the concerns raised and significantly improves the clarity, completeness, and contribution of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.