- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper examines whether multi-agent debate propagates existing correct answers or generates new ones. It compares three instances of the same model with a panel of five different models. Both configurations exchange answers alone or answers with reasoning over three debate rounds. The five-model experiments share 45 contested questions. A separate probe uses 40 questions that all models answered incorrectly during calibration. The findings suggest that debate mainly propagates answers already available within the group. On the contested pool, reasoning-based debate increases verdict accuracy from 80.0% to 84.4%, but reduces the share of questions with any correct member answer from 93.3% to 86.7%.
Critical strengths
The analysis distinguishes agreement from correctness and answer availability. The paper tracks individual accuracy, group verdicts, and whether any member holds a correct answer. Table A3 shows that the five-model panel can improve its verdict accuracy while losing correct answers held by individual members. This is useful evidence that stronger consensus does not necessarily preserve the group’s useful information. It provides insight that final accuracy alone would miss.
The scoring audit addresses errors that directly affect the experimental setup. Appendix A.1 reports that 40 of the 50 code errors initially attributed to one model were actually answer-extraction errors. Correcting these errors changes the model’s measured performance and which questions enter the contested pools. This matters because the study depends on identifying questions that models genuinely fail. The audit strengthens the basis for that selection.
Critical weaknesses
The experiments do not establish whether newly correct answers come from interaction or additional sampling. Section 3.1 reports four creation events in the homogeneous reasoning condition. It interprets them as individual discoveries that debate merely spreads. However, there is no matched no-peer control to separate these explanations. An answer missing from the initial samples is also not necessarily beyond a model’s capability. The creation counts need reconciliation too: the introduction says the all-wrong probe never produces a correct answer, while Appendix A.6 reports one generated item after revised scoring. These issues matter because the paper’s main claim concerns what debate can create. The conclusion should stay tied to the observed answers unless stronger controls support it.
The protocol comparisons do not fully separate the method effect from the message format. Table 4 compares methods that exchange reasoning with an answer-only baseline. Section 3.3 attributes MADC’s entire three-item gain to this format change. However, Table A3 reports 37 correct verdicts for answer-only debate and 38 for reasoning-based debate. MADC reaches 40 in Table 4. The shared successes cited in the text do not fully explain these different totals. A paired comparison with the reasoning baseline is needed to separate gains and losses. This does not establish that MADC is better, but it weakens the claim that reasoning alone explains its gain.
Questions for authors
What are the final creation counts after the scoring corrections, including the generated item reported in Appendix A.6? Is there a matched no-peer comparison that supports attributing newly correct answers to additional sampling rather than interaction?
Can you report the item-level wins and losses between MADC and plain debate with reasoning? How do these results explain the difference between their 40 and 38 correct verdicts?
4 comments
W2: The protocol comparisons do not fully separate the method effect from the message format. Table 4 compares methods that exchange reasoning with an answer-only baseline. Section 3.3 attributes MADC’s entire three-item gain to this format change. However, Table A3 reports 37 correct verdicts for answer-only debate and 38 for reasoning-based debate. MADC reaches 40 in Table 4. The shared successes cited in the text do not fully explain these different totals. A paired comparison with the reasoning baseline is needed to separate gains and losses. This does not establish that MADC is better, but it weakens the claim that reasoning alone explains its gain.
Q2: Can you report the item-level wins and losses between MADC and plain debate with reasoning? How do these results explain the difference between their 40 and 38 correct verdicts?
Following the reviewer's requirements, we supplement a paired comparison against the with-reasoning baseline in original manuscript, see Table R4.
Table R4. Item-level wins and losses between MADC and plain debate in original manuscript.
| Comparison | Items won | Items lost | Net gain | p |
|---|---|---|---|---|
| With reasoning vs Answer-only | omni-147, omni-50, omni-97 | omni-120, omni-60 | +1 | 1.00 |
| MADC vs Answer-only | omni-147, omni-50, omni-97 | None | +3 | 0.25 |
| MADC vs With reasoning | omni-120, omni-60 | None | +2 | 0.50 |
Relative to answer-only, with-reasoning wins omni-147/50/97 and loses omni-120/60 (net +1, p = 1.00); relative to answer-only, MADC wins three items with no losses (net +3, p = 0.25); and relative to with-reasoning, MADC wins omni-120/60 with no losses (net +2, p = 0.50). This precisely explains the gap between 40 and 38: MADC's 40 = 35 (solved by all three) + 3 (solved only by with-reasoning) + 2 (solved only by answer-only). The five additional items MADC solves are those on which the two plain-debate formats complement each other, and the remaining five items defeat all three. In short, MADC recovers the union of items already solvable by the two plain debate formats rather than extrapolating beyond the capability boundary of plain debate.
Regarding whether these results establish that MADC is superior, none of the observed net gains are statistically significant (p = 0.25 and p = 0.50). More fundamentally, our round-level diagnostics reveal that while sharing reasoning traces increases convergence, member accuracy, and final verdict accuracy, it simultaneously lowers oracle accuracy: across multi-round exchanges, a solitary agent that initially answered correctly is frequently swayed by the majority and flips to an incorrect answer, causing the panel to lose its only valid solution. This confirms that debate acts primarily as an amplification mechanism rather than a capability enhancer, and can actively erode accuracy via true-to-false flips.
Finally, regarding finite-sample variance, expanding our contested and all-wrong pools, as well as three independent seeds (0/1/2), we find that model responses fluctuate within their capability range, and the original 45-item sample indeed contains an element of coincidence. Under our expanded evaluation, our primary finding holds: MADC yields no statistically significant improvement over plain debate baselines.
Thank you for the detailed response and additional experiments. I have no major concerns, and I will increase the soundness score.
I am unsure it reached the highest overall score (4 - clear acceptance) though. I maintain my original positive score as 3.
Thanks!
Thank you very much for your thoughtful engagement throughout the discussion, and for raising the soundness score. We would like to add one remark on the broader motivation of this work, which we hope may help in your overall assessment.
Multi-agent debate has become a highly active area in LLM applications, with a steady stream of papers on improving and evaluating MAD systems — even at ICLR 2027, a keyword search for "multi-agent" returns many related studies. However, after carefully analyzing this large body of work, we were left with a set of persistent questions that the literature does not answer: once we set aside the widely used (and largely saturated) benchmarks and small-scale dummy models, how accurate is MAD in realistic settings, and can more rounds and more agents actually solve genuinely hard problems? These basic questions are, to our knowledge, missing from the literature. Although the headline outcome may seem predictable — debate helps surface more correct answers — no complete study had directly established it under controlled, difficulty-calibrated conditions.
Beyond these anticipated findings, our more fine-grained evaluation surfaced phenomena that may hinder the practical adoption of MAD: at the capability boundary, models rapidly converge (within ≤2 rounds) onto a plausible answer and abandon diverse exploration; in this regime, exchanging more context is counterproductive, and adding more rounds or agents only increases computational waste. Existing methods have various limitations and neither identify nor adequately address this failure mode.
We therefore believe this study offers concrete value to the MAD community: it provides the missing controlled measurements, a calibrated evaluation protocol, and actionable evidence about when debate helps, when it merely consumes budget, and why. We hope these contributions merit consideration for a higher overall rating.
Thank you once again for your time and careful consideration.
We thank reviewer 2DD3 for their time and constructive feedback. We address the main concerns below.
Regarding the reviewer's concern about whether debate truly possesses the capacity to create, we respond as follows.
First, we clarify the precise meaning of the capability boundary. We agree that the absence of a correct answer in initial samples is not an absolute proof of impossibility. To make this operational boundary more rigorous, we redesigned Round 0 around self-consistency: each agent generates three independent responses without peer visibility and enters the debate with its majority-voted answer. We have incorporated the details of our self-consistency design into Section 3. Thus, an all-wrong Round-0 indicates that no agent surfaced a valid solution across three independent draws per model, rather than a single-pass omission. Accordingly, we revise the statement in the Introduction from "never" to "rarely". Furthermore, we explicitly scope our conclusions to frontier tasks operating at the models' capability boundary, noting that on easier tasks well within capability limits, models may still generate correct answers.
To ensure greater statistical reliability, we substantially expanded our evaluation dataset, increasing the contested subset from 45 to 340 instances and the all-wrong subset from 40 to 220 instances.
Second, we provide a no-peer control that separates "interaction" from "sampling.":
The two settings also exhibit broadly consistent model-level rates of produced new correct answers and flips. Together, new correct answers are therefore driven by additional sampling, and interaction is responsible only for amplification, not creation.
Third, we correct the counting of creation. R3 reports the updated operational creation counts (items where all agents fail at Round 0 but reach a correct verdict by Round 3) on the expanded dataset after revised scoring. In homogeneous panels evaluated on their respective contested pools, creation counts decrease gradually from M1 to M5 (4 / 3.3 / 3.3 / 3 / 2 under answer-only; 7.3 / 6 / 5.7 / 2 / 2 under with-reasoning). The heterogeneous panel combines complementary model strengths led by M1, raising contested creation above the best single-model baseline from 4 to 5.7 (answer-only) and from 7.3 to 10.7 (with-reasoning). Even on the harder all-wrong pool, the heterogeneous panel integrates cross-model capabilities to achieve 4 creations under answer-only sharing and 8 under with-reasoning sharing.
Table R3. Operational creation counts after scoring corrections.
The single "generated item" previously noted in Appendix A.6 (item omni-60) was a classification artifact: under the corrected scoring, at least one agent already produced a valid answer at Round 0. Consequently, omni-60 represents an instance of amplification rather than creation, and we have reclassified it into the contested pool and updated Appendix A.6 accordingly.