Recoverable Disagreement: When and Why Multi-Agent Debate Outperforms Voting

Multi-Agent Debate (MAD) has been challenged by recent studies showing that it does not consistently outperform voting or inference-time reasoning methods such as chain-of-thought and self-consistency. However, we argue that this conclusion reflects prior studies’ reliance on average-accuracy comparisons, which fail to capture MAD’s potential in the settings where it may be most useful: difficult items where agents disagree and a correct answer exists as a minority signal. We formalize this setting as recoverable disagreement, reframing MAD as a mechanism for recovering correct information from disagreement rather than merely aggregating votes. To improve this recovery process, we introduce Confidence-Aware Debate (CAD), a debate protocol that derives each agent's response stability from repeated responses and makes this information explicit during debate, allowing agents to calibrate when to defend or revise their answers. Through a large-scale evaluation across diverse LLMs, debate groups, and full benchmark test sets, we show that CAD improves the effectiveness of debate relative to voting and existing debate variants, while remaining competitive with inference-time reasoning baselines. Item-level analyses show that CAD's gains concentrate in benchmarks with larger feasible recovery gaps and in harder questions where disagreement is more frequent. In the hardest regime, CAD better recovers correct minority answers and reduces cases where an initially wrong majority persists to the final decision. A targeted persuasion experiment further shows that internally unstable answers are substantially more likely to change under both misleading attacks and true corrections, supporting confidence as a measurable signal for answer revision. Our experimental results show that MAD outperforms voting by recovering useful signals from difficult disagreements where reliable reasoning matters.