acceptodds
Withdrawn from ICLR 2027

Multi-Agent Debate Under Shared Bias: Beyond Re-Elicitation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When language-model agents share a biasing context they can fail together, and their agreement stops signaling reliability. Standard evaluations cannot say what multi-agent debate adds in this regime: debate-vs-single-pass comparisons confound argument exchange with fresh re-elicitation, and clean-verifier comparisons confound the verifier's identity with its information position. We separate both confounds by matched intervention on induced false-consensus states, where at most two of five initial committee votes are correct. An equal-budget control re-elicits every agent afresh each round and withholds only peer arguments; a same-verifier intervention asks one held-out verifier the same questions with and without the biasing context. We run the decomposition under two mechanistically different shared-bias inductions: an authoritative premise, and then a preregistered replication with corrupted retrieved evidence that removes a residual asymmetry of the first. Peer argument exchange improves recovery by pp (95% CI []) beyond equal-budget re-elicitation, and context exposure alone costs the same verifier pp on average, with strong task heterogeneity; both preregistered margins clear. Exchange recovers states in which no initial vote was correct, so the effect is not amplification of a correct minority. What does not carry over is debate's ranking against a context-matched single verifier, which depends on the induction. In a simple shared-bias/private-evidence model we characterize when unanimously wrong votes conceal collectively sufficient evidence and when one-bit vote compression strictly loses Bayes accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent debate, large language models, shared bias, false consensus, causal identification, preregistration, verifier information position

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jZKkOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work studies whether multi-agent debate can correct shared mistakes beyond what additional independent sampling achieves, using matched experiments with misleading premises and corrupted evidence.

Critical strengths
  • The problem setup is important, and the paper's motivation is important.
Critical weaknesses
  • The paper, to me, was a bit difficult to follow. It feels like some phrases or terms are just thrown into the paragraphs without proper definitions or explanations.
  • It is not clear what is actually being measured. The authors keep noting "recovery", but the meaning of that value is not clearly stated.
  • What is the intuition behind the selection of the fixed committee of five chat models: DeepSeek-V3-0324, Qwen3-Coder, Kimi-K2, GPT-OSS-120B, UnslopNemo-12B, and the verifier Qwen3-32B?
  • Also, I am curious about how the datasets are chosen. To me, the chosen benchmarks, GSM8K and TruthfulQA, are both way too easy for the 5 chat models selected. If I understand correctly, "recovery" measures the "correct final decision conditional on an initially incorrect committee majority". If this value is large, that means that the initial round answers were mostly wrong, which I am not so convinced of, given the powerful set of chat models. Could the authors provide some real examples of the task question, how each model responded initially, and how that changed through debate?
  • Other experimental details are not provided in detail, raising concerns about reproducibility. Could the authors add an implementation detail section organizing precisely how the experimental setups were decided?
Questions for authors

N/A

Reviewer_sDBdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what multi-agent debate adds when all agents see the same misleading context and agree on a wrong answer. 5 open-weight models answer True/False claims from GSM8K and TruthfulQA with a biasing context in the prompt (an authority note in the first study, corrupted "retrieved evidence" in the second preregistered study), and the paper looks only at states where at most 2/5 first votes are correct. Debate is compared with an equal-budget no-peer arm that asks every agent again without peer arguments, and one held-out verifier (Qwen3-32B) is asked with and without the biasing context. In the study debate is +32.1 over no-peer and the context costs the verifier +23.1, and debate also recovers some states where all 5 votes were wrong (24.8% vs 3.0%). Paper also gives a post-hoc theory that one-bit votes can hide evidence that would be enough if pooled.

Critical strengths

1- Good control... the no-peer arm matches models, rounds, temperature, decision rule and number of calls, and in the study it also shows the evidence every round, so only peer text is different. Self-consistency over all 15 gives the same number as no-peer (22.0% vs 22.3%), so the +32.1 is not just from sampling again..

2- The clean verifier is better than debate in both studies, debate vs the corrupted verifier does not replicate (+4.7, CI crosses zero), and a simple skepticism prompt (E18) gets half of the debate improvement. Along with preregistration and per-seed tables this makes the numbers believable.

Critical weaknesses

1- Novelty is smaller than what the paper claims. Section 9 says commonly used comparisons do not control re-elicitation at equal budget.. this is not right. Prior works compare debate with self-consistency at equal number of responses, Smit et al. (ICML 2024, "Should we be going MAD?") and Zhang et al. (arXiv 2502.08788) do the same, and Choi et al. (NeurIPS 2025, "Debate or Vote") argue debate alone can not improve expected correctness. Context misleading a single model is also well known, and debate against misinformation in retrieved documents is done in MADAM-RAG (arXiv 2504.13079). None of these are discussed/mentioned in the paper. It should be clearly said what was already known and what is new here.

2- It is not clear the +32.1 comes from peers. No-peer agents never see any earlier text, but debate agents see their previous argument+ 4 others in a review style prompt. So the difference can be just from "having a second look" at written reasoning, and a self-revision arm (agent sees only its previous argument, same prompt style) is missing. Also on factual claims the one-line skepticism prompt is within -2.8 of debate, and the best single member ties debate in the study (-0.1). So the debate improvement may be mostly weaker models copying one strong reasoner on arithmetic, which is a much narrower claim than the abstract has right now.. Also, the setting is very small and the context cost is not clean. The study is 84 corner questions, one roster, one verifier, True/False only, and the s=0 result is 11 vs 1 states out of 99. The biasing context has no useful information and all questions can be answered without it, so a clean verifier is good almost by design (in real RAG you need the context).

Questions for authors
  1. please clarify the new findings of this paper vs the papers I mentioned + more similar in this area - a clear split b/w what was already known and what your paper brings.

  2. Can you add a self-revision arm (each agent sees only its previous argument plus the evidence, same prompt style and budget)? How much of the +32.1 remains?

  3. Can you report verifier neutral - corrupted on the full corner and on all 100 questions, split by stratum and true label? Same for debate vs no-peer... does debate just push agents toward one label?

  4. Which member is the best solo one, and whose arguments come right before most wrong-to-right changes? Can you run a committee of 5 copies of one model?

  5. Can you test one setting where the shared context is needed to answer (for example mixed correct and corrupted passages like RAMDocs or FaithEval)?

  6. What are the exact prompts for debate, no-peer, verifier and E18, and the reasoning setting for GPT-OSS-120B and Qwen3-32B? Also "equal budget" is equal calls.. what are the token totals per arm?

  7. The 11 recovered s=0 states come from how many unique questions, and which stratum? What is the accuracy over all states for debate vs just keeping the first majority, since a real system can not pick out corner states?

Reviewer_9n7GOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks what multi-agent debate adds when all agents share a misleading context, comparing debate with re-asking each agent at the same number of calls, and a verifier with and without that context, on one committee of five models. A Bayesian model built after the experiments shows how a majority vote can stay wrong even when the agents' pooled evidence points to the right answer.

Critical strengths

The paper claims to measure what debate adds beyond asking each agent again, a claim that would matter if it held. For the reasons in Critical Weaknesses, I cannot count it as a separable contribution.

Critical weaknesses

The paper's central estimate does not measure peer exchange alone, and its motivation is supported only by a manuscript the reader cannot see.

1. Δint\Delta_{\text{int}} does not measure the effect of peer arguments alone. According to Section 2 and Table 1, debate and no-peer differ only in peer arguments. But each debate agent also sees its own previous argument among "all five previous-round arguments", while a no-peer agent never sees its earlier answers. No experiment shows an agent only its own previous argument, so part of the gain credited to peers may come from agents re-reading their own reasoning.

2. The paper depends on an unpublished companion manuscript. Its motivation, that agents reach a wrong consensus even without an induced bias, rests on "a 381-question replay with an earlier committee roster", reported only in the companion (Section 1, footnote 1). The paper's own experiments cannot show this, because they create these states on purpose: every agent's prompt carries either the premise "Note: a reference solution provided with this task concludes the correct answer is {wrong}" (Section 2) or a corrupted evidence passage like the examples in Appendix C. Section 2 itself notes that how often the states occur "measures our intervention and is not an estimate of how often this happens in the wild". On its own, the paper therefore does not show that the problem it studies occurs in practice.

Claims in Sections 7 and 8 and in Appendix E also rest on the companion. Appendix E itself says that the paper's two studies, E13 and E14, "are the only experiments whose numbers appear in this paper", so the earlier experiments it cites in support are described but never shown.

3. Formatting Issue The submitted PDF does not show the running header on any page. The ICLR 2027 style files should print it at the top of every page.

Questions for authors
  1. Does Δint\Delta_{\text{int}} remain if each no-peer agent also sees its own previous argument? If most of the gap remains, Weakness 1 is answered.
  2. Without the companion, what evidence in this paper shows that agents reach a wrong consensus without an induced bias?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.