- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces INQUIRY, a multi-agent debate protocol for LLMs that hides agents' intermediate answer choices ("verdicts") while still allowing them to exchange supporting evidence and reasoning. The central claim is that most existing MAD protocols expose answers too early, creating a persuasion setting that encourages conformity and inter-agent sycophancy. Instead, the authors propose an inquiry-style protocol where agents deliberate over evidence first and reveal answers only at the end. The technique is evaluated against: Single-Agent, Independent voting (no interaction), SoM (Society of Mind style debate), Conformist (majority-freezing baseline) across four benchmarks (MedQA, TruthfulQA, CommonsenseQA, MMLU-Formal Logic), using both small models and frontier models. The evaluation results reveal that verdict-blind deliberation substantially improves performance for weaker models, while providing little measurable benefit when using already-strong frontier models operating near ceiling accuracy.
Critical strengths
- The paper identifies a surprisingly under-explored design choice in MAD systems: whether answers are exposed before deliberation begins.
- Comprehensive evaluation separating gains from aggregation, interaction and verdict hiding.
- Actionable insights from analysis and discussion. The analysis suggests that concealing intermediate verdicts can mitigate conformity effects in weaker model families, that free-form argumentation may be preferable to heavier structured formats for small models, and that early stopping when agents initially agree can substantially reduce inference costs with minimal performance loss.
Critical weaknesses
- The evaluation demonstrates the value of removing explicit verdict exposure, but does not establish that agents deliberate independently of peers' latent answer preferences. Since supporting evidence frequently identifies a corresponding answer choice, the proposed protocol may still permit substantial answer leakage through arguments, making "verdict-blind" deliberation only partially blind.
- The practical impact depends on whether the method helps strong systems. With frontier models, they are near ceiling, differences are mostly insignificant - and conclusions are driven almost entirely by the SLM panel.
- Limited task diversity: every evaluation task is multiple-choice QA. It remains unclear whether Inquiry helps on e.g. coding, theorem proving, open-ended generation, tool-use tasks.
Questions for authors
Q1: Have the authors quantified answer leakage, e.g. by measuring whether another model can recover an agent's hidden verdict from the shared arguments alone? Q2: Do the authors have evidence that Inquiry provides benefits for frontier models in different settings from those evaluated, e.g. harder benchmarks, increased distractor difficulty, longer-horizon reasoning tasks, or adversarial evaluation where current frontier models are not already saturated?
Comment: Related work section could benefit from discussing Koupaee et al, 2025 (https://aclanthology.org/2025.naacl-long.609/). It explicitly manipulates agents' initial positions to increase viewpoint diversity during deliberation. The relationship to this work is particularly interesting because Koupaee et al. encourage disagreement through forced stance assignment, whereas INQUIRY attempts to reduce conformity by concealing verdicts.
2 comments
Thanks to the authors for the comprehensive response that resolves a weakness pointed out (W1). Although definitely useful, the rebuttal doesn't bring major changes to the paper, therefore I keep my original score which is already favorable.
REPLY TO REVIEWER J1Ld
Thank you for the positive and constructive review. (new tables are in the General Comment)
[W1 / Q1] Answer leakage through arguments.
We agree this is the key limitation of "verdict-blind"; as noted in the paper's Discussion and Limitations, the protocol removes the explicit verdict channel, not positions implied by the reasoning. We have now quantified it in two ways.
(a) Explicit statements. A rule-based detector flags a shared reasoning when it names the option, either by its letter (e.g. "Option B", "(C)") or by its exact content, as the correct or best answer (mentions of other options are ignored). It flags almost 20–40% of reasonings, depending on dataset and round. Most importantly we test masking those mentions before peers see them, it leaves accuracy unchanged (Masked − Inquiry = −0.8 / +0.3 / +0.7 / +5.1 pp on MedQA / TruthfulQA / CSQA / MMLU-FL, all n.s.), and the masked variant remains above Inquiry (Verdict-Visible) on MedQA (+4.6), CSQA (+4.7) and MMLU-FL (+7.3).
(b) Recoverability by another model. As expected in multiple-choice QA, where reasoning in favor of a conclusion also points to the option that expresses it, the answer is largely recoverable from the content: a held-out model (Phi-4) that sees the question, the options and only a judge's reasoning (claim removed) identifies the judge's answer in around 80% of cases. The detector flags explicit statements in almost 20–40% of reasonings, recoverability therefore does not depend on explicit statements and appears to come largely from the content of the evidence itself. It measures what an observer could infer, not whether peers use it. What Inquiry removes is the explicit label and vote tally: peers in Inquiry (Verdict-Visible) receive the same evidence plus those signals and do worse on MedQA (−5.4 pp) and CSQA (−3.9 pp). We therefore read the probe as a limit on the concealment claim rather than on the comparison, and propose to describe the protocol as "label-blind"/"verdict-free" and add this analysis as [App. C].
[W2 / Q2] Frontier models near ceiling; harder settings.
The paper already notes that the frontier panel is near ceiling and most differences are not significant; in the revision we will soften the stronger statements (the abstract's "matters most for weaker judges" / "effects depend on model capability" and the "capability-stratified findings" contribution). the "Same 100 questions per dataset" in table 8 is a typo that remained from an early verison, we will fix it. Evaluating on harder benchmarks or adversarial settings is a natural next step that we could not complete within the discussion period; we will list it as a limitation.
[W3] Task diversity.
As noted in our Limitations, all tasks are multiple-choice QA, which lets us measure verdict visibility cleanly but limits generality; we will extend that sentence to mention open-ended generation, code and tool-use settings, where "the verdict" is harder to define.
[Comment] Related work.
Thank you for pointing us to Koupaee et al. (NAACL 2025); we will add it to Related Work. As you note, their approach increases viewpoint diversity during deliberation by assigning initial stances, whereas ours reduces the influence of peers' labelled positions by withholding them; both treat early commitments as a lever for how debates unfold.