acceptodds
Under review as a conference paper at ICLR 2027

Hide the Verdict, Keep the Evidence: Inquiry Dialogue for Multi-Agent LLM Debate

Abstract

Multi-Agent Debate (MAD) typically requires agents to publicly disclose their initial answers to peers before any argument is exchanged, creating a persuasion-like setting vulnerable to conformity. We introduce INQUIRY, a verdict-blind, inquiry-inspired protocol in which agents still form a private answer each round, but explicit verdicts are withheld from peers until a final decision round. Across four multiple-choice QA benchmarks, we compare INQUIRY against Single-Agent, Independent (no-debate) aggregation, MAD-SoM, and MAD-Conformist baselines using both a frontier and a small open-weight (SLM) judge panel. On the SLM panel, INQUIRY achieves the highest multi-agent accuracy on every dataset, outperforming Independent by 5.7-20.6 percentage points, while SoM, the dominant early-commitment protocol, is often the weakest multi-agent method; only on CommonsenseQA does INQUIRY fall short of the single strongest individual judge's high solo accuracy. On the frontier panel, near-ceiling accuracies leave far less room for protocols to differ, suggesting verdict exposure matters most for weaker judges. Correct Reasoning Survival and Minority Truth Recovery diagnostics show that INQUIRY better preserves correct majorities and recovers substantially more minority-held correct answers than the other baselines. A further ablation finds that a mandatory argument-structure schema (Claim, Grounds, Warrant) harms small-panel accuracy on three of four datasets but has no detectable effect at frontier scale. These results identify verdict exposure as a consequential MAD design choice whose effects depend on model capability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Debate, Large Language Models, LLM agents, Argumentation, Conformity, Sycophancy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_J1LdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces INQUIRY, a multi-agent debate protocol for LLMs that hides agents' intermediate answer choices ("verdicts") while still allowing them to exchange supporting evidence and reasoning. The central claim is that most existing MAD protocols expose answers too early, creating a persuasion setting that encourages conformity and inter-agent sycophancy. Instead, the authors propose an inquiry-style protocol where agents deliberate over evidence first and reveal answers only at the end. The technique is evaluated against: Single-Agent, Independent voting (no interaction), SoM (Society of Mind style debate), Conformist (majority-freezing baseline) across four benchmarks (MedQA, TruthfulQA, CommonsenseQA, MMLU-Formal Logic), using both small models and frontier models. The evaluation results reveal that verdict-blind deliberation substantially improves performance for weaker models, while providing little measurable benefit when using already-strong frontier models operating near ceiling accuracy.

Critical strengths
  1. The paper identifies a surprisingly under-explored design choice in MAD systems: whether answers are exposed before deliberation begins.
  2. Comprehensive evaluation separating gains from aggregation, interaction and verdict hiding.
  3. Actionable insights from analysis and discussion. The analysis suggests that concealing intermediate verdicts can mitigate conformity effects in weaker model families, that free-form argumentation may be preferable to heavier structured formats for small models, and that early stopping when agents initially agree can substantially reduce inference costs with minimal performance loss.
Critical weaknesses
  1. The evaluation demonstrates the value of removing explicit verdict exposure, but does not establish that agents deliberate independently of peers' latent answer preferences. Since supporting evidence frequently identifies a corresponding answer choice, the proposed protocol may still permit substantial answer leakage through arguments, making "verdict-blind" deliberation only partially blind.
  2. The practical impact depends on whether the method helps strong systems. With frontier models, they are near ceiling, differences are mostly insignificant - and conclusions are driven almost entirely by the SLM panel.
  3. Limited task diversity: every evaluation task is multiple-choice QA. It remains unclear whether Inquiry helps on e.g. coding, theorem proving, open-ended generation, tool-use tasks.
Questions for authors

Q1: Have the authors quantified answer leakage, e.g. by measuring whether another model can recover an agent's hidden verdict from the shared arguments alone? Q2: Do the authors have evidence that Inquiry provides benefits for frontier models in different settings from those evaluated, e.g. harder benchmarks, increased distractor difficulty, longer-horizon reasoning tasks, or adversarial evaluation where current frontier models are not already saturated?

Comment: Related work section could benefit from discussing Koupaee et al, 2025 (https://aclanthology.org/2025.naacl-long.609/). It explicitly manipulates agents' initial positions to increase viewpoint diversity during deliberation. The relationship to this work is particularly interesting because Koupaee et al. encourage disagreement through forced stance assignment, whereas INQUIRY attempts to reduce conformity by concealing verdicts.

2 comments
AuthorsOpenReview ↗

REPLY TO REVIEWER J1Ld

Thank you for the positive and constructive review. (new tables are in the General Comment)

[W1 / Q1] Answer leakage through arguments.
We agree this is the key limitation of "verdict-blind"; as noted in the paper's Discussion and Limitations, the protocol removes the explicit verdict channel, not positions implied by the reasoning. We have now quantified it in two ways.
(a) Explicit statements. A rule-based detector flags a shared reasoning when it names the option, either by its letter (e.g. "Option B", "(C)") or by its exact content, as the correct or best answer (mentions of other options are ignored). It flags almost 20–40% of reasonings, depending on dataset and round. Most importantly we test masking those mentions before peers see them, it leaves accuracy unchanged (Masked − Inquiry = −0.8 / +0.3 / +0.7 / +5.1 pp on MedQA / TruthfulQA / CSQA / MMLU-FL, all n.s.), and the masked variant remains above Inquiry (Verdict-Visible) on MedQA (+4.6), CSQA (+4.7) and MMLU-FL (+7.3).
(b) Recoverability by another model. As expected in multiple-choice QA, where reasoning in favor of a conclusion also points to the option that expresses it, the answer is largely recoverable from the content: a held-out model (Phi-4) that sees the question, the options and only a judge's reasoning (claim removed) identifies the judge's answer in around 80% of cases. The detector flags explicit statements in almost 20–40% of reasonings, recoverability therefore does not depend on explicit statements and appears to come largely from the content of the evidence itself. It measures what an observer could infer, not whether peers use it. What Inquiry removes is the explicit label and vote tally: peers in Inquiry (Verdict-Visible) receive the same evidence plus those signals and do worse on MedQA (−5.4 pp) and CSQA (−3.9 pp). We therefore read the probe as a limit on the concealment claim rather than on the comparison, and propose to describe the protocol as "label-blind"/"verdict-free" and add this analysis as [App. C].

[W2 / Q2] Frontier models near ceiling; harder settings.
The paper already notes that the frontier panel is near ceiling and most differences are not significant; in the revision we will soften the stronger statements (the abstract's "matters most for weaker judges" / "effects depend on model capability" and the "capability-stratified findings" contribution). the "Same 100 questions per dataset" in table 8 is a typo that remained from an early verison, we will fix it. Evaluating on harder benchmarks or adversarial settings is a natural next step that we could not complete within the discussion period; we will list it as a limitation.

[W3] Task diversity.
As noted in our Limitations, all tasks are multiple-choice QA, which lets us measure verdict visibility cleanly but limits generality; we will extend that sentence to mention open-ended generation, code and tool-use settings, where "the verdict" is harder to define.

[Comment] Related work.
Thank you for pointing us to Koupaee et al. (NAACL 2025); we will add it to Related Work. As you note, their approach increases viewpoint diversity during deliberation by assigning initial stances, whereas ours reduces the influence of peers' labelled positions by withholding them; both treat early commitments as a lever for how debates unfold.

Thanks to the authors for the comprehensive response that resolves a weakness pointed out (W1). Although definitely useful, the rebuttal doesn't bring major changes to the paper, therefore I keep my original score which is already favorable.

Reviewer_BdDqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether exposing agents to one another's initial answers causes harmful conformity in multi-agent debate. The authors propose INQUIRY, a protocol in which agents maintain private verdicts while exchanging reasoning and evidence, revealing their final answers only after deliberation.

The method is evaluated against independent aggregation, Society-of-Minds (SoM), and a Conformist debate protocol, using heterogeneous panels of smaller open-weight models and larger proprietary models across four multiple-choice QA benchmarks.

The empirical results are substantial on the smaller-model panel. INQUIRY improves accuracy over independent aggregation by 5.7–20.6 percentage points and outperforms the evaluated debate baselines on all four datasets. The authors additionally introduce Correct Reasoning Survival (CRS) and Minority Truth Recovery (MTR) diagnostics to analyze how deliberation changes collective decisions.

The central idea is sensible, and the empirical results deserve attention. My main objection is that the experiments do not actually isolate the effect of verdict concealment.

INQUIRY changes considerably more than the visibility of peer answers. The initial prompts, reasoning-length constraints, peer-review instructions, and presentation of arguments all differ from the baselines. Consequently, the observed improvements cannot be attributed specifically to hiding verdicts.

The paper demonstrates that a particular evidence-focused deliberation protocol works better than the evaluated alternatives. It does not convincingly establish that verdict concealment is the mechanism responsible.

For a paper whose central contribution is precisely this distinction, that is a serious limitation.

Critical strengths
  1. The research question is interesting and practically relevant.

Multi-agent debate research has frequently treated communication as beneficial by default, despite substantial evidence that additional interaction can reinforce mistakes or produce premature consensus.

The authors identify a plausible design problem: exposing an agent to a peer's explicit answer may encourage it to imitate that answer rather than independently assess the supporting evidence.

Separating verdicts from arguments is a reasonable intervention, and the idea is simple enough to be widely applicable.

  1. The small-model improvements are substantial.

Table 2 reports INQUIRY accuracy of 67.8% on MedQA, 74.9% on TruthfulQA, 80.1% on CommonsenseQA, and 58.7% on MMLU Formal Logic.

These are improvements of 10.8, 20.6, 5.7, and 13.5 percentage points over independent aggregation, respectively.

Unlike many multi-agent systems papers, the contribution is not presented solely through marginal one-point improvements on a composite benchmark. Several of these effects are large enough to warrant serious investigation.

  1. The paper attempts to explain the observed behavior rather than reporting accuracy alone.

The CRS/MTR decomposition is useful. Distinguishing between preserving an initially correct majority and recovering a correct minority position provides a more informative perspective than final accuracy alone.

I also appreciate the authors' acknowledgment that the Conformist protocol achieves perfect CRS mechanically, and that protecting existing consensus is not equivalent to improving collective reasoning.

  1. The appendices are reasonably detailed.

The authors include exact prompt templates, decoding parameters, answer parsing procedures, conditioning-set sizes, and additional ablations.

The short-circuiting results are particularly useful: avoiding unnecessary rounds saves substantial computation without a detectable accuracy reduction.

These details make the paper easier to assess and potentially reproduce.

Critical weaknesses

Major Weaknesses

1. The central causal claim is not experimentally isolated (primary concern).

The paper attributes INQUIRY's improvements to withholding explicit peer verdicts, but the protocols differ in several other important respects. Appendix A.1 shows that INQUIRY uses more explicit evidence-evaluation and critical-reasoning instructions, different Round-1 prompts, and a larger reasoning budget (100 versus 50 words). These differences could independently explain the observed gains. Moreover, the paper lacks a matched multi-round independent-reflection baseline to establish whether peer communication is necessary. The most important missing experiment is a controlled ablation using identical initial responses, deliberation instructions, and token budgets, varying only the visibility of peer verdicts. A matched self-reflection control would further distinguish the benefits of exchanging evidence from those of improved prompting and repeated reconsideration. Without these controls, the central causal interpretation remains insufficiently supported.

2. The CRS/MTR analysis does not control for differences in initial judgments.

The paper interprets higher Correct Retention Share (CRS) and Minority-to-Truth Recovery (MTR) as evidence that INQUIRY produces better revision dynamics. However, because the Round-1 prompts differ, these metrics condition on different populations of questions. For example, Appendix A.4 reports MedQA CRS conditioning sets of 310 examples for INQUIRY versus 284 for SoM, and MTR sets of 238 versus 265. Differences in question difficulty and initial correctness may therefore contribute to the observed effects. A stronger analysis would hold initial judge predictions fixed across protocols and compare subsequent transitions question by question. As presented, these metrics characterize each protocol's behavior but do not isolate the effect of verdict concealment on revision quality.

3. The independent-aggregation baseline is sensitive to an unfavorable decision rule.

Appendix A.3 automatically scores three-way voting splits as incorrect, disproportionately penalizing independent aggregation, which produces such splits on 2.2–15.1% of questions across the smaller-model datasets. Alternative tie-breaking rules could materially change the comparison. This matters particularly because the small-model panel is heterogeneous: Qwen2.5-7B independently achieves 82.6% on CommonsenseQA, exceeding INQUIRY's 80.1%, and 57.9% on MMLU, close to INQUIRY's 58.7%. The paper should report sensitivity to reasonable tie-breaking strategies and include a competence-aware independent-aggregation baseline. Otherwise, the reported gains over independent voting may partly reflect the aggregation rule rather than the value of deliberation.

4. Hiding explicit verdicts does not necessarily conceal agents' preferred answers.

In multiple-choice QA, agents may readily infer a peer's preferred option from its reasoning, even when the explicit answer field is removed. This is especially relevant to the Toulmin variant, where arguments directly support or refute candidate options. Consequently, the intervention may reduce the salience of peer verdicts rather than conceal the underlying positions. The authors should quantify how reliably withheld verdicts can be reconstructed from the shared arguments, for example using an independent model. This would clarify whether the observed benefits arise from genuine information concealment or a change in how peer preferences are presented.

5. Some statistical and generalization claims exceed the available evidence.

The paper's central comparison is INQUIRY versus explicit-verdict debate, yet the main-table significance markers primarily test against Independent rather than directly against SoM and Conformist. Paired comparisons against these protocols, with appropriate consideration of multiple testing, would better support the main conclusions. Smaller improvements also require more cautious interpretation: on the 126-example MMLU subset, INQUIRY exceeds the strongest individual judge by less than one percentage point. Similarly, the claim that verdict concealment matters more for weaker models is difficult to establish because the frontier-model panel uses only 100 questions per dataset, differs in model composition and evaluation samples, and exhibits near-ceiling performance. Matched question sets across model capability levels would provide stronger evidence for this claim. Finally, an accuracy-versus-token-cost comparison under matched reasoning budgets would help establish whether INQUIRY's benefits justify its computational cost.

Questions for authors

Q1. Can the authors rerun the experiment while changing only verdict visibility, holding Round-1 outputs, peer-review prompts, reasoning budgets, and argument formats fixed? How much of INQUIRY's advantage remains?

Q2. How do CRS and MTR compare when all protocols start from identical initial agent predictions? What fraction of the observed differences can be attributed to the different Round-1 prompting procedures?

Q3. How does INQUIRY compare against independent aggregation with a competence-aware voting rule or a predetermined tie-breaker, rather than automatically scoring all three-way splits as incorrect?

Q4. How does the protocol compare against multi-round self-reflection using the same evidence-focused prompting instructions but without peer communication?

Q5. Can the authors measure how accurately the hidden verdict can be inferred from the disclosed argument? Is the intervention actually concealing a position, or merely removing its explicit label?

Q6. Can the authors provide direct paired significance tests against SoM and Conformist, along with a full token-budget comparison and results on matched question subsets across the two model-capability panels?

2 comments
AuthorsOpenReview ↗

REPLY TO REVIEWER BdDq - Part 1/2

Thank you for the thorough review. We agree with your main objection that the original comparison did not isolate verdict concealment, and we have run new controls along the lines you describe. Our answer to your primary concern is in [W1]. (new tables are in the General Comment)

Part 1/2 (W1/Q1, Q4, W2/Q2)
[W1 / Q1] Isolating verdict visibility.
we re-ran all protocols on the SLM panel (all items of all four datasets) from a single shared Round-1 output per judge and question, a 50-word budget in every round. In Rounds 2 and 3 Inquiry uses its own evidence-focused wording, since that wording is the idea under study, Inquiry (Verdict-Visible) isolates the effect of verdict visibility with otherwise identical prompts. We also added the verdict-visible Inquiry variant (identical prompts; peers see claims). Hiding verdicts gives a significant gain on MedQA (+5.4 pp, 60.5 vs 55.1; 95% CI [+3.6, +7.2]) and CSQA (+3.9 pp; [+2.0, +5.8]), and no detectable difference on TruthfulQA (−0.4) or MMLU-FL (+3.2, n=126). The same two effects appear with a 100-word budget (+3.0, +1.8). So a controlled verdict-visibility effect now exists on two of the four datasets. Relative to SoM, Inquiry is +8.2 / +5.0 / +5.2 points on MedQA / TruthfulQA / CSQA (all Holm-significant); this advantage combines verdict concealment with Inquiry's evidence-focused round design, and we will say so explicitly [Sec. 1, Sec. 5] instead of attributing it to concealment alone. We will also reword the contribution as an evidence-focused, verdict-blind deliberation protocol.

[Q4] Self-reflection control.
We added a multi-round self-reflection control with no peers: same shared Round 1, same system prompt, three rounds, 50-word budget, majority vote, with a generic reconsideration prompt in which each judge is shown its own previous answer and reasoning. It performs like SoM and below Inquiry on MedQA (52.8 vs 60.5; p<.001) and CSQA (67.2 vs 73.8; p<.001); TruthfulQA (56.3 vs 58.8) and MMLU-FL (45.2 vs 50.0) are not significant.

[W2 / Q2] CRS/MTR with identical initial judgments.
All protocols now share the same Round-1 predictions, so the conditioning sets are identical for every protocol (MedQA CRS n=287, MTR n=270; TruthfulQA 141/170; CSQA 235/192; MMLU-FL 26/40). Differences between protocols are therefore purely differences in what happens after Round 1. Among all protocols, Inquiry (Verdict-blind) has the highest H on MedQA, CSQA and MMLU-FL, and is within 2 points of Visible on TruthfulQA. This will replace the earlier Table 6, and we will no longer claim the highest H on every dataset.

AuthorsOpenReview ↗

REPLY TO REVIEWER BdDq - Part 2/2

Part 2/2 (W3/Q3, W4/Q5, W5 / Q6)
[W3 / Q3] Tie-breaking and competence-aware aggregation.
Ties are scored wrong in the paper for all protocols. We re-scored every protocol with a random tie-break and will report these numbers in the appendix. Independent rises to 58.6 / 56.3 / 75.4 / 51.3 (MedQA / TruthfulQA / CSQA / MMLU-FL) and Inquiry (Verdict-Blind) to 60.8 / 59.3 / 73.8 / 50.3, so Inquiry's margin over Independent becomes +2.2 / +3.0 / −1.6 / −1.0 pp. We will therefore moderate the claims relative to Independent. About Qwen: in the matched runs it is the strongest single judge and is above all multi-agent protocols on TruthfulQA, CSQA and MMLU-FL (and statistically level with Inquiry on MedQA). We will state in the text that our comparisons concern relative performance among multi-agent protocols and do not claim to beat the best single model in this heterogeneous panel.

[W4 / Q5] Can the hidden verdict be reconstructed?
We looked at this in two ways. (a) Explicit statements. A rule-based detector flags a shared reasoning when it names the option, either by its letter (e.g. "Option B", "(C)") or by its exact content, as the correct or best answer (mentions of other options are ignored). It flags almost 20–40% of reasonings, depending on dataset and round. We then masked those mentions before peers saw them: accuracy is unchanged (Masked − Inquiry = −0.8 / +0.3 / +0.7 / +5.1 pp on MedQA / TruthfulQA / CSQA / MMLU-FL, all n.s.), and the masked variant remains above Inquiry (Verdict-Visible) on MedQA (+4.6), CSQA (+4.7) and MMLU-FL (+7.3).
(b) Recoverability by another model. As expected in multiple-choice QA, where reasoning in favor of a conclusion also points to the option that expresses it, the answer is largely recoverable from the content: a held-out model (Phi-4) that sees the question, the options and only a judge's reasoning (claim removed) identifies the judge's answer in around 80% of cases. The detector flags explicit statements in almost 20–40% of reasonings, recoverability therefore does not depend on explicit statements and appears to come largely from the content of the evidence itself. It measures what an observer could infer, not whether peers use it. What Inquiry removes is the explicit label and vote tally: peers in Inquiry (Verdict-Visible) receive the same evidence plus those signals and do worse on MedQA (−5.4 pp) and CSQA (−3.9 pp). We therefore read the probe as a limit on the concealment claim rather than on the comparison, and propose to describe the protocol as "label-blind"/"verdict-free" and add this analysis as [App. C].

[W5 / Q6] Statistics, MMLU, cost, matched subsets.
Paired tests vs SoM and Conformist. Added for every dataset, paired exact McNemar, Holm-corrected over all 16 comparisons. Inquiry vs SoM: significant on MedQA, TruthfulQA and CSQA after correction; MMLU-FL not significant (n=126). Inquiry vs Conformist: no significant difference on any dataset after correction (+2.1 / +1.6 / −1.5 / 0.0 pp). Conformist reaches its accuracy by freezing agreed judges, so it never changes a correct majority and recovers few minority-correct questions (MTR 3–12% vs 24–35% for Inquiry); we will report this next to the accuracy comparison.
MMLU-FL. We agree that the margin on 126 items is too small to support claims; in the revision we will describe MMLU-FL results as inconclusive and show the confidence intervals.
Cost. Under the matched budgets Inquiry, SoM and the self-reflection control all make 3 judges × 3 rounds = 9 calls per question with the same 50-word budget per call (about 3× Independent). Inquiry's optional round-1 short-circuit removes Rounds 2–3 on questions where the judges already agree (33–72% of SLM questions in the original runs, A.6); an accuracy-versus-token table will be included.
Matched subsets across panels. the " 100 questions per dataset" in table 8 is a typo that remained from an early version, we will fix it. These results come from the original prompts and we did not yet re-run under the matched protocol; they are reported as a ceiling check and we draw no protocol-level conclusions from them. We therefore do not claim a capability effect, and we will soften the abstract accordingly.

Reviewer_mL6LOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether agents in multi-agent debate should see each other's answers while they argue. The proposed protocol, INQUIRY, keeps each judge's claim private. Judges exchange only their reasoning, and all answers are revealed together in the last round. The baselines are Single-Agent, Independent (plurality vote, no interaction), SoM, and Conformist. The datasets are MedQA, TruthfulQA, CommonsenseQA, and MMLU Formal Logic. There is one panel of small open-weight models and one frontier panel. With the small models, INQUIRY gives the best multi-agent accuracy on all four datasets and beats Independent by 5.7 to 20.6 points. With the frontier models, everything sits close to ceiling and most differences aren't significant.

Critical strengths
  1. The idea is simple and easy to state: hide the explicit verdict, keep the arguments. That makes it a clean variable to study.

  2. I liked the Independent baseline, which separates the effect of voting from the effect of interaction. The small-panel experiments use every item of the four benchmarks as preprocessed in Table 1. The comparisons rely on paired exact McNemar tests.

  3. The reporting is honest. All prompts are given verbatim (App. A.1). The null results on the frontier panel and the negative Toulmin result are both reported. The paper also correctly points out that Conformist's 100% CRS comes from its freezing rule.

Critical weaknesses
  1. Prompt confound. Sec. 4 says the five protocols "differ only in M, T, R_P, D" plus two optional mechanisms. The prompts in App. A.1 differ in more ways than that. Single-Agent, Independent, and Round 1 of SoM and Conformist all use "Answer this question." with a 50-word reasoning field. INQUIRY's Round 1 instead says "Think through the evidence for and against each option before answering". INQUIRY also allows 100 words of reasoning in every round. Judging from Fig. 4, this already matters before any interaction happens. Llama-3.1-8B gets 845 MedQA questions right at Round 1 under INQUIRY, versus 730 under SoM. Gemma-3n-E4B gets 458 versus 332 on TruthfulQA. On top of that, INQUIRY's Round-2 prompt asks for "a specific factual counter-argument", while SoM frames the peers' answers as "additional advice". So I don't think the gains can be credited to hiding verdicts alone yet.

  2. CRS definition. Eq. (4) defines M with Σ g ≥ 2. That would include questions where all three judges are right at Round 1. The sizes in Table 6, however, look like they only count questions with exactly two correct judges. For SoM on CommonsenseQA, Table 6 has n = 238. Yet A.6 reports 748 Round-1-unanimous SoM questions at 89.71% Round-1 accuracy, which is roughly 670 unanimous-correct questions.

  3. Frontier panel. Table 8 (App. A.5) indicates 100 questions per dataset. Neither Sec. 4 nor Table 3 gives this number or says how the questions were picked. With one run per protocol and accuracies near ceiling, this panel can't tell us much about capability dependence. The abstract still concludes that the effects of verdict exposure "depend on model capability".

Questions for authors
  1. (W1) Could you report the plurality accuracy of INQUIRY's own Round-1 private claims? It should be computable from the existing logs. Could you also run a verdict-visible version of INQUIRY with otherwise identical prompts? Doing this on the small panel for at least MedQA and TruthfulQA would already help.

  2. (W2) Is M in Eq. (4) supposed to mean exactly two correct judges? Either way, please make Eq. (4) and Table 6 consistent. It would also be useful to report CRS and MTR on the questions that qualify under all three protocols.

  3. (W3) How were the 100 frontier questions per dataset chosen? Please give the sample size in Sec. 4 and Table 3, and add confidence intervals. I'd also bring the capability claim in the abstract in line with the more careful wording in the introduction.

1 comment
AuthorsOpenReview ↗

REPLY TO REVIEWER mL6L

Thank you for the detailed review, and for the very precise pointers. Your W1 was the most important point and led to the main new experiments. (new tables are in the General Comment)

[W1 / Q1] Prompt confound. You are right that the original protocols differed in more than verdict visibility, and that the Round-1 prompt mattered before any interaction. To remove this, we re-ran all protocols on the SLM panel (all items of all four datasets) from a single shared Round-1 output per judge and question, a 50-word budget in every round. In Rounds 2 and 3 Inquiry uses its own evidence-focused wording, since that wording is the idea under study. Inquiry (Verdict-Visible) isolates the effect of verdict visibility with otherwise identical prompts. In the previous runs (before these changes) Inquiry's own Round-1 plurality accuracy was higher than Independent's, which accounts for a substantial part of the original margin. Results are in the general comment.
We also added the verdict-visible Inquiry variant you requested (identical prompts; peers see claims). Under the new matched Round 1, the verdict-hiding effect (in Inquiry) is +5.4 pp on MedQA (95% CI [+3.6, +7.2]) and +3.9 pp on CSQA ([+2.0, +5.8]), and not significant on TruthfulQA (−0.4) or MMLU-FL (+3.2).
The 100-word limit in prompt replication gave the same picture, with no significant change. In the revision, the main tables, Fig. 2 and Fig. 3 will use the new matched runs, and we will replace the sentences in this regard.

[W2 / Q2] CRS definition.
Eq. (4) was a typo: the conditioning set is questions with exactly two correct judges at Round 1 (as in Table 6); we will correct the equation. Because all protocols now share Round 1, the CRS and MTR conditioning sets are identical across protocols (MedQA CRS n=287, MTR n=270; TruthfulQA 141/170; CSQA 235/192; MMLU-FL 26/40), which also addresses the comparability concern. On these shared conditioning sets, Inquiry (Verdict-blind) has the highest H on MedQA, CSQA and MMLU-FL, and is within 2 points of Visible on TruthfulQA. Conformist's CRS of 100% is mechanical, because it freezes agreed judges. Its MTR is only 3–12%, against 24–35% for Inquiry (Verdict-Blind). The MMLU-FL conditioning sets are small (26 and 40 questions), so those figures should be read with caution.

[W3 / Q3] Frontier panel.
the " 100 questions per dataset" in table 8 is a typo that remained from an early version, we will fix it. These results come from the original prompts and we did not yet re-run under the matched protocol; they are reported as a ceiling check and we draw no protocol-level conclusions from them. We agree that some statements were stronger than the evidence, we could soften them to say that the comparison with the frontier panel is suggestive of, but does not establish, a dependence on model capability, consistent with the near-ceiling caveat already in the main text.

We hope the matched experiments resolve the main concern, and we are glad to answer follow-up questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

GENERAL COMMENT

Summary of the author response and revision
We thank the reviewers for careful, constructive reviews. Three main issues were raised: (i) the protocols differ in more than verdict visibility (prompt confound), (ii) hidden verdicts may remain inferable from the shared reasoning, and (iii) statistics and the scope of the frontier panel. We addressed them and we will upload a revised PDF in the coming days.

1. Matched-protocol re-evaluation (all four datasets, all items, SLM panel).

Every protocol now starts from the same Round-1 output of each judge on each question (one shared Round-1 prompt), uses the same system prompt in the debate rounds and the same 50-word reasoning budget in all rounds. We added Inquiry (Verdict-Visible), which is identical to Inquiry except that verdicts are shown to peers. CRS/MTR are now computed on identical conditioning sets for every protocol. Going from 50 to 100 words limit in prompt changes no protocol significantly.

Accuracy % (matched)MedQATruthfulQACSQAMMLU-FL
Independent57.054.174.747.6
SoM52.3 *53.868.6 *44.4
Conformist58.4 *57.2 *75.3 *50.0
Inquiry (Verdict-Visible)55.159.2 *69.9 *46.8
Inquiry (Verdict-Blind)60.5 *58.8 *73.850.0

* p<.05: paired exact McNemar against Independent on the same questions (same convention as the original Table 2). Inquiry (Verdict-Blind) vs Inquiry (Verdict-Visible): MedQA +5.4 * and CSQA +3.9 * are significant; TruthfulQA −0.4 and MMLU-FL +3.2 are not. The Holm-corrected comparisons are in point 2.

2. What the matched evaluation supports (paired exact McNemar, Holm-corrected over 16 comparisons)

Inquiry beats SoM on MedQA (+8.2), TruthfulQA (+5.0) and CSQA (+5.2); beats Independent on MedQA (+3.5) and TruthfulQA (+4.7); and hiding verdicts (Verdict-Blind vs Verdict-Visible) gives a significant gain on MedQA (+5.4) and CSQA (+3.9). Differences from Conformist, and all MMLU-FL differences (n=126), are not significant. The absolute margins are smaller than in the original Table 2, because the earlier comparison also contained Round-1 prompt differences; in the revision we will report the matched numbers as the main results.

Diagnostics versus Conformist. Although accuracy is comparable, the two protocols behave differently. Conformist freezes any judge that agrees with the previous round's majority, so its CRS is 100% by construction and it cannot recover a correct minority (MTR 3–12%). Inquiry (Verdict-Blind) preserves most correct majorities (CRS 73–85%) and recovers minority-held correct answers in 24–35% of cases, so its harmonic mean H is much higher than Conformist's on all four datasets (MedQA 44.8 vs 11.2, TruthfulQA 48.2 vs 21.1, CSQA 36.6 vs 6.1, MMLU-FL 37.3 vs 14.0). These diagnostics use identical conditioning sets for all protocols (small for MMLU-FL: 26/40 questions).

Panel-level CRS / MTR / H (%), matched runsSoMConformistInquiry (Verdict-Visible)Inquiry (Verdict-Blind)
MedQA (n = 287 / 270)61.0 / 18.5 / 28.4100 / 5.9 / 11.270.0 / 23.3 / 35.085.0 / 30.4 / 44.8
TruthfulQA (141 / 170)66.0 / 29.4 / 40.7100 / 11.8 / 21.175.2 / 37.6 / 50.278.7 / 34.7 / 48.2
CSQA (235 / 192)59.1 / 17.2 / 26.6100 / 3.1 / 6.163.8 / 16.1 / 25.877.4 / 24.0 / 36.6
MMLU-FL (26 / 40)53.8 / 17.5 / 26.4100 / 7.5 / 14.073.1 / 10.0 / 17.673.1 / 25.0 / 37.3

(n = conditioning-set sizes for CRS / MTR, identical across protocols; bold = highest H.)

3. Leakage.

We quantified how well an LLM ( a held-out model) can recover a judge's hidden answer from the shared reasoning, and ran a control that masks explicit answer statements in the shared text. Details are in the replies to BdDq (Q5) and J1Ld (Q1) and will be added as [App. C]. In short: as the paper's Limitations already note, the protocol hides the explicit verdict, not the evidence that supports it; we have now quantified this and propose to adjust the wording accordingly.

We respond to each reviewer individually as well.

AuthorsOpenReview ↗

Revised PDF uploaded: summary of changes

We thank the reviewers. The revised PDF reflects our replies, and the main changes are below.

Main text

  • Protocols (Sec. 3): the "differ only in M, T, R_P, D" sentence is replaced. We now state what is shared (Round 1, system prompt and history in the debate rounds, 50-word budget) and what differs (peer-round instructions, what peers see). Inquiry (Verdict-Visible) is defined, and Eq. 4 now uses "exactly two".
  • Table 2 and Figures 2-4: now the shared-Round-1 runs on all questions, with Verdict-Visible added and a single * for p<.05 against Independent. The appendix gives Holm-adjusted pairwise tests against every baseline (16 tests). CRS/MTR use identical conditioning sets.
  • Frontier panel: the sample size and sampling are now stated in Sec. 3, Table 3 and the appendix, with 95% Wilson intervals.
  • Claims: the Abstract, Introduction, Discussion and Conclusion are revised to match. The verdict effect is dataset-dependent, the frontier panel is a ceiling check, capability dependence is a hypothesis, and the strongest single judge is stated explicitly. The Limitations are extended. Koupaee et al. is added.

New appendices: the ablation configuration, leakage controls (detector, masking, recoverability probe), the self-reflection control, tie-break sensitivity, Holm table, confidence intervals, token and latency cost, the 100-word replication, and the corrected implementation details.

Corrections to our earlier replies

  • Frontier panel: we said the "100 questions per dataset" in Table 8 was a typo. That was incorrect. The frontier panel uses a random sample of 100 questions per dataset, identical for all protocols. The accuracies in the tables are unchanged.
  • Masked vs Inquiry on MMLU-FL is +7.1 (not +5.1), and masked vs Visible is +10.3 (not +7.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.