acceptodds
Under review as a conference paper at ICLR 2027

Multi-agent discussion gains less when dissent is withheld

Abstract

Multi-agent systems of LLMs add discussion to majority voting and are therefore expected to be more capable. However, empirical reports conflict on whether discussion improves accuracy or leads to an incorrect consensus. Here, we introduce a parsimonious model that explains when discussion improves accuracy and when it ends in an incorrect consensus, built from four behaviors repeatedly observed in LLM agents: (1) withholding dissent, (2) internalizing a stated answer, (3) reconsidering after seeing dissent, and (4) correcting toward the correct answer. The model shows that discussion can overturn an incorrect initial majority only when the withholding rate is below a critical rate , set by the net correction rate and the internalization rate . We estimate these rates from conversation logs with a Bayesian method and place LLM teams relative to . As the model predicts, the gain from discussion shrinks as withholding rises, across LLMs and on a hidden profile benchmark, HiddenBench, and MedEInst. Instructing agents not to withhold dissent increases this gain. Turning reasoning off also increases the gain, because reasoning raises the internalization rate and keeps agents from reconsidering a minority answer. These findings reconcile the conflicting reports and identify when discussion outperforms majority

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
multiagent systems, phase transition, consensus making

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KSoxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates if and when discussion improves over majority voting in LLM-based multi-agent systems. The proposed model uses four observed behaviors two explain when discussion improves the performance versus when the agents consense on an incorrect answer: withholding dissent, internalizing a stated answer, reconsidering when seeing counter-evidence, and correcting toward the correct answer. Following from the proposed model, discussion can only overturn an initially incorrect majority, if the so-called withholding-rate is below a critical rate (that is based on the correction and the internalization rates).

Critical strengths

The paper is relevant and interesting. It tries to unify the conflicting reports on the performance and behavior of multi-agent systems, which are increasingly used in practice. Therefore, understanding their behavior and failure points is crucial.

Critical weaknesses

The presentation can be improved in several ways: While the paper is generally well-written, the setting for the LLM-agents is only described in the appendix, and important information is not stated as clear as it could be, for example the fact that the agents are not instructed to update their answer in a certain way.

From the paper, it is not clear to me whether the proposed model fits the observed dynamics. The model relies on the agents reconsidering their answer with increasing probability if they are presented with conflicting evidence, which might not be the case in practice.

Questions for authors

Q1: Does the proposed model for updating the belief actually fit the observed dynamics? To me it is not clear where this can be seen in the paper.

Q2: If I understood correctly, the agents only have a binary choice and no way to indicate uncertainty in their reported belief. Could confidence in the stated answer also play a role in the outcome?

3 comments
AuthorsOpenReview ↗

Response to Reviewer KSox*

Thank you for the review. Please find our point-by-point response below.

W1. Setting of the LLM experiments

W1: "…the setting for the LLM-agents is only described in the appendix, and important information is not stated as clear as it could be, for example the fact that the agents are not instructed to update their answer in a certain way."

The reviewer is right that the setting was hard to find and that this instruction was not stated clearly. We revised the text as follows.

  • Setting in the appendix. We moved the experimental setup from the appendix into the main text as a summary table (Table 1, Section 4.1).
  • Instruction on updating. We now state in the main text what the prompts instruct. No prompt describes the four behaviors of the model. The three instructions are the only text that concerns how an agent responds to the answers of the other agents. Instruction A asks an agent to state its honest answer even against all other agents. Instruction B adds nothing to the prompt. Instruction C asks an agent to defer to the emerging consensus unless the agent is certain that the consensus is wrong (Section 4.1, lines 267–270 and 281–283).

W2 and Q1. Model fit

W2: "From the paper, it is not clear to me whether the proposed model fits the observed dynamics. The model relies on the agents reconsidering their answer with increasing probability if they are presented with conflicting evidence, which might not be the case in practice."

Q1: "Does the proposed model for updating the belief actually fit the observed dynamics? To me it is not clear where this can be seen in the paper."

Fit to the observed dynamics (W2, Q1). Thank you for this question. The original manuscript did not say clearly where the fit can be seen. The model fits the ordering of settings by discussion gain rather than the exact magnitude of the gain. Discussion gain falls as the margin c^−c∗\hat c - c^\ast rises (Spearman corr. −0.86-0.86 on the hidden profile task, −0.79-0.79 on HiddenBench and −0.54-0.54 on MedEInst in Figs. 2d and 3). Section 6 now states that the model predicts the ordering, not the exact size of the gain.

Reconsideration that grows with conflicting evidence (W2). The reviewer is right that the proportional rule doesn't always hold. This comment prompted a new check. Our results do not rely on the proportional rule (adopted from Stechly et al., 2025 and Jiang et al., 2025). Replacing the proportional form with a general linear function f(d)=αd+βf(d)=\alpha d+\beta changes the Spearman correlations by at most 0.04, with the margin c^−c∗\hat c - c^\ast still correlating more strongly with gain than γ^\hat\gamma or c^\hat c alone (Appendix B.2). Theoretically, the critical threshold c∗c^\ast depends only on the reconsideration rate under unanimous dissent (d=1d=1), meaning the functional form only affects how this rate is estimated from data.

Q2. Binary choice and confidence

Q2: "If I understood correctly, the agents only have a binary choice and no way to indicate uncertainty in their reported belief. Could confidence in the stated answer also play a role in the outcome?"

Binary choice. Your understanding of the model is correct. We analyze binary questions because they represent the hardest case for majority voting (List and Goodin, 2001). If an agent picks option kk with probability pkp_k (k=1k=1 correct), a large team wins under plurality voting when p1>max⁡k≥2pkp_1 > \max_{k\ge 2} p_k. With K>2K>2 options, wrong votes split across options, allowing the correct answer to win even when p1<1/2p_1 < 1/2 (e.g., p=(0.4,0.2,0.2,0.2)p=(0.4, 0.2, 0.2, 0.2)). With two options, all wrong votes concentrate on one option, requiring p1>1/2p_1 > 1/2 (the strictest condition for any KK). Empirically, wrong answers in our benchmarks also concentrate on a single option, accounting for 92% of wrong answers on MedEInst (average K=27K=27) and 73% on HiddenBench (average K=4K=4), excluding the binary hidden profile task and exact-match MuSiQue.

Confidence. Thank you for this suggestion. Confidence is an interesting factor that the current model does not include. Although we did not measure confidence because agents output only an answer, we can incorporate it into our framework by expressing the withholding, internalization, and reconsideration rates as functions of confidence. Confidence affects discussion because less confident LLM agents are more likely to adopt a presented answer. For example, if agents grow more confident as peers agree with them, they reconsider less and withhold more, pushing the team above the critical threshold and preventing recovery from a wrong consensus. We added this discussion as future work (Section 6, lines 499–502).

Reference: List and Goodin (2001), Epistemic democracy: generalizing the Condorcet jury theorem, J. Political Philosophy 9(3):277–306.

If these answers resolve your concerns, we would be grateful if you could reflect them in your assessment.

Thank you for the detailed reply! Based on this, I will keep my positive score.

AuthorsOpenReview ↗

Acknowledgement

Thank you for taking the time to review, for reading our response carefully, and for keeping your positive score. Your comments led us to improve the manuscript. We sincerely appreciate your feedback.

Reviewer_mHNUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies when discussion among LLM agents improves on majority voting. Its model suggests that discussion helps when agents voice disagreement and remain willing to revise their answers. Experiments generally show that withholding dissent reduces these gains, though the pattern is unclear on MuSiQue. Instructions emphasizing agreement increase withholding, while reducing or disabling reasoning improves discussion gains in the tested models. The findings highlight the importance of sharing contrary evidence and staying open to correction.

Critical strengths
  • The authors separately elicit public and private answers and report held-out analyses, predictive checks, and an intervention with predictions stated in advance. They also acknowledge where the results diverge from the model.

  • Reducing or disabling reasoning improves correction and discussion gains on the hidden-profile task, suggesting that stronger individual reasoning does not necessarily translate into better collaboration.

  • Reasoning models tend to share evidence supporting their own answers while withholding contrary evidence. This offers a useful explanation for why capable agents can still perform poorly as a team.

Critical weaknesses
  • The reported correlations do not establish a general cutoff for gain from discussion. Positive gains occur above the estimated threshold, and its added predictive value over withholding alone remains unclear.

  • The posterior-predictive check underestimates recovery from incorrect initial majorities. The model omits recovery through newly shared evidence, although information pooling is central to the tasks. HiddenBench control and transparent-condition point estimates also exceed the pre-stated prediction ranges.

  • The withholding measure includes cases where the private answer also changes to match the visible majority. It therefore does not isolate suppressed dissent from persuasion or updating based on new evidence.

  • The internalization estimate reflects agreement between separately elicited public and private answers within a round, rather than adoption following a public statement. This does not distinguish internalization from other answer changes or establish it as the cause of gains from disabling reasoning.

Questions for authors
  1. Do the main findings persist when public–private disagreement is analyzed separately from cases where the private answer also shifts to the majority? What evidence isolates internalization’s contribution to the improvement from disabling reasoning?

  2. Can you quantify whether the margin relative to the critical withholding rate predicts gain from discussion better than withholding or net correction alone on held-out tasks, including uncertainty in that comparison?

  3. How were the intervention’s pre-stated prediction ranges constructed, and how do you explain the HiddenBench control and transparent-condition point estimates exceeding those ranges?

5 comments
AuthorsOpenReview ↗

Response to Reviewer mHNU - 1/3

Thank you for the review. Please find our point-by-point response below.

W1, first half. Scope of the cutoff

W1: "The reported correlations do not establish a general cutoff for gain from discussion. Positive gains occur above the estimated threshold, and its added predictive value over withholding alone remains unclear."

Comment of 9 Nov: "The abstract states that discussion can overturn an incorrect initial majority “only when” c < c*, and Section 6 states that gains are “zero above c*.” However, Figs. 2d and 3a show positive gains in settings with a positive estimated margin ĉ − c*."

The reviewer is right that the abstract and Section 6 overstated the cutoff. In our model, c∗c^\ast is the worst-case cutoff for stylized teams starting near the all-wrong state, whereas for LLM teams we claim only that discussion gain falls as the margin c^−c∗\hat c - c^\ast rises. We acknowledge that our original manuscript did not state this boundary clearly. In the stylized model, reaching the correct consensus depends on both the withholding rate cc and the initial fraction pp of correct agents, bounded by the curve p∗(c)p^\ast(c) between the orange and blue regions in Fig. 1c. Because pp is unknown before running a benchmark, we use c∗c^\ast, the most conservative pp-independent threshold. Because c∗c^\ast represents the worst-case initial state, positive gains at positive margins do not contradict the model when some agents start correct. Accordingly, we removed "only when" from the abstract (now stating that "the gain from discussion falls as the margin c^−c∗\hat c - c^\ast rises") and clarified in Section 6 that zero gain above c∗c^\ast holds in the model only near the all-wrong state.

W1 (second half) and Q2. Held-out comparison

Q2: "Can you quantify whether the margin relative to the critical withholding rate predicts gain from discussion better than withholding or net correction alone on held-out tasks, including uncertainty in that comparison?"

Comment of 9 Nov: "I would also appreciate a held-out comparison showing that the margin predicts gain better than ĉ or γ̂ alone, with uncertainty on the difference rather than only the separate correlations in Fig. 7. This seems important because on MuSiQue the margin equals ĉ (Appendix C.1)."

Thank you for requesting this comparison. The three analyses below are new in the revision.

Held-out tasks. The margin c^−c∗\hat c - c^\ast correlates more strongly with held-out task gain than c^\hat c or γ^\hat\gamma alone on the hidden profile task, HiddenBench, and MedEInst. Across 200 random half-splits (estimating rates on one half of tasks and measuring gain on the other), the median Spearman correlations for the margin versus c^\hat c and γ^\hat\gamma are −0.81-0.81 vs. −0.67-0.67 and +0.71+0.71 (hidden profile), −0.75-0.75 vs. −0.70-0.70 and +0.55+0.55 (HiddenBench), and −0.41-0.41 vs. −0.34-0.34 and +0.11+0.11 (MedEInst). On MuSiQue, c∗=0c^\ast = 0 makes the margin equal to c^\hat c, placing every LLM team above the threshold. In this regime, our model predicts that changing the withholding rate does not change the gain when all agents start wrong, matching the weak correlation between c^\hat c and gain (Spearman −0.20-0.20), while γ^\hat\gamma predicts gain better. We added these results to the main text (Section 4.2, lines 315–317 and 403–405) and full details of all three analyses to the appendix (Appendix C.1, lines 1445–1454, Table 4).

Held-out LLMs. The margin also predicts the gain of a held-out LLM with lower error than c^\hat c or γ^\hat\gamma alone on the hidden profile task, HiddenBench, and MedEInst. In leave-one-LLM-out prediction, the RMSEs for the margin versus c^\hat c and γ^\hat\gamma are 0.090 vs. 0.131 and 0.107 (hidden profile), 0.116 vs. 0.133 and 0.169 (HiddenBench) and 0.036 vs. 0.039 and 0.041 (MedEInst). On MuSiQue, where c∗=0c^\ast = 0, γ^\hat\gamma predicts gain better for the reason explained above.

Uncertainty. Regression analyses with uncertainty confirm that c∗c^\ast adds statistically significant explanatory power beyond c^\hat c alone and that the margin adds explanatory power beyond γ^\hat\gamma alone.

In the mixed-effects regression gains=w0+w1c^s+w2cs∗+bm(s)+ϵs\text{gain}_s = w_0 + w_1 \hat{c}_s + w_2 c^\ast_s + b_{m(s)} + \epsilon_s (with setting ss and LLM random intercept bm(s)b_{m(s)}), the coefficient on c∗c^\ast is positive with 95% confidence intervals excluding zero on each benchmark (hidden profile: 0.60 [0.45, 0.76], HiddenBench: 0.52 [0.20, 0.84], MedEInst: 0.17 [0.06, 0.29]). The margin likewise adds information beyond γ^\hat\gamma alone (p≤0.004p \le 0.004 on each benchmark). Both results also hold under a distribution-free rank regression R(gains)=w0+w1R(c^s)+w2R(cs∗)+εsR(\text{gain}_s) = w_0 + w_1 R(\hat{c}_s) + w_2 R(c^\ast_s) + \varepsilon_s (p≤0.035p \le 0.035).

AuthorsOpenReview ↗

Response to Reviewer mHNU - 2/3

Points 1 and 2. Proportional reconsideration rule

Point 1: "Section 6: the premise that reconsideration is proportional to visible dissent cannot be identified from the logs."

Point 2: "Appendix B.2: in replays with shuffled dissent, reconsideration grows in proportion to dissent in only a minority of settings."

We agree with the reviewer that the logs cannot identify the proportional form. This led us to test a more general rule. Our results do not depend on the proportional reconsideration rule. Replacing the proportional rule with a general linear rule f(d)=αd+βf(d)=\alpha d+\beta (Appendix B.2) changes the Spearman correlation between the margin and gain by at most 0.04, with the margin still correlating more strongly with gain than c^\hat c or γ^\hat\gamma on the hidden profile task, HiddenBench, and MedEInst.

Point 3. Wording of the private-answer query

Point 3: "Appendix B.2: â, ρ̂ and r̂ depend on the elicitation method."

The reviewer is right that Appendix B.2 reports dependence on the elicitation. We clarified the text and added a new check. All results in the paper use the separate-query format, whose parameter estimates and margin correlations are stable across query rewordings (Appendix B.2). The sensitivity originally noted in Appendix B.2 reflects the difference between asking for private and public answers in a single reply versus two separate queries. Within the separate-query format used throughout the paper, testing three query rewordings shows high stability:

  • Parameter a^\hat a: Rewording changes a^\hat a by at most 0.05 across seven LLMs and by 0.09 for mixtral-8x22b (comparable to its 0.073 resampling variation under the original wording).
  • Parameters ρ^\hat\rho and r^\hat r: These rates enter c∗c^\ast only through γ^=ρ^ (2r^−1)\hat\gamma = \hat\rho\,(2\hat r - 1), where the share of changed private answers entering γ^\hat\gamma shifts by at most 0.05.

Under both variations, the margin continues to correlate more strongly with gain than c^\hat c or γ^\hat\gamma on all benchmarks except MuSiQue (where the correlation remains weak).

W2. Recovery through shared evidence

W2: "The posterior-predictive check underestimates recovery from incorrect initial majorities. The model omits recovery through newly shared evidence, although information pooling is central to the tasks. HiddenBench control and transparent-condition point estimates also exceed the pre-stated prediction ranges."

Posterior-predictive check and newly shared evidence. The reviewer is right that the model underestimates recovery. Our model captures same-round shared evidence through γ^\hat\gamma but omits evidence that persists across later rounds, making its recovery predictions conservative and explaining why it predicts the ordering of settings rather than exact gain magnitudes. When an agent sees dissent, it reconsiders and reaches the correct answer with probability rr, meaning γ^\hat\gamma estimated from logs includes recovery driven by evidence shared in that round's dissenting statements. Because the model omits evidence that remains visible in subsequent rounds, it underestimates total recovery, which we now state explicitly in Section 6.

Pre-stated prediction ranges. Observed gains exceed the pre-stated prediction ranges for the same reason, as detailed in our response to Q3 below.

W3. Definition of withholding

W3: "The withholding measure includes cases where the private answer also changes to match the visible majority. It therefore does not isolate suppressed dissent from persuasion or updating based on new evidence."

This comment led us to a new experiment on the form of the threshold. Our main findings are unchanged whether withholding is defined as stating the majority answer (c^\hat c) or restricted to cases where the private answer also shifts to the majority (b=c⋅ab = c \cdot a). We revised the text to define c^\hat c explicitly as the rate at which an agent states the visible majority answer when its prior private answer differs (Section 3, lines 232–235), without claiming that c^\hat c isolates suppressed dissent. In the model, concurrent private-answer shifts are captured as internalization (aa). Rewriting the threshold condition c<c∗=γ/(γ+a)c < c^\ast = \gamma/(\gamma + a) as c(γ+a)<γc(\gamma + a) < \gamma yields c⋅a<(1−c)γc \cdot a < (1-c)\gamma, which compares the joint rate b=c⋅ab = c \cdot a (stating the majority answer while also shifting the private answer) against net correction among non-withholding agents. Expressed in terms of bb, the critical threshold is b∗=aγ/(a+γ)b^\ast = a\gamma/(a+\gamma) with margin b−b∗=a(c−c∗)b - b^\ast = a(c - c^\ast), preserving the exact sign of c−c∗c - c^\ast in every setting. Empirically, the Spearman correlation between b^−b∗\hat b - b^\ast and gain is −0.86-0.86 on the hidden profile task, −0.76-0.76 on HiddenBench and −0.56-0.56 on MedEInst (within 0.03 of the original margin and stronger than b^\hat b or c^\hat c alone), while remaining weak on MuSiQue (+0.17+0.17, where c∗=0c^\ast = 0).

AuthorsOpenReview ↗

Response to Reviewer mHNU - 3/3

W4. Internalization and the effect of reasoning

W4: "The internalization estimate reflects agreement between separately elicited public and private answers within a round, rather than adoption following a public statement. This does not distinguish internalization from other answer changes or establish it as the cause of gains from disabling reasoning."

Agreement within a round. We thank the reviewer for pointing out that the original definition was unclear. The internalization rate a^\hat a measures sequential adoption across a round rather than within-round agreement. Specifically, we count an event toward a^\hat a only when the agent's pre-round private answer differs from the majority, the agent publicly states the majority answer, and its post-round private answer shifts to match the majority (Section 3, lines 235–237).

Other answer changes and the cause of the gain. The effect of disabling reasoning on discussion gain is causal, whereas we report the role of internalization in this effect as an association. Because disabling reasoning is an assigned intervention with the LLM and prompt held fixed, its effects on hidden-profile gain (+0.17+0.17) and a^\hat a (−0.11-0.11) are causal, as is the pathway in the stylized model (where lower aa raises c∗c^\ast). For LLM teams, we agree that disabling reasoning can also act through unmodeled routes and that a^\hat a does not separate internalization from evidence-driven private updates, which we now state explicitly in the revised text (Section 3, lines 237–239; Appendix C.1, lines 1556–1559).

Q1. Separating the two kinds of withholding

Q1: "Do the main findings persist when public–private disagreement is analyzed separately from cases where the private answer also shifts to the majority? What evidence isolates internalization’s contribution to the improvement from disabling reasoning?"

Public–private disagreement analyzed separately. Thank you for suggesting this analysis, which is new in the revision. Separating pure public–private disagreement from cases where the private answer shifts to the majority preserves the ordering of discussion gains by our threshold margin. Defining pure public–private disagreement as stating the majority answer while keeping a different post-round private answer yields a Spearman correlation of 0.000.00 with gain on the hidden profile task, whereas the margin still orders the gain (−0.84-0.84) as predicted by the condition on c⋅ac \cdot a. On HiddenBench and MedEInst, pure public–private disagreement also correlates with gain (−0.59-0.59 and −0.44-0.44).

Evidence that isolates internalization. No experimental evidence isolates the contribution of internalization alone. While the gain increase from disabling reasoning is causal by experimental assignment, we report internalization's contribution on the hidden profile task strictly as an association (see W4).

Q2. Answered above under "W1 (second half) and Q2".

Q3. Pre-stated prediction ranges

Q3: "How were the intervention’s pre-stated prediction ranges constructed, and how do you explain the HiddenBench control and transparent-condition point estimates exceeding those ranges?"

Construction. Each pre-stated range is a 90% predictive interval of discussion gain simulated from the estimated behavioral rates.

Point estimates above the ranges. The reviewer is right that the point estimates exceed the ranges. Observed gains exceed the pre-stated ranges upward (with 5 of 24 conditions falling inside) because our model represents a conservative worst case for recovery, while all predicted directional effects hold. Specifically, the model omits evidence that remains visible across later rounds (W2) and uses the worst-case threshold c∗c^\ast near the all-wrong state (W1), matching the reviewer's observation that the model underestimates recovery. The model predicts directional ordering rather than exact gain levels, as confirmed by the collapse of discussion gains when agents are forced to state the majority answer.

Thank you again for the detailed reading and the follow-up comment of 9 Nov. Please let us know if any of these points remain open after these analyses.

Reviewer_QHXGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when multi-agent discussion improves collective decision-making compared with majority voting. The authors propose a mathematical model based on four behavioral mechanisms: withholding dissent, internalizing publicly stated answers, reconsidering decisions after observing disagreement, and correcting toward the right answer.

The model derives a critical withholding threshold, (c^* = \gamma/(\gamma+a)), where (\gamma) represents net correction and (a) represents internalization. The authors estimate these parameters from LLM conversation logs using Bayesian inference and evaluate the framework across nine LLMs and four benchmarks, including hidden-profile tasks, HiddenBench, MuSiQue, and MedEInst. The experiments suggest that higher withholding reduces the benefits of discussion and that disabling reasoning can sometimes improve collective accuracy by reducing internalization and encouraging reconsideration.

Critical strengths
  1. The paper provides a mathematical framework connecting dissent withholding, internalization, and collective decision accuracy. The derived threshold offers an interpretable perspective on when multi-agent discussion can outperform majority voting.
  2. The authors estimate behavioral parameters from actual LLM interactions and evaluate the model's predictions across multiple LLMs, benchmarks, and instruction settings.
  3. The observation that reasoning models may selectively share evidence and internalize majority answers is interesting, highlighting a potential tension between individual reasoning capability and collective performance.
Critical weaknesses
  1. The model assumes binary decisions, independent initial beliefs, local majority-rule interactions, and fixed behavioral transition probabilities. These assumptions may not adequately capture heterogeneous agents, complex reasoning, or realistic multi-agent discussions. The applicability of the derived critical threshold to more general settings remains unclear.
  2. The paper estimates private beliefs using separately elicited LLM responses, which may be sensitive to prompting and sampling variability. Moreover, internalization is operationalized through agreement between public and private responses rather than observing a sequential belief change. This raises concerns about whether the estimated parameters faithfully capture the intended behavioral mechanisms.
  3. The authors report that disabling reasoning improves discussion gains, but changing the reasoning setting can simultaneously affect individual accuracy, answer diversity, evidence sharing, and response behavior. It is therefore difficult to attribute the improvement specifically to reduced internalization.
  4. The experimental protocols primarily involve homogeneous teams, structured information sharing, and relatively short discussions. It remains unclear whether the proposed framework generalizes to heterogeneous teams, open-ended debates, multi-step agentic reasoning, or tool-augmented collaboration.
Questions for authors
  1. How sensitive is the critical threshold to violations of the model's assumptions, particularly correlated initial errors, heterogeneous agent capabilities, and different interaction topologies?

  2. Can the authors provide additional validation showing that separately elicited private responses reliably measure withholding and internalization rather than prompt-induced answer variation?

  3. How accurately does the model predict discussion outcomes on held-out tasks or unseen models, beyond the observed correlation between the estimated threshold margin and discussion gains?

4 comments
AuthorsOpenReview ↗

Response to Reviewer QHXG

Thank you for the review. Please find our point-by-point response below.

W1 and Q1. Model assumptions

W1: "The model assumes binary decisions, independent initial beliefs, local majority-rule interactions, and fixed behavioral transition probabilities."

Q1: "How sensitive is the critical threshold to violations of the model's assumptions, particularly correlated initial errors, heterogeneous agent capabilities, and different interaction topologies?"

Thank you for asking about these assumptions. We answer each of them below.

Binary decisions. We focus on binary questions because they represent the hardest theoretical case for majority voting (List and Goodin, 2001), while our multi-option benchmarks are also effectively binary in practice. If an agent picks option kk with probability pkp_k (k=1k=1 correct), a large team wins under plurality voting when p1>max⁡k≥2pkp_1 > \max_{k\ge 2} p_k. For K>2K>2, wrong votes split across options, allowing the correct answer to win even as a minority (p1<1/2p_1 < 1/2, e.g., p=(0.4,0.2,0.2,0.2)p=(0.4, 0.2, 0.2, 0.2)). For K=2K=2, all wrong votes concentrate on one option, requiring p1>1/2p_1 > 1/2 (the strictest condition for any KK). Empirically, wrong answers concentrate on a single option across our multi-choice benchmarks: 92% on MedEInst (average K=27K=27) and 73% on HiddenBench (average K=4K=4), excluding the binary hidden profile task and exact-match MuSiQue.

Independent initial beliefs and correlated initial errors. The critical threshold c∗c^\ast does not depend on the initial beliefs. We derive c∗c^\ast from the stability of the state in which almost all agents are wrong, which is a property of the update rules and not of the starting point. The derivation therefore does not use the independence of the initial beliefs. The initial beliefs matter for the outcome of a team only through the initial fraction pp of correct agents (Fig. 1c). Correlated initial errors make a low pp more frequent. They do not change c∗c^\ast. Fully correlated errors give the all-wrong start, which is the worst case that c∗c^\ast covers. In our experiments, we measure the number of initially correct agents directly for each task.

Local majority rule, fixed rates, heterogeneous agents, interaction topologies. The reviewer is right that these assumptions simplify the actual system. These simplifying assumptions isolate four core LLM behaviors (withholding, internalization, reconsideration, and correction) to predict the direction of discussion gains across settings rather than their exact magnitude. The revised manuscript clarifies that our theoretical threshold is derived for stylized agents and explains the directional ordering of LLM teams (Section 6, lines 480–484). Furthermore, one can extend fixed rates to time-varying rates by modeling withholding, reconsideration, and internalization as functions of evidence from tool use or peers. We discuss this limitation and its extensions in the revised text (Section 6, lines 499–502), with extensions to heterogeneous agents and topologies detailed under W4 below.

Reference: List and Goodin (2001), Epistemic democracy: generalizing the Condorcet jury theorem, J. Political Philosophy 9(3):277-306.

W2 and Q2. Elicitation of private answers

W2: "The paper estimates private beliefs using separately elicited LLM responses, which may be sensitive to prompting and sampling variability. Moreover, internalization is operationalized through agreement between public and private responses rather than observing a sequential belief change."

Q2: "Can the authors provide additional validation showing that separately elicited private responses reliably measure withholding and internalization rather than prompt-induced answer variation?"

Sensitivity to prompting and sampling variability (W2, Q2). This concern is fair. It led us to a new experiment. Private-answer estimates are stable under both query rewording and resampling (Appendix B.2). Across three rewordings on 500 stored hidden-profile turns for each of eight LLMs, the internalization rate a^\hat a changes by at most 0.050.05 for seven LLMs (0.090.09 for mixtral-8x22b) while the share of changed private answers (affecting γ^\hat\gamma) moves by at most 0.050.05. Resampling with the original wording changes a^\hat a by at most 0.0110.011 for seven LLMs (0.0730.073 for mixtral-8x22b, matching its 0.090.09 rewording variation).

Sequential belief change (W2). The original definition was not clear on this point. We revised the definition. The internalization rate a^\hat a measures sequential belief change from before to after an agent's public statement. Specifically, we count an event toward a^\hat a only when the agent's pre-round private answer differs from the majority, the agent publicly states the majority answer, and its post-round private answer shifts to match the majority (Section 3, lines 235–237).

(Response continues)

(Continues from the previous response)

W3. Effect of turning reasoning off

W3: "…changing the reasoning setting can simultaneously affect individual accuracy, answer diversity, evidence sharing, and response behavior. It is therefore difficult to attribute the improvement specifically to reduced internalization."

Individual accuracy. Thank you for listing these factors. We checked the first one. Individual accuracy does not explain the discussion gain from disabling reasoning. Disabling reasoning barely changes first-round accuracy (−0.005-0.005) while still increasing recovery by 0.17, 0.20, and 0.37 when stratifying by the number of initially correct agents (0, 1, or 2).

Answer diversity, evidence sharing and response behavior. We agree that internalization is not isolated from answer diversity, evidence sharing, or other response behaviors, so we removed the causal attribution to internalization from the abstract.

W4. Generalization

W4: "It remains unclear whether the proposed framework generalizes to heterogeneous teams, open-ended debates, multi-step agentic reasoning, or tool-augmented collaboration."

We thank the reviewer for this question, which helped us clarify the model's scope. Although untested empirically, our model structure extends to heterogeneous teams, general topologies, tool use, and multi-step agentic reasoning because it parameterizes four high-level behaviors (withholding, reconsidering, correcting, and internalizing), whereas open-ended debates remain outside our scope.

  • Heterogeneous teams. Using the agent with the highest withholding and internalization rates yields a conservative threshold condition for discussion gains without altering the model.
  • Topologies (from Q1). Interaction topologies only alter which statements an agent observes, entering exclusively through MqM_q in Eq. (1). Under unequal observation probabilities, a Poisson binomial CDF replaces the uniform binomial distribution (approximated by a normal CDF for large qq) under the mean-field approximation to yield the critical threshold.
  • Tool-augmented collaboration. External tools shift agent accuracy, which the model captures by expressing internalization and reconsideration rates as functions of tool-provided evidence and accuracy.
  • Open-ended debates and multi-step agentic reasoning. Open-ended debates lack a ground-truth answer and fall outside our scope. In contrast, we cover multi-step agentic reasoning because the model parameterizes observable turn-level behavior rather than internal generation steps.

We list all of these settings as limitations in the revised manuscript (Section 6, lines 499–502).

Q3. Held-out prediction

Q3: "How accurately does the model predict discussion outcomes on held-out tasks or unseen models, beyond the observed correlation between the estimated threshold margin and discussion gains?"

Thank you for this question. The held-out analyses below are new in the revision.

Held-out tasks. The threshold margin predicts held-out task gains better than c^\hat c or γ^\hat\gamma alone across 200 random half-splits (estimating rates on one half of tasks and measuring gain on the other).

Unseen models. In leave-one-LLM-out cross-validation, the threshold margin preserves the gain ranking across held-out LLMs, though we did not prospectively evaluate newly released LLMs.

MuSiQue. Our model also explains why MuSiQue is an exception with a weak correlation (−0.20-0.20). On MuSiQue, c∗=0c^\ast = 0 places every LLM team above the threshold, where the model predicts that varying withholding does not change gain for teams that are initially all wrong.

If these answers resolve your concerns, we would be grateful if you could reflect them in your assessment.

Thanks for the clarification, I maintain my positive evaluation.

AuthorsOpenReview ↗

Acknowledgement

Thank you for keeping your positive score and, more importantly, for reading our manuscript and our response carefully. We sincerely appreciate your feedback, which helped us improve the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their constructive feedback and address four shared concerns below, citing the line numbers of the blue-highlighted revisions in the updated PDF.

1. Scope of the cutoff (mHNU W1)

mHNU W1: "The reported correlations do not establish a general cutoff for gain from discussion. Positive gains occur above the estimated threshold…"

We thank Reviewer mHNU for pointing out that the original text overstated the cutoff. The hard cutoff at c∗c^\ast applies to the stylized agents in Section 2 starting near the all-wrong state, whereas LLM teams match the model's direction of change (discussion gains decrease as the margin c^−c∗\hat c - c^\ast increases) rather than its exact magnitude. In the stylized model, reaching the correct consensus depends on both the withholding rate cc and the initial fraction of correct agents pp, bounded by the threshold p∗(c)p^\ast(c) (the black curve separating the orange and blue regions in Fig. 1c). Because pp is unknown before running a benchmark, we use the most conservative, pp-independent worst-case threshold c∗c^\ast. Consequently, even stylized agents can gain from discussion above c∗c^\ast when the initial fraction pp of correct agents is well above 0, which likewise applies to LLM teams. We clarified this distinction in the abstract and Section 6.


2. Proportional reconsideration rule (KSox W2 and Q1, mHNU)

KSox W2: "The model relies on the agents reconsidering their answer with increasing probability if they are presented with conflicting evidence, which might not be the case in practice."

mHNU, comment of 9 Nov: "Section 6: the premise that reconsideration is proportional to visible dissent cannot be identified from the logs."

The reviewers are right that the proportional form is an assumption that the logs cannot identify. This comment prompted a new check. Our results do not depend on the proportional reconsideration rule: replacing f(d)=ρdf(d)=\rho d with a general linear rule f(d)=αd+βf(d)=\alpha d+\beta changes the Spearman correlation between the margin and the gain by no more than 0.04. Theoretically, the critical threshold c∗c^\ast is derived for a single correct agent facing unanimous dissent (d=1d=1), meaning only the reconsideration rate at d=1d=1 affects c∗c^\ast. The functional form only matters when estimating this rate from observations at d<1d < 1 (using f(1)=α+βf(1)=\alpha+\beta in place of ρ\rho). We added this check to Appendix B.2.


3. Wording of the private-answer query (QHXG W2 and Q2, mHNU)

QHXG Q2: "Can the authors provide additional validation showing that separately elicited private responses reliably measure withholding and internalization rather than prompt-induced answer variation?"

mHNU, comment of 9 Nov: "Appendix B.2: â, ρ̂ and r̂ depend on the elicitation method."

We thank the reviewers for this concern, which led us to a new experiment. Private-answer estimates and our main conclusions are stable across three rewordings of the private-answer query across eight LLMs. Rewording changes the internalization rate a^\hat a by at most 0.09 and the share of changed private answers (which enters γ^\hat\gamma) by at most 0.05. Applying random shifts up to these bounds (≤0.09\le 0.09 for a^\hat a and ≤0.05\le 0.05 for γ^\hat\gamma) across all settings over 4,000 draws changes the Spearman correlation between the margin and discussion gain by less than 0.05 in 95% of draws. Across these draws, the margin still correlates with the gain more strongly than c^\hat c or γ^\hat\gamma alone on the hidden profile task, HiddenBench and MedEInst (Appendix B.2).


4. Held-out prediction (QHXG Q3, mHNU Q2)

QHXG Q3: "How accurately does the model predict discussion outcomes on held-out tasks or unseen models, beyond the observed correlation between the estimated threshold margin and discussion gains?"

mHNU Q2: "Can you quantify whether the margin relative to the critical withholding rate predicts gain from discussion better than withholding or net correction alone on held-out tasks, including uncertainty in that comparison?"

We thank the reviewers for asking for a held-out comparison, which we added as a new analysis in the revision. On the hidden profile task, HiddenBench and MedEInst, the margin predicts held-out gain better than either c^\hat c or γ^\hat\gamma alone, with regression analysis confirming that c∗c^\ast adds explanatory power beyond c^\hat c (p≤0.003p \le 0.003 on each benchmark). MuSiQue is the sole exception, which the model also explains. Because c∗c^\ast reaches its minimum (c∗=0c^\ast = 0) in all MuSiQue settings, every LLM team sits above the critical threshold. In this regime, the model predicts that initially all-wrong teams remain at a wrong consensus regardless of withholding, matching the weak Spearman correlation between c^\hat c and gain (−0.20-0.20). We added these results to the paper (Section 4.2, lines 315–317 and 403–405; Table 4; Appendix C.1, lines 1441–1454).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.