- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates if and when discussion improves over majority voting in LLM-based multi-agent systems. The proposed model uses four observed behaviors two explain when discussion improves the performance versus when the agents consense on an incorrect answer: withholding dissent, internalizing a stated answer, reconsidering when seeing counter-evidence, and correcting toward the correct answer. Following from the proposed model, discussion can only overturn an initially incorrect majority, if the so-called withholding-rate is below a critical rate (that is based on the correction and the internalization rates).
Critical strengths
The paper is relevant and interesting. It tries to unify the conflicting reports on the performance and behavior of multi-agent systems, which are increasingly used in practice. Therefore, understanding their behavior and failure points is crucial.
Critical weaknesses
The presentation can be improved in several ways: While the paper is generally well-written, the setting for the LLM-agents is only described in the appendix, and important information is not stated as clear as it could be, for example the fact that the agents are not instructed to update their answer in a certain way.
From the paper, it is not clear to me whether the proposed model fits the observed dynamics. The model relies on the agents reconsidering their answer with increasing probability if they are presented with conflicting evidence, which might not be the case in practice.
Questions for authors
Q1: Does the proposed model for updating the belief actually fit the observed dynamics? To me it is not clear where this can be seen in the paper.
Q2: If I understood correctly, the agents only have a binary choice and no way to indicate uncertainty in their reported belief. Could confidence in the stated answer also play a role in the outcome?
3 comments
Thank you for the detailed reply! Based on this, I will keep my positive score.
Acknowledgement
Thank you for taking the time to review, for reading our response carefully, and for keeping your positive score. Your comments led us to improve the manuscript. We sincerely appreciate your feedback.
Response to Reviewer KSox*
Thank you for the review. Please find our point-by-point response below.
W1. Setting of the LLM experiments
The reviewer is right that the setting was hard to find and that this instruction was not stated clearly. We revised the text as follows.
W2 and Q1. Model fit
Fit to the observed dynamics (W2, Q1). Thank you for this question. The original manuscript did not say clearly where the fit can be seen. The model fits the ordering of settings by discussion gain rather than the exact magnitude of the gain. Discussion gain falls as the margin c^−c∗ rises (Spearman corr. −0.86 on the hidden profile task, −0.79 on HiddenBench and −0.54 on MedEInst in Figs. 2d and 3). Section 6 now states that the model predicts the ordering, not the exact size of the gain.
Reconsideration that grows with conflicting evidence (W2). The reviewer is right that the proportional rule doesn't always hold. This comment prompted a new check. Our results do not rely on the proportional rule (adopted from Stechly et al., 2025 and Jiang et al., 2025). Replacing the proportional form with a general linear function f(d)=αd+β changes the Spearman correlations by at most 0.04, with the margin c^−c∗ still correlating more strongly with gain than γ^ or c^ alone (Appendix B.2). Theoretically, the critical threshold c∗ depends only on the reconsideration rate under unanimous dissent (d=1), meaning the functional form only affects how this rate is estimated from data.
Q2. Binary choice and confidence
Binary choice. Your understanding of the model is correct. We analyze binary questions because they represent the hardest case for majority voting (List and Goodin, 2001). If an agent picks option k with probability pk (k=1 correct), a large team wins under plurality voting when p1>maxk≥2pk. With K>2 options, wrong votes split across options, allowing the correct answer to win even when p1<1/2 (e.g., p=(0.4,0.2,0.2,0.2)). With two options, all wrong votes concentrate on one option, requiring p1>1/2 (the strictest condition for any K). Empirically, wrong answers in our benchmarks also concentrate on a single option, accounting for 92% of wrong answers on MedEInst (average K=27) and 73% on HiddenBench (average K=4), excluding the binary hidden profile task and exact-match MuSiQue.
Confidence. Thank you for this suggestion. Confidence is an interesting factor that the current model does not include. Although we did not measure confidence because agents output only an answer, we can incorporate it into our framework by expressing the withholding, internalization, and reconsideration rates as functions of confidence. Confidence affects discussion because less confident LLM agents are more likely to adopt a presented answer. For example, if agents grow more confident as peers agree with them, they reconsider less and withhold more, pushing the team above the critical threshold and preventing recovery from a wrong consensus. We added this discussion as future work (Section 6, lines 499–502).
Reference: List and Goodin (2001), Epistemic democracy: generalizing the Condorcet jury theorem, J. Political Philosophy 9(3):277–306.
If these answers resolve your concerns, we would be grateful if you could reflect them in your assessment.