- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether agreement among LLM agents remains a reliable signal after interaction. The main finding is that deliberation can improve overall accuracy while substantially increasing the error rate among formed majorities. The paper attributes this to interaction-induced dependence and false consensus, and further studies truth scarcity, peer injection, grounding, and verification as possible mechanisms and boundary conditions.
Critical strengths
The paper asks an interesting question that is different from simply measuring whether multi-agent debate improves accuracy. The distinction between overall performance and the evidential value of agreement is useful. The paper also includes a relatively extensive set of controls, replications, and intervention experiments.
Critical weaknesses
-
The paper seems to conflate the absence of an independently correct answer with the absence of useful distributed information. A truth-scarce case () does not imply that the agents collectively lack information about the truth. Different agents may hold complementary partial evidence that becomes useful only after communication. This matters especially because the agents here appear to share a cooperative objective, unlike classical information-elicitation settings with strategic human participants. The current experiments mainly show a failure of unstructured answer-sharing, but the broader claim that interaction cannot create diagnostic agreement seems too strong.
-
The novelty over recent work on false consensus, diversity collapse, and debate-vs-voting is not yet fully convincing. The paper itself discusses closely related work on manufactured overconfidence, false consensus, confidence/diversity effects, and debate versus voting. The main distinction appears to be the specific notion of “diagnosticity of agreement” and the associated controls. I would like a clearer explanation of which empirical or theoretical conclusions would not already follow from these prior results.
-
Some of the main empirical conclusions depend heavily on conditioning and pool construction. The headline 23.1% versus 74.8% comparison conditions on non-unanimous majorities, but deliberation substantially changes which questions enter that set. In other words, part of the effect is a composition shift caused by manufactured agreement. The paper analyzes this mechanism, but the framing sometimes reads as if deliberation directly makes agreement intrinsically less reliable, rather than changing the process by which agreement is formed. In addition, several results depend on divergent-start or truth-scarce subsets, so the deployment-level interpretation should be stated more cautiously.
Questions for authors
-
How would the main claim change if agents hold complementary partial information that is individually insufficient but jointly sufficient to recover the correct answer? Would such a setting violate the proposed Diagnosticity-of-Agreement Law?
-
What is the strongest empirical distinction between this work and prior work on false consensus, diversity collapse, and debate versus voting? Which of the paper’s main conclusions would not be obtained from those earlier findings?
-
Since deliberation changes the set of questions that form a majority, how much of the diagnosticity gap remains under a comparison that holds the agreement set fixed, rather than conditioning on different post-treatment agreement events?