- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether conflicting peer conclusions disrupt reasoning supported by sufficient evidence. It constructs 500 programmatically validated logical cases and compares matched inputs with identical rules, queries, and evidence, varying the presence of a conflicting opinion. Across 14 models, the added opinion reduces mean accuracy by 30.45 percentage points. Further experiments examine aligned opinions, nonce realizations, presentation variants, evidence re-evaluation, and sequential updates. The study documents substantial sensitivity to conflicting conclusions, including after an initially correct answer.
Critical strengths
The principal strength is the controlled construction. Symbolic validation establishes what follows from the visible evidence, while matched conditions hold the underlying reasoning problem fixed. This makes the accuracy loss more interpretable than failures observed in an entire multi-agent workflow. The breadth of the model evaluation and the sequential repeat-question control are also consequential: the effect is neither confined to one backbone nor adequately explained by ordinary answer drift. Case-level uncertainty and separate analyses of harmful and beneficial changes further strengthen the empirical characterization.
Critical weaknesses
The experiments establish sensitivity to conflicting conclusions more clearly than a specifically peer-induced effect. There is no matched control presenting the same incorrect conclusion without peer attribution, so the contribution of social provenance remains unresolved. Moreover, the hidden proof supporting the peer conclusion is unavailable to the evaluated model; its construction does not distinguish the model-facing treatment from an otherwise identical unsupported answer suggestion. The baseline instruction to classify from “the information provided” also leaves the evidential status of the reported opinion insufficiently explicit. These issues limit the interpretation of the result as displacement of evidence by peer pressure.
External validity remains limited despite the broad model coverage. All cases belong to one synthetic deductive classification setting, and nonce realizations preserve that same task structure. The study therefore provides strong evidence for a controlled failure mode, but does not establish its prevalence in realistic collaboration involving verifiable peer explanations, tools, or genuinely new evidence. The conclusions about collaborative reliability should reflect this boundary.
Questions for authors
For clarification of the mechanism, is there an existing comparison that presents identical conflicting content as a peer opinion versus an unattributed candidate answer? Does the effect persist when instructions explicitly state that only the listed facts and rules determine entailment?
For clarification of reproducibility, can the exact serialized peer messages, source-disclosure instructions, and sequential-update prompts be provided? Their wording is necessary to assess whether the manipulations alter only provenance and delivery or also introduce additional authority or revision cues. Broader task evaluation would be future work; these clarifications could directly change my assessment.
2 comments
Thank you for your response. I will keep my score unchanged.
We thank the reviewer for the thoughtful and constructive feedback. In response, we conducted additional controlled evaluations to examine the roles of peer attribution and evidence-priority instructions, constructed task-grounded examples beyond synthetic classification, and further clarified the experimental prompt specifications.
W1 & Q1: To distinguish the effect of explicit peer attribution from the effect of receiving a conflicting conclusion, we conducted an additional matched evaluation on 100 cases across six models, with three replays per condition. Macro-averaged accuracy is 71.1% under Evidence Only, 46.4% under Peer-Attributed Opinion, and 36.1% under Unattributed Candidate Answer. The degradation therefore persists when the same conflicting conclusion is presented without explicit peer attribution, suggesting that the observed susceptibility is not dependent on identifying the source as another agent.
Our main benchmark further examines the evidential status of peer opinions. The peer conclusion is supported by a competing proof branch, while its supporting premises are hidden from the receiving model. Thus, the peer message introduces a conflicting conclusion without providing additional evidence. We also evaluated a presentation variant exposing the solver-derived rule trace. POE changes from −26.22 to −38.11 pp, with a paired contrast of −11.89 pp. Because this variant changes both visible content and message length, we interpret it as a peer-opinion presentation sensitivity analysis.
As shown in Section 5.3, Source Disclosure alone does not mitigate peer override, whereas Evidence Re-evaluation partially reduces the effect by requiring the model to re-derive its answer from the available evidence. An additional evidence-priority evaluation further reduces the aggregate POE to −4.7 pp, although model-level effects remain heterogeneous. Together, these results characterize the distinct roles of source attribution, conflicting conclusions, and evidence prioritization.
W2: Our proof-graph-based benchmark provides a controlled setting for studying a fundamental problem in collaborative reasoning: whether an agent can preserve a conclusion supported by its own evidence when confronted with a conflicting conclusion from another agent. Unlike end-to-end evaluations, our symbolic construction and programmatic validation ensure that the target proof and competing peer branch are logically grounded. This allows us to examine the influence of conflicting peer conclusions while controlling for errors in benchmark reasoning and ground-truth construction.
To further examine the phenomenon in task-oriented settings, we constructed two additional examples adapted from MATH-500 and DebugBench, covering Mathematical Reasoning and Code Validation. Their simulated peer reports contain intermediate mathematical derivations and code execution traces, respectively, while retaining incorrect final judgments. The resulting accuracy degradation across multiple models provides complementary evidence beyond synthetic classification (Full examples can be foud in our response to Reviewer 5J64, W1 & Q1). We will include these examples and the additional experimental results in the revised Appendix.
Q2: All experimental variants retain the baseline classification task and JSON output contract specified in Appendix B. Only the relevant message structure or instruction clauses are modified. Specifically, the peer-attributed condition appends the statement “Another agent reports: My classification of the query above is: {"label":"<peer_label>"}” to the user message, whereas the unattributed condition removes the source attribution and presents the same conclusion under the heading “Conclusion:”.
For Source Disclosure, the system prompt's Report context clause is modified to either state that no additional information about the report's source is provided or explicitly identify the report as originating from another agent's local information state. For Evidence Re-evaluation, the Decision procedure clause is modified to require independent derivation from the provided facts and rules. In the additional evidence-priority evaluation, these two clauses are replaced with instructions stating that only the listed facts and rules determine logical entailment, that a peer opinion is an unverified proposed conclusion rather than a premise or evidence, and that classification must be determined strictly against the listed facts and rules.
For the two-turn evaluation, Turn 1 uses the baseline prompt and retains the model's response in the conversation. Turn 2 uses the same repeat-question request in both conditions, asking the model to classify the query again and return exactly one JSON object. The Peer Update condition prepends the identical serialized peer-attributed opinion to this request, without explicitly instructing the model to revise its previous answer.