acceptodds
Under review as a conference paper at ICLR 2027

Evidence Under Pressure: The Peer-Override Effect in Collaborative LLM Reasoning

Abstract

Large language models increasingly operate in multi-agent systems where agents exchange not only evidence but also conclusions. Yet when a peer opinion conflicts with the evidence available to a model, it is unclear whether the model preserves its own evidence-grounded reasoning. We introduce a controlled benchmark that isolates this question by holding the rules, query, and model-visible evidence fixed while varying the presence of a conflicting peer opinion. Across 500 cases and 14 LLMs, conflicting peer opinions reduce accuracy by 30.45 percentage points. Among cases answered correctly when sufficient evidence is presented alone, 53.51% are answered incorrectly when the same evidence is accompanied by a conflicting peer opinion. In contrast, only 4.31% of all cases shift from incorrect to correct. We call this systematic displacement the Peer Override Effect. Its magnitude varies by more than 55 percentage points across models with similar reasoning accuracy, revealing a substantial gap between reasoning competence and collaborative reliability. The effect persists across surface realizations and even after a model has already produced a correct answer, while explicit evidence re-evaluation only partially mitigates it. Reliable multi-agent systems must therefore not only reach evidence-supported conclusions, but also preserve them when peer opinions conflict with the available evidence.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Multi-Agent Systems, Collaborative Reasoning, Evidence-Grounded Reasoning, Peer Influence, Reliability

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_YRCHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether conflicting peer conclusions disrupt reasoning supported by sufficient evidence. It constructs 500 programmatically validated logical cases and compares matched inputs with identical rules, queries, and evidence, varying the presence of a conflicting opinion. Across 14 models, the added opinion reduces mean accuracy by 30.45 percentage points. Further experiments examine aligned opinions, nonce realizations, presentation variants, evidence re-evaluation, and sequential updates. The study documents substantial sensitivity to conflicting conclusions, including after an initially correct answer.

Critical strengths

The principal strength is the controlled construction. Symbolic validation establishes what follows from the visible evidence, while matched conditions hold the underlying reasoning problem fixed. This makes the accuracy loss more interpretable than failures observed in an entire multi-agent workflow. The breadth of the model evaluation and the sequential repeat-question control are also consequential: the effect is neither confined to one backbone nor adequately explained by ordinary answer drift. Case-level uncertainty and separate analyses of harmful and beneficial changes further strengthen the empirical characterization.

Critical weaknesses

The experiments establish sensitivity to conflicting conclusions more clearly than a specifically peer-induced effect. There is no matched control presenting the same incorrect conclusion without peer attribution, so the contribution of social provenance remains unresolved. Moreover, the hidden proof supporting the peer conclusion is unavailable to the evaluated model; its construction does not distinguish the model-facing treatment from an otherwise identical unsupported answer suggestion. The baseline instruction to classify from “the information provided” also leaves the evidential status of the reported opinion insufficiently explicit. These issues limit the interpretation of the result as displacement of evidence by peer pressure.

External validity remains limited despite the broad model coverage. All cases belong to one synthetic deductive classification setting, and nonce realizations preserve that same task structure. The study therefore provides strong evidence for a controlled failure mode, but does not establish its prevalence in realistic collaboration involving verifiable peer explanations, tools, or genuinely new evidence. The conclusions about collaborative reliability should reflect this boundary.

Questions for authors

For clarification of the mechanism, is there an existing comparison that presents identical conflicting content as a peer opinion versus an unattributed candidate answer? Does the effect persist when instructions explicitly state that only the listed facts and rules determine entailment?

For clarification of reproducibility, can the exact serialized peer messages, source-disclosure instructions, and sequential-update prompts be provided? Their wording is necessary to assess whether the manipulations alter only provenance and delivery or also introduce additional authority or revision cues. Broader task evaluation would be future work; these clarifications could directly change my assessment.

2 comments

We thank the reviewer for the thoughtful and constructive feedback. In response, we conducted additional controlled evaluations to examine the roles of peer attribution and evidence-priority instructions, constructed task-grounded examples beyond synthetic classification, and further clarified the experimental prompt specifications.

W1 & Q1: To distinguish the effect of explicit peer attribution from the effect of receiving a conflicting conclusion, we conducted an additional matched evaluation on 100 cases across six models, with three replays per condition. Macro-averaged accuracy is 71.1% under Evidence Only, 46.4% under Peer-Attributed Opinion, and 36.1% under Unattributed Candidate Answer. The degradation therefore persists when the same conflicting conclusion is presented without explicit peer attribution, suggesting that the observed susceptibility is not dependent on identifying the source as another agent.

Our main benchmark further examines the evidential status of peer opinions. The peer conclusion is supported by a competing proof branch, while its supporting premises are hidden from the receiving model. Thus, the peer message introduces a conflicting conclusion without providing additional evidence. We also evaluated a presentation variant exposing the solver-derived rule trace. POE changes from −26.22 to −38.11 pp, with a paired contrast of −11.89 pp. Because this variant changes both visible content and message length, we interpret it as a peer-opinion presentation sensitivity analysis.

As shown in Section 5.3, Source Disclosure alone does not mitigate peer override, whereas Evidence Re-evaluation partially reduces the effect by requiring the model to re-derive its answer from the available evidence. An additional evidence-priority evaluation further reduces the aggregate POE to −4.7 pp, although model-level effects remain heterogeneous. Together, these results characterize the distinct roles of source attribution, conflicting conclusions, and evidence prioritization.

W2: Our proof-graph-based benchmark provides a controlled setting for studying a fundamental problem in collaborative reasoning: whether an agent can preserve a conclusion supported by its own evidence when confronted with a conflicting conclusion from another agent. Unlike end-to-end evaluations, our symbolic construction and programmatic validation ensure that the target proof and competing peer branch are logically grounded. This allows us to examine the influence of conflicting peer conclusions while controlling for errors in benchmark reasoning and ground-truth construction.

To further examine the phenomenon in task-oriented settings, we constructed two additional examples adapted from MATH-500 and DebugBench, covering Mathematical Reasoning and Code Validation. Their simulated peer reports contain intermediate mathematical derivations and code execution traces, respectively, while retaining incorrect final judgments. The resulting accuracy degradation across multiple models provides complementary evidence beyond synthetic classification (Full examples can be foud in our response to Reviewer 5J64, W1 & Q1). We will include these examples and the additional experimental results in the revised Appendix.

Q2: All experimental variants retain the baseline classification task and JSON output contract specified in Appendix B. Only the relevant message structure or instruction clauses are modified. Specifically, the peer-attributed condition appends the statement “Another agent reports: My classification of the query above is: {"label":"<peer_label>"}” to the user message, whereas the unattributed condition removes the source attribution and presents the same conclusion under the heading “Conclusion:”.

For Source Disclosure, the system prompt's Report context clause is modified to either state that no additional information about the report's source is provided or explicitly identify the report as originating from another agent's local information state. For Evidence Re-evaluation, the Decision procedure clause is modified to require independent derivation from the provided facts and rules. In the additional evidence-priority evaluation, these two clauses are replaced with instructions stating that only the listed facts and rules determine logical entailment, that a peer opinion is an unverified proposed conclusion rather than a premise or evidence, and that classification must be determined strictly against the listed facts and rules.

For the two-turn evaluation, Turn 1 uses the baseline prompt and retains the model's response in the conversation. Turn 2 uses the same repeat-question request in both conditions, asking the model to classify the query again and return exactly one JSON object. The Peer Update condition prepends the identical serialized peer-attributed opinion to this request, without explicitly instructing the model to revise its previous answer.

Thank you for your response. I will keep my score unchanged.

Reviewer_AoLKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the reliability of multi-agent LLM reasoning when peer opinions conflict with available evidence. It introduces the Peer Override Effect (POE) and constructs a controlled benchmark using programmatically validated proof graphs, where identical reasoning tasks are evaluated with and without conflicting peer opinions. Experiments across 500 cases and 14 LLMs reveal an average accuracy decrease of 30.45 percentage points, demonstrating that even models with strong independent reasoning capabilities can be highly susceptible to peer influence. Additional experiments show that this effect persists across different linguistic representations and sequential interactions, while explicit evidence re-evaluation only partially mitigates it. The findings highlight the importance of preserving evidence-grounded reasoning in multi-agent collaboration.

Critical strengths

1. Clear Motivation and Rigorous Benchmark Design: The paper investigates an important reliability issue in multi-agent LLM collaboration. Its controlled benchmark uses programmatically validated proof graphs and matched experimental conditions to isolate the impact of conflicting peer opinions while keeping the underlying evidence unchanged.

2. Extensive Experiments and Insightful Findings: The evaluation covers 500 cases and 14 LLMs, revealing substantial differences in susceptibility to peer influence across models. Additional analyses of linguistic variations, sequential interactions, and evidence re-evaluation provide valuable insights into the robustness and limitations of collaborative reasoning.

Critical weaknesses

Limited Realism and Generalizability of the Synthetic Benchmark: The benchmark consists entirely of programmatically generated proof graphs expressed through artificial or counterfactual language. While this design enables controlled evaluation, it differs substantially from realistic multi-agent reasoning, where peers generate their own arguments, exchange evidence, and interact over multiple rounds. Moreover, the primary task is restricted to three-way logical classification (entailed, contradicted, or unknown). Without evaluations on real collaborative reasoning tasks, it remains unclear whether the reported 30.45-point accuracy degradation generalizes to practical multi-agent systems.

Potential Ambiguity in the Experimental Prompt and Attribution of Peer Influence: The baseline classification prompt in Appendix B instructs models to determine the answer from the provided information, while peer opinions are presented alongside the facts and rules. However, the protocol does not explicitly establish that peer conclusions should be treated as unverified opinions rather than additional evidence. Consequently, some observed failures may arise from ambiguity in how models interpret conflicting information, rather than from a specific tendency to defer to peers. A controlled comparison with an explicit evidence-priority instruction would help clarify this distinction.

Insufficient Analysis of the Relationship Between Reasoning Accuracy and Override Susceptibility: The paper argues that independent reasoning competence and collaborative reliability are distinct capabilities. However, Evidence Only accuracy varies substantially across the evaluated models, with several achieving only approximately 36–49% despite the availability of sufficient logical evidence (Table 1). Since absolute accuracy degradation is affected by baseline performance and potential floor or ceiling effects, comparing POE values alone may not fully characterize model susceptibility. Additional analyses using baseline-normalized override rates and matched reasoning-difficulty groups would strengthen this conclusion.

Questions for authors
  1. Clarification on the Novelty and Peer-Specific Nature of POE: The proposed Peer Override Effect appears closely related to existing findings on LLM sycophancy, conformity, and susceptibility to misleading information. Although the paper evaluates Source Disclosure, it remains unclear whether the observed degradation is specifically caused by peer influence or simply by the introduction of a conflicting conclusion. Could the authors provide a controlled comparison between conflicting peer opinions and identical misleading statements without peer attribution? Such evidence would help establish whether POE represents a distinct failure mode in multi-agent collaboration.

  2. Clarification on Prompt Design and the Interpretation of Peer Opinions: Appendix B presents peer opinions alongside the available facts and rules, but does not explicitly instruct models to distinguish unverified peer conclusions from logically grounded evidence. Could the authors clarify how this distinction is conveyed to the evaluated models? More importantly, would the substantial accuracy degradation persist if models were explicitly instructed to prioritize verifiable evidence over unsupported peer conclusions? This comparison would help determine whether the observed effect reflects a fundamental limitation in collaborative reasoning or sensitivity to the experimental prompt design.

2 comments

We thank the reviewer for the constructive comments. We further clarify the motivation and construction of our benchmark and report additional controlled evaluations addressing the raised questions.

W1: Existing collaborative-reasoning and MAS benchmarks primarily evaluate end-to-end task performance. Since reasoning, evidence exchange, and multi-agent interaction occur together, it is difficult to isolate why an agent produces an incorrect result or determine whether a failure arises from its own reasoning or conflicting information received from other agents.

Our benchmark addresses this gap by constructing cases from symbolically validated proof graphs. Each case guarantees that the target conclusion follows from the receiving agent's visible facts and rules, while the conflicting peer conclusion is supported by a separate competing proof branch. Programmatic validation ensures the correctness of target labels and proof paths, eliminating hallucinated or unsupported reasoning during benchmark construction as a confounding factor. Matched evaluations then hold the receiving agent's evidence and query fixed, allowing us to isolate the effect of conflicting peer conclusions.

To further examine this phenomenon beyond synthetic classification, we constructed two task-grounded examples adapted from MATH-500 and DebugBench, covering Mathematical Reasoning and Code Validation. Their simulated peer reports contain intermediate derivations or execution traces. See our response to Reviewer 5J64 (W1 & Q1) for the detailed examples and model-level results. We will include them in the revised Appendix.

W2, Q1 & Q2: In Section 5.3, we examine whether peer override can be reduced by changing how the model processes peer input. We consider two prompt-level interventions: Source Disclosure, which explicitly states that the peer opinion comes from another agent, and Evidence Re-evaluation, which requires the model to re-derive its answer from the available evidence before producing the final prediction. Our results show that, under conflicting peer opinions, Source Disclosure changes POE from −28.9 to −42.6 pp, whereas Evidence Re-evaluation reduces it from −28.9 to −22.2 pp. Thus, identifying the source alone does not mitigate override, while explicitly re-deriving the answer from available evidence provides partial mitigation (Figs. 3–4; Appendix E, Table 9).

To further investigate whether explicit peer attribution is necessary, we conducted an additional controlled evaluation on 100 cases across six models, with three replays per condition. Macro-averaged accuracy is 71.1% under Evidence Only, 46.4% under Peer-Attributed Opinion, and 36.1% under Unattributed Candidate Answer. Thus, removing explicit peer attribution does not eliminate the degradation.

We also conducted an additional evaluation introducing an explicit evidence-priority instruction, which specifies that only the listed facts and rules determine entailment and that peer opinions are unverified conclusions rather than evidence. Accuracy changes from 70.8% under Evidence Only to 66.1% under Evidence + Peer, corresponding to a POE of −4.7 pp. Although substantially attenuated, the effect remains heterogeneous across models: Claude Sonnet 5 and Mistral Small retain negative effects (−15.0 and −10.7 pp), whereas Gemma 4 31B shows a positive effect (+8.7 pp). These additional experiments further characterize the roles of peer attribution and evidence prioritization, showing that explicit evidence-priority instructions can reduce, but do not uniformly eliminate, the effect.

W3: We further analyze the relationship between baseline reasoning accuracy and override susceptibility through three complementary analyses.

  • Conditional override rate: Based on the paired decomposition in Section 4.3 (Table 2), the proportion of initially correct Evidence Only cases that are overridden after receiving a conflicting peer opinion is 53.51%. Substantial differences exist even among near-ceiling models: Gemini 3.1 Pro has an override rate of 3.7%, compared with 43.9% for Claude Sonnet 5, 48.9% for GPT-5.6 Terra, and 49.3% for GPT-5.6 Sol. Similar variation appears among lower-baseline models, ranging from 30.8% for Gemma 4 31B to 92.1% for DeepSeek V4 Pro.
  • Baseline accuracy versus POE: Table 1 and Fig. 2 show that baseline accuracy and POE are only weakly aligned across the 14 models, with POE spanning 55.44 pp (−56.88 to −1.44 pp).
  • Structural strata: Appendix C (Fig. 8) reports POE across strata defined by rule count, proof depth, and query-inference depth. POE remains negative across all reported strata, including the shallowest query-inference-depth group (4–8), where accuracy decreases from 66.8% under Evidence Only to 36.8% under Conflicting Peer Opinion.
Reviewer_AoLKOpenReview ↗

Official comment by Reviewer AoLK

Thanks to the author for the reply; I think my concerns are addressed, I keep my positive score.

Reviewer_5J64OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how conflicting peer opinions affect LLM reasoning when sufficient evidence is already available. It introduces a controlled, validated benchmark that holds the facts, rules, and query fixed while varying peer input. Experiments across multiple models show accuracy drops under conflicting opinions, with further analyses covering message presentation, prompting, and two-turn interactions. The results suggest that strong independent reasoning does not ensure robustness to peer influence, while explicit evidence reevaluation provides partial mitigation.

Critical strengths
  • The controlled benchmark provides a clear way to examine how peer input affects reasoning when sufficient evidence is already available. The explicit rules and programmatically verified answers make the analysis more transparent and easier to interpret.

  • The experiments are fairly comprehensive, with detailed analyses across models, message formats, prompting strategies, and interaction settings. These analyses provide strong support for the main findings.

Critical weaknesses

The evaluation is mainly based on synthetic logical tasks with constructed peer opinions. Although the controlled analysis is thorough, it remains unclear how well the findings generalize to realistic multi-agent tasks with natural peer messages. Such validation would strengthen the practical relevance of the findings.

Questions for authors

I wonder how well the failure patterns observed in the synthetic benchmark reflect those in realistic multi-agent interactions. Could you provide some examples from tasks such as collaborative question answering, mathematical reasoning, or code debugging?

2 comments

We thank the reviewer for the constructive feedback and recognition of our controlled benchmark design. Our benchmark isolates a failure pattern relevant to MAS: a receiving agent possesses sufficient local evidence ($ R, F_0 \cup F_+ \vdash y $) but receives a conflicting peer conclusion whose supporting premises are unavailable in its context. To further examine this phenomenon beyond synthetic classification, we constructed two task-grounded examples adapted from MATH-500 and DebugBench, covering Mathematical Reasoning and Code Validation, respectively. Each example was evaluated across six models with three independent calls per condition. The simulated peer reports contain intermediate derivations or execution traces, rather than merely presenting conflicting answers.

  • Mathematical Reasoning (MATH-500): Let AA be the sum of positive divisors of 500. The correct result is A=1092=22×3×7×13A=1092=2^2\times3\times7\times13, so its distinct prime divisors sum to 25. The simulated peer correctly computes A=1092A=1092 but incorrectly factors it as 22×3×7×112^2\times3\times7\times11, obtaining 23 and an incorrect entailed judgment. This factorization actually equals 924, not 1092. Adding this peer report reduces accuracy to 0/3 in four models: GPT-5.6 Luna and Llama 4 Maverick (3/3 to 0/3), Gemma 4 31B (2/3 to 0/3), and Claude Sonnet 5 (1/3 to 0/3).
  • Code Validation (DebugBench): For Solution().minSteps(s='leetcode', t='coats'), the implementation returns 13 by summing the two string lengths, whereas the specification requires a minimum of 7 character additions to equalize their character frequencies. The simulated peer correctly traces the execution (cnt1 contains 8 characters, cnt2 contains 5, and the return value is 13), but incorrectly treats a feasible 13-operation construction as the minimum. With this peer report, Claude Sonnet 5, Llama 4 Maverick, and Mistral Small drop from 3/3 to 0/3.

These examples demonstrate that susceptibility to conflicting peer conclusions also arises when messages contain seemingly informative intermediate reasoning or execution traces. We will include both examples and their evaluation results in the Appendix.

Thank you for the examples and demonstration, my concerns are addressed, I will keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.