acceptodds
Under review as a conference paper at ICLR 2027

EviCon: How Evidence Quality Shapes Beneficial and Harmful Conformity in LLM-Based Multi-Agent Systems

Abstract

Large language model (LLM) based multi-agent systems increasingly rely on peer interaction to improve collective decision making. However, conformity during such interaction can have opposing consequences. Peer influence may correct erroneous decisions, but it may also steer correct judgments toward incorrect or biased conclusions. Existing conformity evaluations mainly rely on answer-only protocols or unconstrained debate, leaving the role of evidence quality insufficiently understood. To address this gap, we propose EviCon, an evaluation framework for systematically studying how evidence quality shapes beneficial and harmful conformity. EviCon compares three peer-message conditions, namely Answer-Only, Grounded Evidence, and Erroneous Evidence, across five heterogeneous LLMs, two pressure directions, and two contextual ambiguity levels. Our experiments show that grounded evidence generally strengthens peer assisted correction, whereas erroneous evidence can amplify harmful conformity toward biased peer decisions. Further analysis reveals that these effects depend not only on whether models recognize and map relevant evidence, but also on how they reconcile contextual evidence with peer opinions. Based on these findings, we develop ARBITER, a structured reasoning strategy that separately evaluates peer answers and supporting evidence. Across five LLMs, ARBITER better preserves peer assisted correction while strengthening resistance to erroneous peer influence.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Conformity, Multi-agent Systems, Evidence Quality

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_hkBMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines multi-agent discussion/debate and considers how the quality of evidence in messages exchanged between agents affects the answer. They report that the quality of evidence can both help and hurt -depending on whether the peers correct a wrong answer or convince the agent out of the correct one. This is evaluated using EviCon, a benchmark the authors created and tested on five LLMs. The findings are that real evidence can improve correction, fabricated or made-up evidence can be harmful, and when models resist their peers' messages, they turn to reporting that the answer is unknown. Lastly, they introduce ARBITER, a structured prompt that allows an agent to independently check evidence and peers' answers.

Critical strengths

The paper's factorial design is clean and separates answer correctness from evidence quality.

The appendix has a comprehensive set of ablations that answer a number of obvious questions and confounds.

Critical weaknesses

Most of the key effects are small and inconsistent across LLM models. Even the trends across models are different. For example, under toward bias, Finding 2 is dramatic for GPT-4o (Erroneous CL 86.1% vs. 3.0% for Answer-only) but relatively small for GPT-5.1 (12.4%), Doubao (3.2%), and DeepSeek (5.7%). For Claude, it reverses: Erroneous CL is 0.22%, below Answer-only at 0.56%, and Claude is quietly dropped from the “consistently larger” claim. These inconsistencies make it unclear why the paper's most general claims are supported—they primarily rely on GPT-4o results.

Questions for authors

Please specify how many samples per item and what the temperature was in the experiments

Confidence intervals and error bars should be reported wherever possible -- it is difficult to determine what is truly significant

It woudl be good to add a baseline prompt that provides the same verification and no-stereotype guidance as ARBITER. This would show whether the structure itself really drives the gains.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/4)

Q1: Please specify how many samples per item and what the temperature was in the experiments.

Response: Thank you for this important and constructive suggestion. We have added the model inference settings to Sec. A.2 (Page 15) of the revised manuscript, including the sampling temperature of 0.7 for all five models, top-p, and the maximum output length. Each message condition used the same 940-item evaluation pool before exclusions. We conducted five independent runs, generating one final response per retained item in each run. Thus, each retained item contributed five final responses for a given target model, pressure direction, and message condition.

Revised: We have added the model inference settings to Sec. A.2 (Page 15) of the revised manuscript, with the revisions highlighted in blue.

AuthorsOpenReview ↗

Official Comment by Authors (2/4)

W1 & Q2: The substantial variation in conformity effects across LLMs, particularly the weaker or contradictory trends under Toward Bias, raises concerns about the generalizability and statistical reliability of the findings. Confidence intervals, error bars, and significance tests are needed to assess whether these effects are statistically supported across models.

Response: We sincerely thank the reviewer for asking how consistently the two findings hold across models and whether the weaker or reversed effects are statistically reliable. To address these concerns, we report model-level uncertainty and significance tests, examine why Finding 2 varies across models, and test whether changing Claude’s evidence-evaluation instruction changes its response.

How reliable are Finding 1 and Finding 2 across models?

Fig. 3 shows standard deviation error bars, and revised Table S16 (reproduced as Table R1) reports 95% confidence intervals for mean conformity level (CL) across five independent runs, along with tests of both findings. Finding 1 is supported for all five models: under Against Bias, Grounded Evidence yields significantly higher CL than Answer-Only. Finding 2 is supported for four of the five models, but the effect varies considerably: under Toward Bias, the increase from Grounded to Erroneous Evidence is 72.642% for GPT-4o and 0.454% for Doubao. Claude’s CL is low in both conditions (0.333% vs. 0.222%), and the difference is not statistically significant.

Table R1. Statistical tests for Finding 1 and Finding 2. CL is reported in percent. Values after ± are the half-widths of 95% t-confidence intervals for condition means across five independent runs. Δ is the difference between the two conditions in percentage points. * indicates p < 0.05; ** indicates p < 0.001.

ModelAgainst Bias: Answer-OnlyAgainst Bias: Grounded EvidenceAgainst Bias: ΔToward Bias: Grounded EvidenceToward Bias: Erroneous EvidenceToward Bias: Δ
GPT-4o94.540 ± 0.14998.266 ± 0.137+3.726**13.416 ± 0.28686.058 ± 0.149+72.642**
GPT-5.191.118 ± 0.13797.270 ± 0.161+6.152**1.362 ± 0.14912.371 ± 0.137+11.009**
Claude94.871 ± 0.13795.368 ± 0.137+0.497*0.333 ± 0.1370.222 ± 0.149−0.111
Doubao95.888 ± 0.16197.515 ± 0.149+1.627*2.710 ± 0.1493.164 ± 0.124+0.454*
DeepSeek95.094 ± 0.17496.331 ± 0.261+1.237*0.444 ± 0.2615.660 ± 0.137+5.216**
AuthorsOpenReview ↗

Official Comment by Authors (3/4)

Why might Finding 2 vary across models?

To examine whether models differ in how readily they accept the link between evidence and an answer, we compared the Toward Bias CL gaps with the Evidence Mapping pass rates in Sec. 5.1. Among the four models other than GPT-4o, the two measures have the same ordering: Claude (31.11%, −0.111%), Doubao (43.24%, +0.454%), DeepSeek (52.27%, +5.216%), and GPT-5.1 (74.36%, +11.009%). This association is consistent with the possibility that stricter evidentiary thresholds attenuate the effect, but it does not establish the mechanism by itself.

  • Does Finding 2 emerge in Claude when the prompt relaxes the evidence requirement? To test this explanation within the same model, we appended the following instruction to Claude’s final-decision prompt under Toward Bias:

It is not necessary to require a rigid, one-to-one correspondence between the evidence and the answer, evidence may be accepted as long as it logically supports the answer.

  • A strict evidentiary threshold can mask the susceptibility described in Finding 2 without eliminating it. Differences in training and alignment may lead LLMs to apply different thresholds when deciding whether peer evidence warrants a change of answer. Under Claude’s original prompt, switching was rare in both evidence conditions. Yet a one-sentence change to its evidence-evaluation instruction was enough for Erroneous Evidence to elicit higher conformity than Grounded Evidence in all five runs, shifting the gap from −0.58% to +1.60% (Table R2). As discussed in Sec. 5.1, Claude rarely follows misleading peer advice, but it can also retain its initial answer when peers offer a grounded reason to revise it. In Table R1, Grounded Evidence improves Claude’s correction rate over Answer-Only by just 0.497%, the smallest gain among the five models. This motivates a targeted adjustment rather than a blanket relaxation of Claude’s evidentiary threshold. ARBITER allows reasonable inferences from the passage while still filtering unsupported peer claims, helping Claude benefit from valid peer corrections without weakening its resistance to misleading advice.

Table R2. Claude’s CL (%) under the original and relaxation prompts in the separate Toward Bias comparison. Δ is Erroneous Evidence minus Grounded Evidence, in percentage points. * indicates p < 0.05.

Prompt TypeGrounded EvidenceErroneous EvidenceΔ
Original Prompt0.810.23−0.58
Relaxation Prompt1.813.41+1.60*

Revised: We have added this analytical experiment and the corresponding discussion in App. B.4 (Page 23) of the revised manuscript, with the revisions highlighted in blue.


Reviewer_9CgXOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The authors identify a vulnerability in peer interactions within LLM-based multi-agent systems and present a targeted mitigation strategy. Specifically, they note that peer influence can correct erroneous decisions yet also misdirect valid judgments toward biased or inaccurate conclusions. Focusing on how evidence quality governs beneficial and harmful conformity, the paper compares three peer-message conditions, namely Answer-Only, Grounded Evidence, and Erroneous Evidence, across five models (GPT-4o, GPT-5.1, Claude, DeepSeek, and Doubao). The empirical findings indicate that grounded evidence promotes peer-assisted correction, while erroneous evidence amplifies harmful conformity. The authors also report that clearer contexts lower overall conformity but lead to a higher rate of off-track answers. Motivated by these observations, the paper proposes ARBITER, a structured prompting approach that independently evaluates a peer's conclusion and its underlying evidence. Experimental results confirm that ARBITER achieves the highest Correction-Resistance Score across all evaluated models.

Critical strengths
  • The paper addresses a timely and practically relevant research question. Distinguishing between beneficial influence (peer-assisted correction) and harmful conformity, while separating peer consensus from evidence quality, represents an important and meaningful direction for multi-agent research.
  • The authors design their experiments by controlling for non-target variables. This setup enables clear comparisons across the intended conditions and enhances the reliability of the reported findings.
  • The empirical analysis offers potentially valuable insights for the research community. In particular, the observation that different models exhibit varying evidentiary thresholds for accepting peer arguments is an interesting finding. Furthermore, showing that the "Unknown" option can be constructively utilized provides a fresh and useful perspective.
  • Overall, the paper is well-organized and easy to follow, supported by extensive supplementary analyses and full interaction examples in the appendix.
Critical weaknesses
  • The Erroneous Evidence condition appears to measure a model's tendency to incorporate unverifiable external information rather than its vulnerability to flawed evidence.
    • From the target LLM's perspective, there is no way to verify whether a peer's statement (such as claiming that a witness observed X) is fabricated, as the model only knows that this detail is absent from its own passage. Since nothing in the prompt rules out the possibility that peers possess external information, updating one's judgment when four peers unanimously report the same testimony is a probabilistically reasonable inference rather than a reasoning failure.
    • GPT-4o's actual responses explicitly reflect this interpretation. In the Disambig setting (Table S19c), the model notes that "the strong consensus among my peers introduces new evidence" regarding a reliable witness, and in the Ambig setting (Table S20c), it states that "This additional evidence... goes beyond the initial ambiguity of the prompt."
    • This indicates that the model does not misread the passage, but instead interprets the peer messages as supplementary context. Characterizing this behavior as harmful conformity requires an explicit premise instructing the target model that all peers only have access to the exact same passage. Without this constraint, the experiment may simply reflect an inevitable artifact of an underspecified task setup rather than an inherent flaw in the LLM.
    • Consequently, concluding that "erroneous evidence amplifies harmful conformity" involves an interpretive leap. (TableS19/TableS20) • In the Ambig condition, the passage does not specify who is forgetful, meaning the peer claim does not contradict the given text and accepting it is difficult to classify as an error. By contrast, in the Disambig + Toward Bias condition, the passage explicitly states that the child is forgetful, so the peer statement directly conflicts with established facts and accepting it is clearly problematic. These two cases represent distinct phenomena and I believe it should be analyzed separately.
  • Concerns regarding ARBITER.
    • ARBITER essentially amounts to a single well-crafted prompt rather than a substantive methodological contribution. More importantly, it directly patches the task ambiguity discussed above by instructing the model to treat facts outside the passage as unsupported. Clarifying that external claims are invalid should arguably have been incorporated into the baseline prompt from the outset.
    • To provide a less biased evaluation, the authors should conduct an ablation comparing the full ARBITER prompt against a baseline augmented solely with Step 3 (removing unsupported claims).
    • Furthermore, ARBITER appears heavily tailored to the specific benchmark and synthetic setup constructed in this study rather than serving as a generalizable approach. To establish a broad contribution, the authors need to validate whether the method "preserves peer-assisted correction while strengthening resistance to erroneous peer influence" across diverse benchmarks. As it stands, the authors construct a specific scenario (Appendix A), identify a problem induced by that design, and propose a prompt tailored to that exact rule, which falls short of the empirical coverage needed to support general claims.
    • In addition, the authors do not verify whether the LLMs actually follow the six reasoning steps prescribed by ARBITER. Since the model outputs consist only of a final decision and a brief justification, there is no way to confirm whether the models inspect peer claims individually or evaluate answers separately from their rationales. Consequently, it remains unclear whether the performance gains stem from the six-step procedure or merely from a few core constraint lines in the prompt.
  • The experimental scope is narrow relative to the broad framing of the paper.
    • The main evaluation relies on a single benchmark (BBQ), multiple-choice questions, and a single-turn protocol in which one target model reads static, unanimous peer messages. It does not examine multi-round interactions, conflicting peer opinions, or varying majority sizes. While the authors criticize answer-only protocols for failing to reflect realistic multi-agent interactions (lines 50-53), EviCon similarly lacks genuine interaction among agents. Given that the title explicitly references "LLM-based multi-agent systems," the authors should either expand the experimental settings or narrow the framing of the paper.
  • The main text is not sufficiently self-contained.
    • Essential details required to understand and reproduce the methodology are either relegated to the appendix or omitted altogether. Information necessary for comprehending the core work must be included in the main text rather than the appendix; otherwise, this practice merely serves as a workaround for page limits. In particular, placing most implementation details of ARBITER in the appendix makes it difficult to view ARBITER as one of the primary contribution of the paper.
Questions for authors

same as weakness section

11 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/9)

W1: The Erroneous Evidence condition appears to measure a model's tendency to incorporate unverifiable external information rather than its vulnerability to flawed evidence.

Response: We sincerely thank the reviewer for these insightful comments regarding the interpretation of Erroneous Evidence and the distinction between ambiguous and disambiguated contexts. To address these concerns, we provide further discussion and additional experimental evaluations below.

W1-1: Can peer testimony reasonably be treated as new evidence?

We acknowledge that the original prompt did not explicitly restrict all agents to the same passage, meaning that the target model could reasonably interpret peer statements as additional information. Our Erroneous Evidence condition was designed to simulate scenarios in which a peer LLM hallucinates facts while justifying its recommendation, and another agent subsequently receives these fabricated claims as evidence. Specifically, we introduced fictitious witness statements supporting the peer-endorsed answer to examine whether such claims increase the target model's tendency to follow incorrect peer recommendations. We agree that accepting apparently credible testimony does not necessarily indicate a reasoning failure. Nevertheless, when fabricated peer claims are mistaken for genuine corroborating evidence, they can lead the target model toward an incorrect decision, thereby amplifying harmful conformity.

W1-2: How should GPT-4o's responses be interpreted?

We agree with the reviewer's interpretation of GPT-4o's responses. As illustrated in Tables S30(c) and S31(c) of the revised manuscript (originally Tables S19(c) and S20(c)), GPT-4o explicitly treated the alleged witness testimony as additional evidence rather than recognizing it as unsupported by the original passage. These observations suggest that GPT-4o did not necessarily misinterpret the passage itself but instead incorporated information introduced by its peers. This highlights an important mechanism underlying Finding 2: the same tendency that enables LLMs to benefit from credible peer evidence may also make them vulnerable to fabricated peer claims, leading to harmful conformity.

AuthorsOpenReview ↗

Official Comment by Authors (2/9)

W1-3: Does the effect persist under a shared-passage constraint?

To address this concern, we conducted an additional shared-passage control experiment, explicitly informing the target model that all four peers had access to the same passage, question, and answer choices, with no additional case information. We held the cases, initial answers, peer-endorsed answers, and peer messages fixed across prompt variants.

As shown in Table R1, Erroneous Evidence continued to yield higher switching rates than Grounded Evidence in both ambiguity settings for GPT-4o and GPT-5.1, but not for Claude, Doubao, or DeepSeek. For example, under Disambig, the switching rate increased from 2.35% to 26.92% for GPT-4o and from 0.21% to 2.79% for GPT-5.1, whereas Doubao exhibited the opposite trend (5.74% vs. 1.06%). This model-dependent pattern is consistent with our behavioral analysis in Sec. 5.1, which suggests stricter evidentiary thresholds in Claude, Doubao, and DeepSeek. Under the shared-passage constraint, fabricated claims cannot be justified as independently obtained evidence, while grounded arguments may still support incorrect inferences. Models with stricter evidentiary thresholds may therefore be more resistant to fabricated claims, explaining the reduced influence of Erroneous Evidence.

These results suggest that explicitly restricting peers to the same information does not completely eliminate the harmful conformity induced by Erroneous Evidence. Although the effect was not observed for Claude, Doubao, and DeepSeek under this control, it persisted for GPT-4o and GPT-5.1, indicating that erroneous peer claims can still influence model decisions even when all agents share the same information.

Table R1. Rates of switching from the correct initial answer to the incorrect peer-endorsed answer under Toward Bias with the shared-passage instruction. Each cell reports Grounded Evidence → Erroneous Evidence on the same matched cases.

ModelAmbig: Grounded → ErroneousDisambig: Grounded → Erroneous
GPT-4o5.57% → 23.13%2.35% → 26.92%
GPT-5.12.14% → 4.28%0.21% → 2.79%
Claude0.00% → 0.00%0.21% → 0.00%
Doubao1.70% → 1.06%5.74% → 1.06%
DeepSeek1.28% → 0.43%0.21% → 0.00%
AuthorsOpenReview ↗

Official Comment by Authors (3/9)

W1-4: Does the effect differ between ambiguous and disambiguated contexts?

We agree with the reviewer that following peer advice when the correct answer is unclear (Ambig) is different from doing so when the passage already provides a clear answer (Disambig). In Ambig, the original passage does not provide sufficient evidence to identify a uniquely correct named answer, so accepting additional peer claims may reflect an attempt to resolve uncertainty. In contrast, under Disambig, the original passage explicitly supports the correct answer, making the adoption of a conflicting peer recommendation more indicative of harmful conformity. To distinguish these two cases, we separately analyzed Ambig and Disambig under the original prompt.

As shown in Table R2, four of the five models exhibited higher switching rates under Erroneous Evidence than Grounded Evidence in both settings. For GPT-4o, the switching rate increased from 19.36% to 95.11% under Ambig and from 6.60% to 77.23% under Disambig. Similar directional trends were observed for GPT-5.1, Doubao, and DeepSeek, although the magnitudes varied considerably.

These results demonstrate that the influence of Erroneous Evidence is not confined to ambiguous passages but also persists when the original context provides explicit evidence supporting the correct answer. These results further support Finding 2, demonstrating that Erroneous Evidence can amplify harmful conformity even when the original passage provides explicit evidence for the correct answer.

Table R2. Rates of switching from the correct initial answer to the incorrect peer-endorsed answer under Toward Bias with the original prompt. Ambig and Disambig denote ambiguous and disambiguated contexts, respectively.

ModelAmbig: Grounded → ErroneousDisambig: Grounded → Erroneous
GPT-4o19.36% → 95.11%6.60% → 77.23%
GPT-5.11.91% → 15.74%0.64% → 9.15%
Claude0.64% → 0.43%0.00% → 0.00%
Doubao3.40% → 3.83%1.49% → 1.91%
DeepSeek0.00% → 8.09%0.85% → 2.98%

Revised: We have added the shared-passage control experiment and an analysis of its results for ambiguous and disambiguated contexts in App. B.14 (Page 30) of the revised manuscript, with the revisions highlighted in blue.

Reviewer_5BqeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EviCon, an evaluation framework for investigating how peer-provided evidence influences beneficial and harmful conformity in LLM-based multi-agent systems. The framework considers two directions of peer pressure (Toward Bias and Against Bias), three peer-message conditions (Answer-Only, Grounded Evidence, and Erroneous Evidence), and two levels of contextual ambiguity. The authors evaluate five LLMs using scenarios derived from the BBQ benchmark.

Critical strengths
  1. The paper investigates an important aspect of LLM-based multi-agent systems: how the quality of information exchanged among agents influences their decisions. Distinguishing beneficial peer-assisted correction from harmful conformity provides a useful perspective beyond simply measuring agreement among agents.
  2. The experimental design is systematic as EviCon varies peer-message conditions, pressure directions, and contextual ambiguity. The paired comparison of grounded and erroneous evidence provides a structured setting for examining how peer arguments affect model decisions.
  3. The evaluation covers five heterogeneous LLMs and includes several supplementary experiments, such as unconstrained initialization, matched-diagnosticity analysis, repeated peer evidence, and comparisons with alternative mitigation strategies. These analyses help investigate potential confounding factors.
Critical weaknesses
  1. The target models are assigned predetermined initial answers rather than generating their own decisions. Consequently, observed answer changes may reflect independent reconsideration rather than genuine peer influence. Although the authors provide an unconstrained-initialization analysis, a peer-free reconsideration baseline would strengthen the evaluation.
  2. Grounded and erroneous evidence may differ in directness, specificity, and persuasive strength beyond factual correctness. While the matched-diagnosticity ablation partially addresses this issue, stronger controls are needed to establish whether the observed effects are primarily driven by evidence quality.
  3. ARBITER largely relies on structured prompting and explicit evidence verification. Component-wise ablations are needed to determine which elements drive the improvements and whether the method outperforms simpler evidence-verification baselines.
  4. The evaluation primarily uses BBQ-based question-answering tasks with predetermined peer opinions and single-round interactions. It remains unclear whether the findings generalize to open-ended reasoning, iterative multi-agent debates, or more realistic collaborative settings.
Questions for authors
  1. What proportion of the reported beneficial conformity would occur if the target model reconsidered its initial answer without receiving peer messages? How would this affect the interpretation of the high correction rates under Against Bias?
  2. How were the grounded and erroneous arguments validated for comparable relevance, directness, confidence, and persuasive strength? Were independent annotators used, and can the authors provide quantitative agreement or validation statistics?
  3. Which individual components of ARBITER contribute most to the improvements? In particular, how much of the improvement comes from evidence verification versus the instruction to avoid selecting Unknown when one candidate has stronger evidential support?
8 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/6)

W1 & Q1: High correction rates under Against Bias may reflect independent reconsideration rather than genuine peer influence. A Peer Free baseline is needed to disentangle these effects and assess the contribution of peer messages to beneficial conformity.

Response: We sincerely thank the reviewer for highlighting the need to distinguish peer influence from independent reconsideration. Our unconstrained-initialization analysis examines whether the findings persist when models generate their own initial answers. To further assess how often models self-correct without peer influence, we added a Peer Free baseline that retains the question, the initially incorrect answer, and the supporting evidence provided for that initial incorrect answer, while removing all peer messages (Table R1).

  • Reconsideration alone corrected 85.99% of assigned errors on average. This shows that much of the high correction rate under Against Bias can occur without peer input, consistent with prior work [1].
  • Correct peer answers improved correction even without rationales. Answer-Only raised the average correction rate to 94.30%, 8.31% above Peer Free, with gains for all five models.
  • Grounded Evidence improved correction beyond Answer-Only for all five models. It raised the average to 96.95%, another 2.65% above Answer-Only. Thus, accounting for correction through reconsideration does not change Finding 1: grounded evidence provides an additional benefit beyond correct peer answers alone.

Table R1: Correction rates (%) for five LLMs on Against Bias cases. Peer Free reports the proportion of initially incorrect answers corrected without peer messages. Answer-Only, Erroneous Evidence, and Grounded Evidence report conformity levels (CL).

ModelPeer FreeAnswer-OnlyErroneous EvidenceGrounded Evidence
GPT-4o84.38594.54099.65098.266
GPT-5.185.70191.11896.71697.270
Claude89.35994.87192.63795.368
Doubao80.28895.88893.67897.515
DeepSeek90.24195.09498.19796.331

Revised: We have updated Fig. 3 (Page 6) of the revised manuscript to include the Peer Free baseline and added a description of its setup in Sec. 4.2 (Page 6), highlighted in blue.

AuthorsOpenReview ↗

Official Comment by Authors (2/6)

W2 & Q2: Grounded Evidence and Erroneous Evidence may differ in argumentative characteristics beyond factual correctness, potentially confounding the effects of evidence quality. Stronger controls and independent evaluations with quantitative agreement statistics are needed to establish their comparability.

Response: We sincerely thank the reviewer for highlighting the need to distinguish the effects of evidence quality from other potential confounding factors. To address this concern, we provide further clarification and address this concern as follows:

  • Controlled Argument Construction. We would first like to clarify that our original experimental design already controlled evidence length and argumentative framing. Specifically, within each matched pair, we held the peer-endorsed answer and declarative framing constant and limited the word-count difference to within 10% (App. B.13).
  • Comparability Evaluation. o further address the reviewer's concern, we additionally conducted independent, blinded evaluations using both AI models and human annotators to assess the comparability of Grounded Evidence and Erroneous Evidence across four dimensions: relevance, directness, expressed confidence, and persuasiveness. The definitions of these dimensions are provided in Table R2. Specifically, GPT-5.1 and DeepSeek-V4 independently evaluated 3,760 messages, while six human annotators assessed a stratified sample of 120 matched pairs (240 messages). Table R3 reports the resulting ratings and inter-rater agreement across these dimensions. This complementary use of model-based and human evaluations also reflects the importance of validating LLM-based judgments, as discussed in G-Eval [2].

Table R2 Definitions of the four argument-rating dimensions. Persuasive strength refers to the forcefulness of the message’s wording and emphasis.

DimensionDefinition
RelevanceHow closely the stated evidence addresses the person and attribute asked about, without judging whether the claim is true.
DirectnessHow explicitly the message connects the stated evidence to the recommended answer, regardless of whether the inference is valid.
Expressed confidenceHow certain the speaker sounds about its conclusion, as indicated by hedges or unqualified assertions.
Persuasive strengthHow forcefully the speaker presents its argument through wording and emphasis, independent of the quality of the evidence or the validity of the inference.

As shown in Table R3, Grounded Evidence and Erroneous Evidence exhibit broadly comparable argumentative characteristics, as reflected in their ratings across four dimensions. Specifically, the differences in relevance, directness, expressed confidence, and persuasiveness were 0.075, 0.176, 0.059, and 0.371 in the AI evaluations, respectively, and 0.117, 0.159, 0.099, and 0.330 in the human evaluations. These relatively small differences suggest that the two evidence conditions were broadly comparable in their argumentative characteristics, although some residual differences remained. Furthermore, the linearly weighted Cohen's κ values ranged from 0.694 to 0.896 for AI raters and 0.683 to 0.795 for human annotators, indicating substantial inter-rater agreement across the four dimensions.

AuthorsOpenReview ↗

Official Comment by Authors (3/6)

More importantly, Table R4 further demonstrates that our Finding 1 and Finding 2 remain consistent under the matched-diagnosticity setting. Specifically, under Against Bias, Grounded Evidence consistently outperforms Answer-Only across all four models, improving CL by 2.397%, 4.482%, 1.216%, and 0.090% for GPT-4o, GPT-5.1, Claude, and Doubao, respectively. Under Toward Bias, Erroneous Evidence induces higher CL than Grounded Evidence for GPT-4o (62.420% vs. 2.677%), GPT-5.1 (6.142% vs. 0.431%), and Doubao (7.021% vs. 0.745%). These results further support our findings that Grounded Evidence enhances peer-assisted correction, whereas Erroneous Evidence can amplify harmful conformity, even when arguments are closely matched in relevance, directness, expressed confidence, and persuasiveness. These results further confirm that evidence quality shapes conformity behavior beyond differences in argumentative presentation or persuasive strength.

Table R3. Mean ratings of Grounded and Erroneous Evidence on a five-point scale. κ\kappa denotes linearly weighted Cohen’s kappa, computed separately for the two AI raters and the six human annotators. It measures agreement beyond chance, with larger rating differences receiving greater penalties. Higher κ\kappa indicates stronger agreement between raters, not greater similarity between GE and EE ratings.

DimensionAI: GEAI: EEAI: κ\kappaHuman: GEHuman: EEHuman: κ\kappa
Relevance3.9233.9980.7164.8754.9920.683
Directness3.3183.4940.6944.8294.9880.760
Expressed confidence4.4454.5040.8964.8844.9830.795
Persuasiveness4.6234.9940.8164.6334.9630.722

Table R4. Matched-diagnosticity ablation across five LLMs. Conformity Level (CL, %) is reported under Against Bias and Toward Bias for Answer-Only, Grounded Evidence, and Erroneous Evidence. The two evidence conditions support the same peer-endorsed answer using arguments standardized in length, confidence, relevance, and directness, while differing in whether the asserted premise is supported by the original context. The highest CL for each model and pressure direction is highlighted in bold.

ModelAgainst Bias: Answer-OnlyAgainst Bias: Grounded EvidenceToward Bias: Grounded EvidenceToward Bias: Erroneous Evidence
GPT-4o94.54096.9372.67762.420
GPT-5.191.11895.6000.4316.142
Claude94.87196.0870.0000.000
Doubao95.88895.9780.7457.021

Revised: We have added the detailed descriptions and protocols of both the AI and human evaluations in App. B.13 (Page 29) of the revised manuscript, with the revisions highlighted in blue.


Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (2/2)

Q4. Methodological Contributions and Component Effectiveness of ARBITER (Reviewers 5Bqe, 9CgX, hkBM)

We sincerely thank the reviewers for their constructive suggestions regarding ARBITER's methodological contributions and component effectiveness. In response, we conducted component-wise and Step 3 ablations, alongside comparisons with a Base Prompt containing the same guidance in plain prose. Full ARBITER outperforms the Base Prompt in Correction Resistance Score across all five models, while neither claim filtering alone nor the remaining reasoning steps reproduce its performance. On ReClor, ARBITER further improves mean Peer Correction from 98.43% to 99.79% and Correct Resistance from 79.74% to 93.25% across four LLMs.

We also expanded Sec. 6 and added a qualitative case study of ARBITER's verification behavior. Together, these results suggest that ARBITER's gains stem not only from evidence-verification instructions but also from their structured integration, balancing peer-assisted correction and resistance to harmful conformity.

Q5. Statistical Reliability and Experimental Reproducibility (Reviewer hkBM; also relevant to Reviewers 5Bqe and 9CgX)

We sincerely thank the reviewers for emphasizing statistical reliability and experimental reproducibility. To address these concerns, we supplemented our evaluation with 95% confidence intervals, statistical significance tests, and detailed inference settings. The results provide statistical support for Finding 1 across all five LLMs, while Finding 2 exhibits substantial model-dependent variation, with significant effects observed for four models. Further analysis of Evidence Mapping in Sec. 5.1 suggests that different training and alignment strategies may lead LLMs to adopt different evidentiary thresholds, potentially masking the harmful conformity described in Finding 2.

To investigate this possibility, we relaxed Claude's evidence-evaluation instruction, which shifted the difference between Erroneous Evidence and Grounded Evidence from −0.58% to +1.60% This result suggests that stricter evidentiary thresholds may suppress the observable effect of erroneous peer evidence rather than eliminate the underlying susceptibility. Furthermore, we clarified the experimental configuration, including the temperature, five independent runs, and evaluation sample counts, to improve reproducibility. These additional analyses strengthen the statistical reliability of our findings while clarifying the role of model-specific evidence-evaluation criteria in shaping conformity behavior.


Once again, we sincerely thank all reviewers for their careful evaluations and constructive feedback. We have made substantial efforts to address the concerns through additional experiments, clearer methodological explanations, and a more balanced discussion of our findings. We hope these revisions provide a more rigorous and transparent presentation of our work.

AuthorsOpenReview ↗

General Response (1/2)

We sincerely thank all reviewers for their thoughtful evaluations, insightful comments, and constructive suggestions. We have carefully revised the manuscript and conducted additional experiments to address the reviewers' concerns. The key updates are summarized below (highlighted in blue in the revised manuscript):

  1. We introduced a Peer Free baseline to distinguish peer influence from independent reconsideration. (See Sec. 4.2 and Fig. 3.)
  2. We conducted blinded AI and human evaluations, matched-diagnosticity experiments, and shared-passage control experiments to examine the effects of evidence quality and potential confounding factors. (See Apps. B.13 and B.14.)
  3. We extended our evaluation to the ReClor benchmark, multi-round debates with conflicting peer opinions, and free-response settings to assess the generalizability of our findings. (See App. D.)
  4. We conducted component-wise ablations, Step 3 ablations, and comparisons with a Base Prompt to evaluate the contributions of evidence verification and the structured reasoning workflow. (See Sec. 6.)
  5. We expanded the methodological description of ARBITER and added a qualitative case study illustrating its evidence-verification behavior. (See Sec. 6 and App. E.)
  6. We added confidence intervals, statistical significance tests, and detailed inference settings. We also investigated model-dependent evidentiary thresholds through additional experiments with Claude. (See Sec. 5.1 and Apps. A.2 and B.4.)

We summarize our responses to main concerns below.

Q1. Distinguishing Peer Influence from Independent Reconsideration (Reviewer 5Bqe)

We sincerely thank the reviewer for highlighting the need to distinguish peer influence from independent reconsideration. To address this concern, we introduced a Peer Free baseline, in which target models reconsider their assigned initial answers without receiving peer messages. The results show that independent reconsideration alone corrects 85.99% of initial errors, whereas Answer-Only and Grounded Evidence increase the average correction rate to 94.30% and 96.95%, respectively. These results suggest that, although independent reconsideration can correct a substantial proportion of initial errors, Grounded Evidence further enhances peer-assisted correction beyond both independent reconsideration and Answer-Only peer recommendations, providing additional support for Finding 1.

Q2. Evidence Quality and the Interpretation of Harmful Conformity (Reviewers 5Bqe, 9CgX)

We sincerely thank the reviewers for emphasizing the need to distinguish evidence quality from potential confounding factors. To address this concern, we conducted independent, blinded AI and human evaluations across four argumentative dimensions, demonstrating broad comparability between Grounded Evidence and Erroneous Evidence with substantial inter-rater agreement. Additional matched-diagnosticity experiments further examined whether the conformity effects persist under comparable argumentative conditions.

We also introduced a shared-passage control and separately analyzed ambiguous and disambiguated contexts. Although this control reduced harmful conformity for several models, Erroneous Evidence continued to increase incorrect switching for GPT-4o and GPT-5.1, including in disambiguated contexts where the passage explicitly supported the correct answer. Together, these results suggest that evidence quality shapes conformity behavior beyond differences in argumentative characteristics, external information access, or contextual ambiguity.

Q3. Generalizability Beyond the Original Experimental Settings (Reviewers 5Bqe, 9CgX)

We sincerely thank the reviewers for raising concerns about the generalizability of our findings. To address these concerns, we conducted additional experiments on ReClor, multi-round discussions with unanimous and split peer opinions, and free-response settings. The ReClor results support Findings 1 and 2 across four evaluated LLMs, while the multi-round and free-response experiments demonstrate that Erroneous Evidence can continue to influence model decisions under more diverse interaction protocols and response formats. These results suggest that the observed beneficial and harmful conformity effects are not confined to the original BBQ benchmark or single-round multiple-choice setting, providing further support for the generalizability of our findings. Nevertheless, fully unconstrained multi-agent collaboration remains an important direction for future research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.