- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines multi-agent discussion/debate and considers how the quality of evidence in messages exchanged between agents affects the answer. They report that the quality of evidence can both help and hurt -depending on whether the peers correct a wrong answer or convince the agent out of the correct one. This is evaluated using EviCon, a benchmark the authors created and tested on five LLMs. The findings are that real evidence can improve correction, fabricated or made-up evidence can be harmful, and when models resist their peers' messages, they turn to reporting that the answer is unknown. Lastly, they introduce ARBITER, a structured prompt that allows an agent to independently check evidence and peers' answers.
Critical strengths
The paper's factorial design is clean and separates answer correctness from evidence quality.
The appendix has a comprehensive set of ablations that answer a number of obvious questions and confounds.
Critical weaknesses
Most of the key effects are small and inconsistent across LLM models. Even the trends across models are different. For example, under toward bias, Finding 2 is dramatic for GPT-4o (Erroneous CL 86.1% vs. 3.0% for Answer-only) but relatively small for GPT-5.1 (12.4%), Doubao (3.2%), and DeepSeek (5.7%). For Claude, it reverses: Erroneous CL is 0.22%, below Answer-only at 0.56%, and Claude is quietly dropped from the “consistently larger” claim. These inconsistencies make it unclear why the paper's most general claims are supported—they primarily rely on GPT-4o results.
Questions for authors
Please specify how many samples per item and what the temperature was in the experiments
Confidence intervals and error bars should be reported wherever possible -- it is difficult to determine what is truly significant
It woudl be good to add a baseline prompt that provides the same verification and no-stereotype guidance as ARBITER. This would show whether the structure itself really drives the gains.
6 comments
Official Comment by Authors (2/4)
W1 & Q2: The substantial variation in conformity effects across LLMs, particularly the weaker or contradictory trends under Toward Bias, raises concerns about the generalizability and statistical reliability of the findings. Confidence intervals, error bars, and significance tests are needed to assess whether these effects are statistically supported across models.
Response: We sincerely thank the reviewer for asking how consistently the two findings hold across models and whether the weaker or reversed effects are statistically reliable. To address these concerns, we report model-level uncertainty and significance tests, examine why Finding 2 varies across models, and test whether changing Claude’s evidence-evaluation instruction changes its response.
How reliable are Finding 1 and Finding 2 across models?
Fig. 3 shows standard deviation error bars, and revised Table S16 (reproduced as Table R1) reports 95% confidence intervals for mean conformity level (CL) across five independent runs, along with tests of both findings. Finding 1 is supported for all five models: under Against Bias, Grounded Evidence yields significantly higher CL than Answer-Only. Finding 2 is supported for four of the five models, but the effect varies considerably: under Toward Bias, the increase from Grounded to Erroneous Evidence is 72.642% for GPT-4o and 0.454% for Doubao. Claude’s CL is low in both conditions (0.333% vs. 0.222%), and the difference is not statistically significant.
Table R1. Statistical tests for Finding 1 and Finding 2. CL is reported in percent. Values after ± are the half-widths of 95% t-confidence intervals for condition means across five independent runs. Δ is the difference between the two conditions in percentage points. * indicates p < 0.05; ** indicates p < 0.001.
| Model | Against Bias: Answer-Only | Against Bias: Grounded Evidence | Against Bias: Δ | Toward Bias: Grounded Evidence | Toward Bias: Erroneous Evidence | Toward Bias: Δ |
|---|---|---|---|---|---|---|
| GPT-4o | 94.540 ± 0.149 | 98.266 ± 0.137 | +3.726** | 13.416 ± 0.286 | 86.058 ± 0.149 | +72.642** |
| GPT-5.1 | 91.118 ± 0.137 | 97.270 ± 0.161 | +6.152** | 1.362 ± 0.149 | 12.371 ± 0.137 | +11.009** |
| Claude | 94.871 ± 0.137 | 95.368 ± 0.137 | +0.497* | 0.333 ± 0.137 | 0.222 ± 0.149 | −0.111 |
| Doubao | 95.888 ± 0.161 | 97.515 ± 0.149 | +1.627* | 2.710 ± 0.149 | 3.164 ± 0.124 | +0.454* |
| DeepSeek | 95.094 ± 0.174 | 96.331 ± 0.261 | +1.237* | 0.444 ± 0.261 | 5.660 ± 0.137 | +5.216** |
Official Comment by Authors (3/4)
Why might Finding 2 vary across models?
To examine whether models differ in how readily they accept the link between evidence and an answer, we compared the Toward Bias CL gaps with the Evidence Mapping pass rates in Sec. 5.1. Among the four models other than GPT-4o, the two measures have the same ordering: Claude (31.11%, −0.111%), Doubao (43.24%, +0.454%), DeepSeek (52.27%, +5.216%), and GPT-5.1 (74.36%, +11.009%). This association is consistent with the possibility that stricter evidentiary thresholds attenuate the effect, but it does not establish the mechanism by itself.
- Does Finding 2 emerge in Claude when the prompt relaxes the evidence requirement? To test this explanation within the same model, we appended the following instruction to Claude’s final-decision prompt under Toward Bias:
It is not necessary to require a rigid, one-to-one correspondence between the evidence and the answer, evidence may be accepted as long as it logically supports the answer.
- A strict evidentiary threshold can mask the susceptibility described in Finding 2 without eliminating it. Differences in training and alignment may lead LLMs to apply different thresholds when deciding whether peer evidence warrants a change of answer. Under Claude’s original prompt, switching was rare in both evidence conditions. Yet a one-sentence change to its evidence-evaluation instruction was enough for Erroneous Evidence to elicit higher conformity than Grounded Evidence in all five runs, shifting the gap from −0.58% to +1.60% (Table R2). As discussed in Sec. 5.1, Claude rarely follows misleading peer advice, but it can also retain its initial answer when peers offer a grounded reason to revise it. In Table R1, Grounded Evidence improves Claude’s correction rate over Answer-Only by just 0.497%, the smallest gain among the five models. This motivates a targeted adjustment rather than a blanket relaxation of Claude’s evidentiary threshold. ARBITER allows reasonable inferences from the passage while still filtering unsupported peer claims, helping Claude benefit from valid peer corrections without weakening its resistance to misleading advice.
Table R2. Claude’s CL (%) under the original and relaxation prompts in the separate Toward Bias comparison. Δ is Erroneous Evidence minus Grounded Evidence, in percentage points. * indicates p < 0.05.
| Prompt Type | Grounded Evidence | Erroneous Evidence | Δ |
|---|---|---|---|
| Original Prompt | 0.81 | 0.23 | −0.58 |
| Relaxation Prompt | 1.81 | 3.41 | +1.60* |
Revised: We have added this analytical experiment and the corresponding discussion in App. B.4 (Page 23) of the revised manuscript, with the revisions highlighted in blue.
Official Comment by Authors (1/4)
Response: Thank you for this important and constructive suggestion. We have added the model inference settings to Sec. A.2 (Page 15) of the revised manuscript, including the sampling temperature of 0.7 for all five models, top-p, and the maximum output length. Each message condition used the same 940-item evaluation pool before exclusions. We conducted five independent runs, generating one final response per retained item in each run. Thus, each retained item contributed five final responses for a given target model, pressure direction, and message condition.
Revised: We have added the model inference settings to Sec. A.2 (Page 15) of the revised manuscript, with the revisions highlighted in blue.