- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper examines why LLM agents change their answers during debate. It compares three conditions: self-reflection, peer answers alone, and peer answers with reasoning. The main experiment uses five GPT-4o agents on MMLU-Pro, with matched comparisons across 5,200 agent–question pairs from questions with initial disagreement. The results show that many changes occur without peer input and that peer adoption can improve or reduce accuracy. In the main experiment, full reasoning improves individual accuracy but does not outperform self-reflection in majority-vote accuracy. Both achieve 77.35% majority-vote accuracy across all 4,900 questions, while full reasoning requires 57% more tokens.
Critical strengths
The matched experimental design provides a useful way to study answer changes. Each agent starts from the same initial answer across the three conditions. The self-reflection control checks whether an answer changes without peer input. This matters because 55.1% of stance-only flips also change under self-reflection, although not always to the same answer (§4.1). The design helps avoid treating every change after peer exposure as conformity. This is the paper’s main contribution.
The paper connects peer adoption to both accuracy and cost. It separates harmful changes, corrections, and changes that remain wrong. It also checks whether individual improvements benefit the final group answer. Full reasoning and self-reflection both reach 77.35% majority-vote accuracy, but full reasoning uses 57% more tokens (§4.4). The targeted intervention also reduces adoption without a detectable accuracy gain (§6.3). These findings show that reducing peer adoption or adding peer reasoning does not necessarily improve group accuracy in this setting.
Critical weaknesses
The reliability of the main event categories remains unclear. A single unchanged self-reflection response makes an example eligible for “strict conformity.” A different response could place it in another category. Appendix B.1 finds substantial variation across repeated initial-answer calls, but it does not directly test whether the full classification stays consistent across repeated calls with fixed inputs. Appendix A acknowledges this limitation. Since the decomposition is the main contribution, a small repeatability check would strengthen confidence in the reported category counts.
The wrong-reasoning experiment has a limited connection to agents abandoning their own correct answers. Section 5.2 gives every target a correct prior answer. However, this matches the target’s actual initial answer in only 253 of 500 cases. In 193 cases, the constructed wrong answer is the target’s original wrong answer (Appendix I). Choosing that answer may partly reflect an existing preference. The paper discloses this setup, and its controlled comparisons remain useful. Still, separate results for targets that originally answered correctly would better show whether wrong peer reasoning causes them to abandon a correct answer.
Questions for authors
If the self-reflection and peer-exposure calls are repeated with the same initial answers and peer materials, how often do the event categories change? Does the overall distribution remain similar?
Do the presentation effects in Table 7 hold for the 253 cases where the assigned correct prior matches the target’s actual initial answer? How do these effects compare with the 193 cases where the constructed wrong answer matches the target’s original answer?
3 comments
Thank you for the response and clarifications. I will increase the soundness score.
I do not think it reached the highest score (4 - clear acceptance) though. I maintain my original positive score as 3.
Thanks!
Thank you for kindly taking the time to respond to our rebuttal! We greatly appreciate your valuable comments and suggestions.
Author Response to Reviewer gAH6
We sincerely thank you for the constructive feedback.
Category shares replicated across the two runs (largest difference 1.32 pp), although individual pairs often changed category. We will therefore treat categories as single-run outcomes and base our conclusions on aggregate quantities. We reused initial answers and peer materials on 200 GPT-4o/MMLU-Pro questions (1,000 pairs). With the original wording, each pair received six replies across two runs (seeds 42 and 43, temperature 0), covering re-answering without peers, seeing peer answers alone, and seeing answers with explanations. Of the 911 pairs with valid choices in all six replies, 552 (60.59%, 95% CI [56.68, 64.36] from resampling questions) kept their category. The strict-conformity share changed by only −0.66 pp [−3.68, 2.36], but only 76 of the 184 pairs that met the strict-conformity rule in the first run met it again (41.30% [33.91, 48.71]).
Per-pair variation is expected here: when we repeated the no-peer call with the same prompt and seed for all original pairs, 18.33% of answers that had stayed unchanged in the original run changed. Aggregate quantities, by contrast, replicate. Besides the category shares above, the increase in switching to peer options when peer answers were shown was 14.06 pp in the original data and 14.65 pp in this experiment. The revision will report the full transition matrix, describe categories as single-run outcomes, and draw conclusions from shares and paired increments rather than from individual labels.
For both comparisons in Table 7, the estimated increases in wrong-option selection were similar in the 253 cases GPT-4o originally answered correctly and the 193 cases where it had originally chosen the wrong option later shown. Each prompt supplied the correct option as the target's previous answer. The table uses the model's actual initial answers to define the groups.
Both intervals for the between-group differences include zero. On the 164 cases where GPT-4o and DeepSeek had both initially answered correctly, the first difference was +14.02 pp for GPT-4o and −9.76 pp for DeepSeek, a paired difference between models of +23.78 pp [11.98, 36.42]. Both models received the same prompts, including the supplied correct previous answer and the same peer material. The results describe these selected cases and presentations, whose lengths differed.