acceptodds
Under review as a conference paper at ICLR 2027

Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

Abstract

Multi-agent debate (MAD) can improve individual answers, but convergence does not by itself show that peer information helped. We separate answer changes into reconsideration instability under information isolation, strict conformity, and reasoning-only peer adoption. In the primary MMLU-Pro setting, 37% of agent-question pairs change under self-reflection alone. Among strict-conformity events, 30.7% are correct-to-wrong, 36.4% are wrong-to-correct, and 32.9% remain wrong. These directions vary with correct-answer availability and peer support. Individual accuracy rises from 36.9% at Round 0 to 45.9% under full reasoning, while majority-vote accuracy is unchanged between self-reflection and full reasoning (77.35%). A controlled, susceptibility-selected information-gradient experiment with an assigned correct prior finds that invalid and question-specific wrong reasoning increase wrong-answer adoption, with the invalid-reasoning contrast reversing across target models. A diagnostic model predicts harmful conformity only in an oracle-eligible setting (AUC = 0.784). An answer-agnostic model predicts peer adoption (AUC = 0.804), but the targeted intervention did not produce a detectable accuracy gain.

Primary area:
generative models
Keywords:
multi-agent debate, large language models, conformity, peer influence, LLM reliability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gAH6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper examines why LLM agents change their answers during debate. It compares three conditions: self-reflection, peer answers alone, and peer answers with reasoning. The main experiment uses five GPT-4o agents on MMLU-Pro, with matched comparisons across 5,200 agent–question pairs from questions with initial disagreement. The results show that many changes occur without peer input and that peer adoption can improve or reduce accuracy. In the main experiment, full reasoning improves individual accuracy but does not outperform self-reflection in majority-vote accuracy. Both achieve 77.35% majority-vote accuracy across all 4,900 questions, while full reasoning requires 57% more tokens.

Critical strengths

The matched experimental design provides a useful way to study answer changes. Each agent starts from the same initial answer across the three conditions. The self-reflection control checks whether an answer changes without peer input. This matters because 55.1% of stance-only flips also change under self-reflection, although not always to the same answer (§4.1). The design helps avoid treating every change after peer exposure as conformity. This is the paper’s main contribution.

The paper connects peer adoption to both accuracy and cost. It separates harmful changes, corrections, and changes that remain wrong. It also checks whether individual improvements benefit the final group answer. Full reasoning and self-reflection both reach 77.35% majority-vote accuracy, but full reasoning uses 57% more tokens (§4.4). The targeted intervention also reduces adoption without a detectable accuracy gain (§6.3). These findings show that reducing peer adoption or adding peer reasoning does not necessarily improve group accuracy in this setting.

Critical weaknesses

The reliability of the main event categories remains unclear. A single unchanged self-reflection response makes an example eligible for “strict conformity.” A different response could place it in another category. Appendix B.1 finds substantial variation across repeated initial-answer calls, but it does not directly test whether the full classification stays consistent across repeated calls with fixed inputs. Appendix A acknowledges this limitation. Since the decomposition is the main contribution, a small repeatability check would strengthen confidence in the reported category counts.

The wrong-reasoning experiment has a limited connection to agents abandoning their own correct answers. Section 5.2 gives every target a correct prior answer. However, this matches the target’s actual initial answer in only 253 of 500 cases. In 193 cases, the constructed wrong answer is the target’s original wrong answer (Appendix I). Choosing that answer may partly reflect an existing preference. The paper discloses this setup, and its controlled comparisons remain useful. Still, separate results for targets that originally answered correctly would better show whether wrong peer reasoning causes them to abandon a correct answer.

Questions for authors

If the self-reflection and peer-exposure calls are repeated with the same initial answers and peer materials, how often do the event categories change? Does the overall distribution remain similar?

Do the presentation effects in Table 7 hold for the 253 cases where the assigned correct prior matches the target’s actual initial answer? How do these effects compare with the 193 cases where the constructed wrong answer matches the target’s original answer?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer gAH6

We sincerely thank you for the constructive feedback.

W1/Q1. How reproducible are the event categories?

Category shares replicated across the two runs (largest difference 1.32 pp), although individual pairs often changed category. We will therefore treat categories as single-run outcomes and base our conclusions on aggregate quantities. We reused initial answers and peer materials on 200 GPT-4o/MMLU-Pro questions (1,000 pairs). With the original wording, each pair received six replies across two runs (seeds 42 and 43, temperature 0), covering re-answering without peers, seeing peer answers alone, and seeing answers with explanations. Of the 911 pairs with valid choices in all six replies, 552 (60.59%, 95% CI [56.68, 64.36] from resampling questions) kept their category. The strict-conformity share changed by only −0.66 pp [−3.68, 2.36], but only 76 of the 184 pairs that met the strict-conformity rule in the first run met it again (41.30% [33.91, 48.71]).

SeedChanged without peersStrict conformitySwitched only with peer explanationsOtherKept answer throughout
4232.93%20.20%11.20%0.66%35.02%
4334.14%19.54%10.87%1.76%33.70%

Per-pair variation is expected here: when we repeated the no-peer call with the same prompt and seed for all original pairs, 18.33% of answers that had stayed unchanged in the original run changed. Aggregate quantities, by contrast, replicate. Besides the category shares above, the increase in switching to peer options when peer answers were shown was 14.06 pp in the original data and 14.65 pp in this experiment. The revision will report the full transition matrix, describe categories as single-run outcomes, and draw conclusions from shares and paired increments rather than from individual labels.

W2/Q2. How do the Table 7 results vary with GPT-4o's initial answer?

For both comparisons in Table 7, the estimated increases in wrong-option selection were similar in the 253 cases GPT-4o originally answered correctly and the 193 cases where it had originally chosen the wrong option later shown. Each prompt supplied the correct option as the target's previous answer. The table uses the model's actual initial answers to define the groups.

Comparison (pp)Initially correct (253)Initially chose the wrong option later shown (193)Initially correct minus initially chose that option
Generic explanation minus assertion+13.44 [5.47, 21.86]+13.99 [8.04, 20.53]−0.55 [−9.07, 7.95]
Question-specific wrong explanation minus generic explanation+7.11 [−0.38, 14.68]+8.29 [1.59, 15.43]−1.18 [−8.23, 5.42]

Both intervals for the between-group differences include zero. On the 164 cases where GPT-4o and DeepSeek had both initially answered correctly, the first difference was +14.02 pp for GPT-4o and −9.76 pp for DeepSeek, a paired difference between models of +23.78 pp [11.98, 36.42]. Both models received the same prompts, including the supplied correct previous answer and the same peer material. The results describe these selected cases and presentations, whose lengths differed.

Thank you for the response and clarifications. I will increase the soundness score.

I do not think it reached the highest score (4 - clear acceptance) though. I maintain my original positive score as 3.

Thanks!

Thank you for kindly taking the time to respond to our rebuttal! We greatly appreciate your valuable comments and suggestions.

Reviewer_c5rBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies why agents change answers in multi-agent debate. 5 GPT-4o agents answer MMLU-Pro questions and on questions where they disagree, each agent is run again under 3 prompts... self-reflection only, peer answers only, and peer answers with reasoning. Using these runs paper split answer changes into change without peers, "strict conformity" and reasoning-only adoption, and they report that 37% of pairs change even with no peers. Paper also find that debate raises individual accuracy but majority vote is same as self-reflection at higher token cost. They also add a controlled test with different kinds of wrong peer reasoning and a risk model + prompt intervention, with some replications on other models and datasets.

Critical strengths

1- Good question to ask... answer change after seeing peers is not proof of peer influence and lot of prior work treat it like that. Running self-reflection, stance-only and full reasoning on the same Round 0 answer is a very good setup.

2- Authors are honest about negative results (rare to see). Full reasoning debate gives no majority vote improvement over self-reflection and using 57% more tokens, and the deployable intervention does not help accuracy. This is useful for people building debate systems.

Critical weaknesses

1- Many findings are already known in literature.. so its very hard for me to "only" find the new knowledge after reading the paper.. if there's any other proposal from the authors, it should be clearly mentioned/highlighted. And already known findings should be clearly declared (something like "<finding> as supported/discussed in X paper" -- that will be more honest for reader to see what are new findings from this paper). For example: models change answers when asked to reconsider (Laban et al. 2023 FlipFlop, Xie et al. ACL 2024), majority vote explains most debate gain (Choi et al. 2025a), and wrong or even empty reasoning persuades more than a bare answer (Kim & Khashabi 2025, Zhao et al. 2025). The Cost of Consensus (arXiv 2605.00914) already uses a no-peer self-correction control and an unrelated reasoning control in debate. None of these are discussed/mentioned in the paper. Please clarify.

2- Strict conformity uses only one self-reflection call per pair (the authors say this in limitations). Outputs are random even at temperature 0... the authors check this and it shows 3 identical calls disagree on 22.4% of pairs. So a pair can look "stable" in one call and change in the next, and on a disagreement question the new answer is very likely one of the peers answers. This gets counted as conformity. What the design actually measures is the diff b/w the 2 conditions (about 10%), not the 29.3% in the abstract. So, this is misleading..

3- All agents are the same model with same prompt, and only the 1,048 disagreement questions are used (Round 0 accuracy 36.9 vs 75.1 overall). So the "peers" are just other samples of the target model and the sample is picked where the model is most unstable. Appendix J shows GPT-4o drops a correct answer only 5% of time under 3 vs 1 wrong pressure on a wider set, but around 42% on the selected set. These rates dont carry over to normal debate..

Questions for authors
  1. Can you run a second self-reflection call on all pairs and report how often a "stable" pair changes to a peer answer with no peers shown? How much of the 29.3% remains after removing this?
  2. Of the self-reflection changes, how many are on an answer some peer already had?
  3. Can you split the RQ2 results into the 253 matched cases, the 193 cases where the wrong answer is the target's own answer, and the other 54?
  4. Why do the two 500-case RQ2 runs give +13.8 and +2.0? Which one was planned first?
  5. Can you report majority vote only on the 1,040 disagreement questions, add a self-consistency baseline with same token cost, and try 2-3 rounds?
  6. Lastly, apart from above experiments, please clarify all the novelties/new findings from these papers - a clear distinguish b/w what was already known and what your paper brings.
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer c5rB (1/3)

We are grateful for the reviewer's valuable feedback.

W1/Q6. What does this study add to prior work?

We measure how often a model independently switches to an answer another agent gave, and how much that rate changes when the same answers are shown. We will make this contribution explicit in Section 2:

  • The Cost of Consensus compared three-round debate among ten 7–8B agents with isolated self-correction and an unrelated-rationale control, and found that debate used 2.1–3.4 times as many tokens at comparable or lower accuracy. Our one-round GPT-4o result agrees: exchanging peer explanations used 57% more tokens than answering again without peers, with the same vote accuracy (Table 5). What we add is a matched measurement: from the same initial answers, we count switches to peers' initial answers with no peer information, peer answers alone, or peer answers and explanations. This shows how often a no-peer change already lands on a peer's answer (Q2 below), which their design does not report.
  • Laban et al. and Xie et al. showed that follow-up questions can induce changes. We quantify how often those changes already match another agent's initial answer (Q2 below).
  • As noted in Section 2, Choi et al. found that voting on initial answers accounted for most debate gains in their tests. On our 1,040 disagreement questions, exchanging explanations improved individual accuracy over answering again without peers, while 107 group votes became correct and 107 became incorrect.
  • Kim and Khashabi showed that detailed user rebuttals can persuade models even when their conclusions are wrong. Zhao et al. found that padding and repetition can also persuade. We hold the wrong option fixed and compare a generic explanation with a bare assertion that the option is correct. This difference was positive for GPT-4o and negative for DeepSeek-V4-Flash under the presentations tested, which differed in length (Section 5.2, Table 7).

W2/Q1. What remains after a second no-peer call?

A second no-peer call lowers the strict-conformity rate from 31.28% to 26.91%, but showing peer answers still raises switching to peer answers by about 14–17 pp after subtracting the no-peer baseline. We repeated the no-peer prompt for all 5,200 original pairs and analyzed the 2,449 that had valid initial choices matching the original prompts and kept their answer on the first no-peer call. Of these, 31.28% had switched to a peer's answer when shown peer answers alone, compared with 14.54% on the second no-peer call (+16.74 pp [14.32, 19.02]). Among pairs that kept their answer on both no-peer calls, the strict-conformity rate was 26.91%, and 69% (529/766) of the original strict-conformity cases still qualified. The new no-peer calls changed answers at nearly the same rate as the original ones (30.32% vs. 30.04%; +0.28 pp [−1.39, 2.01]), so comparing new calls with the original responses is reasonable.

To avoid comparing original and new responses, we also ran all conditions at the same time on 200 questions (1,000 pairs, reusing the original initial answers and peer materials; also reported for Fh8E W1 and gAH6 Q1). Among pairs that kept their answer on a first no-peer call, showing peer answers raised switching to those answers by 14.09 pp [10.35, 18.32] relative to a second no-peer call. Unlike the 29.30% strict-conformity rate (Section 4.1, no baseline subtracted) and the 9.87 pp in Table 1 (any answer change), this paired difference subtracts the no-peer baseline and counts only switches to peer answers. We agree that 29.30% should not be read as the peer effect. The abstract will instead read: "In the primary MMLU-Pro setting, 37% of agent–question pairs change their answer when asked to reconsider without peers, and 73% of these changes land on an answer another agent gave. Showing peer answers raises switching to a peer's answer from 27.1% to 41.2% (+14.1 pp, 95% CI [12.5, 15.7])."

AuthorsOpenReview ↗

Author Response to Reviewer c5rB (2/3)

W3. Which questions do the rates describe?

We agree that the main answer-switching rates apply to questions selected for initial disagreement among five calls to the same model. When all five answers agree, peers offer no alternative to switch to. Of 1,048 questions whose initial answers differed, 1,040 had follow-up data. Initial individual accuracy was 36.9% on these questions and 75.1% across all 4,900. Table 5 covers the full set. As Q2 shows, the model often switches to an answer another instance gave, even without seeing it.

We believe the 42% refers to Table 23. Among 238 initially correct targets that kept their answer without peers and had exactly one agreeing peer, 99 switched to a dissenting peer's answer when shown peer answers. When all three dissenters gave the same answer, 70/148 (47.3%) switched to it. Appendix J starts with cases where both GPT-4o and Qwen3-Plus answered correctly. Facing three peers backing one wrong answer and one backing the correct answer, GPT-4o abandoned the correct answer in 5.05% of the 14,055 cases and chose the wrong majority in 4.05%.

Under the constructed 3:1 majority in Appendix J, GPT-4o adopted the wrong answer in 43.14% of cases (393/911) on questions where its initial answers disagreed, close to the 47.3% (70/148) observed with a natural 3:1 split in Table 23. Another GPT-4o instance had independently chosen the displayed wrong option in 898 of those 911 cases. On questions where all five GPT-4o initial answers were correct, adoption was 1.34% overall. Within this group, GPT-4o adopted a wrong option more often if Qwen had previously chosen it (7.82% vs. 1.18% when neither model had chosen it). This pattern is consistent with more plausible alternatives being adopted more often. These rates are conditional on initial disagreement and are not per-question rates over a full benchmark. Full-benchmark effects are in Table 5. We will state this in the limitations.

Q2. How often does a switch without peers match a peer's answer?

Nearly three quarters of the recorded changes without peers matched a peer's initial answer (1,410/1,923, 73.32% [70.65, 75.97]). This shows why we measure the same kind of switch with and without peer answers. Showing peer answers increased switching to those answers by 14.06 pp [12.46, 15.67] across all 5,200 original pairs (27.12% without peers and 41.17% with peer answers alone). The new 200-question experiment with the original wording gave a similar increase of 14.65 pp [11.70, 17.65].

The pattern also held on 3,503 pairs with valid choices throughout and initial answers matching those used in the prompts. In this subset, 79.61% (847/1,064) of no-peer changes matched a peer's answer, and the paired difference was +16.61 pp [14.68, 18.55].

Q3. What do the 253, 193, and 54 cases show?

Even when GPT-4o had originally answered correctly, a generic explanation elicited more selections of the wrong option than a bare assertion that it was correct. The 500 cases came from 100 questions selected because the original experiment had observed switches from correct to wrong after showing peer explanations. For each case, the new prompt supplied the correct option as the target's previous answer. The groups below use the answer GPT-4o actually gave in the original experiment.

Actual initial answer (n)Wrong option aloneAsserts it is correctGeneric explanationQuestion-specific wrong explanation
Correct (253)66.4%65.2%78.7%85.8%
Same wrong option later shown by the peer (193)61.1%64.8%78.8%87.0%
A different wrong option (54)66.7%63.0%74.1%92.6%

After excluding cases where the responding model had originally selected the wrong option shown, the generic-explanation versus assertion difference remained negative for DeepSeek (−10.90 pp [−16.95, −5.58], 413 pairs) and was uncertain for Qwen (−0.93 pp [−5.88, 4.29], 428 pairs). These models received the same prompts and peer material as GPT-4o. The rates describe the selected questions and the tested presentations, whose lengths differed.

AuthorsOpenReview ↗

Author Response to Reviewer c5rB (3/3)

Q4. Why did the two 500-case studies differ, and which came first?

The Appendix E study planned first. The increase from adding a generic explanation was +2.0 pp [−2.4, 6.6] in that study and +13.8 pp [7.4, 20.4] in the later four-condition study. Selecting the wrong option when only the option was shown was similar (63.6% and 64.4%). The larger difference appeared when a generic explanation was added (65.6% and 78.2%). In both studies, a question-specific wrong explanation raised wrong-option selection over a generic one (+12.2 pp in Appendix E, +8.8 pp [2.2, 15.6] in the later study).

The later study revisited the same 500 cases with one peer instead of several, used "answer again", and added a condition that explicitly asserted the option was correct without giving a reason. Several parts of the prompt changed together, so we will describe the increase as sensitive to how the option and explanation were presented.

Q5a. Voting on the 1,040 disagreement questions

Showing peer answers and explanations yielded no detectable vote-accuracy gain over answering again without peers. Both reached 47.21% (491/1,040), with a paired difference of 0.00 pp [−2.79, 2.79]. Voting over the initial answers reached 41.92%, and voting after seeing peer answers alone reached 46.44%.

Q5b. Self-consistency at a similar GPT-4o token budget

At a similar GPT-4o token budget, self-consistency and one round of debate each got 86 of the 198 MMLU-Pro disagreement questions with complete results right (43.43%), so exchanging explanations did no better than answering independently. For self-consistency, we asked GPT-4o to answer each question independently 23 times and voted over those answers. For debate, five agents first answered independently, then each answered again after seeing the other four agents' answers and explanations. We voted over those five revised answers. At temperature 0.5, self-consistency averaged 19,811 GPT-4o input and output tokens per question, and debate averaged 20,295 across both stages. We used temperature 0.5 because self-consistency needs sampling diversity, and 23 samples because that number matches the debate token budget.

Q5c. Does a second exchange improve the group vote?

No condition improved its vote accuracy in a second round. In the rebuttal experiment, we extended each of the three GPT-4o conditions from its own original Round 1 responses. Of the 1,040 MMLU-Pro disagreement questions, 1,024 had all five Round 2 replies in every condition (16 were excluded because the provider's content filter blocked some replies). The table compares Round 1 and Round 2 vote accuracy on those same 1,024 questions in each condition.

Answer information shown in both roundsRound 1Round 2Round 2 minus Round 1 (pp)
Their own previous answer only47.46%45.12%−2.34 [−4.69, −0.20]
Their own and their peers' previous answers46.88%47.75%+0.88 [−1.07, 2.73]
Their own previous answer and their peers' answers and explanations47.66%47.27%−0.39 [−1.95, 1.17]

From Round 1 to Round 2, vote accuracy changed by 3.22 percentage points more with peer answers alone than without peers [0.20, 6.15]. With peer answers and explanations, the corresponding difference was +1.95 pp [−0.68, 4.39]. We will therefore restrict the accuracy–cost comparison in Section 4.4 to one exchange; rounds beyond the second were not tested.

Reviewer_Fh8EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a lot of what looks like peer conformity in multi-agent debate would actually happen anyway, even without any peer input. Using a matched setup where the same agent is tested under self-reflection alone, peer stances only, and full peer reasoning, the authors split answer changes into reconsideration instability, strict conformity, and reasoning-driven peer adoption, and show these sources push in different directions depending on whether a correct peer is available. They also run a controlled experiment on how richer peer material affects wrong-answer adoption, build risk models for harmful conformity, and test an intervention that cuts down on conformity without producing any measurable gain in accuracy.

Critical strengths
  1. This paper separates "the agent changed its mind" from "the agent was persuaded by a peer". It targets a real blind spot in how conformity gets measured in this literature. Showing that self-reflection alone changes a large share of answers puts a real number on how inflated past conformity estimates might be.
  2. The statistical practice is careful throughout. Confidence intervals are reported for nearly every comparison, one key experiment is pre-registered before results are seen, and the authors run a wide range of sensitivity checks rather than leaning on one setup.
  3. The experimental design doesn't just count how often answers change, it tracks whether a change moves a correct answer to wrong or a wrong one to correct, and pairs this with an experiment that varies how many correct peers are present.
Critical weaknesses
  1. The self-reflection condition is supposed to be a clean baseline that isolates reasoning instability from any social pressure. But a separate check shows that just changing the wording of the reconsideration prompt, from a neutral "answer again" to "please reconsider," roughly doubles how often the answer flips. Since the main self-reflection condition uses that same "reconsider" wording, its headline flip rate is probably a mix of genuine reasoning instability, sycophantic deference to an implied suggestion that the answer might be wrong, and simple compliance with the word "reconsider" itself. The comparisons between conditions still hold up, since all three share the same wording, but the baseline isn't the clean, mechanism-isolated control the paper's argument needs it to be, and the appendix evidence that could help untangle this is never used in the main analysis.
  2. Prediction and mitigation are pitched as one of the paper's three main contributions, but the only version of the intervention that could actually be deployed fails to produce a real accuracy gain, it just suppresses both harmful and helpful peer adoption together. The version that does show a real improvement needs information that isn't available at deployment time. So the practical takeaway ends up being "we can rank conformity risk but not safely reduce it," which is a fair finding but falls short of what the mitigation framing promises.
  3. Nearly all of the paper's central numbers come from one model on one benchmark, and the cross-model checks don't really back this up, they mostly reveal how shaky it is. Self-reflection flip rates swing wildly across models on the same benchmark, and the number of disagreement cases each model produces varies just as much, which is why only the primary model-benchmark pairing gives a confidence interval tight enough to mean anything. The directional claims that anchor the paper, like beneficial peer influence being at least as common as harmful influence, are really only established for that one pairing. The other combinations read more like inconclusive spot checks than independent confirmations.
Questions for authors

Please answer the questions in the weaknesses.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer Fh8E

We sincerely thank you for the detailed review.

W1. Does "reconsider" inflate the baseline or the peer comparison?

"Answer again" reduced switching without peers, and the increase in switching when peer answers were shown was slightly larger with the neutral wording. In Section 4.1 and Appendix B.1, on the same 594 GPQA-Diamond initial answers, the no-peer switch rate was 8.8% with "answer again" and 16.7% with "reconsider" (difference of 7.91 percentage points, 95% CI [4.38, 11.62] from resampling questions).

We extended this test to 200 GPT-4o/MMLU-Pro disagreement questions (1,000 agent–question pairs). Each question had five valid initial choices that matched the answers used in the original prompts. We reused the same initial answers and peer materials, changed only the final instruction, and averaged the two runs per pair (temperature 0, seeds 42 and 43).

Outcome"Reconsider""Answer again"Change with "answer again" (pp)
Switch to another valid option without seeing peers32.75%24.80%−7.95 [−10.35, −5.40]
Increase in switching to a peer's initial answer when peer answers are shown+14.65 pp+17.90 pp+3.25 [0.35, 6.15]

Even with the neutral wording, about a quarter of pairs (24.80%) still switched without seeing peers, so the no-peer baseline is not an artifact of the word "reconsider". The GPQA check is already in Section 4.1, and we will add this MMLU-Pro check next to it. The self-reflection condition measures how often the model changes its answer when asked again without seeing peers, including any response to the prompt wording.

W2. What does the intervention achieve?

The risk score helps rank cases likely to switch to a peer's answer. Targeting those cases yielded no detectable accuracy gain. Compared with the original responses that saw peer answers alone, accuracy changed by −0.77 pp [−3.46, 2.02] in the 20% of cases ranked most likely to switch to a peer's answer and by −0.15 pp [−0.73, 0.40] across all 5,200 pairs (submission Table 8). In that 20%, switches from right to wrong and from wrong to right both fell in the point estimates (−1.44 pp [−2.79, −0.10] and −2.12 pp [−4.33, 0.00]).

The score used for targeting predicts any switch to a peer's answer (ROC-AUC 0.804). A separate model using the same features available before the first revision ranks harmful switches with ROC-AUC 0.661 [0.629, 0.692] (Section 6.1). The +14.63 pp accuracy gain [8.2, 21.0] in Figure 3 applies to the high-risk group of a diagnostic experiment that selected targets using the known correct answer and their unchanged no-peer response (Section 6.2). We will frame the contribution around risk ranking and the observed limits of the intervention.

W3. How far does the result generalize across models?

Switches without peers occur across the tested settings. Whether switches to peer answers more often correct errors than introduce them remains uncertain across settings. On MMLU-Pro, recorded changes without peers ranged from 11.91% for DeepSeek-V4-Flash to 74.07% for Qwen3-Plus. The models also contributed different numbers of disagreement questions, including only 141 for DeepSeek-V4-Flash (Appendix B.2).

In the primary setting, corrections were 54.2% [50.5, 57.9] of strict-conformity events that changed correctness (349 of 644). When restricted to cases with valid options in all initial and follow-up responses, the share was 52.80% [48.74, 56.93], an interval that includes 50%. The intervals for the other model–benchmark settings include 50% as well (Tables 10–11). After standardizing over peer support, harmful switches were less frequent than beneficial ones in the primary setting (−8.75 pp [−13.44, −4.27], Table 2). We will report the direction descriptively and give the sample size and no-peer baseline for each setting.

Thanks for the detailed response and the additional experiment. My remaining concerns are W1 and W3. On W1, it would help to see the full decomposition under the neutral wording, not just the flip rate. On W3, it seems, at the event share level, the directional claim no longer holds significantly under the stricter filter you disclosed. I hope these can be addressed in the revision. I'll keep my current score.

AuthorsOpenReview ↗

Author Response to Reviewer Fh8E

Thank you for the follow-up questions.

W1. What does the full decomposition of answer changes look like under the neutral wording?

"Answer again" lowered the no-peer change rate, but among observations that kept their answer without peers, the strict-conformity rate was unchanged (30.66% vs. 30.63%; difference −0.03 pp [−2.92, +2.95]). The wording mainly shifts the no-peer baseline.

We applied the paper's full decomposition to the same 200 GPT-4o/MMLU-Pro disagreement questions used in our previous response, with the category names from our table for Reviewer gAH6 Q1. For each of the 1000 agent–question pairs, we held the initial answer and peer materials fixed and evaluated all three conditions under both wordings and two seeds, giving 2000 observations per wording. An observation is classifiable when all three conditions return a valid choice; unclassifiable observations remain in the denominator.

CategoryReconsider, n (%)Answer again, n (%)Difference, pp [95% CI]
Changed without peers640 (32.00%)479 (23.95%)−8.05 [−10.45, −5.75]
Strict conformity386 (19.30%)438 (21.90%)+2.60 [+0.35, +4.75]
Switched only with peer explanations206 (10.30%)215 (10.75%)+0.45 [−1.20, +2.15]
Other24 (1.20%)36 (1.80%)+0.60 [−0.20, +1.45]
Kept answer throughout643 (32.15%)741 (37.05%)+4.90 [+2.80, +7.05]
Unclassifiable101 (5.05%)91 (4.55%)−0.50 [−1.85, +0.80]

Differences are "answer again" minus "reconsider", with paired confidence intervals from resampling questions. The 32.75% and 24.80% in our previous response used the no-peer response alone rather than the full decomposition.

Under "answer again", fewer observations changed without peers and more kept their answer in all three conditions. More observations therefore met the no-peer stability requirement (1430 vs. 1259), which raised the share of strict conformity in the table. Conditional on that requirement, the strict-conformity rate did not change: 386 of 1259 under "reconsider" and 438 of 1430 under "answer again". The share that switched only with peer explanations also changed little.

Counts by correctness transition for the three main categories:

CategoryWordingCorrect -> wrongWrong -> correctWrong -> different wrong
Changed without peersReconsider196223221
Answer again150179150
Strict conformityReconsider135127124
Answer again139159140
Switched only with peer explanationsReconsider746468
Answer again656981

We will add both tables to the revision alongside the wording comparison in Section 4.1.

W3. Does the event share still support a directional claim under the stricter filter?

We agree that the direction remains uncertain under the stricter filter, including in the primary setting. In the original GPT-4o/MMLU-Pro experiment, when we require valid options in all five initial answers and the target's three follow-up answers, 283 strict-conformity events corrected an error and 253 introduced one. Corrections accounted for 52.80% of the events that changed correctness (95% CI [48.74%, 56.93%]). Additionally requiring the five initial choices to match those used in the original prompts gives an interval that also includes 50% (280 of 524, 53.44%, 95% CI [49.26%, 57.60%]). The new runs do not resolve it. The share was 53.4% (159 of 298) under "answer again" and 48.5% (127 of 262) under "reconsider", on either side of 50%.

In the revision, we will:

  • report these event counts and intervals, under both filters, in Sections 4.2–4.3;
  • remove the statement that beneficial transitions are at least as frequent as harmful ones from the discussion and conclusion;
  • describe the direction of strict-conformity events as uncertain in every tested setting, and present the transition counts as descriptive.

The main measurements hold under both the neutral wording and the stricter filter: 23.95% of observations still changed without peers under "answer again", and showing peer answers raised switching to peer answers by 17.90 pp under "answer again" and by 16.61 pp on the filtered original data.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Remarks

Dear Area Chairs and Reviewers,

Contribution

The key contribution of this work is a matched measurement of peer influence in multi-agent debate: from the same initial answers, agents switch to a peer's answer in 27% of cases without seeing peers and in 41% with peer answers shown, and the additional experiments confirm this 14 pp increase. Full-reasoning debate does not improve the majority vote over self-reflection.

Revision Summary

We have responded to all review comments and revised the paper accordingly. Please refer to the updated PDF for the revised sections. Below we summarize the revisions.

Fh8E

  • W1 We compared "reconsider" with a neutral "answer again" instruction on MMLU-Pro and report the full decomposition under both wordings (Section 4.1, Tables 4 and 5).
  • W2 Section 6 now frames the contribution as risk ranking and reports that targeting likely adopters alone does not raise accuracy.
  • W3 We report the direction of strict-conformity events with counts and intervals under both answer filters, and the sample size and no-peer baseline for each setting (Section 4.2 and Appendix E).

c5rB

  • W1, Q6 Section 2 states the relation to each cited study and what our matched measurement adds.
  • W2, Q1, Q2 The paired increase over the no-peer baseline is now the main number in the abstract, Table 2, and Figure 1, and it holds under a second no-peer call and concurrent repeated runs (Table 3).
  • W3 We report rates over all pairs next to the disagreement-question rates (Table 2) and report how adoption of a wrong majority varies with the model's initial answers to the question (Appendix M).
  • Q3, Q4 The increase from a question-specific wrong explanation holds for all three models and both protocols, and the generic-text contrast is reported as descriptive (Section 5, Appendices K and L).
  • Q5 We added vote accuracy on disagreement questions, self-consistency at a similar token budget, and a second round (Section 4.4 and Appendix D).

gAH6

  • W1, Q1 We report the category transition matrix across repeated runs and base conclusions on category shares and paired increases (Appendix C).
  • W2, Q2 The increase from generic text over an assertion remains for targets that originally answered correctly, at a similar size as in the other group (Section 5 and Appendix L).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.