acceptodds
Under review as a conference paper at ICLR 2027

When Order Matters: First-Speaker Bias and Mitigation through Personality in Sequential Multi-Agent Debate

Abstract

Multi-agent debate (MAD) is often used to improve large language model (LLM) reasoning, but sequential debate is rarely a neutral aggregator of agents' opinions. We show that sequential MAD suffers from a pronounced first-speaker bias: agents disproportionately shape the final answer when they speak first. As a result, placing a stronger model after weaker ones can substantially offset its reasoning advantage. We then focus on the disadvantaged strong-agent-last setting and ask whether personality prompting can mitigate this imbalance. Drawing on the Big Five model, we study agreeableness and extraversion as behavioral interventions applied to either the strong or weak side. We find that their effects are trait-specific. Influence consistently shifts in the direction of lower agreeableness, and assigning low agreeableness to the stronger agent helps restore its lost influence and improves final accuracy. Extraversion, by contrast, produces less systematic changes in influence and accuracy, with its clearest effect appearing in agents' verbosity. These findings show that effective MAD design depends not only on model capability, but also on how speaking order and induced interaction behavior shape the debate process.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
multi-agent debate, large language models, first-speaker bias, personality prompting, Big Five, LLM agents, speaking order

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qTFaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper examines how speaking order affects sequential multi-agent debate. Each team contains one stronger model and two instances of a weaker model. Across four debate rounds, the authors compare placing the stronger agent first or last. Experiments cover ten model combinations and four benchmarks. Placing the stronger agent first increases average accuracy by 2.34 percentage points. The paper then tests personality prompts in the stronger-agent-last setting. A low-agreeableness prompt for the stronger agent improves accuracy by 1.64 points over the unprompted baseline, while extraversion produces less consistent effects.

Critical strengths

The speaking-order comparison provides useful evidence about the debate protocol. The authors compare Strong-Weak-Weak with Weak-Weak-Strong while keeping the models and questions fixed. They also preserve the order of the two weak agents. Across ten model combinations and four benchmarks, putting the strong agent first improves average accuracy by 2.34 percentage points (Table 6). This shows that speaking order can affect performance even when the team members stay the same.

The paper shows that greater strong-agent influence does not always improve accuracy. Giving the strong agent a low-agreeableness prompt improves both influence and accuracy. However, also making the weak agents more agreeable increases strong-agent influence further but lowers accuracy relative to the strong-agent-only intervention (Appendix D). This is a useful finding. It shows why encouraging all agents to follow the stronger model is not necessarily the best strategy.

Critical weaknesses

The influence measure can compare different questions across conditions. Section 4.1 includes a question only when the standalone answers disagree and the final answer matches one of them. This excludes debates that end with a third answer. Since the final answer changes across conditions, the included questions can change too. Personality prompts also change the standalone answers used as references. The paper reports similar eligible proportions, but this does not show that the same questions qualify. A comparison using fixed questions and reference answers would make the influence shifts easier to interpret. This concern does not invalidate the accuracy results on all questions.

The personality experiments do not directly show that the prompt reduces the speaking-order gap. All personality interventions use Weak-Weak-Strong (§3.4). The low-agreeableness prompt improves this setting, but it might also improve Strong-Weak-Weak. Without testing both orders, it is unclear whether the prompt specifically reduces the disadvantage of speaking last or provides a more general benefit. The standalone checks in Appendix A do not answer this question. A comparison of both orders with and without the prompt would support the mitigation claim more directly. Otherwise, the conclusion should focus on improving the tested strong-agent-last setting.

Questions for authors

Do the influence differences remain when all conditions use the same disagreement questions and fixed standalone reference answers? How often does each condition produce a third answer that the current metric excludes?

Does the strong agent’s low-agreeableness prompt reduce the accuracy gap between Strong-Weak-Weak and Weak-Weak-Strong, or does it improve both orders similarly?

3 comments

We thank the reviewer for recognising the value of our controlled comparison and for the constructive suggestions. We have conducted additional analyses using fixed-reference influence measures and complete outcome distributions, and evaluated low agreeableness under both speaking orders. We address each concern below.

Q1: Fixed-reference influence and third-answer outcomes

We recalculated strong-agent influence using fixed no-personality standalone reference answers across all personality conditions, holding the initial disagreement questions constant within each benchmark–model combination. The directions of the original influence shifts are preserved. In particular, under W-W-S, n_n_al increases strong-agent influence by 9.18 pp, compared with 9.55 pp under the original calculation (Appendix M.1).

However, fixed references alone do not eliminate condition-dependent exclusions when final answers differ from both references. We therefore additionally analysed the complete set of 7,980 question–model-pair observations across all 40 benchmark–model combinations, using the same denominator for every condition. This includes questions where the standalone references agree or disagree. Final outcomes are classified into five mutually exclusive categories: strong-only, weak-only, both-match, third-answer, or no-final-majority.

Under W-W-S, n_n_al increases strong-only outcomes from 26.47% to 30.43% (+3.96 pp) and decreases weak-only outcomes from 18.46% to 13.97% (−4.49 pp). Third-answer outcomes increase modestly from 11.74% to 12.79% (+1.05 pp), with rates ranging from 10.53% to 14.90% across all reported conditions.

The agreeableness conditions preserve the directional pattern of the original influence analysis: the strong-only share increases and the weak-only share decreases when the weak agents receive high agreeableness or the strong agent receives low agreeableness. Under the opposite assignments, the strong-only share decreases, while the weak-only share increases except under al_al_n, where it decreases slightly, from 18.46% to 18.22%. Extraversion does not exhibit a similarly clear trait-dependent pattern, consistent with the less systematic effects observed in our main experiments.

The full-denominator analysis is consistent with the direction of the original influence findings, while measuring outcome shares rather than conditional influence. Appendix M.1 reports the original and fixed-reference influence calculations alongside the complete per-condition outcome distributions, including third-answer and no-final-answer rates.

Q2: Low agreeableness under both speaking orders

We additionally evaluated low agreeableness for the strong agent under S-W-W across all ten model combinations and four benchmarks, allowing a direct comparison with the existing W-W-S results (Appendix L.2).

The results show distinct accuracy effects across the two orders:

W-W-S: Accuracy increases from 66.43% to 68.08% (+1.64 pp), while strong-agent influence increases from 58.83% to 68.38% (+9.55 pp).

S-W-W: Accuracy decreases from 68.78% to 68.08% (−0.70 pp), although strong-agent influence increases from 79.84% to 83.35% (+3.51 pp).

These results suggest that the accuracy benefit of low agreeableness is specific to the disadvantaged W-W-S setting, rather than reflecting a general improvement across speaking orders. One plausible explanation is that under W-W-S, low agreeableness helps the strong agent resist abandoning its independent position after exposure to preceding weak-agent responses, thereby improving final accuracy. In contrast, under S-W-W, where the strong agent already benefits from speaking first, further increasing its resistance does not improve accuracy. We have clarified this targeted mitigation interpretation in the revised manuscript.

Thank you for your time for the rebuttal. Your response partially solved my concerns, but a few points remain open:

  1. Influence measure. Table 29 helps, but the main influence results (e.g., +9.55 pp) still rely on the conditional metric. That metric excludes third-answer cases, so the question sets still differ across conditions. It would help to note this in the main text, or to report the main results on a fixed question set.

  2. Order-specificity. Table 25 does not include a statistical test for the S-W-W effect (−0.70 pp) or for the order × prompt interaction, which is what would support the claim that the benefit is specific to W-W-S. Also, both W-W-S n_n_al and S-W-W al_n_n report 68.08%. Could you confirm this?

I increased the soundness score to 3.

We thank the reviewer for the follow-up and for increasing the soundness score.

Q1: Full-denominator influence analysis

We agree that the original conditional influence measure excludes third-answer outcomes, potentially leading to different eligible question sets across conditions. We have included a discussion of this limitation and a reference to the full-denominator analysis in Section 4.1, and have now further highlighted the corresponding robustness results in Section 5.2.

Specifically, Table 30 (previously Table 29) reports outcome distributions over the same fixed set of 7,980 question-model-combination observations across conditions. Under n_n_al, strong-only outcomes (where the final answer matches only the strong agent's standalone answer) increase from 26.47% to 30.43%, while weak-only outcomes decrease from 18.46% to 13.97%. These results preserve the directional pattern of the original influence findings without condition-dependent exclusions.

Q2: Order-specificity and statistical significance

We have conducted two additional analyses using the 40 matched model-combination–benchmark cells, following our existing paired-testing framework.

S-W-W intervention effect. Under S-W-W, assigning low agreeableness to the strong agent produces a small, non-significant decrease in accuracy (−0.69 pp; two-sided paired t-test p = 0.1311, Wilcoxon p = 0.2082, sign test p = 0.4050).

Order × intervention interaction. To directly test whether the intervention effect differs across speaking orders, we computed the within-cell difference between the W-W-S intervention effect and the S-W-W intervention effect. The interaction is positive and statistically significant (+2.34 pp; two-sided paired t-test p = 0.0010, Wilcoxon p = 0.0018, sign test p = 0.0385).

These findings provide statistical support for our interpretation that the accuracy benefit of low agreeableness depends on speaking order, with a positive effect under W-W-S but no corresponding improvement under S-W-W. We have added these results to Appendix L.2 and clarified the interpretation in Section 5.2.

Finally, we confirm that both W-W-S n_n_al and S-W-W al_n_n achieve 68.08% aggregate accuracy. This reflects their different no-personality baselines (66.43% and 68.78%, respectively) and is not a reporting error.

Reviewer_zRdmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines speaking order and personality prompting in sequential multi-agent debate(MAD). Teams contain one stronger model and two instances of a weaker model, evaluated across 10 model pairs and four QA benchmarks. Moving the stronger agent from first (S-W-W) to last (W-W-S) reduces its influence from 79.84% to 58.83% and debate accuracy from 68.78% to 66.43%. In the strong-agent-last setting, agreeableness prompts shift influence toward the less agreeable side; low agreeableness for the stronger agent (n_n_al) raises accuracy by 1.64 percentage points, while extraversion mainly changes response length. The authors analyze these results using paired tests and mixed-effects models over 40 pair-by-benchmark cells and argue that both speaking order and prompted interaction behavior matter for MAD performance.

Critical strengths
  1. The comparison gives clear evidence that speaking order affects which model prevails when models of different capabilities debate on graded tasks. The models, questions and prompts are held fixed while the stronger agent's position relative to the two weaker agents is reversed (Sec. 3.4). Its influence is higher in S-W-W in all 40 cells in Table 12. The differences range from a near tie (61.54% versus 61.40% for mistral-small/2.5-flash on AQuA-RAT) to a large gap (80.49% versus 21.95% for 2.5-flash-lite/2.5-flash on SuperGPQA). Accuracy is higher in S-W-W in 30 of 40 cells (Table 11), and the aggregate accuracy gap is positive for each of the five weaker models, which come from four vendors. Under W-W-S, the stronger agent loses most disagreements on SuperGPQA despite its higher standalone accuracy (mean influence 45.23%). Round-robin is a common MAD protocol, so this is a useful result for anyone choosing a debate order. This is the main reason I do not recommend clear rejection.

  2. The detailed reporting makes the results easy to check. It also shows that greater influence does not always lead to higher accuracy. It provides the full prompts (App. F), per-cell results (App. J) and standalone accuracy for each persona (Table 5). For example, ah_ah_al yields greater influence than n_n_al (71.43% versus 68.38%) but lower accuracy (67.23% versus 68.08%; Table 8); ah_ah_n raises influence by 3.76 points without changing accuracy (Table 2). These examples show why maximizing the stronger agent's influence alone may be a poor design rule. The 0.85-point accuracy difference, however, is not tested statistically.

Critical weaknesses
  1. The order effect is clear in this setup, but its cause and generality are less clear.

(1-a) Under matching prompts and temperature 0, the first speaker's initial answer should largely match its standalone answer, although identical outputs are not guaranteed (Apps. F.4-F.5). In W-W-S, the strong agent's standalone answer a_S may never be voiced, as in Table 10. Since influence is measured against a_S, exposure may explain part of the gap. Conformity to a two-to-one consensus is another possibility when the weaker agents agree. Without W-S-W or a simultaneous first round, the evidence for general first-speaker bias remains limited to two orders and the tested Gemini setup.

(1-b) The gap varies substantially by benchmark. From Tables 11-12, influence gaps are 35.8, 21.1, 19.1 and 8.0 points on SuperGPQA, GPQA, MMLU-Pro and AQuA-RAT; accuracy gaps are 4.65, 2.12, 1.10 and 1.50 points. AQuA-RAT accuracy is already around 84-85%, which may compress its gap. Figure 1 and Appendix J report these differences, but the paper does not test whether the order effect varies reliably across benchmarks. Dependence on difficulty or uncertainty remains a plausible explanation.

(1-c) The claim that weaker agents contribute useful information needs a stronger baseline. W-W-S averages 71.24% for Gemini-3-Flash pairs, below that model alone at 72.04%, whereas Gemini-2.5-Flash pairs average 61.63% versus 52.60% alone (Tables 5, 11). Thus, the pooled advantage does not hold for both strong models. Debate also gives the strong agent four debate turns, compared with one standalone call; there is no compute-matched self-revision or three-strong-agent baseline.

(1-d) ChatEval, already cited, proposed simultaneous-talk among several strategies partly to remove order effects. Zhang et al. (2026), also cited, report greater influence for later-positioned viewpoints. The paper distinguishes sequential MAD from that setting (Sec. 2.2), but the discussion could explain which protocol differences might account for the opposite directions: the strong agent here has more influence when it speaks first.

  1. The persona effect is not clearly trait-specific, and the robustness of the accuracy gain remains uncertain.

(2-a) n_n_al requires knowing which model is stronger. Given that information, simply using S-W-W achieves 68.78% accuracy versus 68.08% for n_n_al, winning in 24 of 40 cells, losing in 14 and tying in two. The two interventions are not combined, and no capability-agnostic variant is tested.

(2-b) The low-agreeableness prompt explicitly says to "resist updating your answer unless strong counter-evidence is provided" (F.3), while the baseline has no persona template. Prompting one strong agent also differs from prompting two weak agents. The agreeableness effects (+9.55 for n_n_al and -11.07 for n_n_ah) exceed the generic shifts from adding extraversion prompts, so their direction is credible. But the added value of Big Five framing over a direct anti-conformity instruction is untested, and influence is not separated by which side was correct.

(2-c) The headline accuracy gain of 1.64 points is the best of eight unadjusted tests. After Bonferroni correction, the sign test no longer reaches 0.05 (about 0.055), although the t-test, Wilcoxon and mixed-effects results remain significant. One-tailed tests also report p = 1.0000 for harms, including -5.13 points for n_n_ah (Tables 2-3). Each condition appears to be run once, and the 40 cells share only two strong models. Gains are concentrated in Gemini-3-Flash pairs (+2.98 versus +0.30 points). In the alternative-prompt check, n_n_al scores 74.17%, below the corresponding W-W-S baseline of 74.43% (App. G; Tables 11-19).

Questions for authors
  1. In W-W-S, how often does the strong agent voice a_S in the first round, and what is its influence conditional on that? This would help separate exposure from primacy.

  2. Does the claimed benefit over standalone performance hold by strong model and benchmark, especially for Gemini-3-Flash? How does debate compare with a compute-matched single-agent baseline?

  3. Please report two-sided, multiplicity-adjusted accuracy tests and question-level uncertainty, with persona effects by strong model. Does low agreeableness selectively preserve correct answers, or reinforce incorrect ones too?

  4. (regarding suggestion/future work) Adding W-S-W and a simultaneous first round may help distinguish explanations for the order effect. A direct "resist updating" prompt would test whether Big Five framing adds anything beyond that instruction.

5 comments

We thank the reviewer for the detailed assessment and constructive suggestions. In response, we have expanded our analyses to examine the mechanisms underlying speaking-order effects, including the exposure of the strong agent's independent answer and the additional W-S-W configuration. We have also strengthened the evaluation of personality interventions through a direct resistance-prompt baseline, correctness-dependent influence analysis, multiplicity-adjusted statistical tests, and analyses of strong-model heterogeneity. Finally, we have clarified the comparison with standalone performance and the scope of our claims. We address these points below.

Q1: Speaking-order mechanisms, exposure, and related work

We agree that exposure and conformity are plausible mechanisms underlying the observed speaking-order effect, rather than competing explanations. In our strictly sequential debate protocol, speaking order determines what information each agent encounters before generating its response. Under S-W-W, the strong agent expresses its initial position without exposure to the weak agents' opinions. Under W-W-S, it responds after both weak agents, potentially facing an established consensus before expressing its own position. Thus, speaking order shapes both exposure to preceding opinions and the potential for conformity. Our study examines the total effect of speaking order on debate outcomes, rather than attempting to isolate these mechanisms' individual contributions. To further examine whether the effect is driven specifically by placing the strong agent after two consecutive weak agents, we additionally evaluated W-S-W (Q4).

Exposure analysis (Weakness 1-a). On questions with standalone disagreement, the strong agent voices its independent answer (a_S) in Round 1 on 56.91% of W-W-S debates versus 91.88% under S-W-W. Only 2.84% and 1.62%, respectively, first voice a_S in later rounds. Consequently, a_S is never voiced in 40.25% of W-W-S debates, compared with 6.51% under S-W-W (Appendix K.2).

Conditional on voicing a_S in Round 1, strong-agent influence remains high under both orders (87.65% for W-W-S versus 82.52% for S-W-W), calculated over influence-eligible questions. These findings suggest that the observed order effect may operate through whether the strong agent expresses its independent answer in the first place. However, this observational comparison does not causally isolate exposure from other interaction mechanisms.

Related work (Weakness 1-d). ChatEval (Chan et al., 2023) compares communication strategies for LLM-based evaluation using homogeneous-model teams with distinct roles. Its simultaneous-talk strategy removes within-round sequential dependencies but does not directly examine individual speaking-position effects. In contrast, our study focuses specifically on speaking-order effects within a strictly sequential debate protocol, where agents condition their responses on preceding agents' outputs.

Regarding Zhang et al. (2026), their study independently generates agents' initial viewpoints before arranging their positions within the debate context, finding greater influence for later-positioned viewpoints. Their setting is therefore more closely related to information-presentation order effects, examining how the placement of pre-generated viewpoints shapes subsequent outcomes. In contrast, our study examines speaking-order effects on debate dynamics, where agents generate responses sequentially, conditioned on preceding agents' outputs. Thus, speaking order affects not only which viewpoints are presented first, but also how agents form, express, and revise their positions throughout the debate. These differences in protocols may explain the contrasting findings.

Q2: Standalone performance, benchmark heterogeneity, and model capability Standalone performance (Weakness 1-c). We agree that our original claim regarding debate outperforming the standalone strong model was too general. Appendix K.3 provides comparisons by strong model and benchmark. For Gemini-2.5-Flash, both W-W-S (61.63%) and S-W-W (63.48%) outperform standalone accuracy (52.60%) across all four benchmarks. For Gemini-3-Flash, however, W-W-S (71.24%) slightly underperforms standalone (72.04%), while S-W-W improves to 74.07%. In particular, W-W-S underperforms standalone Gemini-3-Flash on GPQA (66.57% vs. 67.17%) and SuperGPQA (56.00% vs. 61.00%). We have revised our discussion to reflect this heterogeneity rather than suggest a universal benefit from debate.

Repeated-inference baseline. We additionally evaluated four-round self-revision using the standalone strong model: it generates an initial answer, then reviews and potentially revises it over three additional rounds. Self-revision achieves 65.51% accuracy for Gemini-2.5-Flash and 75.93% for Gemini-3-Flash, outperforming the corresponding mixed-capability debate averages under both speaking orders.

These results show that repeated strong-model inference can outperform mixed-capability debate. However, self-revision uses a different inference allocation and interaction structure and is not directly comparable to our mixed-capability debate setting. Our primary objective is to understand how speaking order shapes interaction dynamics and whether personality-based behavioural interventions can mitigate positional disadvantages in sequential MAD, rather than establish the efficiency or superiority of multi-agent collaboration over single-agent inference. We have clarified this limitation in the updated manuscript.

Benchmark heterogeneity (Weakness 1-b). We included four benchmarks spanning different domains and difficulty levels to assess whether the speaking-order effect extends beyond a particular task. The direction is consistent across all four, although the influence gap varies from 35.8 pp on SuperGPQA to 8.0 pp on AQuA-RAT (Figure 1; Appendix J). Differences in difficulty and baseline accuracy, including potential ceiling effects on AQuA-RAT, may contribute to this variation. However, our analysis does not establish whether the differences across benchmarks are statistically significant or isolate their causes.

Capability identification and intervention combination (Weakness 2-a). Our design assumes that relative model capabilities can be estimated in advance. As discussed in Section 3.3, this can be informed by known differences in model scale and generation, or by standalone performance on a validation set. Such information is often available when constructing a multi-agent team.

We agree that placing the strong agent first is simpler and achieves higher aggregate accuracy than applying low agreeableness under W-W-S (68.78% vs. 68.08%). We do not position personality prompting as a replacement for favourable speaking-order design. Rather, our objective is to investigate whether behavioural interventions can improve outcomes when the strong agent occupies a disadvantaged position.

We also evaluated low agreeableness under S-W-W (Appendix L.2). It increases strong-agent influence from 79.84% to 83.35%, but reduces accuracy from 68.78% to 68.08%. In contrast, under W-W-S, the same intervention improves accuracy by 1.64 pp. These results demonstrate that assigning low agreeableness to the strong agent is a targeted intervention for addressing the identified first-speaker bias in the disadvantaged W-W-S configuration, rather than a universally effective strategy for improving mixed-capability debates. Capability-agnostic interventions remain for future work.

Q3: Statistical robustness and personality effects

Statistical robustness (Weakness 2-c). We re-evaluated all personality conditions against the no-personality baseline using matched questions. To account for dependence across model combinations, we conducted 10,000 dataset-stratified question-level bootstrap resamples, retaining matched outcomes across model combinations and conditions. We report pointwise 95% confidence intervals and two-sided z-tests with dataset-stratified question-cluster standard errors, applying Holm correction across the eight personality comparisons (Appendix M.2).

The accuracy improvement under n_n_al remains significant (+1.64 pp, 95% CI [0.79, 2.48], Holm-adjusted p < 0.001). In contrast, the smaller improvement under el_el_n (+0.78 pp) is no longer significant after adjustment (p = 0.1815). The two-sided analyses also identify significant accuracy reductions under several other configurations.

Strong-model heterogeneity. Disaggregating by strong model, the accuracy improvement under n_n_al is significant for Gemini-3-Flash (+2.98 pp, Holm-adjusted p < 0.001), but smaller and not significant for Gemini-2.5-Flash (+0.30 pp; Appendix M.3). One possible explanation is the difference in capability gaps: Gemini-3-Flash achieves 72.04% standalone accuracy versus 52.60% for Gemini-2.5-Flash, compared with 41.29% on average for the weak models. Under W-W-S, Gemini-3-Flash teams already underperform their standalone strong model (71.24% vs. 72.04%), whereas Gemini-2.5-Flash teams outperform theirs (61.63% vs. 52.60%). These differences suggest that the benefit of preserving the strong agent's position may depend on relative model capabilities, although this explanation remains tentative.

Correctness-dependent influence (Weakness 2-b). We additionally stratified strong-agent influence by whether its standalone answer was correct (Appendix L.4). Under n_n_al, influence increases from 74.09% to 81.90% (+7.81 pp) when the strong agent is correct, and from 32.24% to 44.79% (+12.56 pp) when it is incorrect. Thus, low agreeableness does not selectively preserve correct answers; it reinforces the strong agent's independent position regardless of correctness. Because the strong model is more accurate on average (62.32% versus 41.29% for weak models), this can still produce a net accuracy improvement. However, greater influence does not necessarily imply higher accuracy, as illustrated by ah_ah_al (71.43% influence, 67.23% accuracy) versus n_n_al (68.38%, 68.08%).

Alternative-prompt robustness. Finally, we acknowledge that the simplified personality prompt in Appendix G does not improve accuracy over its corresponding baseline (74.17% vs. 74.43%). This experiment supports the directional relationship between agreeableness and influence, but does not establish the robustness of the accuracy gain to prompt wording.

Reviewer_ohHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines if speaking order affects the outcomes of sequential multi-agent debate (MAD), and if personality prompting based on the Big Five traits can mitigate this imbalance. There are two weak agents and one strong agent, and the authors compare weak-weak-strong against strong-weak-weak across various model families and reasoning benchmarks, with some additional experimental manipulation.

Critical strengths
  1. Controlled investigation of speaker-order effects, as well as exploration of mitigation methods. Results demonstrate that changing the order of otherwise identical agents affects which agent's initial answer prevails. This means that for multi-agent systems, placing a more capable model later in debate can influence collective reasoning performance substantially.
Critical weaknesses
  1. Personality intervention does not have a strong baseline. To my understanding, the personality prompting based on Big Five traits is supposed to manipulate agreeableness, and therefore influence reasoning performance. However, having a simpler baseline such as just telling the agent to make more independent verifications and resist unjustified consensus could be utilized as well. Without a true comparison between the two methods, the personality prompting based on Big Five traits could be unnecessarily elaborate or even redundant, and also would have little novelty.

  2. Influence metric is flawed. The influence metric could conflate resistance to changing one's own answer, with the ability to changing other agents' answers, which are totally different mechanisms. There needs to be a round-by-round analysis to separate these effects.

Questions for authors
  1. Can you justify that the personality prompting framework is necessary, with supporting evidence? Would a simpler intervention such as resisting unjustified consensus work? It seems that the personality framework contributes at most, anti-conformity instructions.

  2. How general is the speaking order effect, and why is the weak-strong-weak condition omitted? I am concerned if the observed effect actually reflects a first-speaker advantage, or if the effect is just because you're placing a strong agent before or after a block of weaker agents.

  3. For the influence shift, what is actually driving it? There needs to be deeper analysis and presentation of the prompting effects, which is why the round-by-round responses need to be shown. For example, the strong agent could make the weaker agents change their answer, or it could be more stubborn in holding on to its original position. This needs to be explored.

2 comments

We thank the reviewer for recognising the value of our controlled investigation of speaking-order effects and for the constructive feedback. Following the reviewer’s suggestions, we have (1) evaluated the previously omitted W-S-W speaking order, (2) compared our low-agreeableness intervention against a simpler resistance-only instruction, and (3) examined round-by-round responses to distinguish strong-agent answer retention from weak-agent adoption. We address each concern below.

Q1: Personality prompting and simpler baselines

We agree that a direct behavioural prompt is an important comparator. We adopt the Big Five framework as a theory-grounded perspective for systematically examining how different behavioural tendencies shape multi-agent debate. Specifically, agreeableness concerns accommodation versus resistance, while extraversion concerns assertiveness and expressiveness. We operationalise these behavioural tendencies through prompting, without assuming that LLMs possess stable human-like personalities.

Importantly, the proposed anti-conformity instruction is closely related to low agreeableness, which encompasses a tendency to challenge others' views rather than readily accommodate them. This is consistent with established practices in LLM personality prompting, where traits are operationalised through behavioural descriptions (Jiang et al., 2023, arXiv:2206.07550). We therefore do not assume that Big Five framing is necessary or uniquely effective.

To test a simpler alternative, we replaced the strong agent's full low-agreeableness prompt with “You resist updating your position during disagreements,” across all ten model combinations and four benchmarks under W-W-S. All other experimental settings remained unchanged. This setting achieved 64.87% accuracy and 60.49% strong-agent influence, compared with 66.43% and 58.83% for the no-personality baseline and 68.08% and 68.38% for the full low-agreeableness prompt (Appendix L.1). The tested resistance-only instruction increased influence by 1.66 pp but reduced accuracy by 1.56 pp, whereas the full low-agreeableness prompt increased influence by 9.55 pp and accuracy by 1.64 pp. While this does not establish that Big Five framing is necessary, it suggests that the observed benefits cannot be attributed solely to this simple resistance instruction.

Q2: Generality of speaking-order effects

Our original comparison kept the two weak agents adjacent to preserve their relative interaction structure (Section 3.4). To examine the generality of the speaking-order effect, we additionally evaluated W-S-W across all ten model combinations and four benchmarks with no personality prompting (Appendix K.1).

Both accuracy and strong-agent influence decrease monotonically from S-W-W (68.78%, 79.84%) to W-S-W (67.12%, 62.25%) to W-W-S (66.43%, 58.83%). Notably, moving the strong agent from first to second position reduces its influence by 17.59 pp, compared with only 3.42 pp from second to third. This suggests a pronounced first-speaker advantage, rather than an effect driven solely by placing the strong agent after two weak agents.

One plausible explanation is that S-W-W allows the strong agent to generate its first response independently, without exposure to preceding weak-agent opinions, potentially allowing its initial position to shape subsequent discussion more effectively. While the precise mechanism remains to be established, these results provide further evidence of a pronounced first-speaker advantage in sequential debate.

Q3: Answer retention versus weak-agent adoption

Our original influence metric measures which standalone answer ultimately prevails, without separating strong-agent answer retention from weak-agent adoption. To examine these behaviours separately, we conducted an additional round-by-round analysis comparing the low-agreeableness intervention (n_n_al) with the no-personality baseline (n_n_n). For questions where the strong and weak agents disagree in their standalone responses, we track three measures across all four rounds: (1) the proportion where the strong agent retains its standalone answer, (2) the proportion where at least one weak agent selects the strong agent's standalone answer, and (3) the proportion where both weak agents select that answer (Appendix L.3).

Under n_n_al, the strong agent exhibits higher answer retention from Round 1 (62.60% vs. 56.91%) through Round 4 (59.02% vs. 52.71%). Moreover, following exposure to the strong agent's reasoning, the weak agents also become more likely to select the strong agent’s standalone answer. By Round 4, this proportion increases from 53.31% to 61.37% for at least one weak agent, and from 51.31% to 59.87% for both weak agents.

These findings show that the increased influence under low agreeableness is associated with both greater strong-agent answer retention and greater subsequent alignment of the weak agents with the strong agent’s standalone answer. Both patterns are consistent with the observed increase in strong-agent influence.

Comments on the paper

1 comment

We thank all three reviewers for their constructive feedback. In response, we have strengthened our empirical evaluation and clarified the scope of our findings through additional experiments and analyses:

  1. Speaking-order effects and mechanisms. We evaluated the previously omitted W-S-W order and analysed the exposure of the strong agent's independent answer, providing further evidence of a pronounced first-speaker advantage while clarifying the roles of exposure and conformity.

  2. Intervention baselines and generalisability. We compared our personality intervention against a direct resistance-only prompt and evaluated low agreeableness under both speaking orders. These results support its effectiveness as a targeted mitigation for the disadvantaged W-W-S setting, rather than a universally beneficial intervention.

  3. Measurement and statistical robustness. We conducted fixed-reference and full-denominator influence analyses, examined round-by-round answer retention and weak-agent adoption, and added correctness-dependent analyses, two-sided multiplicity-adjusted tests, question-level uncertainty estimates, and strong-model heterogeneity analyses.

We have incorporated these findings into the revised manuscript and appendices, alongside clearer discussion of the study's limitations. The additional evidence reinforces our central findings: speaking order substantially shapes sequential multi-agent debate outcomes, and targeted behavioural interventions can partially mitigate the resulting positional disadvantage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.