- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper examines how speaking order affects sequential multi-agent debate. Each team contains one stronger model and two instances of a weaker model. Across four debate rounds, the authors compare placing the stronger agent first or last. Experiments cover ten model combinations and four benchmarks. Placing the stronger agent first increases average accuracy by 2.34 percentage points. The paper then tests personality prompts in the stronger-agent-last setting. A low-agreeableness prompt for the stronger agent improves accuracy by 1.64 points over the unprompted baseline, while extraversion produces less consistent effects.
Critical strengths
The speaking-order comparison provides useful evidence about the debate protocol. The authors compare Strong-Weak-Weak with Weak-Weak-Strong while keeping the models and questions fixed. They also preserve the order of the two weak agents. Across ten model combinations and four benchmarks, putting the strong agent first improves average accuracy by 2.34 percentage points (Table 6). This shows that speaking order can affect performance even when the team members stay the same.
The paper shows that greater strong-agent influence does not always improve accuracy. Giving the strong agent a low-agreeableness prompt improves both influence and accuracy. However, also making the weak agents more agreeable increases strong-agent influence further but lowers accuracy relative to the strong-agent-only intervention (Appendix D). This is a useful finding. It shows why encouraging all agents to follow the stronger model is not necessarily the best strategy.
Critical weaknesses
The influence measure can compare different questions across conditions. Section 4.1 includes a question only when the standalone answers disagree and the final answer matches one of them. This excludes debates that end with a third answer. Since the final answer changes across conditions, the included questions can change too. Personality prompts also change the standalone answers used as references. The paper reports similar eligible proportions, but this does not show that the same questions qualify. A comparison using fixed questions and reference answers would make the influence shifts easier to interpret. This concern does not invalidate the accuracy results on all questions.
The personality experiments do not directly show that the prompt reduces the speaking-order gap. All personality interventions use Weak-Weak-Strong (§3.4). The low-agreeableness prompt improves this setting, but it might also improve Strong-Weak-Weak. Without testing both orders, it is unclear whether the prompt specifically reduces the disadvantage of speaking last or provides a more general benefit. The standalone checks in Appendix A do not answer this question. A comparison of both orders with and without the prompt would support the mitigation claim more directly. Otherwise, the conclusion should focus on improving the tested strong-agent-last setting.
Questions for authors
Do the influence differences remain when all conditions use the same disagreement questions and fixed standalone reference answers? How often does each condition produce a third answer that the current metric excludes?
Does the strong agent’s low-agreeableness prompt reduce the accuracy gap between Strong-Weak-Weak and Weak-Weak-Strong, or does it improve both orders similarly?
3 comments
Thank you for your time for the rebuttal. Your response partially solved my concerns, but a few points remain open:
-
Influence measure. Table 29 helps, but the main influence results (e.g., +9.55 pp) still rely on the conditional metric. That metric excludes third-answer cases, so the question sets still differ across conditions. It would help to note this in the main text, or to report the main results on a fixed question set.
-
Order-specificity. Table 25 does not include a statistical test for the S-W-W effect (−0.70 pp) or for the order × prompt interaction, which is what would support the claim that the benefit is specific to W-W-S. Also, both W-W-S n_n_al and S-W-W al_n_n report 68.08%. Could you confirm this?
I increased the soundness score to 3.
We thank the reviewer for the follow-up and for increasing the soundness score.
Q1: Full-denominator influence analysis
We agree that the original conditional influence measure excludes third-answer outcomes, potentially leading to different eligible question sets across conditions. We have included a discussion of this limitation and a reference to the full-denominator analysis in Section 4.1, and have now further highlighted the corresponding robustness results in Section 5.2.
Specifically, Table 30 (previously Table 29) reports outcome distributions over the same fixed set of 7,980 question-model-combination observations across conditions. Under n_n_al, strong-only outcomes (where the final answer matches only the strong agent's standalone answer) increase from 26.47% to 30.43%, while weak-only outcomes decrease from 18.46% to 13.97%. These results preserve the directional pattern of the original influence findings without condition-dependent exclusions.
Q2: Order-specificity and statistical significance
We have conducted two additional analyses using the 40 matched model-combination–benchmark cells, following our existing paired-testing framework.
S-W-W intervention effect. Under S-W-W, assigning low agreeableness to the strong agent produces a small, non-significant decrease in accuracy (−0.69 pp; two-sided paired t-test p = 0.1311, Wilcoxon p = 0.2082, sign test p = 0.4050).
Order × intervention interaction. To directly test whether the intervention effect differs across speaking orders, we computed the within-cell difference between the W-W-S intervention effect and the S-W-W intervention effect. The interaction is positive and statistically significant (+2.34 pp; two-sided paired t-test p = 0.0010, Wilcoxon p = 0.0018, sign test p = 0.0385).
These findings provide statistical support for our interpretation that the accuracy benefit of low agreeableness depends on speaking order, with a positive effect under W-W-S but no corresponding improvement under S-W-W. We have added these results to Appendix L.2 and clarified the interpretation in Section 5.2.
Finally, we confirm that both W-W-S n_n_al and S-W-W al_n_n achieve 68.08% aggregate accuracy. This reflects their different no-personality baselines (66.43% and 68.78%, respectively) and is not a reporting error.
We thank the reviewer for recognising the value of our controlled comparison and for the constructive suggestions. We have conducted additional analyses using fixed-reference influence measures and complete outcome distributions, and evaluated low agreeableness under both speaking orders. We address each concern below.
Q1: Fixed-reference influence and third-answer outcomes
We recalculated strong-agent influence using fixed no-personality standalone reference answers across all personality conditions, holding the initial disagreement questions constant within each benchmark–model combination. The directions of the original influence shifts are preserved. In particular, under W-W-S, n_n_al increases strong-agent influence by 9.18 pp, compared with 9.55 pp under the original calculation (Appendix M.1).
However, fixed references alone do not eliminate condition-dependent exclusions when final answers differ from both references. We therefore additionally analysed the complete set of 7,980 question–model-pair observations across all 40 benchmark–model combinations, using the same denominator for every condition. This includes questions where the standalone references agree or disagree. Final outcomes are classified into five mutually exclusive categories: strong-only, weak-only, both-match, third-answer, or no-final-majority.
Under W-W-S, n_n_al increases strong-only outcomes from 26.47% to 30.43% (+3.96 pp) and decreases weak-only outcomes from 18.46% to 13.97% (−4.49 pp). Third-answer outcomes increase modestly from 11.74% to 12.79% (+1.05 pp), with rates ranging from 10.53% to 14.90% across all reported conditions.
The agreeableness conditions preserve the directional pattern of the original influence analysis: the strong-only share increases and the weak-only share decreases when the weak agents receive high agreeableness or the strong agent receives low agreeableness. Under the opposite assignments, the strong-only share decreases, while the weak-only share increases except under al_al_n, where it decreases slightly, from 18.46% to 18.22%. Extraversion does not exhibit a similarly clear trait-dependent pattern, consistent with the less systematic effects observed in our main experiments.
The full-denominator analysis is consistent with the direction of the original influence findings, while measuring outcome shares rather than conditional influence. Appendix M.1 reports the original and fixed-reference influence calculations alongside the complete per-condition outcome distributions, including third-answer and no-final-answer rates.
Q2: Low agreeableness under both speaking orders
We additionally evaluated low agreeableness for the strong agent under S-W-W across all ten model combinations and four benchmarks, allowing a direct comparison with the existing W-W-S results (Appendix L.2).
The results show distinct accuracy effects across the two orders:
W-W-S: Accuracy increases from 66.43% to 68.08% (+1.64 pp), while strong-agent influence increases from 58.83% to 68.38% (+9.55 pp).
S-W-W: Accuracy decreases from 68.78% to 68.08% (−0.70 pp), although strong-agent influence increases from 79.84% to 83.35% (+3.51 pp).
These results suggest that the accuracy benefit of low agreeableness is specific to the disadvantaged W-W-S setting, rather than reflecting a general improvement across speaking orders. One plausible explanation is that under W-W-S, low agreeableness helps the strong agent resist abandoning its independent position after exposure to preceding weak-agent responses, thereby improving final accuracy. In contrast, under S-W-W, where the strong agent already benefits from speaking first, further increasing its resistance does not improve accuracy. We have clarified this targeted mitigation interpretation in the revised manuscript.