- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how non-semantic biases affect final-answer aggregation in multi-agent LLM reasoning systems. The authors propose a training-free and validation-free framework that combines bias control and bias injection to improve answer selection. Bias control reduces frequency and stylistic biases by retaining one reasoning trace per distinct answer and normalizing its presentation, while bias injection deliberately increases the frequency of competing answers to assess the stability of an agent's initial decision. The framework then combines decision stability and cross-agent agreement to determine the final answer. Experiments across three mathematical reasoning benchmarks and four heterogeneous model combinations demonstrate improvements in accuracy and token efficiency over several aggregation baselines, with additional results on multi-agent debate and Mixture-of-Agents.
Critical strengths
- The paper explores how deliberately introducing bias can reveal the reliability of an LLM's decision. The observation that incorrect decisions are more susceptible to adverse-frequency perturbations provides an interesting motivation for the proposed aggregation framework.
- The proposed method requires neither additional training nor validation data or prior knowledge of individual model strengths. It operates directly on existing reasoning traces and can be integrated into different multi-agent frameworks without modifying their upstream reasoning processes.
- The proposed method achieves higher average accuracy than the evaluated non-oracle baselines across all four heterogeneous model combinations, with improvements of 3.2–6.2 percentage points over the strongest baseline. The authors also consider inference costs and analyze both the recovery of correct answers missed by majority voting and the retention of already-correct decisions.
Critical weaknesses
- The final aggregation procedure relies on a manually designed combination of decision persistence, second-pass majority support, and fallback voting rules. Although Table 10 shows that decisions satisfying both persistence and majority-support conditions have higher conditional accuracy, this does not directly establish the effectiveness of the specific aggregation rule.
- Although the method balances answer frequencies and normalizes reasoning styles, candidates are still ordered according to their original frequencies, potentially introducing a correlation between position and majority status. Moreover, the source-attribution analysis shows that model identity remains highly distinguishable after normalization, with 97.2% attribution accuracy in the judge representation space (Table 9).
Questions for authors
N/A
2 comments
Dear Reviewer Dmr2,
Thank you again for your feedback. Since our full response is long, here are the main concerns and the new results and revisions we added.
Summary of our response
1. Effectiveness of the final aggregation rule (W1)
-
A simple plurality voting rule achieves nearly the same accuracy. Replacing our final rule with plurality voting over the same decisions yields 76.6% accuracy, compared with 76.9% for our rule.
-
Table 10 shows how persistence and majority support relate to correctness. We added a plurality-voting comparison to evaluate the final rule and do not claim a separate accuracy gain from the specific rule (Appendix D.3).
2. Candidate ordering and original answer frequency (W2)
-
Higher observed accuracy remains under random ordering. With randomized candidate order, our method achieves 77.73% pooled accuracy, compared with 75.00% for Combined MV and 73.64% for SLM-MUX.
-
We acknowledge the observed 1.36-point decrease from default ordering, which suggests that frequency-based ordering may contribute to performance. The new randomized-order comparison is reported in Appendix D.8.
3. Style normalization and model identity (W2)
-
Normalization targets presentation-driven preferences in answer selection, not the removal of model-identity information. Model identity can remain recoverable even when sensitivity to presentation style is reduced.
-
New direct test of presentation sensitivity: We rewrote a competing incorrect trace in an authoritative tone while keeping candidate answers, frequencies, and order fixed.
-
Without normalization, the rewrite increased selection of the targeted incorrect answer by 1.31 percentage points (95% CI [0.16, 2.51]); with normalization, the change was −0.15 points (95% CI [−1.46, 1.18]). These results are consistent with reduced sensitivity to this style intervention (Added in Appendix D.2; Figure 14).
Thank you for the positive assessment and for your comments on the aggregation rule, candidate ordering, and style normalization.
W1. Role and effectiveness of the final aggregation rule
Response: Replacing the final aggregation rule with plurality voting over the same decisions yields 76.6% benchmark-averaged accuracy, compared with 76.9% for our rule, using the same averaging convention as Table 2. We do not claim a separate accuracy gain from the specific rule. It makes persistence under bias injection and cross-agent support explicit, and Table 10 reports how these conditions relate to correctness. This plurality-voting comparison is now included in Appendix D.3.
W2. Candidate ordering and style normalization
Response: We agree that the default ordering retains a correlation between candidate position and original answer frequency, even after displayed answer counts are balanced. We compared the default ordering with random candidate ordering in one paired run covering all 220 problems across the four model combinations. Pooled accuracy was 79.09% under the default ordering and 77.73% under random ordering (randomized minus default: −1.36 percentage points; 95% problem-level cluster-bootstrap CI [−2.73, −0.11]; two-sided sign-flip permutation p = 0.064). Both analyses keep the four model combinations for each problem together. The observed 1.36-point decrease suggests some benefit from frequency-based ordering. Although the bootstrap CI excludes zero, the difference is not significant at the 5% level under the permutation test. With random ordering, Ours still achieves higher pooled accuracy than the budget-matched Combined MV and SLM-MUX baselines in Table 2 (77.73% vs. 75.00% and 73.64%, respectively).
Response: The 97.2% attribution accuracy confirms that model identity remains recoverable after normalization. Our goal is to reduce presentation-driven preferences in answer selection. We examined this through a GPT-5 rewrite of a competing incorrect trace in an authoritative tone, keeping the candidate answers, frequencies, and order fixed. Without normalization, the rewrite increased selection of the targeted answer by 1.31 percentage points (95% CI: [0.16, 2.51]); with normalization, the change was −0.15 points [−1.46, 1.18]. These estimates are consistent with reduced sensitivity to this particular presentation intervention despite retained source information. Appendix D.2 now reports this analysis, with the rewrite prompt provided in Figure 14.
The revised manuscript includes the plurality-voting comparison (Appendix D.3), randomized-order comparison (Appendix D.8), and style-attack analysis (Appendix D.2; Figure 14), while acknowledging that model identity remains recoverable after normalization. We believe these additional analyses and revisions address your concerns, and we respectfully ask you to raise your rating to 4.