acceptodds
Under review as a conference paper at ICLR 2027

Bias Control and Injection for Multi-Agent Aggregation

Abstract

Multi-agent reasoning with large language models (LLMs) raises a central question for collective decision-making: when agents disagree, whose decision should be trusted? Resolving such disagreements is difficult because agents' decisions are susceptible to non-semantic biases such as majority pressure and response-style preferences, which can obscure the quality of the underlying reasoning. Our analysis shows how bias can be both mitigated and exploited: bias control improves initial answer selection, while bias injection elicits decision instability that serves as a reliability signal. Building on these findings, we introduce a test-time aggregation framework that requires neither validation data nor prior knowledge of relative model strength. The framework implements bias control by presenting one style-normalized reasoning trace per distinct answer, and bias injection by placing each aggregating agent’s initially selected answer in the minority. Overall, our framework achieves higher average accuracy and better token efficiency than competing non-oracle aggregation methods across three mathematical reasoning benchmarks and four heterogeneous LLM combinations. It also improves final-answer accuracy when applied to multi-agent debate and Mixture-of-Agents, without modifying their interaction or synthesis processes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent reasoning, LLM reasoning, reasoning aggregation, frequency bias, style bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dmr2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how non-semantic biases affect final-answer aggregation in multi-agent LLM reasoning systems. The authors propose a training-free and validation-free framework that combines bias control and bias injection to improve answer selection. Bias control reduces frequency and stylistic biases by retaining one reasoning trace per distinct answer and normalizing its presentation, while bias injection deliberately increases the frequency of competing answers to assess the stability of an agent's initial decision. The framework then combines decision stability and cross-agent agreement to determine the final answer. Experiments across three mathematical reasoning benchmarks and four heterogeneous model combinations demonstrate improvements in accuracy and token efficiency over several aggregation baselines, with additional results on multi-agent debate and Mixture-of-Agents.

Critical strengths
  1. The paper explores how deliberately introducing bias can reveal the reliability of an LLM's decision. The observation that incorrect decisions are more susceptible to adverse-frequency perturbations provides an interesting motivation for the proposed aggregation framework.
  2. The proposed method requires neither additional training nor validation data or prior knowledge of individual model strengths. It operates directly on existing reasoning traces and can be integrated into different multi-agent frameworks without modifying their upstream reasoning processes.
  3. The proposed method achieves higher average accuracy than the evaluated non-oracle baselines across all four heterogeneous model combinations, with improvements of 3.2–6.2 percentage points over the strongest baseline. The authors also consider inference costs and analyze both the recovery of correct answers missed by majority voting and the retention of already-correct decisions.
Critical weaknesses
  1. The final aggregation procedure relies on a manually designed combination of decision persistence, second-pass majority support, and fallback voting rules. Although Table 10 shows that decisions satisfying both persistence and majority-support conditions have higher conditional accuracy, this does not directly establish the effectiveness of the specific aggregation rule.
  2. Although the method balances answer frequencies and normalizes reasoning styles, candidates are still ordered according to their original frequencies, potentially introducing a correlation between position and majority status. Moreover, the source-attribution analysis shows that model identity remains highly distinguishable after normalization, with 97.2% attribution accuracy in the judge representation space (Table 9).
Questions for authors

N/A

2 comments

Thank you for the positive assessment and for your comments on the aggregation rule, candidate ordering, and style normalization.


W1. Role and effectiveness of the final aggregation rule

Response: Replacing the final aggregation rule with plurality voting over the same decisions yields 76.6% benchmark-averaged accuracy, compared with 76.9% for our rule, using the same averaging convention as Table 2. We do not claim a separate accuracy gain from the specific rule. It makes persistence under bias injection and cross-agent support explicit, and Table 10 reports how these conditions relate to correctness. This plurality-voting comparison is now included in Appendix D.3.


W2. Candidate ordering and style normalization

Comment: "Although the method balances answer frequencies and normalizes reasoning styles, candidates are still ordered according to their original frequencies, potentially introducing a correlation between position and majority status."

Response: We agree that the default ordering retains a correlation between candidate position and original answer frequency, even after displayed answer counts are balanced. We compared the default ordering with random candidate ordering in one paired run covering all 220 problems across the four model combinations. Pooled accuracy was 79.09% under the default ordering and 77.73% under random ordering (randomized minus default: −1.36 percentage points; 95% problem-level cluster-bootstrap CI [−2.73, −0.11]; two-sided sign-flip permutation p = 0.064). Both analyses keep the four model combinations for each problem together. The observed 1.36-point decrease suggests some benefit from frequency-based ordering. Although the bootstrap CI excludes zero, the difference is not significant at the 5% level under the permutation test. With random ordering, Ours still achieves higher pooled accuracy than the budget-matched Combined MV and SLM-MUX baselines in Table 2 (77.73% vs. 75.00% and 73.64%, respectively).

Comment: "Moreover, the source-attribution analysis shows that model identity remains highly distinguishable after normalization, with 97.2% attribution accuracy in the judge representation space (Table 9)."

Response: The 97.2% attribution accuracy confirms that model identity remains recoverable after normalization. Our goal is to reduce presentation-driven preferences in answer selection. We examined this through a GPT-5 rewrite of a competing incorrect trace in an authoritative tone, keeping the candidate answers, frequencies, and order fixed. Without normalization, the rewrite increased selection of the targeted answer by 1.31 percentage points (95% CI: [0.16, 2.51]); with normalization, the change was −0.15 points [−1.46, 1.18]. These estimates are consistent with reduced sensitivity to this particular presentation intervention despite retained source information. Appendix D.2 now reports this analysis, with the rewrite prompt provided in Figure 14.


The revised manuscript includes the plurality-voting comparison (Appendix D.3), randomized-order comparison (Appendix D.8), and style-attack analysis (Appendix D.2; Figure 14), while acknowledging that model identity remains recoverable after normalization. We believe these additional analyses and revisions address your concerns, and we respectfully ask you to raise your rating to 4.

Dear Reviewer Dmr2,

Thank you again for your feedback. Since our full response is long, here are the main concerns and the new results and revisions we added.


Summary of our response

1. Effectiveness of the final aggregation rule (W1)

  • A simple plurality voting rule achieves nearly the same accuracy. Replacing our final rule with plurality voting over the same decisions yields 76.6% accuracy, compared with 76.9% for our rule.

  • Table 10 shows how persistence and majority support relate to correctness. We added a plurality-voting comparison to evaluate the final rule and do not claim a separate accuracy gain from the specific rule (Appendix D.3).

2. Candidate ordering and original answer frequency (W2)

  • Higher observed accuracy remains under random ordering. With randomized candidate order, our method achieves 77.73% pooled accuracy, compared with 75.00% for Combined MV and 73.64% for SLM-MUX.

  • We acknowledge the observed 1.36-point decrease from default ordering, which suggests that frequency-based ordering may contribute to performance. The new randomized-order comparison is reported in Appendix D.8.

3. Style normalization and model identity (W2)

  • Normalization targets presentation-driven preferences in answer selection, not the removal of model-identity information. Model identity can remain recoverable even when sensitivity to presentation style is reduced.

  • New direct test of presentation sensitivity: We rewrote a competing incorrect trace in an authoritative tone while keeping candidate answers, frequencies, and order fixed.

  • Without normalization, the rewrite increased selection of the targeted incorrect answer by 1.31 percentage points (95% CI [0.16, 2.51]); with normalization, the change was −0.15 points (95% CI [−1.46, 1.18]). These results are consistent with reduced sensitivity to this style intervention (Added in Appendix D.2; Figure 14).


As the author–reviewer discussion phase draws to a close, we would like to follow up and see if you have any remaining concerns or questions that we could clarify. A brief acknowledgement that you have read our response would be appreciated. Thank you for your time!

Reviewer_W7VpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a test-time aggregation scheme for heterogeneous multi-agent reasoning that works in two stages: (1) Bias control: keep a single trace for each distinct answer, rewrite every trace into a uniform plain-prose style, and have J agents pick. (2) Bias injection: if the agents disagree, re-query each of them with its own chosen answer positioned as the minority (the strongest rival shown twice). A decision that "holds" is then treated as trustworthy. A rule over hold and support counts produces the final answer. On 220 math problems with four 8-14B reasoning models, the approach outperforms MV, SLM-MUX, and MAV by 3-6 pp while using fewer tokens, and it also improves Debate and MoA when applied post hoc.

Critical strengths
  • Using an adverse perturbation as a stress test on a decision, rather than only as a debiasing mechanism, is a clean and well-motivated idea (Fig. 3, Table 3).
  • The scheme requires no validation data and no prior assumptions about model strength, and it builds on the reasoning history that is already at hand.
  • Parts of the evaluation are rigorous: budget-matched comparisons, instance-level pairing in Table 1, an exact-duplication control (Table 4), and a robustness check that swaps the normalizer.
Critical weaknesses

1. The gains are not attributed to the two proposed components, and they may largely stem from plain LLM selection over distinct answers. The pipeline contains three ingredients:

  • (i) an agreement gate plus LLM-judge selection over one trace per distinct answer, which is essentially the one-rationale-per-answer construction of Wang et al. (2026b) with multiple judges;
  • (ii) style normalization;
  • (iii) adverse-frequency injection.

The ablations (Fig. 9) are computed only on the subsets that reach each stage. On instances the gate does not resolve, removing normalization costs 2.1 pp; on instances where the judges disagree, injection buys about 2.4 pp over repeated inference. The paper never states what fraction of the 220 problems reaches each stage. If, say, 25% reach stage 2 and a third of those reach stage 3, then injection contributes well under 1 pp to the headline numbers, and most of the 3-6 pp edge over MV would come from (i).

2. The statistical support is thin relative to the claimed effect sizes. The full benchmark contains only 220 problems, and AIME and HMMT hold 60 each, so a single problem is worth 1.7 pp in those columns. The 3-6 pp main-table differences and the ~2 pp component effects are reported without paired significance tests or CIs across problems. The ± values are run-to-run standard deviation of the judge sampling, which does not capture problem-level variance.

3. The injection effect is confounded with extra reasoning content. In the default injection, the rival answer is shown twice with a distinct trace, so the second pass sees new evidence about the rival, not just higher frequency. The cleaner exact-copy control (Table 4) leaves a gap only 3 pp above position swap (37.3 vs. 34.3), with no CI. The claim that it is frequency bias that makes the signal selective is therefore only weakly supported.

There is also a simpler competing explanation: any second pass with a higher overall flip rate will tend to show a larger absolute incorrect−correct gap. A more appropriate comparison is one matched on hold rate, such as repeated inference with several re-queries thresholded to the same hold rate, or the AUROC of the stability signal. Table 3's held accuracy (46.9% vs. 42.4%) is reported at different hold rates (37.5% vs. 52.4%), and keeping fewer decisions mechanically raises the precision of those kept.

Questions for authors
  1. Per model combination, what fraction of problems reach (a) the bias-control stage and (b) the injection stage?
  2. Do the reported token costs include the normalizer's tokens and the second-pass judges' tokens?
  3. Was the final rule (H(a) > 0 ∧ S(a) > J/2, then fallbacks) chosen a priori, or tuned on these benchmarks? Without a validation set, any rule selection on the test problems is a concern.
  4. MAV underperforms Combined MV by about 3-5 pp. Is this expected for 8-14B verifiers, or a sign of a weak reimplementation?
4 comments

We appreciate your detailed comments on the contributions of the proposed components and the statistical evidence supporting our results.


W1. Contributions of the proposed components

Response: In our experiments, 54.0% of instances reach bias control and 23.3% reach injection. Evaluated over all benchmark problems, normalization and injection together improve accuracy by 2.1 percentage points (95% CI [1.25, 3.06], p < 0.001) over ingredient (i): the agreement gate with LLM selection over one trace per distinct answer. Relative to the same budget-matched Combined MV reference in Table 2, ingredient (i) gains 2.7 points, while the full framework gains 4.8 points. The full-benchmark normalization ablation shows a gain of 1.2 points (95% CI [0.24, 2.17], p = 0.018). Injection adds 0.9 points over omitting the second decision pass (95% CI [0.33, 1.51], p = 0.002), and 0.6 points over repeated inference (95% CI [0.04, 1.23], p = 0.050).

The ablations use identical k=3 initial response pools. Accuracy differences are averaged equally across benchmarks and model combinations, as in Table 2; stage-reach rates are pooled over all evaluation instances. We use problem-level paired cluster-bootstrap CIs and sign-flip permutation tests. The table below shows the combined gain from normalization and injection over all problems in each benchmark: gains are positive in 11 of the 12 benchmark–combination pairs and larger on average on HMMT. Section 6.3 reports the overall stage-entry rates and combined full-benchmark gain. Appendix D.6 now includes the separate component effects and gains by benchmark and model combination (Tables 18 and 19), together with the statistical procedures and confidence intervals.

Model combinationOlymMATHAIMEHMMT
Qwen–Phi–GLM+0.7+0.6+3.9
Qwen–Phi–Nemotron+2.7+1.1+1.7
Qwen–GLM–Nemotron+3.3−1.1+1.1
Phi–GLM–Nemotron+1.7+2.8+6.7

W2. Statistical support for the reported gains

Response: The ± values in Table 2 are standard deviations across runs. We computed paired 95% confidence intervals with a problem-level cluster bootstrap and two-sided p-values with a sign-flip permutation test. All results for the same problem are kept together across model combinations and runs. The analysis includes all 220 problems and follows Table 2: we average the three benchmarks equally, then average across the four model combinations.

Comparison: Ours minusAccuracy difference (pp)95% CIp-value
Combined MV+4.8[2.3, 7.5]<0.001
SLM-MUX+6.1[3.4, 9.1]<0.001
MAV+8.2[5.6, 11.1]<0.001
Single-Best-SC (oracle)+3.0[−0.1, 6.1]0.071
Without normalization and injection+2.1[1.3, 3.1]<0.001

The gains over the three non-oracle baselines and the variant without normalization and injection are significant (all p < 0.001). Against repeated inference on the common injection subset in Figure 9, the gain is 2.4 percentage points (95% CI [0.0, 5.0]; p = 0.076). This difference and the oracle comparison are not significant at the 5% level. Appendix D.6 now reports the confidence intervals and significance tests for the five comparisons above (Table 17) and the injection-subset comparison, together with the statistical procedures. The Table 2 caption specifies that ± denotes the standard deviation across three runs.

W3. Interpretation and evaluation of the bias-injection signal

Comment: "In the default injection, the rival answer is shown twice with a distinct trace, so the second pass sees new evidence about the rival, not just higher frequency. The cleaner exact-copy control (Table 4) leaves a gap only 3 pp above position swap (37.3 vs. 34.3), with no CI. The claim that it is frequency bias that makes the signal selective is therefore only weakly supported."

Response: The exact-copy control exceeds position swap in the incorrect–correct flip-rate gap by 3.0 percentage points, but this difference is not statistically significant (95% CI [−4.4, 10.3]; p = 0.43). We added a paraphrase control that re-expresses the existing competing trace with instructions to preserve its reasoning content. On the common subset of 2,172 decisions with available paraphrases, its gap exceeds position swap by 8.3 points (95% CI [1.7, 14.7]; p = 0.009). However, paraphrasing also changes wording, while the default intervention introduces distinct reasoning content. These controls do not fully separate the contributions of frequency manipulation and additional reasoning, so we do not attribute the observed selectivity solely to frequency bias. Table 4 includes the paraphrase control, and Section 6.2 now reports the confidence intervals for both the exact-copy and paraphrase comparisons with position swap. Sections 3.2 and 6.2 clarify this interpretation.

Comment: "There is also a simpler competing explanation: any second pass with a higher overall flip rate will tend to show a larger absolute incorrect−correct gap. A more appropriate comparison is one matched on hold rate, such as repeated inference with several re-queries thresholded to the same hold rate, or the AUROC of the stability signal. Table 3's held accuracy (46.9% vs. 42.4%) is reported at different hold rates (37.5% vs. 52.4%), and keeping fewer decisions mechanically raises the precision of those kept."

Response: We agree that higher held accuracy at lower coverage does not by itself establish better discrimination. Following your suggestion, we compare injection with a repeated-inference control that retains a decision only when both additional queries preserve the initial answer. We also report self-consistency over the initial decision and two re-queries. Hold rates and held accuracies use the same 967 of Table 3’s 990 decision slots with both re-query outcomes available. Token counts include all stages and are averaged over all 2,640 evaluation instances (220 problems × four model combinations × three runs).

ConditionTokens / problemHold rate (%)Held accuracy (%)
Repeated inference219.9k53.741.0
Repeated inference ×2, majority of three232.3k70.337.9
Repeated inference ×2, both hold232.3k40.145.6
Adverse-frequency injection219.8k37.547.7

At these nearby hold rates, injection has numerically higher held accuracy (47.7% vs. 45.6%; 95% CI for the difference [−3.1, 7.4] percentage points). On the 615 injection-stage instances, final-answer accuracy is also numerically higher: 41.6% for injection, compared with 39.2% for a single re-query and 40.2% for three-sample self-consistency. These accuracy differences are not statistically significant. In practical terms, injection reaches a similar observed coverage–accuracy trade-off to the both-hold control at a lower mean total token cost (219.8k vs. 232.3k per evaluation instance), close to that of a single re-query (219.9k). The analyses are now included in Appendix D.3 (Table 11), and the discussion of Table 3 explicitly acknowledges the coverage–accuracy trade-off.


Q1. Per model combination, what fraction of problems reach (a) the bias-control stage and (b) the injection stage?

Response: The table reports stage-entry rates for each model combination. Both columns use all 220 problems as the denominator, with percentages averaged across three runs. Overall averages the four model combinations.

Model combination(a) Bias control(b) Bias injection
Qwen–Phi–GLM54.1%22.6%
Qwen–Phi–Nemotron53.2%22.7%
Qwen–GLM–Nemotron52.7%26.1%
Phi–GLM–Nemotron55.9%21.8%
Overall54.0%23.3%

Averaged across model combinations, HMMT has the highest stage-entry rates: 73.8% for bias control and 38.9% for bias injection. Appendix D.5 now reports the stage-entry rates (Table 16), while Appendix D.6 reports the corresponding full-benchmark component gains and paired confidence intervals.

Q2. Do the reported token costs include the normalizer's tokens and the second-pass judges' tokens?

Response: The costs in the submitted manuscript included style normalization and both decision passes, but omitted the normalizer’s thinking tokens and output tokens from failed attempts. Including these raises Ours’ cost from 212.3k to 219.8k tokens per problem (+3.5%). These corrections are reflected in Tables 2 and 8 and Figures 5 and 7 of the revised manuscript. Reapplying the budget-matching rule leaves the baseline configurations in Table 2 unchanged and updates those for two model combinations in Table 8. In both tables, Ours retains the highest average accuracy and lowest token cost among the compared configurations in all four model combinations.


Q3. Was the final rule (H(a) > 0 ∧ S(a) > J/2, then fallbacks) chosen a priori, or tuned on these benchmarks? Without a validation set, any rule selection on the test problems is a concern.

Response: The rule was specified a priori and was not selected or tuned on these benchmarks. Replacing only the final aggregation rule with plurality voting over the same decisions yields 76.6% benchmark-averaged accuracy, compared with 76.9% for our rule. We do not claim a separate accuracy gain from the specific rule. It makes decision persistence and cross-agent support explicit, and Table 10 reports how these conditions relate to correctness. Appendix D.3 now includes this plurality-voting comparison.


Q4. MAV underperforms Combined MV by about 3-5 pp. Is this expected for 8-14B verifiers, or a sign of a weak reimplementation?

Response: The lower accuracy is consistent with the high false-positive rate observed in our verifiers. On instances with multiple candidate answers, individual verifiers approve 93.0% of correct candidates but also 61.0% of incorrect candidates under the final-answer correctness aspect. Even retrospectively selecting the best of the three evaluated aspect configurations for each model combination yields 69.2% average accuracy, below Combined MV’s 72.1% in Table 2. Our implementation retains MAV’s binary verification and selection by total approvals, with three adaptations: the verifiers are the three participating models rather than GPT-4o-mini and Gemini-1.5-Flash; each distinct answer is verified through one representative trace rather than every sampled output; and we use a fixed, adapted aspect set—final-answer correctness, reasoning validity, and solution completeness—without the original validation-based verifier selection. Appendix C.2 now documents these implementation adaptations and verification diagnostics.


The revision now reports stage-entry rates, component gains, and problem-level statistical analyses (Section 6.3; Appendices D.5 and D.6), coverage and plurality comparisons (Appendix D.3), and MAV diagnostics (Appendix C.2). It also includes corrected token accounting (Tables 2 and 8; Figure 5) and the revised mechanism interpretation (Sections 3.2 and 6.2). With all of your concerns and questions now addressed in the revision, we kindly ask you to raise your rating to 4.

Dear Reviewer W7Vp,

Thank you again for your feedback. Since our full response is long, here are the main concerns and the new results and revisions we added.


Summary of our response

1. Contribution beyond plain LLM selection (W1/Q1)

  • Normalization and injection provide an additional gain over plain selection. Across all benchmark problems, they add 2.1 percentage points over the agreement gate with LLM selection alone (p < 0.001), using identical reasoning pools.

  • The combined gain is positive in 11 of 12 benchmark–combination pairs. We now report full-benchmark component effects and stage-entry rates: 54.0% reach bias control and 23.3% reach injection (Revised/added in Section 6.3; Appendices D.5–D.6).

2. Statistical support for the reported gains (W2)

  • New problem-level paired analyses: Our method improves accuracy over Combined MV by 4.8 percentage points (95% CI [2.3, 7.5]), SLM-MUX by 6.1 points, and MAV by 8.2 points; all three comparisons have p < 0.001.

  • These analyses keep results for the same problem together across model combinations and runs, addressing problem-level uncertainty beyond the run-to-run standard deviations (Added in Appendix D.6).

3. Frequency effects and the coverage–accuracy trade-off (W3)

  • New paraphrase control: We added a control designed to increase competing-answer frequency while preserving the reasoning content of an existing trace. On the common subset of decisions, its incorrect–correct flip-rate gap exceeds that of position swap by 8.3 percentage points (95% CI [1.7, 14.7]). We also clarify that these controls do not fully separate frequency and reasoning-content effects (Revised/added in Sections 3.2 and 6.2).

  • New repeated-inference control at nearby hold rates: Injection reaches a similar observed coverage–accuracy trade-off with 219.8k vs. 232.3k mean total tokens. Held accuracy is numerically higher (47.7% vs. 45.6%), although the difference is not statistically significant (Added in Appendix D.3).

4. Token accounting, rule selection, and MAV implementation (Q2–Q4)

  • The reported accuracy–cost advantage remains after correcting token accounting. The corrected total is 219.8k tokens per instance (+3.5%); our method retains the highest average accuracy and lowest token cost among the compared configurations in Tables 2 and 8.

  • The final rule was specified a priori, without tuning on these benchmarks. We also added a plurality-voting comparison and documented the MAV implementation adaptations and verifier diagnostics (Added in Appendices C.2 and D.3).


As the author–reviewer discussion phase draws to a close, we would like to follow up and see if you have any remaining concerns or questions that we could clarify. A brief acknowledgement that you have read our response would be appreciated. Thank you for your time!

Reviewer_UJFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-agent aggregation framework that both mitigates and exploits biases in LLM decision-making. It first applies bias control by balancing answer frequencies and normalizing reasoning styles, then injects adverse-frequency bias to assess decision reliability through decision stability. Final answers are selected based on retained decisions and cross-agent support. Experiments across three mathematical reasoning benchmarks demonstrate improved accuracy and token efficiency compared to existing aggregation methods.

Critical strengths
  1. The idea of deliberately injecting bias to elicit decision instability as a reliability signal is interesting and intuitive.

  2. The experimental evaluation is relatively comprehensive, including multiple model combinations, budget-matched comparisons, ablations, and applications to existing multi-agent frameworks.

Critical weaknesses

1. Limited justification for the complexity and reliability of bias injection.

While the authors demonstrate that bias injection can distinguish correct and incorrect decisions through their different flip rates, its incremental benefit over simpler re-evaluation strategies remains relatively modest. Although the exact-copy ablation partially isolates frequency bias from additional reasoning content, their respective contributions to final-answer accuracy remain unclear. Furthermore, decision stability does not necessarily imply correctness, as 19.1% of initially correct decisions are flipped while 40.2% of incorrect decisions remain unchanged.

2. Unclear practical efficiency and generalizability.

The framework requires additional LLM-based style normalization and multiple judging passes without directly improving the underlying reasoning process. Although the budget-matched comparisons show favorable accuracy–cost trade-offs, the incremental overhead of aggregation given identical initial response pools is not sufficiently characterized. Furthermore, the evaluation focuses on relatively small reasoning models, leaving the benefits for stronger models uncertain.

Questions for authors

Q1. Bias injection vs. additional reasoning

Although Table 4 includes an exact-copy ablation, could the authors further distinguish the contributions of frequency manipulation and additional reasoning content to final-answer accuracy, rather than only decision flip rates?

Q2. Decision reliability

Among initially incorrect decisions that flip under bias injection, how many actually become correct? Conversely, how often does bias injection cause the final aggregation to lose an initially correct answer?

Q3. Efficiency

Could the authors provide a stage-wise breakdown of token usage and latency? In particular, how does the framework compare with simpler aggregation methods using identical initial reasoning pools, rather than only budget-matched configurations with different sampling budgets?

Q4. Stronger reasoning models

Do the accuracy and efficiency gains persist when using substantially stronger, more recent reasoning models (e.g., Qwen3.8)?

4 comments

Thank you for the thoughtful review and for your questions about the benefits of bias injection, decision reliability, and aggregation costs.


W1. Justification for the complexity and reliability of bias injection

Comment: “While the authors demonstrate that bias injection can distinguish correct and incorrect decisions through their different flip rates, its incremental benefit over simpler re-evaluation strategies remains relatively modest. Although the exact-copy ablation partially isolates frequency bias from additional reasoning content, their respective contributions to final-answer accuracy remain unclear.”

"Q1. Bias injection vs. additional reasoning: Although Table 4 includes an exact-copy ablation, could the authors further distinguish the contributions of frequency manipulation and additional reasoning content to final-answer accuracy, rather than only decision flip rates?"

Response: We added a paraphrase control that repeats an existing competing trace in different wording, with instructions to preserve its reasoning content. Our default intervention uses a distinct trace already available in the initial pool, requiring no additional solution generation. The table reports pooled final-answer accuracy on the common subset requiring bias-injected inference.

ConditionAdditional competing traceAccuracy (%)
No second pass—37.6
Repeated inferenceNone39.2
Position swapNone39.3
Adverse frequency: exact copyDuplicate of an existing trace39.2
Adverse frequency: paraphraseParaphrase of an existing trace40.3
Adverse frequency: distinct trace (Ours)Distinct trace from the initial pool41.6

Both paraphrased and distinct traces yield higher observed accuracy than repeated inference, with the distinct trace performing best. The gains of Ours over repeated inference and paraphrasing are not statistically significant (p = 0.076 and 0.20, respectively). Across all benchmark problems, injection improves benchmark-averaged accuracy over repeated inference by 0.6 percentage points (95% CI [0.04, 1.23]; p = 0.050; Appendix D.6, Table 18). Mean total token costs are similar: 219.8k for injection versus 219.9k for repeated inference per evaluation instance (Appendix D.3, Table 11).

These results suggest that both increased competing-answer frequency and additional reasoning content may contribute. Our controls do not fully separate their effects, so we do not attribute the observed gains solely to frequency bias. We have clarified this interpretation in Sections 3.2 and 6.2 and report the final-answer comparisons in Figure 9 and the paraphrase prompt in Figure 13.

Comment: "Furthermore, decision stability does not necessarily imply correctness, as 19.1% of initially correct decisions are flipped while 40.2% of incorrect decisions remain unchanged."

"Q2. Decision reliability: Among initially incorrect decisions that flip under bias injection, how many actually become correct? Conversely, how often does bias injection cause the final aggregation to lose an initially correct answer?"

Response: We agree that individual decision stability does not guarantee correctness. Our method aggregates multiple agents’ decisions, so a single agent’s flip need not change the final answer. Among instances reaching bias injection, aggregating the initial decisions alone yields 231 correct answers. Injection changes 14 of these answers to incorrect ones (6.1%), compared with 24 (10.4%) under repeated inference. This difference in loss rates is not statistically significant (p = 0.121). In the analysis underlying Table 4, 41.4% of initially incorrect agent decisions that flip become correct (42.5% under repeated inference). Because injection flips more incorrect decisions, it corrects 24.8% of all initially incorrect decisions, compared with 18.8% under repeated inference. This decision-recovery and retention analysis is now reported in Appendix D.3.

W2. Practical efficiency and generalizability

Comment: "The framework requires additional LLM-based style normalization and multiple judging passes without directly improving the underlying reasoning process. Although the budget-matched comparisons show favorable accuracy–cost trade-offs, the incremental overhead of aggregation given identical initial response pools is not sufficiently characterized."

"Q3. Efficiency: Could the authors provide a stage-wise breakdown of token usage and latency? In particular, how does the framework compare with simpler aggregation methods using identical initial reasoning pools, rather than only budget-matched configurations with different sampling budgets?"

Response: We compare methods on identical initial pools with k=3 responses per model, alongside k=4 voting baselines using additional CoT sampling. “Agreement gate + LLM selection” retains the agreement gate and one trace per distinct answer, but omits normalization and injection. Accuracy is averaged equally across the three benchmarks and then across the four model combinations, as in Table 2; costs are per-instance means covering initial sampling, normalization of candidate and injected traces, and all decision calls and retries. All methods use the same single-GPU, fully serial latency assumption.

MethodkAccuracy (%, benchmark average)Total tokensEstimated serial latency (s)
Combined MV372.2178.2k5,086
SLM-MUX370.9178.2k5,086
Agreement gate + LLM selection374.8199.0k5,514
Ours376.9219.8k5,930
Combined MV472.1239.6k6,846
SLM-MUX470.8239.6k6,846

Unlike the budget-matched comparison in Table 2, this comparison uses identical k=3 pools. Ours improves accuracy over Combined MV by 4.7 percentage points, adding 41.6k tokens (+23.3%) and 844 s of estimated serial latency (+16.6%). Compared with the agreement gate and LLM selection alone, normalization and injection improve accuracy by 2.1 points. Ours also outperforms the k=4 voting baselines with 8.3% fewer tokens and 13.4% lower estimated serial latency.

Additional computation is selective: 46.0% of instances require no additional LLM calls. The table reports overall mean costs, with conditional means for instances reaching each stage in parentheses. Conditional values are derived from the reported means and execution rates.

Aggregation stageExecution rateTokensEstimated serial latency (s)
Style normalization, including injected traces54.0%7.8k (14.4k)106 (196)
Bias-controlled decisions54.0%22.1k (40.9k)474 (878)
Bias-injected decisions23.3%11.7k (50.2k)263 (1,129)
Aggregation total54.0%41.6k (77.0k)844 (1,563)

Latency is estimated by summing per-call processing times derived from actual input/output token counts and model-specific processing rates measured on an RTX 6000 Ada, assuming sequential execution of all calls and retries (Totals may differ from sums of rounded entries.). Our cost audit identified previously omitted normalizer’s thinking tokens and output tokens from failed attempts; including them revises Ours’ total from 212.3k to 219.8k (+3.5%). All costs above use the corrected accounting. These identical-pool comparisons and stage-wise overall and conditional mean costs are now reported in Appendix D.5 (Tables 14 and 15), together with the token-accounting and latency-estimation procedures.

Comment: "Furthermore, the evaluation focuses on relatively small reasoning models, leaving the benefits for stronger models uncertain."

"Q4. Stronger reasoning models: Do the accuracy and efficiency gains persist when using substantially stronger, more recent reasoning models (e.g., Qwen3.8)?"

Response: We evaluated two stronger-model combinations on all 60 HMMT problems, the most challenging benchmark in our original experiments. Both combinations include Qwen3.8-27B and Qwen3.6-35B-A3B, together with either gemma-4-31B-it (Gemma4) or NVIDIA-Nemotron-3.5-Lightning-30B-A3B (Nemotron3.5). We compare Ours with Combined MV at nearby token budgets.

Model combinationMethodkAccuracy (%)Tokens/problem
Qwen3.8–Qwen3.6–Gemma4Combined MV576.7231.9k
Ours380.0250.8k
Qwen3.8–Qwen3.6–Nemotron3.5Combined MV473.3259.4k
Ours378.3251.7k

SLM-MUX yields the same accuracy and token cost as Combined MV under the reported settings in both combinations. For Nemotron3.5, we follow the budget-matching rule in Table 2. For Gemma4, the largest available voting pool uses slightly fewer tokens than Ours (7.5% lower). Ours achieves higher observed accuracy by 3.3 and 5.0 percentage points, respectively, although neither difference is statistically significant (p = 0.51 and 0.25). Token efficiency varies across the two combinations: Ours uses more tokens in the Gemma4 combination and 3.0% fewer in the Nemotron3.5 combination. These results and budget-matching details are now reported in Appendix D.7 (Table 20).


The additional controls, decision-reliability analysis, and stronger-model results provide a fuller assessment of the method’s benefits and limitations. The revised manuscript includes the paraphrase comparisons and prompt (Figures 9 and 13), decision-reliability analysis (Appendix D.3), efficiency analysis (Appendix D.5), and stronger-model results (Appendix D.7), alongside the revised interpretation of bias injection (Sections 3.2 and 6.2). We kindly ask you to raise your rating in light of these additional results and revisions.

Dear Reviewer UJFT,

Thank you again for your feedback. Since our full response is long, here are the main concerns and the new results and revisions we added.


Summary of our response

1. Contribution of bias injection and decision reliability (W1/Q1–Q2)

  • New paraphrase control: To better distinguish frequency and reasoning-content effects, we added a paraphrase control designed to increase competing-answer frequency while preserving the reasoning content of an existing trace. On the common injection-stage subset, final-answer accuracy is 39.2% with repeated inference, 40.3% with paraphrasing, and 41.6% with a distinct trace. The distinct-trace gains over the other two conditions are not statistically significant. We also clarify that these controls do not fully separate frequency and reasoning-content effects (Revised/added in Sections 3.2 and 6.2; Appendix D.3).

  • More initially incorrect decisions are corrected. In the decision-recovery analysis, injection corrects 24.8% of initially incorrect agent decisions, compared with 18.8% under repeated inference.

2. Efficiency with identical initial reasoning pools (W2/Q3)

  • Higher accuracy from the same initial responses: Using identical k = 3 response pools, our method achieves 76.9% accuracy vs. 72.2% for Combined MV, with 23.3% more total tokens.

  • Higher accuracy at lower cost than additional CoT sampling: Compared with the k = 4 voting baselines, our method achieves higher accuracy with 8.3% fewer tokens and 13.4% lower estimated serial latency.

  • New stage-wise breakdown of aggregation costs: We now report execution rates, token usage, and estimated serial latency for each aggregation stage, using corrected token accounting (Added in Appendix D.5).

3. “Stronger reasoning models” (W2/Q4)

  • New evaluation with two stronger-model combinations including Qwen3.8: On all 60 HMMT problems, our method achieves higher observed accuracy than Combined MV in both combinations: 80.0% vs. 76.7% and 78.3% vs. 73.3%, at nearby token budgets (these differences are not statistically significant). Appendix D.7 reports the results and budget details.


As the author–reviewer discussion phase draws to a close, we would like to follow up and see if you have any remaining concerns or questions that we could clarify. A brief acknowledgement that you have read our response would be appreciated. Thank you for your time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We deeply appreciate the time and effort the reviewers devoted to our manuscript. The reviewers noted that our framework resolves disagreements among LLM agents without validation data or prior knowledge of which model is strongest (W7Vp, Dmr2). They also highlighted our use of adverse-frequency injection to test decision stability and help identify which answers to trust (all three reviewers). Across three mathematical reasoning benchmarks, the full framework achieves higher average accuracy while using fewer tokens. Below, we summarize how the additional experiments address the reviewers’ questions and concerns.

How we addressed the concerns

  1. How does the full framework compare with majority voting over a larger pool of solutions? (UJFT Q3, W7Vp Q2) Across three benchmarks and four model combinations, our framework achieves 76.9% accuracy using three responses per model, versus 72.1% for Combined MV using four. It does so with 8.3% fewer total tokens, including normalization and retries.

  2. Does the gain extend beyond plain LLM selection? (W7Vp W1–W2) With identical initial responses, adding normalization and injection to the agreement gate and LLM selection improves accuracy by +2.1 [+1.25, +3.06] percentage points. The gain is positive in 11 of 12 benchmark–model combinations.

  3. What does injection add over re-asking? (UJFT Q1, W7Vp W3) On the 615 instances reaching the second pass, a distinct competing trace gives 41.6% accuracy, versus 39.2% for both re-asking and exact duplication. The gain over re-asking is modest and not statistically significant (p = .076). Since the distinct trace changes both answer frequency and reasoning content, we do not attribute the gain solely to frequency bias.

  4. Does the result extend to stronger models? (UJFT Q4) On HMMT, both stronger-model combinations achieved higher observed accuracy than voting, though neither difference was statistically significant.

Where to find the changes in the revision (Changes are marked in orange)

  • Main text: Sections 3.2 and 6.2 clarify the interpretation of injection. Table 4 and Figure 9 report the new paraphrase control. Section 6.3 reports component gains. Tables 2 and 8 and Figure 5 use corrected token counts.
  • Appendix: C.2 and D.2–D.3 cover verifier diagnostics, style sensitivity, decision reliability, and plurality voting. D.5–D.8 report additional results/analyses including stage-wise costs, paired statistical analyses, stronger-model results, and the candidate-ordering control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.