acceptodds
Under review as a conference paper at ICLR 2027

Global-to-Local Decision Projection for Collaborative Reasoning among Heterogeneous Large Language Models

Abstract

Complementarity among heterogeneous large language models (LLMs) underpins collaborative reasoning, yet reaching a reliable collective decision remains challenging when initial answers disagree. Majority support may favor an incorrect answer, while dispersed support may leave the choice unclear. We observe that models with incorrect initial answers may shift their support to the correct answer when directly comparing disputed candidates, but these judgments may conflict or provide clear support for only some candidate pairs. To this end, we introduce PairRecon, a global-to-local decision projection framework for collaborative reasoning among heterogeneous LLMs. PairRecon organizes collaboration using the candidates and the distribution of model support revealed by initial disagreement. Guided by this structure, it projects global candidate selection into local pairwise comparisons, enabling models to provide additional judgments on specific disputes. For each candidate pair, judgments that agree across both presentation orders are retained as directed relations supporting one candidate and opposing the other. These retained relations, together with initial support, enable candidate reconstruction despite unresolved local disagreements. Depending on the disagreement pattern, candidate admission or staged arbitration further constrains answer updates to reduce harmful revisions. PairRecon requires no training or parameter updates and allocates additional inference only to initial disagreement cases. Across nine benchmarks, PairRecon improves collective decision accuracy. In the primary evaluation, it improves overall and disagreement-case accuracy over direct three-model aggregation by 1.50% and 6.89%, respectively. It also achieves the greatest net gain and the lowest inference cost among evaluated methods requiring additional inference. Accuracy gains extend to smaller models and open-ended tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Collaborative Reasoning, Collective Decision-Making

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_k29zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PairRecon, a training-free method for resolving disagreement among heterogeneous language models. Initial answers determine candidates and their support. Models compare candidate pairs under order-balanced presentations, and only consistent judgments become directed support and opposition relations. The method reconstructs a provisional answer by minimizing opposition, breaking ties with initial aggregation scores, and then applies candidate-admission rules or staged arbitration. Extra inference is used only for instances without unanimous initial agreement. Experiments cover six multiple-choice and three open-ended benchmarks, with further validation using a smaller model ensemble. On the primary multiple-choice test pool, PairRecon improves aggregate accuracy by 1.50 percentage points over direct aggregation of three models and by 0.57 percentage points over the strongest competing baseline. Ablations and correction/error-induction analyses examine local judgments and conservative revision.

Critical strengths
  1. The method makes effective use of incomplete and conflicting pairwise judgments instead of forcing a complete ranking or unconstrained debate. Explicitly retaining unresolved comparisons is an interpretable design choice.
  2. The empirical evaluation spans nine benchmarks, two model scales, and numerical, mathematical, and program-output tasks. Main comparisons use the same initial disagreement partition and report both accuracy and inference cost.
  3. Correction/error-induction decomposition and reconstruction analysis provide useful evidence beyond aggregate accuracy. On the main test pool, PairRecon corrects 285 initial errors while introducing 139; paired bootstrap intervals support the benefit of reconstruction over score-only selection.
Critical weaknesses
  1. Comparisons primarily cover sampling and debate methods, without directly testing alternative pairwise aggregation rules on the same collected judgments. Comparing minimum-opposition reconstruction to simple net-win voting or standard pairwise ranking rules would clarify what the reconstruction method adds beyond obtaining extra binary decisions.
  2. The admission and arbitration protocol contains several consequential heuristics, including a fixed score-gap threshold, minimum support threshold, and early-retention rule, but no sensitivity study is reported. Moreover, in the 1-1-1 path, the two orders are evaluated by different models, conflating model disagreement with order sensitivity; the single-order ablation does not modify this path.
  3. The principal margin over the strongest baseline is small, and uncertainty for the final method-versus-baseline comparison is not reported. The gain over Direct9 is 0.57 percentage points on the main test, and over ReConcile 0.23 points in the smaller ensemble; some component gains are below 0.1 points. Paired confidence intervals and repeated inference runs would clarify robustness.
  4. Generalization on open-ended tasks is compared only with Direct3, whereas broader sampling and debate comparisons are limited to multiple-choice tasks. Because open-ended arbitration is restricted to existing candidates and uses different input conventions, competitive baselines and cost reporting on these tasks are needed to establish a practical advantage.
Questions for authors

See Weaknesses.

Reviewer_DNqBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims at addressing the problem of collective decision-making among heterogeneous large language models when their initial answers disagree. The authors propose PairRecon, a decision protocol motivated by the observation that a model giving an incorrect initial answer may still provide a correct local judgment when directly comparing disputed candidates. PairRecon detects initial disagreement, projects the global decision into targeted local pairwise comparisons, retains consistent relational judgments, and constructs a partial relation graph for candidate reconstruction. Candidate admission and staged arbitration are further utilized to control potentially harmful revisions. This method is evaluated on nine benchmarks covering multiple-choice and open-ended tasks. Notably, PairRecon improves accuracy from 54.39% to 61.28% on the 2118 initial-disagreement cases, corresponding to a 6.89-percentage-point gain over Direct3. The method also achieves favorable inference efficiency relative to other methods requiring additional inference.

Critical strengths

This paper aims to address a meaningful and practically relevant problem in heterogeneous LLM collaboration and proposes a novel and interesting protocol for resolving model disagreement. When initial answers provided by different LLMs disagree, reaching a reliable collective decision can be challenging. PairRecon aims at addressing this problem and is motivated by the empirical observation that a model with an incorrect initial answer may still provide a useful local pairwise judgment when directly comparing disputed candidates. This observation offers a reasonable basis for allocating additional inference to disagreement cases through targeted local comparisons, rather than uniformly increasing computation for all examples. This design makes better use of disagreement information, helps reduce unnecessary inference on consensus cases, and limits potentially harmful revisions.

The experimental evaluation is relatively comprehensive and goes beyond reporting aggregate accuracy. The authors evaluate PairRecon on nine benchmarks spanning multiple-choice and open-ended reasoning tasks and compare it with several representative collaborative reasoning baselines. The results show strong performance against the evaluated baselines, with particularly large gains in the challenging 1-1-1 disagreement cases. In addition to reporting aggregate performance, the paper analyzes the effects of decision revisions and presents ablation studies of the main components. It also evaluates generalization across model scales and inference efficiency. These analyses provide a well-rounded empirical validation of the proposed method and help clarify where the gains come from and how the revision mechanism affects final decisions.

Critical weaknesses

Some design choices in the candidate admission and revision procedure are not fully justified. While the ablation studies support the usefulness of the overall components, some specific choices, such as the fixed thresholds used in the decision update procedure, appear heuristic. The paper would benefit from clearer justification for these choices and, where possible, sensitivity analyses to show that the reported gains are not overly dependent on particular settings.

Questions for authors

How were the support threshold τ=2 and the score-gap threshold "δ=10" determined? Were they tuned on a validation set or chosen heuristically? How sensitive is the performance to these values?

Reviewer_muaaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PairRecon, a training-free collaborative reasoning procedure for heterogeneous LLMs. It converts disagreement among initial answers into counterbalanced pairwise candidate judgments, retains only order-consistent directed relations, reconstructs a candidate from the resulting partial graph, and conditionally revises the initial aggregate decision through admission rules or staged arbitration.

Critical strengths
  1. The paper identifies a useful distinction between a model's global initial answer and its ability to make a local relative judgment between two candidate answers. This is a plausible and practically useful observation for heterogeneous collaboration, especially because PairRecon only invokes extra inference in disagreement cases.

  2. Many models and datasets were used in the experiment, making the conclusions rigorous.

Critical weaknesses
  1. The thresholds τ and δ are not adequately justified or ablated.

  2. From the experimental results, the performance improvement is not significant, especially in Table 1.

  3. The staged arbitration procedure remains underspecified in the main text. This information is consequential because early stopping changes both decisions and inference cost.

Questions for authors
  1. Can you conduct further ablation experiments on τ and δ?
  2. Can you elaborate on the staged arbitration procedure in more detail?
1 comment

Response to muaa [Part 1] — Parameter Choice and Threshold Sensitivity

● Main revisions and findings

Thank you for your specific comments on parameter settings, performance gains, and staged arbitration. We have clarified the method and added threshold sensitivity analyses for both ensembles, two complete inference reruns, and a control without early stopping. Key mechanisms and evidence now appear in §§3.3 and 4.1–4.4, with full results in Appendices B.1 and B.3.

This comment addresses parameter choice and sensitivity: both ensembles retain positive gains across all tested settings, and the defaults balance error correction and damage control. Part 2 reports statistical support for the gains and arbitration efficiency.

1. Parameter choice: defaults correspond to explicit decision conditions.

Sections 3.3 and 4.1 clarify that τ=2 and δ=10 are empirical defaults established during development, unchanged across both MCQ ensembles and the additional runs.

τ governs support-threshold admission on the 2-1 route and arbitration-based revision on the 1-1-1 route. τ=2 is the smallest threshold requiring repeated support: in these branches, 2-1 admission requires at least two retained supporting relations and no opposing relations; 1-1-1 revision requires at least two votes for an alternative and no votes supporting the provisional answer.

δ controls only the single-relation admission exception for MCQ tasks: replacement is allowed when the reconstructed candidate has one supporting relation, no opposition, and an aggregate log-score deficit relative to the initial answer of at most δ. δ=10 is the empirical setting for this tolerance.

2. Support threshold τ: repeated support controls rescues and damages.

With model judgments and other decision rules fixed, we evaluate 21 configurations per ensemble: 18 combinations of τ∈{1,2,3} and δ∈{0,5,10,15,20,∞\infty}, plus disabling the single-relation exception for each τ. ∞\infty removes only the exception's score-gap bound.

All configurations retain gains over Direct3: disagreement accuracy gains range from 6.33%–6.89% for the main ensemble and 4.89%–6.50% for the smaller ensemble. At δ=10:

τMain disagr. accuracyRescues/Damages/NetSmaller disagr. accuracyRescues/Damages/Net
161.19%306/162/+14461.76%453/256/+197
261.28%285/139/+14662.27%407/193/+214
361.00%249/109/+14061.15%333/156/+177

Rescues, damages, and net gains are relative to Direct3. Increasing τ reduces both rescues and damages, showing that the support requirement controls revision behavior. At δ=10, τ=2 achieves the largest net gain in both ensembles.

3. Score tolerance δ: stable gains near the default, with additional rescues from single-relation admission.

At τ=2, the full δ sweep gives:

δMain disagr. accuracyRescues/Damages/NetSmaller disagr. accuracyRescues/Damages/Net
Disabled61.00%276/136/+14062.18%404/193/+211
061.00%276/136/+14062.18%404/193/+211
561.10%279/137/+14262.27%407/193/+214
1061.28%285/139/+14662.27%407/193/+214
1561.10%285/143/+14262.24%409/196/+213
2061.00%287/147/+14062.27%412/198/+214
∞\infty61.19%306/162/+14461.79%450/252/+198

Relaxing δ increases both rescues and damages. The main ensemble gains 142, 146, and 142 net correct answers at δ=5,10,15; the smaller ensemble gains 214 at each of δ=5,10,20, showing stable gains near the default.

Even with the single-relation exception disabled, the ensembles retain net gains of 140 and 211 correct answers, versus 146 and 214 under the defaults. The main gains persist without the exception, while single-relation admission provides additional opportunities for correction. Figure 5 shows parameter responses; Table 11 in Appendix B.3 reports the full grids for both ensembles.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.