- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PairRecon, a training-free method for resolving disagreement among heterogeneous language models. Initial answers determine candidates and their support. Models compare candidate pairs under order-balanced presentations, and only consistent judgments become directed support and opposition relations. The method reconstructs a provisional answer by minimizing opposition, breaking ties with initial aggregation scores, and then applies candidate-admission rules or staged arbitration. Extra inference is used only for instances without unanimous initial agreement. Experiments cover six multiple-choice and three open-ended benchmarks, with further validation using a smaller model ensemble. On the primary multiple-choice test pool, PairRecon improves aggregate accuracy by 1.50 percentage points over direct aggregation of three models and by 0.57 percentage points over the strongest competing baseline. Ablations and correction/error-induction analyses examine local judgments and conservative revision.
Critical strengths
- The method makes effective use of incomplete and conflicting pairwise judgments instead of forcing a complete ranking or unconstrained debate. Explicitly retaining unresolved comparisons is an interpretable design choice.
- The empirical evaluation spans nine benchmarks, two model scales, and numerical, mathematical, and program-output tasks. Main comparisons use the same initial disagreement partition and report both accuracy and inference cost.
- Correction/error-induction decomposition and reconstruction analysis provide useful evidence beyond aggregate accuracy. On the main test pool, PairRecon corrects 285 initial errors while introducing 139; paired bootstrap intervals support the benefit of reconstruction over score-only selection.
Critical weaknesses
- Comparisons primarily cover sampling and debate methods, without directly testing alternative pairwise aggregation rules on the same collected judgments. Comparing minimum-opposition reconstruction to simple net-win voting or standard pairwise ranking rules would clarify what the reconstruction method adds beyond obtaining extra binary decisions.
- The admission and arbitration protocol contains several consequential heuristics, including a fixed score-gap threshold, minimum support threshold, and early-retention rule, but no sensitivity study is reported. Moreover, in the 1-1-1 path, the two orders are evaluated by different models, conflating model disagreement with order sensitivity; the single-order ablation does not modify this path.
- The principal margin over the strongest baseline is small, and uncertainty for the final method-versus-baseline comparison is not reported. The gain over Direct9 is 0.57 percentage points on the main test, and over ReConcile 0.23 points in the smaller ensemble; some component gains are below 0.1 points. Paired confidence intervals and repeated inference runs would clarify robustness.
- Generalization on open-ended tasks is compared only with Direct3, whereas broader sampling and debate comparisons are limited to multiple-choice tasks. Because open-ended arbitration is restricted to existing candidates and uses different input conventions, competitive baselines and cost reporting on these tasks are needed to establish a practical advantage.
Questions for authors
See Weaknesses.
Response to muaa [Part 1] — Parameter Choice and Threshold Sensitivity
● Main revisions and findings
Thank you for your specific comments on parameter settings, performance gains, and staged arbitration. We have clarified the method and added threshold sensitivity analyses for both ensembles, two complete inference reruns, and a control without early stopping. Key mechanisms and evidence now appear in §§3.3 and 4.1–4.4, with full results in Appendices B.1 and B.3.
This comment addresses parameter choice and sensitivity: both ensembles retain positive gains across all tested settings, and the defaults balance error correction and damage control. Part 2 reports statistical support for the gains and arbitration efficiency.
1. Parameter choice: defaults correspond to explicit decision conditions.
Sections 3.3 and 4.1 clarify that τ=2 and δ=10 are empirical defaults established during development, unchanged across both MCQ ensembles and the additional runs.
τ governs support-threshold admission on the 2-1 route and arbitration-based revision on the 1-1-1 route. τ=2 is the smallest threshold requiring repeated support: in these branches, 2-1 admission requires at least two retained supporting relations and no opposing relations; 1-1-1 revision requires at least two votes for an alternative and no votes supporting the provisional answer.
δ controls only the single-relation admission exception for MCQ tasks: replacement is allowed when the reconstructed candidate has one supporting relation, no opposition, and an aggregate log-score deficit relative to the initial answer of at most δ. δ=10 is the empirical setting for this tolerance.
2. Support threshold τ: repeated support controls rescues and damages.
With model judgments and other decision rules fixed, we evaluate 21 configurations per ensemble: 18 combinations of τ∈{1,2,3} and δ∈{0,5,10,15,20,∞}, plus disabling the single-relation exception for each τ. ∞ removes only the exception's score-gap bound.
All configurations retain gains over Direct3: disagreement accuracy gains range from 6.33%–6.89% for the main ensemble and 4.89%–6.50% for the smaller ensemble. At δ=10:
Rescues, damages, and net gains are relative to Direct3. Increasing τ reduces both rescues and damages, showing that the support requirement controls revision behavior. At δ=10, τ=2 achieves the largest net gain in both ensembles.
3. Score tolerance δ: stable gains near the default, with additional rescues from single-relation admission.
At τ=2, the full δ sweep gives:
Relaxing δ increases both rescues and damages. The main ensemble gains 142, 146, and 142 net correct answers at δ=5,10,15; the smaller ensemble gains 214 at each of δ=5,10,20, showing stable gains near the default.
Even with the single-relation exception disabled, the ensembles retain net gains of 140 and 211 correct answers, versus 146 and 214 under the defaults. The main gains persist without the exception, while single-relation admission provides additional opportunities for correction. Figure 5 shows parameter responses; Table 11 in Appendix B.3 reports the full grids for both ensembles.