acceptodds
Under review as a conference paper at ICLR 2027

Agreement Before Diversity: Verification-First Complementarity for Heterogeneous Language-Model Coordination

Abstract

Heterogeneous language-model ensembles expand the space of candidate responses, yet they lack a principled criterion for when a newly generated answer should supersede an already supported one. We decouple candidate headroom from replacement authority, rendering the latter as an explicit, auditable object. Our proposed method, Agreement-Before-Diversity (ABD), is a frozen, label-free decision rule: an anchor answer is retained if two additional trusted samples corroborate it under a fixed equivalence relation; otherwise, it is replaced by a heterogeneous synthesis. For this gating mechanism, we prove two exact identities. The first shows that the accuracy gap relative to unconditional synthesis is determined jointly by the agreement coverage and the anchor's advantage on the protected subset. The second shows that the gap relative to never synthesizing reflects a contrast between authorized recovery and authorized destruction. Neither identity assumes independence or calibrated confidence, and the expected inference cost is approximately eight minus five times the coverage in number of calls. Under blind, exact-ID evaluation, ABD achieves 59.43% on the complete LiveCodeBench-v6 (vs. 52.57% for Single and 52.00% for HAC; n = 175) and 75.00% on an untouched GPQA-Diamond split (both controls at 72.78%; n = 180). Furthermore, these identities localize every aggregate difference to an enumerable protected stratum: no discordant items occur among the 3 protected cases on LiveCodeBench, where coverage bounds the gate's contribution to 1.71 points a priori; 13 versus 8 discordant cases among 132 on GPQA-Diamond; and 12 versus 0 among 71 under a frozen anchor perturbation. Diversity supplies potential; verification structure supplies authority.

Primary area:
optimization
Keywords:
large language models, test-time scaling, model ensembles, multi-agent systems, verification, selective prediction, model routing, mixture of agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8Ms6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a decision rule for using ensembles of LLM reasoning results. Stated simply, run a model 3 times. If the 3 answers are not equivalent, run other models 4 further times and synthesize an answer from those results. The authors test under-described models on two benchmarks and find modest gains for their method, at substantially increased inference costs.

Critical strengths
  • Paper proposes a way to increase the reliability of LLM answers.
Critical weaknesses
  • Submitted to optimization track. This paper does not handle optimization in the machine learning sense, it is entirely dedicated to deciding between using 3 LLM calls or 8.
  • Decision on number of ensemble calls is arbitrary and unsupported.
  • Choice of Anchor answer, to which other answers are compared, seems arbitrary and is not explored.
  • Models used are not stated in the main paper and only briefly mentioned in appendix. Since this paper proposes to explore alternatives to test time scaling, it is egregious to not discuss choice of model and capability.
  • Excessive dense notation used throughout, which disguises straightforward ideas and lack clarity. For example, the "fixed equivalence relation" means that answers are case and whitespace normalized to check string equality.
  • Devotes large amounts of paper space to superfluous proofs and analysis rather than checking experiments that would make the paper more interesting. For example, Section 3 is a long-winded proof of a simple decision tree with 2 branches: to submit the anchor answer or a synthesized one.
  • Section 5 examines win rates over two benchmarks rather than exploring other models, thinking time, or inference costs. Actual performance differences are very small.
  • Method requires running multiple models multiple times, which is highly inefficient for inference, rather than considering "depth-wise" test time compute. The synthesis answer also may result in extremely long sequences for inference, and repeated model calls mean that KV caching does not help.

Paper also states:

Executable results were checked through task-specific evaluators and indepen- dent replay; finite mathematical claims were verified by exact or exhaustive computation; and formal results were checked in Lean where indicated.

However there are no Lean checks distributed. This is indicative of over-reliance on AI assistance.

Questions for authors

N/A

Reviewer_JosSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ABD presents an interesting idea on coordinating heterogenous language models. The idea is clear, preserve an existing answer when additional samples corroborate it and invoke it otherwise. The validation is transparent which also include negative results.

Critical strengths

ABD requires no model fine-tuning, confidence calibration, or labeled training data. The approach can be applied to existing LLMs using a simple decision rule, making it relatively easy to reproduce and deploy.

Critical weaknesses

My concerns are: (1) The dependence of LLMs caused by distillation, sharing training data could weak the foundation of proposed approach. (2) Why additional 2 and why additional 4? The numbers sounds arbitary without theoretical guidance (3) The approach is fairly simple and the empirical evidence is insufficient to show significant benefits (4) Agreement does not equal to verification. LLM might repeatedly give same incorrect answer because of systematic bias. This will lead to incorrect answer without consuting heterogenous models. (5) The theoretical contribution is correct but not that novel.

Questions for authors

(1) Provide additional datasets while also showing there is significantly improvement from statistical perspective (2) Provide theoretical foundations for why addtional 2 for self-consistency.

Reviewer_PyrGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ABD, which keeps an anchor answer when three samples from the same model agree and otherwise invokes heterogeneous models for synthesis. It separates gains from preserving correct answers and correcting wrong ones, and evaluates the approach on coding and science questions.

Critical strengths

Avoiding unnecessary changes to a correct answer is a useful and practical objective. The proposed rule is simple to implement. The paired comparisons against the same synthesis pipeline help distinguish the contribution of the gate from that of heterogeneous generation.

Critical weaknesses

The main experiments provide limited evidence for the added gate. On LiveCodeBench, the gate protects only 3 of 175 examples, and ABD has exactly the same accuracy as always-on synthesis, at 59.43%. Its nominal average call count is approximately 7.91 rather than 6. On the original GPQA set, the comparison gives 13 wins and 8 losses, a net gain of 2.78 percentage points. These results are encouraging in places, but do not establish a reliable accuracy–cost advantage for the gate.

The distinction from existing agreement-based cascades is overstated. Agreement-Based Cascading already routes examples using agreement among model outputs [1], and Semantic Agreement Cascades use semantic consistency as a training-free deferral signal [2]. These methods make decisions after observing outputs, contrary to the paper’s broad characterization of earlier cascades. ABD’s asymmetric anchor protection and heterogeneous synthesis are meaningful differences, but their added value needs a direct comparison. [1] Kolawole et al. Agreement-Based Cascading for Efficient Inference. TMLR, 2025. [2] Soiffer, Kolawole, and Smith. Semantic Agreement Enables Efficient Open-Ended LLM Cascades. EMNLP Industry Track, 2025.

Questions for authors
  1. Does the gate retain an advantage on independent test sets when inference cost is matched and paired uncertainty is reported?
  2. Compared with agreement-based cascades, what improvement specifically comes from preserving a fixed anchor and escalating to heterogeneous synthesis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.