acceptodds
Under review as a conference paper at ICLR 2027

Who Is Worth Waiting For? Decision Closure in Heterogeneous Language Model Ensembles

Abstract

A heterogeneous language model ensemble can reach a fixed decision before all its models finish: individual outputs may remain uncertain even when every compatible combination gives the same answer. We use this condition to stop unfinished calls while preserving the complete-panel decision. Split conformal prediction converts native token prefixes into nested sets of possible terminal outputs. The controller stops when these sets imply a unique decision and, before then, cancels individual calls that can no longer change it. We prove reference-decision fidelity on the joint coverage event for any fixed deterministic rule over finite outputs. Experiments with ten open models on six public benchmarks show wall-time reductions of 34.9% to 53.6% in paired low-load trials. On two complete science-question test splits totaling 1,672 questions, both panels save 49.0% to 71.0% of generated tokens and retain both reference decisions on 99.8% of questions. Weighted voting and strict majority with deferral also benefit from the same stopping condition.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Computation, Language Model Ensembles, Adaptive Inference, Conformal Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3WVDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a controller that stops heterogeneous LLM ensemble early while preserving its fixed-rule decision. It maps token prefix of running call to a nested split-conformal set of possible outputs, stops once all compatible combinations give the same decision and cancels calls that can no longer change it. The authors provide a statistical guarantee of decision preservation and report substantial token savings with nearly perfect fidelity on multiple choice benchmarks.

Critical strengths
  • The formulation provides a simple stopping and cancellation criterion for any fixed deterministic aggregation rule and separates exact decision logic from conformal coverage.
  • The evaluation is detailed and transparent, covering complete test splits and itemizing every reference disagreement.
Critical weaknesses
  • The evaluation is limited to multiple-choice tasks with generation capped at 128 tokens. It is unclear whether the gains carry over to long-reasoning models where the answer appears late.
  • The method is not compared end-to-end with existing early-stopping methods so it is unclear whether it offers any practical advantage over these alternatives.
Questions for authors
  • Could the authors evaluate the method on long-reasoning models where the answer appears late?
  • Please clarify what finite decision representation the approach would use for open-ended generation.
  • Could the authors provide end-to-end comparison with existing early-stopping methods, such as MARS or Parallel-Probe, under a matched generation budget?
2 comments

This is a short update on your three questions. The revised paper is uploaded. Since our first response we measured wall time directly and compared the stopping rules on identical observations. Changes made for your review are marked in green.

Long reasoning with late answers (Weakness 1, Question 1)

We raised the cap from 128 to 4,096 tokens and used 128 fresh GSM8K questions. Answers do come late: the first answer that can be parsed from a prefix appears at a median of 1,280 tokens, and 92 of 640 calls reach the cap. Five-sample self-consistency still reaches its decision 28.2% sooner in wall time (95% interval 23.4 to 33.1), and none of the 128 decisions changes. The token saving is smaller, 13.2%, because the decision forms near the end of the reasoning. The time saving is larger because the complete panel must wait for its slowest call and the stopped panel does not. On BFCL tool calls with three models the time saving is 31.9%, with 0 of 104 decisions changed. Both are measured with one query at a time on one GPU, running the complete and the stopped execution of each question back to back (Table 3, Appendix S).

Representation for open-ended outputs (Question 2)

A fixed parser maps each response to an action: a number, a complete tool call with its arguments, or abstention. Over an unbounded vocabulary, the outputs still possible for a call form a finite set or the complement of one, and the stopping test stays exact without listing strings (Appendix P). For free-form dialogue the application must first declare the action. Agreement of surface text is not an action.

MARS and Parallel-Probe under a matched budget (Weakness 2, Question 3)

All methods use the same prompts, seeds, five calls and 4,096-token caps. To compare the stopping rules alone, we gave every rule the same observations: the answers visible in the same prefixes at the same reviews of the same 128 questions. The complete panel answers 113 correctly.

Rule, same observationsCorrectDecisions changedTokens savedDecode steps saved
Ours, with selective cancellation113015.3%25.2%
Parallel-Probe rule, released settings114111.9%23.0%
Parallel-Probe rule, most aggressive setting114229.9%42.5%
MARS rule, calibrated on development questions113329.7%42.3%

Our rule is the only one that changes no decision, and it saves more than Parallel-Probe at its released settings. The rules that save more stop on the current leader before the unfinished calls have shown an answer, and they change two or three decisions.

Run live with their own forced probes, Parallel-Probe saves 36.4% of tokens and changes 7 of 128 decisions. MARS saves 68.6% and changes 6, but its 16 warm-up traces per question raise its total token cost to about six times that of the complete panel when only five calls are used. On the Qwen3-8B traces released by the Parallel-Probe authors for AIME24, AIME25 and HMMT25, their policies save 20% to 29% and change 8% to 19% of decisions. Our exact rule changes none and saves 6.0% to 12.4% (Appendix T).

In practice the three rules differ in what they promise. Parallel-Probe targets accuracy and cost and does not bound how many answers change. The guarantee of MARS needs warm-up traces for every question. Our rule kept every answer in these tests and still cut latency by about a quarter, with no probes and no per-question preparation.

Do these results answer your three questions? If so, we would be grateful if you would update your score. If something is still open, please tell us what.

Reviewer_3WVDOpenReview ↗

Reviewer response

Thank you for detailed experiments and thorough response that address most of the concerns. I will keep my original score.

Reviewer_QmebOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Based on the current generation prefix of each model, the author estimates the set of potential final outputs for that model. These sets are then incorporated into a fixed voting rule: if all possible combinations yield the same outcome, the system returns the result early; if the final outcome remains undetermined, any model invocations that can no longer influence the ultimate decision are terminated.

Critical strengths

This paper proposes an early-stopping method for voting on multiple-choice questions using a heterogeneous ensemble of large models. By determining whether the remaining, unfinished models could still alter the final voting outcome, the approach terminates unnecessary model invocations early, thereby saving on generated tokens and reducing response time.

Critical weaknesses

1.Multi-model ensembling inherently introduces significant computational and deployment overhead; however, the paper fails to sufficiently demonstrate that these additional costs yield commensurate performance gains.

2.All experiments are confined to traditional multiple-choice datasets and do not encompass more representative large-model applicationssuch as open-ended dialogue, multi-turn agent interactions, or tool use leaving the practical scope of the proposed method unclear.

3.Furthermore, the paper lacks comparisons against a single, high-performing model operating under equivalent constraints (in terms of compute, token usage, or time budget).

Questions for authors

no question

2 comments

This is a short update. The revised paper is uploaded, and the three comparisons you asked for are done and reported in it. Changes made for your review are marked in orange.

Tool use and long reasoning (Weakness 2)

We ran the method on tool calling and on long mathematical reasoning. On BFCL v4 the decision is the complete tool call with its arguments, scored by the official AST checker. On GSM8K each call may reason for up to 4,096 tokens. We measured wall time with one query at a time on one GPU, running the complete and the stopped execution of each question back to back.

Task (held-out questions)Decisions changedTokens savedWall time saved (95% interval)
GSM8K, five Qwen3-1.7B samples (128)013.2%28.2% (23.4 to 33.1)
BFCL tool calls, three models (104)029.1%31.9% (27.2 to 36.5)

The stopping test works on any parsed action. Appendix P shows that it stays exact for numbers and tool calls without listing all possible strings. We did not evaluate dialogue or multi-turn agents. They need a declared action for each turn, such as the selected tool call, and Section 7 now states this limit.

A strong single model under the same budget (Weakness 3)

We gave single models exactly the time that the stopped panel took on each question, on the same GPU and the same 128 fresh GSM8K questions. At the deadline a model had to answer.

SystemCorrect in the panel's timeCorrect with no time limitTime with no limit
Five Qwen3-1.7B samples, stopped1071079.6 s (13.4 s without stopping)
One Qwen3-1.7B108108about 9.0 s
One Qwen3-4B117113about 17.0 s
One Qwen3-8B104118about 23.4 s

This result does not favor the ensemble. In equal time one Qwen3-4B beats five Qwen3-1.7B samples, 117 to 107 (exact McNemar p = 0.02). Qwen3-8B is too slow for this budget. Under the other budgets you listed the picture is mixed. With equal maximum tokens, the three-model panel beats the validation-selected single model 116 to 104, while one Qwen3-1.7B call with the allowance of five calls matches the five-sample panel, 114 to 113. With equal compute and a forced answer, Qwen3-8B scores between 116 and 124 on mathematics against 115 for the panel, and between 106 and 182 on tool calls against 193 (Appendices R and U). The 124 in our first response is the upper end of that range; that analysis used knowledge of where each answer ends.

Do the costs of an ensemble pay off? (Weakness 1)

It depends on the task, and Section 7 now says so. On the complete OpenBookQA and ARC-Challenge tests, Panel B answers 1,438 of 1,672 questions and the best single model chosen on validation data answers 1,409. On GSM8K in equal time, a single Qwen3-4B is better. Our method takes no side in that choice. It applies once someone has decided to run a panel, whether for accuracy, for the stability of self-consistency, or for robustness across different models. It then removes the part of the cost that cannot change the answer. Run to completion, Panel B generates 2.64 times the tokens of that single model. With stopping it generates 1.31 times and answers 1,439 (Appendix O). Self-consistency hosts one model only, so the deployment overhead in your first point does not arise there.

Do these results answer the three points? If a comparison you had in mind is still missing, please tell us which model and which budget. Otherwise we ask you to reconsider the rating.

Dear Reviewer Qmeb, the discussion closes very soon, so we ask once more. The three comparisons in your review are in the revised paper and are summarized in our comment above: tool calls and long reasoning (Table 3), a strong single model given the same time, tokens or compute (Appendices S and U), and the cost of the ensemble against its accuracy gain (Appendix O). One of the results goes against ensembles, and we report it as it is.

A one-line reply would help us and the Area Chair: do these results address your three points, or is something still missing?

Reviewer_AawUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents an early stop for a ensemble of LLMs when some individual LLMs are not yet finishing but sufficient models are given the same answers. This idea appears to be reasonable at the first glance but when look closely, it has several problems. a known problem with Ensemble learning is the failure cases when all LLMs in the ensemble agree but all wrong or the majority of LLMs agree and certain but with strong agreement and high confidence on the wrong result.

Critical strengths

(+) Ensemble of LLMs efficiently is an interesting topic.

(+) The paper presents an early stop for a ensemble of LLMs when some individual LLMs are not yet finishing by using so called reference decision fidelity on the joint coverage event.

(+) The experiments are conducted on five cases of agreements/disagreements: task decision disagreement, task raw coverage failure, same trace decision fidelity, live to shadow decision fidelity and closure/full accuracy.

Critical weaknesses

(-) The idea of early stop for a ensemble of LLMs when some individual LLMs are not yet finishing but sufficient models are given the same answers, though reasonable at the first glance but it fails to address a known problem with Ensemble learning is the failure cases when all LLMs in the ensemble agree but all wrong or the majority of LLMs agree and certain but with strong agreement and high confidence on the wrong result.

(-) The paper did not provide comparison to LLM ensemble literature with known LLM benchmark datasets and model pool.

(-) The methodology presented in Sections 3 & 4 (2 pages) fails to cover the cases where all LLMs in the ensemble agree to an wrong result and the cases where the guarantee is based on the large number of LLMs agree rather than whether they agree upon is not the correct result but the wrong result.

(-) The experimental evaluation is weak and insufficient given the large literature in recent years in NLP conferences and ICML/ICLR/NeurIPS conferences.

Questions for authors

Would appreciate your answers to those in the weakness section.

1 comment

We are following up because the revised paper is now uploaded and because one new count tests the exact failure case in your review. Changes made for your review are marked in blue.

Ensembles that agree on a wrong answer (Weaknesses 1 and 3)

The method does not treat agreement as evidence of correctness, and it does not try to repair a wrong ensemble. Its objective is stated in Section 3.1: "The reference action is D(Y1,…,Ym)D(Y_1,\ldots,Y_m) after all computations finish. Our objective is to reproduce this action earlier." It is an accelerator of the same kind as speculative decoding or caching. It returns what the complete panel would have returned, only sooner. When the panel agrees on a wrong answer, the stopped system returns that same wrong answer, and no new error is made.

We counted how often this happens. On the complete OpenBookQA and ARC-Challenge tests, with two panels, there are 3,344 panel-question cases. The complete panel is wrong in 653 of them. In 354 of these, at least three of the five models give the same wrong answer, and in 24 all five do. Stopping returns the identical answer in all 354, and in 651 of the 653. The two exceptions are split votes; in one of them stopping happens to return the correct answer.

The consequence for accuracy takes one line. Let AA be the complete decision, BB the stopped decision and ZZ the truth. BB can be right where AA is wrong, or wrong where AA is right, only when A≠BA\ne B. So ∣Pr⁡(B=Z)−Pr⁡(A=Z)∣≤Pr⁡(A≠B)≤α|\Pr(B=Z)-\Pr(A=Z)|\le\Pr(A\ne B)\le\alpha, however correlated the errors of the models are. This is now stated beside Theorem 1 in Section 4 and proved in Appendix O.

"the guarantee is based on the large number of LLMs agree" (Weakness 3)

The review summarizes the method as stopping when "sufficient models are given the same answers". That is not the test the paper uses. The stopping test does not count agreeing models. It asks whether every output still possible for the unfinished calls leads to the same decision (Definition 1). Two small cases show the difference. First, suppose two calls have finished with C and three calls are still unconstrained. A count would say that C leads, but we do not stop, because the three could all return B. Second, suppose the five calls can still end in {C}\lbrace C\rbrace, {C}\lbrace C\rbrace, {A,C}\lbrace A,C\rbrace, {B,C}\lbrace B,C\rbrace and {A,B,C}\lbrace A,B,C\rbrace, with ties going to the first call. All 12 possible outcomes give C, so we stop, although three calls are still undecided.

Comparison with the ensemble literature and the evaluation (Weaknesses 2 and 4)

The paper does not propose a new ensemble, so there is no new ensemble to rank against others. It speeds up voting ensembles that already exist: self-consistency (Wang et al., 2023), sampling-and-voting (Li et al., 2024), plurality over different models, weighted voting, and voting with deferral. Methods in which another model writes a fused answer, such as Mixture-of-Agents, have no fixed voting rule and are outside its scope. The comparable prior work is early stopping for parallel reasoning. The revision compares with Parallel-Probe and MARS on the same traces, and ours is the only rule that changes no decision (Appendix T).

The benchmarks are standard ones: ARC-Challenge, HellaSwag, MMLU, TruthfulQA, CommonsenseQA and OpenBookQA, with ten open models from seven families. The revision adds GSM8K with 4,096-token reasoning and BFCL tool calls. There, measured wall time falls by 28.2% and 31.9%, and none of the 232 held-out decisions changes (Table 3).

If you still see a soundness problem, could you point to the step of Theorem 1 or of its proof that fails, or name the method you would like us to compare with? We will answer promptly. If the points above settle the concern, we ask you to update the soundness score and the rating.

Comments on the paper

2 comments

We thank the reviewers. Each review has a self-contained response; this note summarizes the evidence they share.

What the method guarantees. The controller is an output-preserving accelerator for a fixed ensemble: it returns the decision that running every model to completion would return, right or wrong, with probability at least 1−α1-\alpha. Because correctness can change only where decisions change, stopping alters accuracy by at most that probability, for arbitrarily correlated model errors. Choosing an accurate ensemble is a separate design decision, which the method makes cheaper to deploy.

Measured latency on long reasoning and tool calls. One query at a time on one GPU, with the complete and stopped executions of each held-out question run back to back:

Setting (questions)Decisions changedGenerated tokens savedMeasured wall time saved (95% CI)
GSM8K, 5-sample self-consistency, 4,096-token reasoning (128 fresh)013.2%28.2% (23.4-33.1)
BFCL v4 tool calls with arguments, three models (104)029.1%31.9% (27.2-36.5)

Late answers reduce token savings, but the complete panel still waits for its slowest call, so latency savings remain.

Early-stopping baselines. Given identical native-prefix observations of the same 128 x 5 traces, our rule changes no decision and saves 25.2% of decode steps, versus 23.0% with one change for Parallel-Probe's released default rule; rules that save more change 2-7 of 128 decisions. On Parallel-Probe's own released traces, its policies change 8%-19% of decisions, while our exact rule changes none.

Single models. At the stopped panel's measured latency on the same GPU, a single Qwen3-4B answers more fresh GSM8K questions than five Qwen3-1.7B samples (117 vs 107), whereas a time-limited Qwen3-8B does not (104). Whether to deploy an ensemble is therefore task-dependent; we report every outcome, and closure lowers the cost of whichever panel is deployed.

Revision. The revised PDF states the accuracy bound beside the main theorem (Section 4, Appendix O), adds the latency results to the main experiments (Table 3), discusses ensemble choice (Section 7), and reports protocols, replays, timing records, costs, and coverage in Appendices R to U. The supplement adds the one-cell program and every new record. Changes are colored by reviewer (AawU blue, Qmeb orange, 3WVD green, several reviewers purple), each with a margin tag naming the addressed point; a revision map precedes Appendix A.

Thank you for replying. We answer the two points and the remark on novelty, and then return to soundness.

(1) Comparison with state-of-the-art LLM ensemble methods

Decision closure is not an ensemble method, so it has no accuracy of its own to compare. It is applied to a voting ensemble and returns that ensemble's answer sooner. Its accuracy is therefore the accuracy of that ensemble, up to the risk level α\alpha (Section 4, Appendix O). Putting it next to another ensemble method in an accuracy table would compare the two underlying ensembles and say nothing about our method. The comparison that does test it is each ensemble with and without it. Appendix B of the revised paper now collects these in Table 6:

EnsembleModels and taskDecisions unchangedSaving
Self-consistency, five samplesQwen3-1.7B, GSM8K, 4,096-token reasoning128 of 12828.2% of wall time
Plurality over five different modelstwo panels, complete OpenBookQA and ARC-Challenge3,341 of 3,34449.0% to 71.0% of tokens
Weighted votingsame panels and tests3,342 of 3,34452.6% to 68.4% of tokens
Majority with deferralsame panels and tests3,342 of 3,34456.5% and 57.9% of tokens
Plurality over three models, tool callsBFCL v4104 of 10431.9% of wall time

The methods that address the same problem as ours, stopping a running panel early, are Parallel-Probe (Zheng et al., 2026) and MARS (Chen et al., 2026). We compare with both on the same traces (Appendix T). We asked in both earlier responses which ensemble method is missing, and the reply still names none. If you name a method that aggregates answers by a fixed rule, we can add its row.

(2) More benchmarks and strong closed-source single models

The revised paper uses eight benchmarks and thirteen open models. We did not test closed-source models, and Appendix B now says so. A frontier closed-source model will certainly answer more of these questions than a panel of open models with one to eight billion parameters. The paper does not claim otherwise. Where we compared with stronger single models at equal time, a single Qwen3-4B beat the five-sample panel, and Section 7 reports it. Our claim is narrower: a panel that someone has chosen to run can return the same answers sooner, by 28.2% to 53.6% of wall time in our low-load measurements.

Novelty

The weaknesses in the review did not mention novelty, and the reply cites no earlier work, so we can only state what we believe is new. First, the stopping test is on the decision and not on the single outputs: we stop when every output still possible leads to the same decision, and this is exact for any fixed rule. Second, the same test tells which single running calls can no longer change the decision, and we prove that such calls can be cancelled together. Third, the calibration treats a whole question as one unit, which bounds the probability of changing the answer although the stopping time depends on the data. Parallel-Probe and MARS stop pools of samples from one model. Parallel-Probe uses a consensus heuristic and does not bound how many answers change. MARS bounds that risk, but it needs warm-up traces for each question and stops only the whole pool.

Soundness

Your reply no longer raises the two points on which the soundness score of 1 rested: ensembles that agree on a wrong answer, and a guarantee based on the number of agreeing models. It also names no step of Theorem 1 that fails. If these two points are settled, we ask you to update the soundness score, whatever your overall rating is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.