acceptodds
Under review as a conference paper at ICLR 2027

Arrival-Conditioned Voting Power in Heterogeneous Language Model Ensembles

Abstract

Large language model (LLM) ensembles can reduce latency by voting before every response finishes. With heterogeneous models, completion order also determines whose responses count. We derive how this selection changes the response distribution and final decision, separating changes in model participation from selection within each model. Across two disjoint five-model panels spanning seven benchmarks, voting on the first three responses lowers selected-response accuracy by 5.09 and 11.84 percentage points. Changes in model participation explain most of both losses, and exhaustive subset controls distinguish arrival selection from the effect of using fewer votes. Changing delivery times alone realizes all 128 predicted decision changes with fixed responses. We then derive a completion-invariant rule: commit when every possible completion gives the same full-panel decision. This rule matches all 672 full-panel decisions while reducing mean latency by 40.83% and 22.49% on the two panels. Completion order thus changes effective voting weights, but some latency savings remain available without changing the full-panel decision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM ensembles, Selection bias, Early stopping, Inference efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Bs9jOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates arrival-conditioned voting in heterogeneous LLM ensembles, showing that selecting the earliest model responses can introduce systematic participation bias and degrade ensemble accuracy. The authors derive a probabilistic decomposition separating identity reweighting from within-model selection effects and propose a completion-invariant early stopping rule that preserves the full-panel plurality decision. Experiments across two five-model panels and seven benchmarks demonstrate the impact of arrival-based selection and report latency reductions of 40.83% and 22.49% while preserving full-panel decisions.

Critical strengths
  1. A relevant and practical problem. The paper highlights an often-overlooked issue in latency-aware LLM ensembles: early stopping can systematically favor faster but less accurate models, even when nominal voting weights are equal.

  2. Careful experimental analysis. The experiments include exhaustive subset comparisons, a held-out model panel, delay-only interventions, and shared-GPU evaluations, providing multiple perspectives on arrival-conditioned selection.

  3. Decision-preserving early stopping. The completion-invariant stopping rule provides an exact guarantee of agreement with full-panel plurality while achieving measurable latency reductions.

Critical weaknesses
  1. Unnecessarily complicated writing and poor organization. The paper relies heavily on abstract mathematical formalism without sufficient intuitive explanations. For example, Section 3.1 introduces marked arrival processes and filtrations before explaining the basic first-k voting scenario. Sections 3.2–3.3 present multiple probability laws and kernels without concrete examples illustrating their practical meaning. Moreover, Section 3.4 combines delay-based decision manipulation and completion-invariant early stopping, two conceptually different topics, without a clear transition. The practically important plurality certificate is deferred to Appendix B.10, leaving the main-text method difficult to understand.

  2. Limited theoretical novelty. Equations (3)–(5) are essentially conditioning and covariance identities, as the authors explicitly acknowledge in Appendix H.3. Their exact numerical reconstruction in the experiments is therefore expected and does not independently validate a new theoretical model. Similarly, the correctness of the completion-invariant stopping condition in Equation (7) follows directly from its definition. While Proposition 6 provides an efficient implementation, the overall theoretical contribution appears relatively incremental.

  3. Imprecise mathematical presentation of latency and delay guarantees. The reported latency reductions average per-request relative reductions (Equation 33), which are not generally equivalent to percentage reductions in mean latency, as suggested by the Abstract. Additionally, the discussion of Equation (6) refers to a minimum delay, whereas the derived quantity is an infimum that may not be attained under certain tie-breaking configurations. These distinctions should be clarified.

Questions for authors
  1. Could the authors better distinguish the substantive theoretical contributions from standard probabilistic identities? In particular, beyond the exact decomposition in Equation (4), what new predictive or explanatory capability does the proposed formalism provide?

  2. Could the authors clarify the latency metric and report absolute latency measurements? Specifically, how do the reported results change when latency reduction is calculated from the ratio of mean completion times rather than the mean of per-request relative reductions?

2 comments

Thank you for the detailed review. Your points on presentation and precision were correct, and we have fixed each of them. New and rewritten text is blue in the revised PDF.

W1. "Unnecessarily complicated writing and poor organization."

Section 3 has been rewritten (lines 123–196). It now opens with the first-k scenario on a measured vote sequence, 6, 12, 3, 12, 12. Voting after three arrivals returns 6, and waiting for all five returns 12. First-k is then defined in plain notation. Marked arrival processes, filtrations and kernels appear only in Appendix B.

Before Eq. 1 there is a two-model example. With accuracies 0.6 and 0.9 and inclusion probabilities 0.9 and 0.1, the selected response is correct with probability 0.63 instead of 0.75. Swapping the inclusion probabilities gives 0.87.

Early stopping and delay are now separate subsections. Section 3.2 asks when the full decision is already determined. Section 3.3 opens by saying that delay is a separate question in which the answers are held fixed.

The certificate that was in Appendix B.10 is now Eq. 3 in the main text. It states the check the controller runs at each arrival: the current winner must still win after any rival is given all the pending weight and the best pending tie-break rank. Section 5.2 works through it on the example sequence (line 370).

W2. "Limited theoretical novelty."

We agree, and the paper no longer presents these results as new theory. The former Eqs. 3–5 are called standard conditioning and covariance identities and are attributed to Horvitz–Thompson and Price. That they reproduce the observed shift is now described as an arithmetic check (line 269). The former Eq. 7 is identified as necessary-winner determination (Konczak & Lang 2005; Xia & Conitzer 2011), and we cite Li & Li (arXiv:2610.01014) for exact locking of weighted LLM votes. Propositions 6 and 7 are presented as the implementation of that test for identity-priority ties, open-ended answers and abstentions.

What we claim is empirical. Completion order changes which models count. An exhaustive random-three control separates this from the effect of using fewer votes. Delivery delay alone changes decisions while the responses are held fixed. An early commit with real cancellation saves measured time and tokens. Our answer to Q1 describes what the formalism adds beyond the algebra.

W3. "Imprecise mathematical presentation of latency and delay guarantees."

Both remarks are correct.

Write C for the commit time, F for the full completion time and r = 1 − C/F. The reduction in mean latency is 1 − E[C]/E[F] = E[r] + Cov(F, r)/E[F]. It equals the mean per-request reduction E[r] only when r is uncorrelated with F. Our abstract used the first wording for the second quantity.

The abstract, Figure 1 and the conclusion now report the reduction in mean latency. Tables 1 and 2 keep the earlier values under the label "per-request reduction". Section 4 defines both quantities (line 229), and Appendix I.2 lists both with absolute times. The numbers are under Q2.

For the delay, the text now says "infimum" throughout, and Section 3.3 notes that it need not be attained (line 194). The argument is in Appendix B. Delaying each outsider to t_S + ε achieves the target subset for every ε > 0, but at exactly t_S the arrival tie may be resolved against the target.

Q1. "what new predictive or explanatory capability does the proposed formalism provide?"

You are right that an identity which reproduces the data it was computed from predicts nothing. The revision adds tests that the decomposition could have failed.

The most direct one is an out-of-sample forecast (new Appendix J.11). We compute the identity term on calibration questions and use it to forecast the shift on 672 unseen test questions, for every subset of two to five models and every k. That gives 98 configurations. No test label or test time enters the forecast. Forecast and measured shift correlate at r = 0.58 (95% CI 0.34 to 0.69). Their signs agree in 75.5% of configurations: 85.7% on choice and 65.3% on reasoning.

The misses on reasoning have one cause, and the second term of the decomposition names it. The within-model term is negative in 48 of the 49 reasoning configurations, −3.59 pp on average, so a forecast from the identity term alone is too optimistic there. We checked this term on its own by sampling several responses from a single model, where the identity term is zero. The first three of five samples are less accurate by 6.67 and 5.21 pp (1,536 new responses).

The decomposition also says when the sign should flip. In the new collection the most accurate reasoning model enters 60.07% of first-three sets, against 0% originally. The identity term turns positive (+2.63 pp) and the response shift is zero (CI −1.64 to 1.62). Reversing the speed ranks on the original choice panel turns −11.84 pp into +14.55 pp.

Finally, the analysis shows what response-level quantities cannot settle. Inclusion rates and accuracies fix the sign of the response shift on the original choice panel, [−17.92, −7.50] pp, but leave the decision shift open, [−88.28, +19.01] pp. Adding the recorded answer and arrival patterns identifies a negative decision shift on both original panels (Appendices J.8 and J.9).

There are limits. An earlier leave-one-benchmark-out forecast has the wrong sign on the MATH subset under the Appendix G protocol, and with only a third of the labels no sign is identified.

Q2. "Could the authors clarify the latency metric and report absolute latency measurements?"

RegimeFull voteEarly commitMean per-request reduction, E[1 − C/F]Reduction in mean latency, 1 − E[C]/E[F] (95% CI)
Reasoning, original27.06 s16.30 s40.83%39.76% (34.70 to 44.67)
Choice, original23.14 s17.81 s22.49%23.03% (19.56 to 26.59)
Shared GPU50.44 s44.83 s12.91%11.11% (9.27 to 12.38)

The ratio-of-means values are slightly lower on reasoning and on the shared GPU and slightly higher on choice. No conclusion changes.

The original submission had no physical execution with cancellation, so we added it. Five models share one GPU with first-in first-out queues, and a commit cancels running calls and skips queued ones. Mean commit time falls from 25.591 s to 23.016 s on reasoning (10.06%, CI 5.58 to 15.19) and from 5.656 s to 4.008 s on choice (29.13%, CI 23.12 to 35.63), with 6.80% and 15.48% fewer generated tokens. All 192 early decisions equal the full vote. The abstract now reports these numbers. Tables 30, 32–35, 43–44 and 47–51 give seconds for every policy and every single model.

Beyond your questions, the revision adds 672 new test questions with all policies fixed on separate calibration questions, and learned baselines (RouteLLM, FrugalGPT, Adaptive-Consistency). A weighted vote calibrated on those questions is more accurate than the best single model on reasoning by 6.94 pp (CI 3.47 to 10.76) and less accurate on choice by 4.69 pp. The general response gives the overview.

Reviewer_CodTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how completion-order-based selection changes the composition and decisions of heterogeneous LLM ensembles. It derives a selected-response distribution and an exact decomposition of response-quality changes into identity reweighting and within-model selection. It also characterizes the nonnegative delivery delay needed to realize a target first-k subset and develops a completion-invariant stopping rule that preserves the full-panel plurality decision. Experiments use two disjoint five-model panels across seven benchmarks, exhaustive three-model subset controls, response-preserving latency remappings, and fixed-payload ingress replay. Experimental results demonstrate that the stopping criterion proposed in this paper—while maintaining decisions identical to those of the fully integrated system—reduces the average relative latency per task by 40.83% and 22.49%, respectively.

Critical strengths
  1. The exhaustive uniform-random-three baseline is an important strength: it separates the effect of using fewer votes from the additional effect of selecting votes by arrival order. The paper also correctly avoids inferring nonlinear decision behavior from selected-response accuracy alone.

  2. Careful, transparent experimental hygiene. The paper uses full shadow traces, task-cluster bootstrap, sharp bounds for unresolved answers, a parser frozen before held-out generation, all 120 tie-priority orders, all values of k, and a full artifact inventory.

Critical weaknesses
  1. Limited theoretical novelty and insufficient positioning relative to prior work.

Eqs. 3–5 follow directly from standard conditioning and covariance identities, while Eq. 6 follows from an elementary threshold argument. These results may be useful diagnostics, but their mathematical novelty appears limited. More importantly, the plurality instantiation of completion invariance closely matches necessary-winner determination under incomplete voting information, for which polynomial-time results are established (Konczak & Lang, 2005; Xia & Conitzer, 2011). The paper should clarify what Proposition 6 adds beyond this literature.

  1. The deployment value of the preserved target is not sufficiently established. Full-five plurality has lower observed accuracy than the best constituent model in both panels: 56.94% versus 59.72% in the reasoning panel, and 69.27% versus 79.17% in the held-out panel. This does not invalidate the decision-equivalence guarantee, but it leaves unclear whether accelerating this target offers a competitive deployment trade-off. The paper should report constituent-model accuracy–latency operating points and compare against validation-selected single-model and accuracy-weighted alternatives, clearly separating these system-level comparisons from fixed-target stopping comparisons.
Questions for authors
  1. The original reasoning experiment and Appendix G use the same task set, yet full-five accuracy changes from 56.94% to 59.03%, and Llama’s first-three inclusion changes from 0% to 28.82% even under the paired isolated condition. Could you quantify the differences in normalized model-task outputs between these experiments and clarify the relevant differences in decoding configuration, chat templates, batching, and timing measurement? Which deployment regimes do the two protocols represent, and how should their substantially different latency reductions be interpreted?

  2. Could you report constituent-model accuracy and latency under comparable execution settings, alongside completion-invariant commit, to establish whether the proposed policy offers a competitive accuracy–latency trade-off?

3 comments

Thank you for the precise review and for the pointer to the necessary-winner literature. New and rewritten text is blue in the revised PDF.

W1. "Limited theoretical novelty and insufficient positioning relative to prior work."

We agree with this assessment, and the positioning has changed. Sections 2 and 3.2 now state that completion-invariant commit is necessary-winner determination under incomplete votes, and cite Konczak & Lang (2005) and Xia & Conitzer (2011) where the rule is defined. We claim neither the principle nor its tractability. The former Eqs. 3–5 are described as standard conditioning and covariance identities, and the fact that they reproduce the observed shift is now called an arithmetic check. We also cite Li & Li (arXiv:2610.01014), posted after our submission, which studies exact locking for LLM votes, including weighted ones.

You asked what Proposition 6 adds beyond this literature. Together with the new Proposition 7, it is the form of the necessary-winner test that this controller needs, and no more than that. Ties in our panels are broken by a fixed order over the models, so a pending model brings both a weight and a tie-breaking rank, and the test becomes the lexicographic comparison in Eq. 3. Answers are open-ended, so the set of labels is not known in advance; one hypothetical unseen label is enough to cover it. Abstentions can be excluded from the eligible answers. The test runs in O(n+m) and is checked against full enumeration.

Your review notes that the paper avoids inferring decision behavior from response accuracy alone. We tried to make this precise (Appendices J.8 and J.9). Given only each model's inclusion rate and accuracy, sharp bounds fix the sign of the response shift on the original choice panel, [−17.92, −7.50] pp. The same information leaves the decision shift open, [−88.28, +19.01] pp. Once the recorded answer and arrival patterns are added, the decision shift is identified as negative on both original panels, [−26.04, −11.04] and [−17.36, −2.66] pp. The sign is not identified in the shared-execution or new-question regimes.

W2. "The deployment value of the preserved target is not sufficiently established."

We agree that unweighted full-five voting is not a competitive target on these panels. Following your suggestion, the revision keeps system-level comparisons apart from fixed-target stopping comparisons (Sections 3.2 and 5.5, Appendices I.3 and J). All policies are fixed on calibration questions.

EvaluationWeighted vote, early commitBest single model, chosen on calibrationAccuracy difference (95% CI)
Reasoning, original 144 questions67.36%, 21.66 s59.72%, 27.04 s+7.64 pp (2.08 to 13.19)
Reasoning, 288 new questions68.75%, 50.92 s61.81%, 43.29 s+6.94 pp (3.47 to 10.76)
Choice, original 192 questions76.56%, 18.99 s79.17%, 21.19 s−2.60 pp (−6.77 to 1.56)
Choice, 384 new questions67.19%, 15.49 s71.88%, 11.89 s−4.69 pp (−7.81 to −1.56)

With calibrated weights the vote is more accurate than the selected single model on reasoning in both evaluations. It is faster in the original run and slower in the new collection. On choice the single model is preferable.

The stopping test applies to whichever target is chosen, and all 1,344 early commits on the new questions agree with their own full targets. Times come from parallel replay of isolated workers, and the ensemble uses more total computation than one model. Constituent models and the other baselines are under Q2.

Q1. "Could you quantify the differences in normalized model-task outputs between these experiments"

We compared the two runs row by row (Appendix I.7, Table 41). Of the 1,440 aligned generations, 672 token sequences, 142 normalized answer labels and 62 correctness values differ. That is 336, 71 and 31 of the 720 model-question pairs.

On configuration, both runs use the same checkpoint revisions, the same prompt text rendered with each model's own chat template (Qwen3 in non-thinking mode), bfloat16 arithmetic and the same attention implementation. Prompt token lengths are identical in every row. The runs differ in decoding path, batching and clocks. The original run decodes with an explicit cached argmax loop, loads one model at a time and uses batch size 144; it records completion from device events. The Appendix G run uses native greedy generation with all five models resident and batch size 24; it records completion from host end-of-sequence callbacks. These factors change together, so we do not attribute the output differences to any one of them.

Two comparisons are clean. Within Appendix G, isolated and shared execution produce identical tokens, so the move of Llama's inclusion from 28.82% to 36.11% comes from completion order alone. And when we cross the two response sets with the two clock sets, clocks account for 3.47 to 4.51 pp of the change in first-three accuracy and responses for 1.04 pp (Appendix J.7). The full-vote accuracies of 56.94% and 59.03% therefore belong to different sets of outputs.

As for the regimes, the original protocol corresponds to each model running on its own device. Completion times are then far apart, and an early commit saves the most. Shared execution corresponds to five models competing for one device, which brings their completion times closer together.

RegimeFull voteEarly commitReduction in mean latency
Original, one device per model (reasoning)27.06 s16.30 s39.76%
Appendix G, shared GPU50.44 s44.83 s11.11%
New: shared GPU, queues and real cancellation (reasoning)25.591 s23.016 s10.06%
New: same setup, choice5.656 s4.008 s29.13%

The saving depends on how far apart the completions are, so we report each regime on its own. In every regime the early decisions match the full target.

Q2. "Could you report constituent-model accuracy and latency under comparable execution settings"

They are now reported for every regime: Tables 32–35 for the original and Appendix G runs, Tables 43–44 for the new questions, and Tables 47–48 and 50–51 for single models served on their own. On the 288 new reasoning questions:

PolicyAccuracyTime
Llama 3.1 8B (best single model)61.81%43.29 s
Qwen3 0.6B51.74%33.70 s
Full original plurality56.60%64.35 s
Original plurality, early commit56.60%51.83 s
Weighted vote, early commit68.75%50.92 s
Fallback cascade, best model first70.14%46.63 s
Calibration-selected subset64.58%53.10 s
RouteLLM, calibrated59.38%41.62 s

For a fixed target, the early commit lowers latency and never changes a decision. Whether the accelerated target is competitive depends on the target and the task. On reasoning, the weighted vote and a simple fallback cascade are both more accurate than the best single model, and both are slower. On choice the best single model is more accurate, and in the new collection it is also faster. In the small physical cohort of 24 questions per panel, a single model served alone is faster and at least as accurate (Table 3).

The revised paper states this trade-off in Section 5.5 and no longer suggests that plurality should be the deployed target.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. However, a few follow-up question remain.

  1. In the W2 table of the rebuttal, Resolved weighted, early commit (68.75%, 50.92s) outperforms the best single model Llama 3.1 8B (61.81%, 43.29s) by +6.94 pp. However, in the Q2 table (and Table 59), a simple Fallback cascade (Parallel Endpoint Fallback) reaches 70.14% accuracy in 46.63s, which strictly dominates your calibrated weighted voting in both accuracy and latency. Why is that? Is the ensemble gain here mainly due to abstentions caused by the 256-token truncation limit?

  2. In Table 43, why is the 0.5B model actually slower than the 8B model?

Reviewer_aeb2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLM ensembles often vote on whichever responses finish first, and when the models differ, the fast ones get counted far more often than the slow ones. Therefore, the authors come up with arrival-conditioned selection, which treats each response as a (completion time, answer, model) event and determine voting power based on selected response law, effective voting power, designing a decision strategy with delay-only intervention and safe early commit. The experiments show that completion invariance saves more time and maintaining high decision accuracy.

Critical strengths

(1) The paper focuses on a previously overlooked problem and shows that in latency-aware ensembles, finish order quietly sets each model’s real voting weight, even when every model nominally has one vote.

(2) The authors have developed practical early-stopping rule to alleviate the problem.

(3) The proposed strategies have matched all 672 full-panel decisions while cutting mean latency by 40.83% and 22.49%, and it is also immune to the delay-based manipulation the paper demonstrates.

Critical weaknesses

(1) The headline accuracy loss is largely connected to the panel design: in both panels, the strongest models are always the last to finish, and the authors’ own reverse-rank remapping (Table 20) turns the -11.84 loss into a +14.55 gain. Therefore the "arrival first-k hurts" conclusion are tied to the panel design rather than a general finding. The authors should consider test out different strategies on panels where speed and quality are not tied to size.

(2) While the paper diagnoses why early arrival hurts, the proposed options offer little in the way of a fix. All three options it discusses either accept the bias or fall back to full-panel voting, which is itself worse than the best single model here. Nothing gives the stronger models more weight or helps them get counted.

(3) There are some simple baselines that the authors should consider: such as accuracy-weighted voting, preselecting the best models on a validation set, routing or cascades, and existing adaptive stopping methods.

(4) The theorems offer limited novelty but just heavy formalism. For example, Eq 4 is just mean-propensity and covariance decomposition behind Horvitz-Thompson and Price-equation decompositions, which are not novel results.

(5) Each benchmark contributes only 48 questions. That creates much noise and wide confidence intervals. For example, the reasoning panel, 8.12 arrival-vs-random effect has a CI between 3.68 and 12.71.

Questions for authors

(1) The problem itself seems not well-motivated. For the reasoning panel, the full-5 plurality decision accuracy reaches 56.94% but the single best model reaches 59.72%.The best single model beats full-5 plurality decision making on both accuracy and latency. Full-5 must wait for the slowest model, and Llama is never in the first three, so it is always among the last.

(2) Why do the authors use unrealistic latency evaluation, by measuring the time from models run in isolation and replaying as if they have started together. The early commit does not consider compute savings either, due to no cancellation and no queueing.

(3) In the experiments, the two greedy executions produce identical outputs, so the 288 and 384 traces correspond to 144 and 192 independent tasks? Greedy decoding means each model gives one answer per task. Do the authors test within-model term for settings such as sampled self-consistency?

(6) Isn't Eq.6 also very idealized and unrealistic setting, as Eq.6 is computed from complted traces and attackers already know every answer and finish time?

3 comments

Thank you for the careful review. Your comments asked for specific tests, and we ran them. New and rewritten text is blue in the revised PDF.

W1. "The headline accuracy loss is largely connected to the panel design"

We agree that the sign is not universal. The revised Section 3.1 says that Eq. 1 is "a diagnostic identity, not a universal prediction of harm". What we claim to be general is narrower: voting on early arrivals reweights the models, and the direction follows how speed and accuracy line up in the panel. We tested this where speed is not tied to model size.

First, we collected 672 new test questions on a different device and batch setting. There Llama 3.1 8B, the largest and most accurate reasoning model, enters 60.07% of first-three sets, against 0% in the original run. The identity term of Eq. 1 turns positive (+2.63 pp), a within-model term of −2.63 pp cancels it, and the response shift is zero. The choice panel keeps a shift of −4.98 pp (95% CI −6.58 to −3.42).

Second, we went through every sub-panel. For each subset of two to five models and each k, we computed the identity term on separate calibration questions and compared it with the shift measured on the test questions. No test label or test time enters this forecast. Across the 98 configurations the two correlate at r = 0.58 (CI 0.34 to 0.69) and have the same sign in 75.5% of cases: 85.7% on choice and 65.3% on reasoning. Twenty-two of the measured shifts are positive. This is the new Appendix J.11 (Table 60).

Third, we removed model size altogether by sampling several responses from one model: 1,536 new responses on 96 new GSM8K questions. Taking the first three of five samples still lowers response accuracy, by 6.67 pp (CI 3.75 to 9.65) for Qwen2.5 0.5B and 5.21 pp (CI 2.57 to 7.99) for Qwen3 0.6B.

W2. "Nothing gives the stronger models more weight or helps them get counted."

The revision adds this. The stopping test, now Eq. 3 in the main text, works for any nonnegative weights fixed before the request arrives. Stronger models can therefore carry more weight, and the controller can still commit early without ever disagreeing with the full weighted vote. We set the weights to calibration accuracy and do not count abstentions as votes.

EvaluationWeighted vote, early commitBest single model, chosen on calibrationDifference (95% CI)
Reasoning, original 144 questions67.36%59.72%+7.64 pp (2.08 to 13.19)
Reasoning, 288 new questions68.75%61.81%+6.94 pp (3.47 to 10.76)
Choice, original 192 questions76.56%79.17%−2.60 pp (−6.77 to 1.56)
Choice, 384 new questions67.19%71.88%−4.69 pp (−7.81 to −1.56)

The weighted vote is more accurate than the best single model on reasoning in both evaluations. It is less accurate on choice, where one model is far ahead of the others. All 1,344 early commits on the new questions agree with their full votes.

W3. "There are some simple baselines that the authors should consider"

The revision includes each of them, and every choice they need is made on calibration questions only: accuracy-weighted voting, a calibration-selected single model and subset, RouteLLM with its released router, a FrugalGPT cascade with learned scorers, and Adaptive-Consistency with its published stopping rule.

On the new reasoning questions the accuracies are 68.75% for the weighted vote, 64.58% for the selected subset, 61.81% for the selected single model, 59.38% for RouteLLM, 56.60% for Adaptive-Consistency and 45.14% for FrugalGPT. A plain fallback cascade, which asks the best model first and moves on only when it gives no answer, reaches 70.14%. It is the strongest alternative we found, and Section 5.5 reports it. Tables 43–44 list every policy.

Among stopping rules, Adaptive-Consistency is the closest to ours. It never disagreed with the full vote in our data, but it stops later. It saves 7.00% and 2.91% of mean time on the two panels, against 19.47% and 22.21% for our early commit.

W4. "The theorems offer limited novelty but just heavy formalism."

We agree. The revision attributes Eq. 1 to standard covariance algebra (Horvitz–Thompson, Price) and the stopping rule to necessary-winner determination (Konczak & Lang 2005; Xia & Conitzer 2011). The point-process formalism has moved to Appendix B, and Section 3 now starts from a measured example. The contribution we claim is empirical.

W5. "Each benchmark contributes only 48 questions."

We added 96 unused test questions per benchmark, 672 in total, kept separate from the 336 calibration questions. Table 45 gives intervals per benchmark. The choice result holds up: first-three loses 5.31 pp of decision accuracy against random-three (CI 2.37 to 8.36). The reasoning result does not recur in the new collection, for the reason given under W1.

Q1. "The problem itself seems not well-motivated."

You are right that the best single model beats unweighted full-five voting on these panels, and the revision no longer treats that vote as the system to deploy. The motivation rests on two other points.

The selection effect does not depend on plurality being a good policy. Any controller that decides from the responses that arrive first, whether by first-k, a quorum or hedged requests, changes which models count. This holds whether or not the full vote beats a single model.

Choosing a target and deciding when its result is fixed are also separate steps (Sections 3.2 and 5.5). With calibrated weights the target is more accurate than the best single model on reasoning, as in the table under W2. In the original run it was also faster, 21.66 s against 27.04 s, because the best single model there is the slowest one. In the new collection it is slower, 50.92 s against 43.29 s, and on choice the single model is the better option. An ensemble also runs five models, and we do not claim that it uses less computation than one.

Q2. "Why do the authors use unrealistic latency evaluation"

This was a fair criticism, and we now measure it directly. Five models stay loaded on one GPU, each with a first-in first-out queue. When the controller commits, running calls are cancelled and queued calls never start. The run has 216 execution units and 864 decisions.

PanelFull voteEarly commitReduction in mean latency (95% CI)Fewer generated tokensShorter workload duration
Reasoning25.591 s23.016 s10.06% (5.58 to 15.19)6.80%8.63%
Choice5.656 s4.008 s29.13% (23.12 to 35.63)15.48%15.14%

All 192 early decisions equal their full votes. The abstract and the conclusion now report these measured numbers in place of the replay numbers.

The cohort is small: 24 questions per panel, in six workloads that are each run twice. We therefore use it to measure saved work and not accuracy. A single strong model served alone is faster on this cohort (Table 3). The earlier shared-GPU experiment, which had contention but no cancellation, went from 50.44 s to 44.83 s, a reduction of 11.11%.

Q3. "Do the authors test within-model term for settings such as sampled self-consistency?"

On the number of tasks you are right. The independent units are 144 and 192 questions, every interval resamples questions, and Section 4 now says so. The 672 new questions have no repeats.

We also ran the sampling experiment. Two models each produce eight sampled responses, at temperature 0.7, to 96 new GSM8K questions. All samples of a model come from one response distribution, so the identity term is zero and only selection within the model remains. The first three of five samples are less accurate by 6.67 and 5.21 pp, and the first three of eight by 11.41 and 7.77 pp (Table 40).

The same pattern appears across models. The within-model term is negative in 48 of the 49 reasoning sub-panels (Appendix J.11). On these reasoning tasks, a response that finishes early is more often wrong.

Q6. "Isn't Eq.6 also very idealized and unrealistic setting"

Yes. It needs the complete trace, and the revision presents it as an offline measure of which decisions are reachable, not as an attack (Section 3.3).

To test something that could run online, we added a policy that never sees answers or future completion times. Calibration questions decide which models to hold back, and a held response is released once three others have arrived. In physical execution this changes 33.33% of reasoning decisions and 45.83% of choice decisions.

When the held models are chosen to lower accuracy, replay on the 672 new questions shows a loss of 8.33 pp (CI 4.17 to 12.50) on reasoning and 3.91 pp (CI 1.30 to 6.51) on choice. In the small physical cohort the same policy gives −4.17 pp on reasoning, with an interval that reaches zero, and no change on choice.

So delivery timing alone changes decisions without an oracle, while a reliable loss of accuracy is shown only in replay. Section 5.5 is worded accordingly. The early commit rule is not affected by delivery order, because it waits until no remaining response can change the result.

I thank the authors for the extensive revision and the candor of the rebuttal. However, the authors' response largely confirm my original concerns.

  1. Previously W4: The original paper's theory is weak, and it is now mainly empirical findings. The authors admitted their math is standard: the decomposition is textbook covariance, and the stopping rule is an existing voting-theory method.

  2. Previously W1/W5: The experiments are weaker than claimed. The paper’s headline is that voting on whichever answers arrive first costs accuracy: 5.09 points on choice tasks and 11.84 on reasoning. On the 672 new questions, the reasoning loss disappears. On smaller panels, it points the expected way only 65% of the time.

  3. Previously Q1: Simpler alternatives win in the authors’ own results. On new reasoning questions, a fallback cascade is both more accurate ((70.14% vs. 68.75%) and faster than their method (46.63 s vs. 50.92 s). On choice tasks and in real execution, the best single model is at least as accurate and faster.

  4. Previously W3: Adaptive-stopping baselines remain incomplete. Adaptive-Consistency is the only adaptive-stopping comparison.

Comments on the paper

1 comment

We thank the three reviewers. All three recognized the participation problem, the exhaustive subset control and decision-preserving stopping. All three also raised the same concerns: the mathematics is standard, the vote we preserve may not be worth preserving, and the latency evaluation was not realistic. We agree with each, and the revision responds to them. A revised PDF is uploaded, with new and rewritten text in blue. The main text is 10 pages, as allowed in the discussion phase.

What the paper claims after revision

Completion order changes which models count in an early vote. This effect can be measured, separated from the effect of using fewer votes, and forecast from calibration data. Once a target vote is chosen, committing early without changing its decision saves measured time and computation. We no longer present the identities or the stopping rule as new theory, and we no longer suggest that unweighted plurality should be the deployed target.

The mathematics is standard (aeb2 W4, CodT W1, Bs9j W2)

We agree. The decomposition is attributed to standard covariance algebra (Horvitz–Thompson, Price) and the stopping rule to necessary-winner determination (Konczak & Lang 2005; Xia & Conitzer 2011). Section 3 now starts from a measured example and gives the computable stopping test in the main text; the general formalism is in Appendix B. To check that the decomposition says something about unseen data, a new analysis forecasts the selection shift on 672 new questions from calibration data alone, over all 98 sub-panels. Forecast and measurement correlate at r = 0.58 (95% CI 0.34 to 0.69) and agree in sign in 75.5% of cases (Appendix J.11).

Is the preserved target useful? (aeb2 W2, W3, Q1; CodT W2, Q2)

We added 336 calibration questions and 672 unused test questions, and fixed every policy on the calibration set. The baselines now include accuracy-weighted voting, a calibration-selected single model and subset, RouteLLM, FrugalGPT, Adaptive-Consistency and a fallback cascade. The stopping test applies to any nonnegative weighted vote. With calibrated weights, the vote is more accurate than the best single model on reasoning, by 7.64 pp on the original questions and 6.94 pp (CI 3.47 to 10.76) on the new ones. It is less accurate on choice, by 2.60 and 4.69 pp. All 1,344 early commits on the new questions agree with their full votes.

Realism of the latency evaluation (aeb2 Q2; CodT Q1; Bs9j W3, Q2)

We added physical execution with five models loaded on one GPU, first-in first-out queues and real cancellation: 216 execution units and 864 decisions. Early commit lowers mean commit time by 10.06% and 29.13%, generated tokens by 6.80% and 15.48%, and workload duration by 8.63% and 15.14%. All 192 early decisions agree with their full votes. We also corrected the latency wording. The abstract now reports the reduction in mean latency, and both latency summaries appear with absolute seconds.

Other additions

Sampling several responses from one model (1,536 new responses) isolates selection within a model, with no variation in model size. A row-by-row audit quantifies how the two decoding protocols differ in output. A delivery policy that sees only arrival events tests delay sensitivity without access to answers or future times.

Limits stated in the paper

On choice tasks the best single model is more accurate than the ensemble votes. In the small physical cohort of 24 questions per panel, a single model served alone is faster and at least as accurate. The selection effect on reasoning is near zero in the new collection, where the most accurate model is no longer slow, and the decomposition accounts for this. Delivery control changes decisions, but a reliable loss of accuracy is not established in physical execution. Energy and production throughput are not measured.

Where to find the changes

TopicLocation in the revised PDF
Rewritten Section 3 and positioningSection 3 (lines 123–196), Section 2 (lines 103–111)
New questions, calibrated votes, queued executionSection 5.5, Table 3
Absolute latency and both reduction metricsAppendix I.2, Table 30
All baselines and constituent modelsAppendix I.3 (Tables 31–39), Appendix J.1–J.3 (Tables 43–46)
Sampling within one modelAppendix I.5, Table 40
Sub-panels and the direction of the effectAppendix I.6, Appendix J.11 (Table 60)
Audit of the two decoding protocolsAppendix I.7 (Table 41), Appendix J.7
Queues, cancellation, standalone modelsAppendix J.4–J.5 (Tables 47–53)
Delivery control from arrival eventsAppendix I.8, Appendix J.6
What marginals and answer patterns identifyAppendix J.8–J.9

The Supplementary Material contains the code, raw generations, execution logs and all analysis outputs. Replies to each reviewer follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.