- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates arrival-conditioned voting in heterogeneous LLM ensembles, showing that selecting the earliest model responses can introduce systematic participation bias and degrade ensemble accuracy. The authors derive a probabilistic decomposition separating identity reweighting from within-model selection effects and propose a completion-invariant early stopping rule that preserves the full-panel plurality decision. Experiments across two five-model panels and seven benchmarks demonstrate the impact of arrival-based selection and report latency reductions of 40.83% and 22.49% while preserving full-panel decisions.
Critical strengths
-
A relevant and practical problem. The paper highlights an often-overlooked issue in latency-aware LLM ensembles: early stopping can systematically favor faster but less accurate models, even when nominal voting weights are equal.
-
Careful experimental analysis. The experiments include exhaustive subset comparisons, a held-out model panel, delay-only interventions, and shared-GPU evaluations, providing multiple perspectives on arrival-conditioned selection.
-
Decision-preserving early stopping. The completion-invariant stopping rule provides an exact guarantee of agreement with full-panel plurality while achieving measurable latency reductions.
Critical weaknesses
-
Unnecessarily complicated writing and poor organization. The paper relies heavily on abstract mathematical formalism without sufficient intuitive explanations. For example, Section 3.1 introduces marked arrival processes and filtrations before explaining the basic first-k voting scenario. Sections 3.2–3.3 present multiple probability laws and kernels without concrete examples illustrating their practical meaning. Moreover, Section 3.4 combines delay-based decision manipulation and completion-invariant early stopping, two conceptually different topics, without a clear transition. The practically important plurality certificate is deferred to Appendix B.10, leaving the main-text method difficult to understand.
-
Limited theoretical novelty. Equations (3)–(5) are essentially conditioning and covariance identities, as the authors explicitly acknowledge in Appendix H.3. Their exact numerical reconstruction in the experiments is therefore expected and does not independently validate a new theoretical model. Similarly, the correctness of the completion-invariant stopping condition in Equation (7) follows directly from its definition. While Proposition 6 provides an efficient implementation, the overall theoretical contribution appears relatively incremental.
-
Imprecise mathematical presentation of latency and delay guarantees. The reported latency reductions average per-request relative reductions (Equation 33), which are not generally equivalent to percentage reductions in mean latency, as suggested by the Abstract. Additionally, the discussion of Equation (6) refers to a minimum delay, whereas the derived quantity is an infimum that may not be attained under certain tie-breaking configurations. These distinctions should be clarified.
Questions for authors
-
Could the authors better distinguish the substantive theoretical contributions from standard probabilistic identities? In particular, beyond the exact decomposition in Equation (4), what new predictive or explanatory capability does the proposed formalism provide?
-
Could the authors clarify the latency metric and report absolute latency measurements? Specifically, how do the reported results change when latency reduction is calculated from the ratio of mean completion times rather than the mean of per-request relative reductions?
2 comments
Q1. "what new predictive or explanatory capability does the proposed formalism provide?"
You are right that an identity which reproduces the data it was computed from predicts nothing. The revision adds tests that the decomposition could have failed.
The most direct one is an out-of-sample forecast (new Appendix J.11). We compute the identity term on calibration questions and use it to forecast the shift on 672 unseen test questions, for every subset of two to five models and every k. That gives 98 configurations. No test label or test time enters the forecast. Forecast and measured shift correlate at r = 0.58 (95% CI 0.34 to 0.69). Their signs agree in 75.5% of configurations: 85.7% on choice and 65.3% on reasoning.
The misses on reasoning have one cause, and the second term of the decomposition names it. The within-model term is negative in 48 of the 49 reasoning configurations, −3.59 pp on average, so a forecast from the identity term alone is too optimistic there. We checked this term on its own by sampling several responses from a single model, where the identity term is zero. The first three of five samples are less accurate by 6.67 and 5.21 pp (1,536 new responses).
The decomposition also says when the sign should flip. In the new collection the most accurate reasoning model enters 60.07% of first-three sets, against 0% originally. The identity term turns positive (+2.63 pp) and the response shift is zero (CI −1.64 to 1.62). Reversing the speed ranks on the original choice panel turns −11.84 pp into +14.55 pp.
Finally, the analysis shows what response-level quantities cannot settle. Inclusion rates and accuracies fix the sign of the response shift on the original choice panel, [−17.92, −7.50] pp, but leave the decision shift open, [−88.28, +19.01] pp. Adding the recorded answer and arrival patterns identifies a negative decision shift on both original panels (Appendices J.8 and J.9).
There are limits. An earlier leave-one-benchmark-out forecast has the wrong sign on the MATH subset under the Appendix G protocol, and with only a third of the labels no sign is identified.
Q2. "Could the authors clarify the latency metric and report absolute latency measurements?"
| Regime | Full vote | Early commit | Mean per-request reduction, E[1 − C/F] | Reduction in mean latency, 1 − E[C]/E[F] (95% CI) |
|---|---|---|---|---|
| Reasoning, original | 27.06 s | 16.30 s | 40.83% | 39.76% (34.70 to 44.67) |
| Choice, original | 23.14 s | 17.81 s | 22.49% | 23.03% (19.56 to 26.59) |
| Shared GPU | 50.44 s | 44.83 s | 12.91% | 11.11% (9.27 to 12.38) |
The ratio-of-means values are slightly lower on reasoning and on the shared GPU and slightly higher on choice. No conclusion changes.
The original submission had no physical execution with cancellation, so we added it. Five models share one GPU with first-in first-out queues, and a commit cancels running calls and skips queued ones. Mean commit time falls from 25.591 s to 23.016 s on reasoning (10.06%, CI 5.58 to 15.19) and from 5.656 s to 4.008 s on choice (29.13%, CI 23.12 to 35.63), with 6.80% and 15.48% fewer generated tokens. All 192 early decisions equal the full vote. The abstract now reports these numbers. Tables 30, 32–35, 43–44 and 47–51 give seconds for every policy and every single model.
Beyond your questions, the revision adds 672 new test questions with all policies fixed on separate calibration questions, and learned baselines (RouteLLM, FrugalGPT, Adaptive-Consistency). A weighted vote calibrated on those questions is more accurate than the best single model on reasoning by 6.94 pp (CI 3.47 to 10.76) and less accurate on choice by 4.69 pp. The general response gives the overview.
Thank you for the detailed review. Your points on presentation and precision were correct, and we have fixed each of them. New and rewritten text is blue in the revised PDF.
W1. "Unnecessarily complicated writing and poor organization."
Section 3 has been rewritten (lines 123–196). It now opens with the first-k scenario on a measured vote sequence, 6, 12, 3, 12, 12. Voting after three arrivals returns 6, and waiting for all five returns 12. First-k is then defined in plain notation. Marked arrival processes, filtrations and kernels appear only in Appendix B.
Before Eq. 1 there is a two-model example. With accuracies 0.6 and 0.9 and inclusion probabilities 0.9 and 0.1, the selected response is correct with probability 0.63 instead of 0.75. Swapping the inclusion probabilities gives 0.87.
Early stopping and delay are now separate subsections. Section 3.2 asks when the full decision is already determined. Section 3.3 opens by saying that delay is a separate question in which the answers are held fixed.
The certificate that was in Appendix B.10 is now Eq. 3 in the main text. It states the check the controller runs at each arrival: the current winner must still win after any rival is given all the pending weight and the best pending tie-break rank. Section 5.2 works through it on the example sequence (line 370).
W2. "Limited theoretical novelty."
We agree, and the paper no longer presents these results as new theory. The former Eqs. 3–5 are called standard conditioning and covariance identities and are attributed to Horvitz–Thompson and Price. That they reproduce the observed shift is now described as an arithmetic check (line 269). The former Eq. 7 is identified as necessary-winner determination (Konczak & Lang 2005; Xia & Conitzer 2011), and we cite Li & Li (arXiv:2610.01014) for exact locking of weighted LLM votes. Propositions 6 and 7 are presented as the implementation of that test for identity-priority ties, open-ended answers and abstentions.
What we claim is empirical. Completion order changes which models count. An exhaustive random-three control separates this from the effect of using fewer votes. Delivery delay alone changes decisions while the responses are held fixed. An early commit with real cancellation saves measured time and tokens. Our answer to Q1 describes what the formalism adds beyond the algebra.
W3. "Imprecise mathematical presentation of latency and delay guarantees."
Both remarks are correct.
Write C for the commit time, F for the full completion time and r = 1 − C/F. The reduction in mean latency is 1 − E[C]/E[F] = E[r] + Cov(F, r)/E[F]. It equals the mean per-request reduction E[r] only when r is uncorrelated with F. Our abstract used the first wording for the second quantity.
The abstract, Figure 1 and the conclusion now report the reduction in mean latency. Tables 1 and 2 keep the earlier values under the label "per-request reduction". Section 4 defines both quantities (line 229), and Appendix I.2 lists both with absolute times. The numbers are under Q2.
For the delay, the text now says "infimum" throughout, and Section 3.3 notes that it need not be attained (line 194). The argument is in Appendix B. Delaying each outsider to t_S + ε achieves the target subset for every ε > 0, but at exactly t_S the arrival tie may be resolved against the target.