- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines plurality voting for repeated LLM responses with a fixed call limit. It distinguishes between two events: a correct answer occurs in the sample set, and the voting process selects that answer as the final output. The authors use the current vote totals and the number of remaining calls to calculate whether each candidate can still win. They show that, as sampling continues, the set of candidates that can win can only decrease. The authors also examine how the distribution of incorrect answers affects plurality accuracy. They propose a change to the input order for each call. This method increases the variation in incorrect answers.
Critical strengths
The empirical evaluation is complete and clear. It includes multiple models, tasks, and studies that the authors specified before the evaluation. The authors also report results that are inconclusive or negative.
Critical weaknesses
- The reachability threshold compares the differences between vote totals with the remaining call budget. The authors state that this method is the classical deterministic curtailment method from sequential analysis.
- The claim about the earliest safe stop follows directly from the definition of the certificate. The lock-time law uses standard renewal theory.
- The ensemble-learning literature has established that repeated and correlated incorrect answers decrease the accuracy of majority voting. Thus, the error-competition analysis gives little new conceptual information.
- The method operates only after sampling. It does not help the model find correct answers. Finding correct answers is the main unresolved problem in test-time scaling.
- Input permutation gives a large improvement on one synthetic task. However, this improvement is small on new MMLU-Pro inputs. Thus, the results do not show that this method applies generally.
Questions for authors
Please address the concerns in the Weaknesses section. Specifically, clearly explain how the concept differs from classical deterministic curtailment. Also explain how the framework can improve the sampling process and not only the aggregation process. These explanations could significantly change my assessment.
3 comments
Weakness 2: Earliest safe stop and lock-time law
The claim about the earliest safe stop follows directly from the definition of the certificate. The lock-time law uses standard renewal theory.
Response: We agree that the earliest-stop characterization is direct and that the renewal argument is standard, as the paper notes. The results matter for what they settle in practice.
Under the retained-unit-vote information model, the earliest-stop result gives a hard limit: no universally output-preserving prefix rule can save more than half of the calls (Theorem 5.3(iii)). For users of adaptive self-consistency (ASC) or early-stopping self-consistency (ESC), which save more, it settles what is traded: in the known-universe setting, any rule that stops before the certificate gives up the universal guarantee that its output equals the full-budget answer (Remark 5.2).
The law turns the saving into a prediction from a reference sample of the response law. Predictions match public GSM8K runs within 0.1 points at every budget from 16 to 1,024 (Appendix H.2, Table 23). The law also explains why savings on public MATH level off at 15.8% and 26.3%: the margins there are small (means 0.24 and 0.45), and the per-problem asymptotes average 15.75% and 26.25% (Table 24).
In LLM voting the savings of this rule have so far been reported from experiments alone, to our knowledge. What the paper adds for this setting is the almost-sure form, a central limit theorem with explicit variance , budget-consuming abstentions, the open-universe rule, validated finite-budget predictions and, since submission, dependent votes.
Weakness 3: Error competition and the ensemble literature
The ensemble-learning literature ...
Response: The ensemble literature already establishes the average statement: correlated errors hurt majority voting. Proposition 4.2 is sharper: whether concentrating errors hurts depends on the current vote state, and the effect can go in either direction.
Take one remaining call and probabilities for the correct answer and two wrong answers and . Moving 0.1 of probability from to makes the errors more diverse. It lowers the probability that the correct answer wins from 0.9 to 0.8 at vote counts and raises it from 0.4 to 0.5 at counts . No statistic of the response distribution alone, such as diversity, correlation or collision probability, can therefore order accuracy.
The data also caution against interpreting concentration alone. A validity filter on Word16 removed 85% of wrong votes and made the remaining wrong answers more concentrated (collision probability up by 0.24), yet accuracy rose from 74.2% to 92.2% and no input was damaged.
The state-dependent account is also predictive: where forecasts frozen before generation predicted an effect above 0.02 in magnitude, the sign was right in 29 of 32 states. More directly, the full-state forecast improves squared error over a control that preserves the wrong-answer probability spectrum and collision but shuffles the identities carrying that mass by 0.0030 [0.0004, 0.0068] (Appendix F.5, Table 12).
The revised Section 4.1 states this contrast with the ensemble literature directly after Proposition 4.2, and the related work repeats it.
Weakness 4 and Question: Improving the sampling process
Weakness 4: The method operates only after sampling. It does not help the model find correct answers. Finding correct answers is the main unresolved problem in test-time scaling.
Question: Also explain how the framework can improve the sampling process and not only the aggregation process. These explanations could significantly change my assessment.
Response: The framework changes what a sampler should be optimized for, in four concrete ways.
- Finding the correct answer is often not the binding step. At , between 2% and 46% of trajectories that sample the correct answer still end with a wrong output. On public MATH samples from Llama-3-70B, the correct answer is sampled in 80.0% of trajectories at a budget of 16 and in 95.2% at 1,024, while plurality accuracy moves from 58.6% to 60.4% (Appendix E.4, Table 6). At 1,024 calls, 4.8 points are lost to never finding the answer and 34.8 points to finding it and not selecting it. A sampler tuned for coverage alone leaves this gap untouched.
- The sampler gets a second target: which wrong answers it produces. At a fixed single-call accuracy of 0.3, splitting one wrong answer into 32 raises 16-call accuracy from 7.4% to 88.7%. Input permutation is a change on the sampling side. On Word16 it raises accuracy by 21.1 points [14.1, 28.1] with no change in mean single-call accuracy, which is consistent with this second target. On 384 fresh MMLU-Pro inputs it also raises the share of inputs on which the correct answer is sampled, from 84.9% to 89.1% ( points , Appendix G.4, Table 17).
- A sampling change can be scored from a reference sample. In a controlled recoding, the state forecast predicted that dispersing the leading wrong answer would raise accuracy by 1.03 points, and the observed gain was 1.06. For permutation, where forecasts predicted an effect, the sign was right in 29 of 32 states.
- Samples can be directed. Calls released by the lock on settled inputs can go to undecided ones. In a replay run after submission, this raises accuracy over a fixed 16-call budget by 0.6 points [0.2, 1.0] on GSM-Symbolic and 1.2 points [0.5, 1.7] on public MATH within the same total budget.
We agree that finding correct answers is central. In a large share of failures the answer was found and then lost, and changing the sampler's response distribution can affect that loss.
Weakness 5: Generality of input permutation
Input permutation gives a large improvement on one synthetic task. However, this improvement is small on new MMLU-Pro inputs. Thus, the results do not show that this method applies generally.
Response: We agree, and the paper makes no claim that permutation is a general method. It reports the transfer directly, including points on the 768-input MMLU-Pro hold-out. In the paper permutation is a controlled perturbation of error competition: it shifts the vote distribution while mean single-call accuracy stays the same.
Proposition 4.2 characterizes the effect of wrong-answer probability transfers under its stated assumptions, and we test whether the state-based calculation predicts intervention effects. A controlled recoding of the wrong answers on MMLU-Pro was predicted to raise accuracy by 1.03 points and raised it by 1.06 [0.46, 1.82]. In a prospective test on 128 MMLU-Pro inputs, forecasts of the permutation effect were frozen before generation. Where an effect was predicted, the sign was right in 29 of 32 states, and the other three states did not move. The same forecasts put the effect within in 224 of the 256 states, where the mean observed effect was 0.000 (Appendix F.5, Table 13). The small average effect on MMLU-Pro is what the state-based account predicts for these inputs. Reference samples of the two response laws, together with the gold label, therefore support offline forecasts of where a sampling change of this kind may help.
Section 4.2 of the revised paper now ends with a paragraph that states this scope.
If any part of the two explanations at the top of this response remains unclear, we would be glad to elaborate.
Short answers to the two explanations you ask for.
How does the concept differ from classical deterministic curtailment? The curtailment stopping criterion asks whether the current leader can still be overtaken. The paper tracks the set of answers that can still win, for every candidate at every call (Proposition 2.1). The stopping rule of Section 5 is the special case in which that set has one element, and there the two coincide. The additional analyses concern when candidates lose reachability and how their endpoint probabilities respond to changes in the response law. Of 553 failures in which the correct answer was sampled, 479 still had a reachable correct answer when it was found and lost it during the remaining calls (Appendix E.1, Table 3).
How can the framework improve sampling? It shows where the loss is and gives the sampler a target for it. On public MATH samples from Llama-3-70B at a budget of 16, 20.0 points of accuracy are lost because the correct answer is never sampled and 21.4 points because it is sampled and then not selected. At a budget of 1,024, more sampling has cut the first loss to 4.8 points, and the second has grown to 34.8 points (Appendix E.4, Table 6). Sampling more closes most of the discovery gap on this benchmark and leaves the larger loss in place. That loss depends on which wrong answers the sampler produces and how often (Section 4). A sampling-side change aimed at this target, input permutation, raised Word16 accuracy by 21.1 points [14.1,28.1], with a small transfer to MMLU-Pro that we discuss under Weakness 5. On 384 fresh MMLU-Pro inputs it also raised the share of inputs on which the correct answer is sampled, from 84.9% to 89.1% (+4.2 points [1.8,6.8], Appendix G.4, Table 17).
The responses below give the details for each weakness.
Weakness 1 and Question: Difference from classical deterministic curtailment
Response: The difference is the object of study. Curtailment is a stopping rule: it asks at each call whether the current leader can still be overtaken. Our analysis tracks, for every candidate answer at every call, whether that answer can still win. The stopping rule of Section 5 is the special case in which one answer remains, and it coincides with classical deterministic curtailment, as the paper states in its related-work section.
Tracking every candidate yields the set of answers that can still win and, for each answer, the last call at which it can still be rescued. Four results follow from this wider view.
Section 5 of the revised paper now names the rule as deterministic curtailment where the certificate is introduced, and Appendix J.1 (Table 32) contains the comparison above.