acceptodds
Under review as a conference paper at ICLR 2027

You Cannot Vote Your Way to the Truth: The Rank Law of Verification-Free Selection

Abstract

Repeated sampling is a standard way to spend inference-time compute: coverage, the probability that at least one of samples is correct, can keep increasing with . Yet a deployed system must return one answer without a gold label, and single-answer accuracy often saturates after only a few dozen samples. We characterize this selection ceiling by a single integer for each problem, : the mass rank of the correct answer in the model's per-problem output distribution. The count-ranked shortlist frontier is then a functional of the distribution of : self-consistency converges to , a top- shortlist to , and coverage to , with exponential finite- convergence. This turns the gap into an information price: an ideal target-aware channel with messages is worth exactly the Bayes top- posterior mass, the count-ranked shortlist is a measurable realization of that capacity and coincides with it under rank calibration, so resolving a top- count-ranked ambiguity costs oracle bits, which a real verifier need not collect. Across 10 model–task pairs, three model families, and budgets to , masses estimated from half of each problem's samples predict held-out selection accuracy across budgets and shortlist depths with median absolute error 0.002 and no fitted parameter. Across 12 bridgeable difficulty strata, a count-ranked four-answer shortlist plus two such ideal bits would recover 77–100% of the gap because, when plurality fails, the truth is usually the runner-up. Yet extracting those bits from the model is difficult: no count-based selector we test beats plurality, and only 1 of 26 implementable verifier conditions does; even there, spending the same tokens on a stronger generator performs better. Repeated sampling can therefore make the truth available without making it identifiable. The selection gap is not a decoding problem; it is missing information, and it can be priced.

Primary area:
learning theory
Keywords:
test-time compute, self-consistency, repeated sampling, reasoning, verification, answer selection, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_729qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies verifier-free repeated sampling from LLMs and the limits of answer-count based selection rules such as majority voting. The key quantity is R*, the rank of the correct answer in the sorted probability distribution over answers. The authors prove that self-consistency is always upper bounded by coverage, and the best count-based rule's utility sits in between both. Empirically, amongst other experiments based on the large language monkey paper's data, they plot the gap between coverage and self consistency, showing that it widens as the number of samples increases.

Critical strengths

Figure 1c) is a nice illustration of the limits of self-consistency

Critical weaknesses
  • The paper is exceptionally difficult to parse. For example, theorems and propositions are often quite long and technical, without being contextualized at all. Similarly, the abstract starts highly technical without providing a good overview of the papers' main conceptual ideas, and figure 1 has a very lengthy and difficult to read caption.
  • Many of the paper's claims seem trivial. For example, assumption 1 should be true by default for proper repeated sampling, but the paper highlights "testing it". Similarly, it is pretty obvious that verifier-free selection from answer counts on their own can in general not achieve perfect accuracy. Overall, I find it hard to identify particularly useful insights that have not been present in prior work.
Questions for authors
Reviewer_gxN9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the justification behind the improvement in the answer coverage as we increase the inference-time sampling while the accuracy of self-consistency voting does not necessarily improve. It characterizes the asymptotic success of count-ranked selection using the probability-mass rank of the correct answer and introduces an information-theoretic interpretation of the coverage–selection gap.

Critical strengths

Experiments are conducted across 10 model–task configurations, with up to 10^4 samples, provide substantial empirical support. The paper systematically check if alternative selection strategies and external verifiers can overcome the voting ceiling. The evaluation of 26 verifier conditions, including compute-matched comparisons, provides useful evidence about the limitations of current verification approaches.

Critical weaknesses

The paper suffers from the following serious issue:

(1) There are a couple of papers that show several of the paper's theoretical findings. The paper [1] already characterizes the easy vs. hard query distinction underlying the voting ceiling. This shows additional samples can systematically reinforce incorrect answers and proposing a model to predict voting performance from limited samples. Another work [2] analyzes self-consistency as a mode estimation and investigate its scaling behavior while [3] derives finite-sample guarantees for recovering the modal answer. These closely related results are not discussed. I find the the new elements of the paper more focused on the top-L shortlist extension, difficulty-stratified rank measurements, and empirical verifier analysis.

[1] Chen, L., et al. (2024). Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems. NeurIPS 2024.

[2] Feng, A., et al. (2026). Optimal Self-Consistency for Efficient Reasoning with Large Language Models. ICML 2026.

[3] Cordero-Encinar, P., & Duncan, A. B. (2025). Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs.

Questions for authors

Please check the Critical Weaknesses section.

Reviewer_KSrbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Repeated sampling from a LLM increases the probability that at least one sample is correct (coverage) when sample k increases, whereas the probability of choosing the correct answer without labels (selection accuracy) often saturates after only a few dozen draws. The gap between coverage and selection accuracy is governed by the mass rank R* of the correct answer. Closing such gap costs external information whose price can be measured in bits. In the empirical study, across 10 model-task pairs spanning three model families (Llama-3, Gemma, Qwen2.5) and two benchmarks (GSM8K, MATH), R* that comes from half of each problem’s samples can predict held-out selection accuracy with small median absolute error and requires no fitted parameter. Across 12 bridgeable difficulty strata, a count-ranked four-answer shortlist plus two such ideal bits would recover 77–100% of the gap because, when plurality fails, the truth is usually the runner-up.

Critical strengths

S-1: The paper provides a principled characterization of the coverage-selection gap through the mass rank R* and the guessing curve.

S-2: The zero-free-parameter empirical validation fits on half of a problem's samples and predicts the other half, which achieves median absolute error of 0.002 across 496 cell × budget × depth points with no fitted parameters. The predictive accuracy is remarkably strong.

Critical weaknesses

W-1: The paper tests only two math benchmarks (GSM8K and MATH/MATH-500) with models up to 70B parameters. Extending tasks to open-ended generation, coding, multi-turn conversations, or complex agentic workflows remains unvalidated.

W-2: The paper diagnoses the coverage-selection gap but offers no constructive method to actually close it. In practice, the authors’ solution merely shifts the problem to an external oracle. Exposing shortlists for external checks provides no mechanism on how to obtain oracle bits cheaply. Process reward models, latent verifiers could provide these bits, but the paper does not engage with this topic.

Questions for authors

Suggestions for the future work: The rank law assumes that outputs can be partitioned into discrete answer classes, exactly one of which is correct, so that R* is the count rank of that class. In tasks without a canonical answer form, such as code synthesis, two outputs can be judged equivalent at different granularities (e.g., exact match, or agreement on a weak versus a strong test set), and the resulting R* can change with this choice. For example, a coarse partition may merge correct and incorrect programs into one class. How would you define R* in this setting, and how to pick one answer from the class?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript: Presentation, Novelty, and Verification Clarifications

Thank you to all reviewers for their detailed feedback. We have uploaded a revised manuscript addressing the concerns raised during the discussion.

We have substantially improved the readability of the abstract, introduction, and theoretical statements while preserving the original results and structure. We also retained Figure 1(c), which illustrates the central intuition, and shortened its caption.

The revision explicitly distinguishes our contributions from prior voting theory, including Chen et al. (2024), Feng et al. (2026), and Cordero-Encinar and Duncan (2025). We have also clarified the distinction between shortlist inclusion, ideal oracle capacity, and achievable verification, incorporating the additional analyses and statistical qualifications discussed in our rebuttals.

The main changes appear in Sections 1–2 and 4–9, with supporting details in the appendices. We hope the revised presentation makes the paper's contributions easier to evaluate.

We appreciate the reviewers' consideration of the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.