- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies verifier-free repeated sampling from LLMs and the limits of answer-count based selection rules such as majority voting. The key quantity is R*, the rank of the correct answer in the sorted probability distribution over answers. The authors prove that self-consistency is always upper bounded by coverage, and the best count-based rule's utility sits in between both. Empirically, amongst other experiments based on the large language monkey paper's data, they plot the gap between coverage and self consistency, showing that it widens as the number of samples increases.
Critical strengths
Figure 1c) is a nice illustration of the limits of self-consistency
Critical weaknesses
- The paper is exceptionally difficult to parse. For example, theorems and propositions are often quite long and technical, without being contextualized at all. Similarly, the abstract starts highly technical without providing a good overview of the papers' main conceptual ideas, and figure 1 has a very lengthy and difficult to read caption.
- Many of the paper's claims seem trivial. For example, assumption 1 should be true by default for proper repeated sampling, but the paper highlights "testing it". Similarly, it is pretty obvious that verifier-free selection from answer counts on their own can in general not achieve perfect accuracy. Overall, I find it hard to identify particularly useful insights that have not been present in prior work.
Revised Manuscript: Presentation, Novelty, and Verification Clarifications
Thank you to all reviewers for their detailed feedback. We have uploaded a revised manuscript addressing the concerns raised during the discussion.
We have substantially improved the readability of the abstract, introduction, and theoretical statements while preserving the original results and structure. We also retained Figure 1(c), which illustrates the central intuition, and shortened its caption.
The revision explicitly distinguishes our contributions from prior voting theory, including Chen et al. (2024), Feng et al. (2026), and Cordero-Encinar and Duncan (2025). We have also clarified the distinction between shortlist inclusion, ideal oracle capacity, and achievable verification, incorporating the additional analyses and statistical qualifications discussed in our rebuttals.
The main changes appear in Sections 1–2 and 4–9, with supporting details in the appendices. We hope the revised presentation makes the paper's contributions easier to evaluate.
We appreciate the reviewers' consideration of the updated manuscript.