- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper aims to characterize how aggregating (by unweighted majority vote) multiple predictions at test time can affect the results of the votes. The characterization is done through a mathematical classification of the accuracy curves that can arise as the number of votes increases. The paper provides a rigorous mathematical framework for it, and derives a voting signature formula that fully determines the curves studied.
Critical strengths
The mathematical framework developed for the study at hand looks very well-thought and very solid.
Critical weaknesses
The paper is only theoretical and makes little effort to explain how useful it could be in practical scenarios. In that vein, the title is a bit misleading, as it teases the reader with a practical question that is never really answered. Hence, although the math looks good, the motivation of the paper and its potential applications are underdeveloped and may prevent it from reaching a wide audience, as the contributions and take-home messages are unclear.
Questions for authors
Although the paper is exclusively theoretical, and likely good in that regard, I believe it would be much more impactful if it was clarified in various ways and if it was connected to practical applications and examples for the general machine learning / data science community. Can you please clarify, for a less-specialized audience:
- The goal of the paper: what problem is targeted by the paper?
- The motivation of the paper: why exactly is the paper important and needed?
- The contributions of the paper: what exactly is novel and is provided in the paper to achieve its goal?
- The practical uses of the paper: how exactly can/should the paper be used in practice?
5 comments
What was removed, and why not using page 10?
I thank the authors for their clear answers to my questions and to other reviewers' questions. I also appreciate the changes made in the paper in that regard, although they are quite minimalistic (maybe due to lack of space?).
I simply would like to draw the attention of the authors that the revised version of the paper, even in the rebuttal phase, can contain 10 pages. Given that the previous manuscript was already at 9 pages, and the current revision is still at 9, I am wondering which parts disappeared, and why?
Stated differently, I strongly encourage authors to keep their initial content if it was relevant, and to use fully the extra page allowed so that the clarifications asked by the reviewers are completely handled, and not too compressed (as it is a bit the case now).
I do not think that this comment will be difficult to address, and I am willing to increase my score once done, because I feel that the answers provided in the rebuttal are satisfying but do not yet fully transfer to the manuscript.
Thank you for the prompt and positive follow-up.
The revision now uses all ten pages. All theorems, propositions, equations and figures of the submission remain in the main text with their original numbers, and the few original sentences shortened in the first revision are restored (introduction; Sections 2, 4 and 5). The clarifications are now in the main text:
- Table 1 (Section 3) collects the notation.
- Section 5 explains the curves of Figure 1, what the infinite-reversal example means, and the source of the total-variation bound.
- Section 6 adds the budget comparison from repeated calls: subset estimator, paired gain, simultaneous lower bound and adoption rule.
- Appendix G adds the covariance, the finite-sample bound and an illustration on the repeated-call records (G.1).
New text is in blue with red margin notes; restored original sentences are in black. We will keep the manuscript up to date as the discussion continues.
We would be grateful if you would consider raising your score.
Thank you for your careful reading. We appreciate your concrete questions.
We have uploaded a revised version: changed text is in blue, new appendix sections have blue titles, and red margin notes name the review point each change addresses.
W1 (only theoretical), Q1 (goal)
The goal is to answer the title question exactly for a fixed predictor on binary questions under unweighted majority: how accuracy can change as votes are added, and what such a curve reveals about the population of questions (Sections 3–5). Figure 1 shows five exact curves that all start at 75% one-call accuracy and then stay constant, fall quickly, dip and later surpass the starting value, rise and then fall, or rise slowly. The revised introduction states the goal, motivation, contributions and practical use in these plain terms.
Q2 (motivation)
More votes do not always help: Chen et al. observed GPT-3.5 majority accuracy on MMLU Physics first rising and then falling with more calls, and Khan et al. a slight decline as GPT-4-Turbo judge votes increased (revised introduction). The paper characterizes exactly when a curve rises, falls or reverses (Section 5). One-call accuracy and small-budget gains do not settle whether more votes will help, and even the whole curve may not identify the population: an equal mixture of questions with q=0.8 and q=0.2 has the same constant 50% curve as questions that all have q=0.5. The signature identifies exactly which differences voting observes and which it discards, and so what a scaling experiment can reveal.
Q3 (contributions)
There are four results. We say exactly which differences between populations of questions voting can never detect: those symmetric under exchanging q and 1−q (Theorem 3.2). We describe every accuracy-versus-budget curve that can occur and every population producing it (Proposition 4.2), and give the exact condition under which only one population produces a curve (Corollary 4.4). Finally, we show that J repeated calls per question determine the accuracies at budgets up to J and, in general, no further (Section 6).
Q4 (practical uses)
A user with J labeled calls per question can apply two results without estimating the signature. Section 6 shows that these calls determine the accuracy at every odd budget up to J and, in general, at no larger budget, so claims about larger budgets or limiting accuracy require more calls per question or an assumption on the population near q=1/2. Within the identified range, the accuracy of an m-vote majority (odd m≤J) is estimated by the average majority correctness over m-call subsets, and a larger budget is adopted when a simultaneous lower confidence bound on its gain, from paired per-question differences, exceeds the cost of the extra calls (Appendix G of the revised version). The signature fixes which gains and curve shapes can occur at budgets beyond the data.
Repeated calls from three LLMs on QNLI and QQP illustrate the rule. For QNLI with Llama-3.1-8B at temperature 1, three calls gain 1.388 points over one (standard error 0.100), whereas for Qwen2.5-7B the gain is 0.077 points (standard error 0.040); the reply to Reviewer G3nk gives all 18 conditions.
We hope these responses address your concerns, and we would be grateful if you would consider raising your score.