acceptodds
Under review as a conference paper at ICLR 2027

When to Overrule the Majority Vote? Precision Under Disagreement in Test-Time Reasoning

Abstract

LLM-as-a-judge, re-solving, and other correction mechanisms are often used to improve self-consistency by overruling its majority vote. These methods are usually judged by overall accuracy, which is a crude measure because a net gain can hide the right answers they break behind the wrong ones they fix. We instead measure a method's precision under disagreement, meaning how often it is right when it overrules the vote, and compare it with how often the overruled vote was right on those same questions. We study three open reasoning models on competition math, focusing on questions where their samples split. On the test contests, the correct answer is among the five most common answers for only 38% of wrong votes, so choosing among candidates has limited headroom. Unanimous same-model verification is 79.5% precise overall but right on only 17% of its overrides, while the votes it overruled were right on 50% of them. Used alone, the model's own executed programs fix about as many answers as they break, but when they all agree on a different answer, they are almost always right. Overriding the vote only in that case raises test accuracy by 1.6 points and breaks a right answer only once across all contests.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self consistency, test time scaling, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_y3oHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks when a correction method should be allowed to overrule a majority vote in LLM reasoning. Instead of reporting the net accuracy alone, it argues for also looking at cases where the method changes the answer and counting how often it fixes a wrong answer or breaks a right one. On competition math problems, the authors find that letting the model write and execute several programs can fix some wrong answers while not breaking right ones much if their outputs agree with each other, while other methods, such as self-verification or a larger judge, often break nearly as many answers as they fix.

Critical strengths

Improving on majority voting has been an important problem for test-time scaling. The paper compares a wide range of correction methods and presents some useful findings, for example that similar accuracy gains can hide very different numbers of broken answers, and that on hard questions the correct answer is often not among the top candidates at all.

The experiments and ablations are very extensive, and results are reported with significance tests. Prompts, costs, and random seeds are also reported, which is good for reproducibility.

Critical weaknesses

The paper is difficult to follow, especially when comparisons are made and many numbers are listed. Consider alternative ways to visualize or present the numbers (which for example could be a figure showing fixed vs broken answers per method), and add a summary or closing sentence near or after them so that readers can quickly get the intuitions behind the comparisons.

The guidance the paper offers is limited. The headline unanimous program rule improves test accuracy by 1.6 percentage points, but this gain comes from only five distinct problems, mostly involving computations that a short program can enumerate. Also, for the gpt-oss-20b model that could run code during its reasoning, doing so was more accurate than the majority vote even without any agreement rule, which raises the question of whether the proposed rule is needed when such tools are available. Regarding novelty, it is well known that running code helps LLMs with math, and checking whether reasoning and code agree before deciding whom to trust has been tried before (Yue et al., 2024, cited as the closest work), so the main contribution appears to be counting how often a method fixes versus breaks answers.

However, the authors themselves note that this kind of counting is standard in earlier work on when a model should defer to another source, and similar counts appear in prior self-correction studies. It would be helpful if the authors could explain more what their analysis shows that these earlier works do not.

Questions for authors

It would be helpful if the authors could explain more what their analysis shows that these earlier works do not.

2 comments
AuthorsOpenReview ↗

Response to Reviewer y3oH (Part 1/2)

Dear Reviewer y3oH,

Thank you for your review of our paper. We have rewritten it to be more clear in terms of our claims and contributions. We respond to individual points below.

Summary: The main analysis now applies one agreement rule to every method on the same 150 questions, with a fixed 16-response majority vote: we replace the majority-vote answer only when at least k of 8 additional responses from either the same model, executed programs, or a helper model agree on a different answer. This sort of rule has known precedents in work on LLM cascades (Yue et al., 2024), where an answer is accepted once the share of sampled answers that agree on it reaches a threshold. We report all four thresholds (k = 5, 6, 7, 8) instead of selecting one.

“The paper is difficult to follow, especially when comparisons are made and many numbers are listed.”

We agree that the original manuscript reads more like a collection of experiments and tables. Following your suggestions, we have rewritten it to focus on a smaller set of empirical questions, replaced tables with figures that more clearly show the comparisons between methods, and shortened paragraphs to focus on basic takeaways from each comparison.

“The guidance the paper offers is limited.”

Under the revised framing, the results are admittedly mostly negative. Although cross-model overrides can improve a weaker base model, our evaluations do not show a clear advantage over using the stronger helper model alone. The practical implication is that using a stronger model directly may be preferred over adding model-based overrides to a weaker model. We have revised the paper to make this implication clearer and do not extend our claims beyond the evaluated models and tasks.

“The headline unanimous program rule improves test accuracy by 1.6 percentage points, but this gain comes from only five distinct problems, mostly involving computations that a short program can enumerate.”

“Also, for the gpt-oss-20b model that could run code during its reasoning, doing so was more accurate than the majority vote even without any agreement rule, which raises the question of whether the proposed rule is needed when such tools are available.”

These comments helped us reconsider the paper’s framing. The original presentation emphasized the unanimous program rule and its accuracy gain, which gave a lot of credit to a result supported by few distinct questions. The revised paper asks whether overriding a majority vote is worthwhile and reports largely negative findings across methods and agreement thresholds, which we believe to be more useful.

Regarding the earlier tool-integrated result, that experiment used the most frequent answer among eight tool-integrated responses without imposing an agreement threshold on those responses. Here, replacement was restricted to cases where baseline agreement remained below 80% after expanded sampling. Within those cases, the most frequent tool-integrated answer improved observed accuracy without an additional agreement threshold. This result mostly speaks to conditional replacement under the original protocol, so its relevance to tool use alone across all questions is limited.

The revised paper places greater emphasis on direct standalone comparisons. Cross-model overrides can improve a weaker base, but our evaluations do not establish an advantage over using the stronger helper alone, even with only the eight helper responses already supplied to the combined method. Program execution provides useful corrections in some cases, while strict agreement often leaves few or no overrides. These findings offer limited support for adding an override stage and motivate considering the stronger model directly.

AuthorsOpenReview ↗

Response to Reviewer y3oH (Part 2/2)

“Regarding novelty, it is well known that running code helps LLMs with math, and checking whether reasoning and code agree before deciding whom to trust has been tried before (Yue et al., 2024, cited as the closest work), so the main contribution appears to be counting how often a method fixes versus breaks answers.”

“However, the authors themselves note that this kind of counting is standard in earlier work on when a model should defer to another source, and similar counts appear in prior self-correction studies.”

“"It would be helpful if the authors could explain more what their analysis shows that these earlier works do not.”

Our central contribution is questioning whether overriding a majority vote is worthwhile. We have revised the paper to make this empirical question clearer. Program execution, agreement-based acceptance rules, and separate accounting of corrected and introduced errors provide established tools for investigating it. Yue et al. (2024) supplies a relevant precedent for using agreement to guide acceptance and escalation.

The revised analysis examines whether improvements over a weaker base justify combining models. Cross-model overrides can improve the base model’s accuracy, but our comparisons do not establish an advantage over the stronger helper even when only using the same eight helper responses. Additional same-model reasoning rarely overrides, while executed programs provide useful corrections in some cases. Increasing the agreement requirement often leaves few or no answer changes.

These largely negative findings suggest that, for the evaluated models and questions, using the stronger model directly may be preferable to adding model-based overrides. We have reorganized the manuscript around this practical implication and the evidence supporting it.

We appreciate your careful reading and feedback. We look forward to continuing the discussion and hearing how we can further strengthen the paper and its analyses.

Reviewer_LMb1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper seeks to quantitatively analyze when an overriding rule (like a call to an LLM to judge many candidates and produce one answer) helps compared to a simple majority voting scheme. The main conclusion is that even if there is an overall accuracy improvement, it can mask a poor override rule. The authors claim that most override rules have poor accuracy (<50%) on the set of problems that they decide to override and produce a different answer.

Critical strengths

The problem setting and motivation is interesting. It is important to understand whether it actually helps to use an overriding model call when aggregating many possible solutions.

Critical weaknesses
  1. The models (Deepseek-R1, GPT-OSS 20b, Qwen3-30b) and the datasets (high school math competitions) that are used in this work are outdated and it is unclear whether the results transfer to frontier or more recent open-source models and more challenging math datasets.
  2. In Table 3, which is presumably the table with the main results, the confidence intervals for the override accuracy are very large indicating that the dataset size considered is insufficient. Moreover, the main conclusion seems to be that overrides rules are often not accurate on the examples they override, but this does not correspond to the results in Table 3 where it seems that most rules have πdis>\pi_{dis} > 50%
  3. The paper is very difficult to read and contains many grammatical errors. The terminology introduced is unintuitive. For example, the term "source" is used to refer to the answer produced by the override rule. This is not an appropriate term. Similarly, the terms "test" and "development" do not make sense in the context which they are used. The tables are also hard to comprehend with no guidance given in the table captions. It will need significant rewriting to fix.
Questions for authors
  1. Could you justify why the experimental results are an important contribution considering that the models and the datasets are outdated?
  2. Could you justify the decision to ignore the large confidence intervals and still draw conclusions from the results in Table 3?
1 comment
AuthorsOpenReview ↗

Response to Reviewer LMb1

Dear Reviewer LMb1, Thank you for your feedback. We have rewritten the manuscript, revised the analysis, and clarified the contribution and our conclusions. We address your concerns below.

Models and datasets (Question 1). The evaluated models remain below saturation on these benchmarks. Their majority votes are incorrect on approximately 16%–30% of questions, leaving headroom to evaluate whether additional reasoning can correct their errors. The competition questions also provide reference answers for evaluating both correct and incorrect changes. Three of the five contests took place after the evaluated model versions were released. Our central contribution is questioning whether overriding a majority vote is worthwhile. In particular, improving a weaker base does not establish that combining models is preferable to using the stronger helper directly. The revised analysis compares these alternatives, including a baseline using the exact eight helper responses supplied to the combined method. Our evaluations do not establish an accuracy advantage for the combined method over the stronger helper alone. These findings concern the evaluated models and tasks. We have restricted our conclusions accordingly. Newer models and other benchmarks would require separate evaluation.

Confidence intervals in Table 3 (Question 2). We have removed the broad claim that override accuracy is usually below 50%. The revised analysis reports corrected and introduced errors separately. We use question-bootstrap confidence intervals for accuracy differences, comparing methods on the same resampled questions. Our conclusions now focus on whether overrides improve on the relevant standalone baselines. We also report how many distinct questions have overrides and explain that sparse overrides limit conclusions about reliability.

Readability and terminology. We have rewritten the main text around the empirical question of whether overrides are worthwhile. We replaced ambiguous terms such as “source,” defined override, rescue, and regression, and added a diagram explaining the methods and their shared agreement rule. The results now use focused figures and shorter explanations, with captions describing the displayed quantities. The appendix documents the original development/test protocol and subsequent analysis choices. We also corrected grammatical problems and moved secondary details out of the main text.

Thank you for taking the time to review our work. Your comments helped us clarify the paper’s contribution and presentation. We look forward to your response and any additional suggestions.

Reviewer_ZcLnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates when a correction mechanism should override the majority-vote answer in test-time LLM reasoning. The authors introduce precision under disagreement, which measures how often a correction method is correct when it disagrees with majority voting, while also tracking how often the original vote was correct. They evaluate same-model verification, re-solving, executed programs, cross-model reasoning, and stronger LLM judges across three open-source reasoning models on competition mathematics benchmarks. The results show that conventional verification and stronger judges can introduce incorrect overrides, whereas unanimous agreement among executed programs provides a more reliable correction signal. The proposed program-agreement rule improves test accuracy by 1.6 percentage points with very few observed regressions.

Critical strengths

A correction method that is accurate overall may still be unreliable when it disagrees with the majority vote. The papers detailed comparisons and explicit accounting of harmful overrides are valuable.

  1. Meaningful evaluation of correction reliability: The proposed disagreement-based analysis distinguishes corrections that fix incorrect answers from those that break previously correct answers. This provides useful information that overall accuracy alone can obscure.

  2. Comprehensive empirical comparison: The paper evaluates multiple correction strategies, including self-verification, re-solving, program execution, cross-model evidence, and stronger judges. The results demonstrate that strict agreement among executed programs can improve reliability, with additional evaluations on held-out models and contests.

Critical weaknesses

The proposed method does not fundamentally resolve the problem of correlated errors in self-consistency. Agreement among executed programs may also reinforce incorrect answers when the programs share the same flawed assumptions. Therefore, the method appears to be a task-specific correction strategy rather than a general solution.

  1. Limited novelty: The proposed evaluation metric mainly measures how often a correction method fixes an incorrect answer versus changing a correct answer into an incorrect one. While useful, this is a straightforward analysis, and the paper does not clearly establish a significant methodological contribution.

  2. Limited generalization: The experiments focus on mathematical problems that can be solved using executable programs. Additionally, the reported 1.6 percentage-point accuracy improvement comes from only five rescued test problems, making it difficult to assess the reliability of the approach.

Questions for authors
  1. Novelty: Beyond measuring correct and incorrect overrides, what is the main technical contribution of the proposed evaluation framework?

  2. Role of execution: Is the improvement primarily due to executing Python programs, or could similar results be achieved through agreement among independently generated reasoning solutions?

  3. Generalization: How would the proposed approach handle reasoning tasks where the answer cannot be computed or verified using executable programs?

  4. Robustness: Does the reported improvement remain consistent across different random seeds and independently generated program samples?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZcLn

Dear Reviewer ZcLn,

Thank you for your careful reading and feedback. We have rewritten the paper around the question of whether overriding a majority vote is worthwhile. We address your concerns below.

Correlated errors. Additional responses can share the same mistaken assumptions and agree on an incorrect answer, whether they come from further reasoning, executed programs, or another model. This motivates examining whether agreement provides a useful basis for changing the vote. The revision compares these methods under a common agreement rule and reports both the errors corrected and those introduced.

W1 / Q1 — Novelty and contribution. Our central contribution is testing whether overrides justify their additional computation. Improving a weaker base model can appear promising while still falling short of using the stronger helper directly. Our revised comparisons do not establish an advantage for the combined method over the stronger helper alone, even when its vote uses only the same eight helper responses. Together with the limited gains from additional same-model reasoning, these largely negative findings question the practical value of adding an override stage. The agreement rule has precedents in LLM cascades (Yue et al., 2024), and we report all thresholds.

Q2 — Role of execution. Additional same-model reasoning did not produce gains comparable to executed programs in our evaluation. We have revised the framing to make execution one comparison in assessing whether overrides are worthwhile. Useful corrections from programs alone provide limited grounds for recommending an override method, particularly when strict agreement leaves few answer changes.

W2 / Q3 — Generalization and few corrections. These concerns helped us reconsider the original emphasis on the unanimous program rule. The revised paper examines whether several forms of additional reasoning justify changing the majority vote. Additional reasoning and cross-model reasoning can be evaluated on tasks without executable answers, but their effectiveness beyond our setting remains untested. We have narrowed our conclusions and replaced the headline emphasis on five corrected problems with results across all thresholds, reporting uncertainty and the number of distinct questions with overrides.

Q4 — Robustness. The revised analysis uses three disjoint sets of 16 baseline responses per model and question. Results are also similar on the balanced 150-question set and the overlapping set of all 183 eligible questions (Table 4). These repetitions share program samples, so they do not establish robustness to independently regenerated programs. The original submission included a separate experiment with fresh programs for each run on two models, which also produced gains, although that evidence remains limited in scale.

Thank you again for your time and feedback. We look forward to your thoughts on the revision and welcome further suggestions for improving the paper and analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.