- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper asks when a correction method should be allowed to overrule a majority vote in LLM reasoning. Instead of reporting the net accuracy alone, it argues for also looking at cases where the method changes the answer and counting how often it fixes a wrong answer or breaks a right one. On competition math problems, the authors find that letting the model write and execute several programs can fix some wrong answers while not breaking right ones much if their outputs agree with each other, while other methods, such as self-verification or a larger judge, often break nearly as many answers as they fix.
Critical strengths
Improving on majority voting has been an important problem for test-time scaling. The paper compares a wide range of correction methods and presents some useful findings, for example that similar accuracy gains can hide very different numbers of broken answers, and that on hard questions the correct answer is often not among the top candidates at all.
The experiments and ablations are very extensive, and results are reported with significance tests. Prompts, costs, and random seeds are also reported, which is good for reproducibility.
Critical weaknesses
The paper is difficult to follow, especially when comparisons are made and many numbers are listed. Consider alternative ways to visualize or present the numbers (which for example could be a figure showing fixed vs broken answers per method), and add a summary or closing sentence near or after them so that readers can quickly get the intuitions behind the comparisons.
The guidance the paper offers is limited. The headline unanimous program rule improves test accuracy by 1.6 percentage points, but this gain comes from only five distinct problems, mostly involving computations that a short program can enumerate. Also, for the gpt-oss-20b model that could run code during its reasoning, doing so was more accurate than the majority vote even without any agreement rule, which raises the question of whether the proposed rule is needed when such tools are available. Regarding novelty, it is well known that running code helps LLMs with math, and checking whether reasoning and code agree before deciding whom to trust has been tried before (Yue et al., 2024, cited as the closest work), so the main contribution appears to be counting how often a method fixes versus breaks answers.
However, the authors themselves note that this kind of counting is standard in earlier work on when a model should defer to another source, and similar counts appear in prior self-correction studies. It would be helpful if the authors could explain more what their analysis shows that these earlier works do not.
Questions for authors
It would be helpful if the authors could explain more what their analysis shows that these earlier works do not.
2 comments
Response to Reviewer y3oH (Part 2/2)
“Regarding novelty, it is well known that running code helps LLMs with math, and checking whether reasoning and code agree before deciding whom to trust has been tried before (Yue et al., 2024, cited as the closest work), so the main contribution appears to be counting how often a method fixes versus breaks answers.”
“However, the authors themselves note that this kind of counting is standard in earlier work on when a model should defer to another source, and similar counts appear in prior self-correction studies.”
“"It would be helpful if the authors could explain more what their analysis shows that these earlier works do not.”
Our central contribution is questioning whether overriding a majority vote is worthwhile. We have revised the paper to make this empirical question clearer. Program execution, agreement-based acceptance rules, and separate accounting of corrected and introduced errors provide established tools for investigating it. Yue et al. (2024) supplies a relevant precedent for using agreement to guide acceptance and escalation.
The revised analysis examines whether improvements over a weaker base justify combining models. Cross-model overrides can improve the base model’s accuracy, but our comparisons do not establish an advantage over the stronger helper even when only using the same eight helper responses. Additional same-model reasoning rarely overrides, while executed programs provide useful corrections in some cases. Increasing the agreement requirement often leaves few or no answer changes.
These largely negative findings suggest that, for the evaluated models and questions, using the stronger model directly may be preferable to adding model-based overrides. We have reorganized the manuscript around this practical implication and the evidence supporting it.
We appreciate your careful reading and feedback. We look forward to continuing the discussion and hearing how we can further strengthen the paper and its analyses.
Response to Reviewer y3oH (Part 1/2)
Dear Reviewer y3oH,
Thank you for your review of our paper. We have rewritten it to be more clear in terms of our claims and contributions. We respond to individual points below.
Summary: The main analysis now applies one agreement rule to every method on the same 150 questions, with a fixed 16-response majority vote: we replace the majority-vote answer only when at least k of 8 additional responses from either the same model, executed programs, or a helper model agree on a different answer. This sort of rule has known precedents in work on LLM cascades (Yue et al., 2024), where an answer is accepted once the share of sampled answers that agree on it reaches a threshold. We report all four thresholds (k = 5, 6, 7, 8) instead of selecting one.
We agree that the original manuscript reads more like a collection of experiments and tables. Following your suggestions, we have rewritten it to focus on a smaller set of empirical questions, replaced tables with figures that more clearly show the comparisons between methods, and shortened paragraphs to focus on basic takeaways from each comparison.
Under the revised framing, the results are admittedly mostly negative. Although cross-model overrides can improve a weaker base model, our evaluations do not show a clear advantage over using the stronger helper model alone. The practical implication is that using a stronger model directly may be preferred over adding model-based overrides to a weaker model. We have revised the paper to make this implication clearer and do not extend our claims beyond the evaluated models and tasks.
These comments helped us reconsider the paper’s framing. The original presentation emphasized the unanimous program rule and its accuracy gain, which gave a lot of credit to a result supported by few distinct questions. The revised paper asks whether overriding a majority vote is worthwhile and reports largely negative findings across methods and agreement thresholds, which we believe to be more useful.
Regarding the earlier tool-integrated result, that experiment used the most frequent answer among eight tool-integrated responses without imposing an agreement threshold on those responses. Here, replacement was restricted to cases where baseline agreement remained below 80% after expanded sampling. Within those cases, the most frequent tool-integrated answer improved observed accuracy without an additional agreement threshold. This result mostly speaks to conditional replacement under the original protocol, so its relevance to tool use alone across all questions is limited.
The revised paper places greater emphasis on direct standalone comparisons. Cross-model overrides can improve a weaker base, but our evaluations do not establish an advantage over using the stronger helper alone, even with only the eight helper responses already supplied to the combined method. Program execution provides useful corrections in some cases, while strict agreement often leaves few or no overrides. These findings offer limited support for adding an override stage and motivate considering the stronger model directly.