- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work proposes cross-model consensus as a training-free best-of-N selector. The method starts with a single Generator model generating N candidate answers. Then, we use an LLM-jury (a panel of LLM jurors) where each juror independently generates their own answer without knowing the N candidate answers. Each of the N candidate answers is then compared against the answers from the LLM-jury, and the candidate answer that matches the largest number of jurors is selected. The authors report that this selector outperforms self-consistency and a single-model LLM-verifier, and matches or exceeds four trained verifiers. The authors use Qwen3-235B as the generator and evaluate on seven benchmarks. Additionally, the paper introduces a parameter-free law that predicts how reliable the consensus will be on a given domain: its accuracy, the accuracy-coverage tradeoff when abstaining on low agreement, and a "shared-error floor" (the rate at which the panel is unanimously wrong) that caps the accuracy any agreement threshold can achieve.
Critical strengths
- The idea of the method is simple and elegant.
- The shared-error floor gives an interpretable upper bound on selective accuracy, and the reported operating points match it (e.g., 99.5% accuracy at 85% coverage on MATH-500 when the panel unanimously agrees).
- Honest about limitations. The main paper clarifies its limitations openly.
- The paper is generally easy to follow and flows logically.
Critical weaknesses
- Only two models are used as generators, which is still a small number.
- No baseline of the best single member. The Limitations section discloses that the vote does not beat the strongest member when member strength is unequal. Leave-one-member-out on AIME-24 shows that member strength actually drives the result, since dropping Claude or Kimi lowers accuracy from 0.70 to 0.50, while dropping Qwen or DeepSeek raises it to 0.77. This result contradicts Appendix E's claim that the signal does not derive from a single dominant model.
- The LLM-verifier baseline is self-scoring only. It is always the generator scoring its own candidates, and the authors did not specify how ties are broken. Also, no cross-model judge is tested, so the claim that "a model cannot adjudicate its own candidates" is never compared with a different model adjudicating them.
- The law's practical value is unclear. Its inputs must be measured on a labeled sample from the target domain, but with such a sample, consensus accuracy and the accuracy-coverage curve can be measured directly, so why bother with the law?
- "Parameter-free" is misleading, since the three inputs are estimated from the same labeled benchmark the law predicts.
- No baseline predictors reported. Performance metrics such as the mean member accuracy and the best-member accuracy are not reported.
- Inconsistencies:
- Figure 1 shows that the generator (Qwen3-235B) is also inside the LLM-jury, while Appendix K states that the generator is never a voting juror.
- Appendix D reports 0/530 unanimously wrong cases on math, while MATH-500 and GSM8k show 0.004 and 0.014 in Table 2 under the “shared-error floor” column.
- Overclaims:
- The abstract says “on competition math it closes the entire gap to an oracle selector”, but that’s true only on AIME-24, as shown in table 1 where 56.7 matches 56.7 of the oracle. The gap does not close under the other columns.
- Section 5.3 is titled “CROSS-MODEL CONSENSUS OUTPERFORMS TRAINED REWARD MODELS”, but that conclusion overclaims because it relies on a non-significant result that the authors themselves acknowledged (“0.966 vs. 0.956, not significant at p=0.142”).
- Limited novelty. The paper acknowledges that "voting helps" is known. What remains largely reframes existing ideas, and the closest prior work is not cited: consensus and ensembling across diverse LLMs [1, 2]; models of majority-vote accuracy under correlated voters and varying item difficulty [3, 4]; and direct measurements of correlated errors across LLMs [5, 6]. The main new element is the wrong-answer attractor in the predictive law.
References
[1] Chen et al., ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs, ACL 2024.
[2] Tekin et al., LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity, Findings of EMNLP 2024.
[3] Kuncheva et al., Limits on the Majority Vote Accuracy in Classifier Fusion, Pattern Analysis and Applications, 2003.
[4] Chen et al., Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems, NeurIPS 2024.
[5] Kim et al., Correlated Errors in Large Language Models, ICML 2025.
[6] Goel et al., Great Models Think Alike and This Undermines AI Oversight, ICML 2025.
Questions for authors
- What is each panel member's standalone accuracy on each benchmark? How does the jury compare with simply taking the strongest member's answer?
- Is the generator itself inside the LLM-jury or not?
- The law's inputs require labels from the target domain, and with those labels, consensus accuracy can be measured directly. What does the law add beyond that?
6 comments
2nd Response to Author Rebuttal
Thank you for the thorough follow-up and for running all the analyses I asked for. The judge with a matched tie-break, the per-benchmark leave-one-out, the third generator, and the extrapolation test make the 30K results much more convincing, and I’ve raised my score to weak reject (2).
A few concerns stop me from going higher, listed in descending order of severity:
-
At full budget, the evidence that agreement across several models is crucial is still limited: on the main pool, selecting by Sonnet’s answer alone matches the jury (91.1 vs 91.2), and the gains over the other baselines are modest.
-
The PRM comparison in the title and the decorrelation experiments in Section 5.4 still rely only on the truncated 4K runs.
-
The extrapolation test validates a modified version of the law rather than the original one in Section 3.2, and its accuracy gain over the simpler correlated-errors baseline is small.
-
The left panel of Figure 3 still shows the stated law, which stays nearly flat while measured accuracy rises as panel size grows. Since the figure uses a different tie rule, I suggest recreating it with the validated model and a consistent tie rule.
More generally, the paper has changed substantially since submission, and I think the revised paper should be evaluated as a new submission. With all experiments run on the full budget (30K token cap), I think it could become a solid paper.
Response to Reviewer KCmL (second follow-up, 1/2)
We thank the reviewer for the continued engagement and for raising the score. We deeply appreciate the reviewer's active engagement during the discussion phase, and we are committed to addressing these concerns and improving the paper regardless of the outcome. We address the remaining points in the reviewer's order.
Agreement across several models at the full budget. Sonnet-match ties the jury only on the Qwen3-235B pool (91.1 vs. 91.2), and only because Sonnet is chosen as the best single juror using the test labels. A deployment must choose the single juror without test labels, so we selected the best single juror on a labeled calibration split of 10–30% of each benchmark and evaluated both selectors on the remaining problems: over 500 random splits, the jury exceeds the calibrated single juror by +0.65 to +0.76 points on average and is ahead in every split. The advantage also holds across generators. Pooling both 30K-cap generators (Qwen3-235B and gpt-oss-120b, 7,502 problem pools over the seven benchmarks), the jury significantly outperforms every single-juror selector, even when that juror is chosen with hindsight: +0.5 [+0.2, +0.8] over Sonnet-match, +1.0 [+0.7, +1.3] over Kimi-match and +1.4 [+1.1, +1.8] over DeepSeek-match, and +0.9 [+0.4, +1.4] over Sonnet-match with the gpt-oss-120b pool alone. Over self-consistency, the jury gains +1.6 [+1.0, +2.1] with the Qwen3-235B pool and +3.3 [+2.6, +4.0] with the gpt-oss-120b pool, most where the pool leaves headroom (GPQA +4.5 and +9.6), using three single answers rather than a model that reads every candidate.
PRM comparison and the decorrelation experiments at 30K. We re-ran both at the 30K cap. All four trained verifiers (Qwen2.5-Math-PRM-7B/72B, AceMath-72B-RM, ThinkPRM-14B) now score the 30K Qwen3-235B pools with the same self-check gate and the same pool-frequency tie-break as the jury, and the revised §5.3 and its table now report these results, with the 4K table moved to Appendix B. The jury is at or above every verifier on every benchmark: on MATH-500 it significantly outperforms both discriminative PRMs (98.0 vs. 97.0, p = 0.006), while AceMath-72B and ThinkPRM, whose scores there fall below the self-check gate, revert to self-consistency (97.8; ungated 97.2 and 97.8); on GPQA it is ahead of all four (84.3 vs. 74.7–81.3), significantly so for three (p < 0.001); and on AIME-24/25 it is the strongest selector (93.3 and 86.7, vs. at most 90.0 and 80.0). Pooled over the four benchmarks, it significantly outperforms every verifier, including all three process reward models: +3.8 [+2.4, +5.4] over each Qwen2.5-Math-PRM, +2.9 [+1.6, +4.2] over AceMath-72B and +1.3 [+0.3, +2.5] over ThinkPRM (p ≤ 0.011). For §5.4, pairwise error correlation at the 30K cap is 0.75 within one model, 0.54 within a family (gpt-oss-120b and gpt-oss-20b on all seven benchmarks; 0.58 for Qwen3-235B and Qwen3.8-27B) and 0.42 across families, the same ordering as at 4K with a wider gap. The matched-accuracy isolation also holds at 30K, with exactly matched member accuracy by construction: on the same problems, a panel of k answers drawn half from Qwen3-235B and half from gpt-oss-120b beats k answers from either model alone by +1.0 [+0.8, +1.1], +1.3 [+1.1, +1.5], +1.4 [+1.1, +1.6] and +1.5 [+1.2, +1.9] pooled at k = 4, 6, 8 and 12. The combinatorial sweep also holds at 30K once member accuracy is controlled: among panels whose members are within 8 points of each other in accuracy, consensus gain over the panel's best member correlates with lower error correlation (r = +0.17 to +0.75 on the unsaturated benchmarks, pooled within-benchmark r = +0.55). The jury also improves over self-consistency at every pool size (+1.4, +1.4 and +1.6 at N = 4, 8 and 12).
| Trained verifiers, 30K pools | MATH-500 | AIME-24 | AIME-25 | GPQA |
|---|---|---|---|---|
| Self-consistency | 97.8 | 86.7 | 80.0 | 79.8 |
| Qwen2.5-Math-PRM-7B | 97.0 | 83.3 | 80.0 | 74.7 |
| Qwen2.5-Math-PRM-72B | 97.0 | 86.7 | 76.7 | 74.7 |
| AceMath-72B-RM | 97.8† | 90.0 | 80.0 | 75.3 |
| ThinkPRM-14B | 97.8† | 90.0 | 80.0 | 81.3 |
| Jury | 98.0 | 93.3 | 86.7 | 84.3 |
† Below the self-check gate (AUROC < 0.6); falls back to self-consistency.
Gentle reminder: author–reviewer discussion period ending in a few days
Dear Reviewer KCmL,
We wanted to send a brief reminder that the author–reviewer discussion period concludes in a few days.
In our follow-up response, we ran the analyses you suggested at the 30K cap: calibrating the law on three-member panels and testing its extrapolation to larger panels, where it predicts both the accuracy and the shared-error floor of larger panels better than every simpler predictor in the test; per-benchmark leave-one-member-out; the Sonnet judge on all seven benchmarks with the same pool-frequency tie-break as the jury; juries ranging from one to six members across six model families; and a third generator (gpt-oss-120b). With the third generator, the jury improves over self-consistency by 3.3 points [+2.6, +4.0] and exceeds every single-juror selector.
If these updates address your concerns, we would be glad to hear it, and we would be grateful if you would consider reassessing your rating accordingly. If there are any remaining questions or concerns, we would be happy to address them during the remaining discussion period.
Thank you again for the time and effort you put into reviewing our work.
Best regards,
Authors