acceptodds
Under review as a conference paper at ICLR 2027

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

11 views

Abstract

Selecting the correct answer from a pool of candidate reasoning chains is the engine of test-time scaling, yet the standard selectors each carry a cost: self-consistency inherits the errors of the single model it resamples, and trained reward models need labeled data and transfer poorly off-distribution. We study a third signal, free at inference time: cross-model consensus, the degree to which independently trained models, each solving the problem once, agree on a final answer. We treat the panel as an LLM-jury, in which the verification signal is the structure of agreement itself, with no model scoring another's work. Across seven benchmarks it selects correct answers better than self-consistency and far better than a model scoring its own candidates: on competition math it closes the entire gap to an oracle selector, while self-scoring closes almost none. The mechanism is error decorrelation: independently trained models err differently, so their wrong answers scatter while the correct one accumulates agreement. We make this precise with a parameter-free law, derived in closed form, that predicts consensus accuracy from three measured panel statistics to a mean absolute error of and exposes the method's ceiling: a shared-error floor where models share a misconception, near zero on math but non-trivial on science. Against four trained verifiers spanning discriminative, outcome, and generative reward models, the free LLM-jury matches the strongest inside their math training domain and is the top selector outside it. Cross-model consensus is thus a verifier we can characterize in advance: a law that says when to trust it, and a floor that marks where it cannot.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, verification, cross-model consensus, self-consistency, process reward models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KCmLOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes cross-model consensus as a training-free best-of-N selector. The method starts with a single Generator model generating N candidate answers. Then, we use an LLM-jury (a panel of LLM jurors) where each juror independently generates their own answer without knowing the N candidate answers. Each of the N candidate answers is then compared against the answers from the LLM-jury, and the candidate answer that matches the largest number of jurors is selected. The authors report that this selector outperforms self-consistency and a single-model LLM-verifier, and matches or exceeds four trained verifiers. The authors use Qwen3-235B as the generator and evaluate on seven benchmarks. Additionally, the paper introduces a parameter-free law that predicts how reliable the consensus will be on a given domain: its accuracy, the accuracy-coverage tradeoff when abstaining on low agreement, and a "shared-error floor" (the rate at which the panel is unanimously wrong) that caps the accuracy any agreement threshold can achieve.

Critical strengths
  • The idea of the method is simple and elegant.
  • The shared-error floor gives an interpretable upper bound on selective accuracy, and the reported operating points match it (e.g., 99.5% accuracy at 85% coverage on MATH-500 when the panel unanimously agrees).
  • Honest about limitations. The main paper clarifies its limitations openly.
  • The paper is generally easy to follow and flows logically.
Critical weaknesses
  • Only two models are used as generators, which is still a small number.
  • No baseline of the best single member. The Limitations section discloses that the vote does not beat the strongest member when member strength is unequal. Leave-one-member-out on AIME-24 shows that member strength actually drives the result, since dropping Claude or Kimi lowers accuracy from 0.70 to 0.50, while dropping Qwen or DeepSeek raises it to 0.77. This result contradicts Appendix E's claim that the signal does not derive from a single dominant model.
  • The LLM-verifier baseline is self-scoring only. It is always the generator scoring its own candidates, and the authors did not specify how ties are broken. Also, no cross-model judge is tested, so the claim that "a model cannot adjudicate its own candidates" is never compared with a different model adjudicating them.
  • The law's practical value is unclear. Its inputs must be measured on a labeled sample from the target domain, but with such a sample, consensus accuracy and the accuracy-coverage curve can be measured directly, so why bother with the law?
  • "Parameter-free" is misleading, since the three inputs are estimated from the same labeled benchmark the law predicts.
  • No baseline predictors reported. Performance metrics such as the mean member accuracy and the best-member accuracy are not reported.
  • Inconsistencies:
    • Figure 1 shows that the generator (Qwen3-235B) is also inside the LLM-jury, while Appendix K states that the generator is never a voting juror.
    • Appendix D reports 0/530 unanimously wrong cases on math, while MATH-500 and GSM8k show 0.004 and 0.014 in Table 2 under the “shared-error floor” column.
  • Overclaims:
    • The abstract says “on competition math it closes the entire gap to an oracle selector”, but that’s true only on AIME-24, as shown in table 1 where 56.7 matches 56.7 of the oracle. The gap does not close under the other columns.
    • Section 5.3 is titled “CROSS-MODEL CONSENSUS OUTPERFORMS TRAINED REWARD MODELS”, but that conclusion overclaims because it relies on a non-significant result that the authors themselves acknowledged (“0.966 vs. 0.956, not significant at p=0.142”).
  • Limited novelty. The paper acknowledges that "voting helps" is known. What remains largely reframes existing ideas, and the closest prior work is not cited: consensus and ensembling across diverse LLMs [1, 2]; models of majority-vote accuracy under correlated voters and varying item difficulty [3, 4]; and direct measurements of correlated errors across LLMs [5, 6]. The main new element is the wrong-answer attractor in the predictive law.

References

[1] Chen et al., ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs, ACL 2024.

[2] Tekin et al., LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity, Findings of EMNLP 2024.

[3] Kuncheva et al., Limits on the Majority Vote Accuracy in Classifier Fusion, Pattern Analysis and Applications, 2003.

[4] Chen et al., Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems, NeurIPS 2024.

[5] Kim et al., Correlated Errors in Large Language Models, ICML 2025.

[6] Goel et al., Great Models Think Alike and This Undermines AI Oversight, ICML 2025.

Questions for authors
  • What is each panel member's standalone accuracy on each benchmark? How does the jury compare with simply taking the strongest member's answer?
  • Is the generator itself inside the LLM-jury or not?
  • The law's inputs require labels from the target domain, and with those labels, consensus accuracy can be measured directly. What does the law add beyond that?
6 comments
AuthorsOpenReview ↗

Gentle reminder: author–reviewer discussion period ending in a few days

Dear Reviewer KCmL,

We wanted to send a brief reminder that the author–reviewer discussion period concludes in a few days.

In our follow-up response, we ran the analyses you suggested at the 30K cap: calibrating the law on three-member panels and testing its extrapolation to larger panels, where it predicts both the accuracy and the shared-error floor of larger panels better than every simpler predictor in the test; per-benchmark leave-one-member-out; the Sonnet judge on all seven benchmarks with the same pool-frequency tie-break as the jury; juries ranging from one to six members across six model families; and a third generator (gpt-oss-120b). With the third generator, the jury improves over self-consistency by 3.3 points [+2.6, +4.0] and exceeds every single-juror selector.

If these updates address your concerns, we would be glad to hear it, and we would be grateful if you would consider reassessing your rating accordingly. If there are any remaining questions or concerns, we would be happy to address them during the remaining discussion period.

Thank you again for the time and effort you put into reviewing our work.

Best regards,

Authors

Reviewer_KCmLOpenReview ↗

2nd Response to Author Rebuttal

Thank you for the thorough follow-up and for running all the analyses I asked for. The judge with a matched tie-break, the per-benchmark leave-one-out, the third generator, and the extrapolation test make the 30K results much more convincing, and I’ve raised my score to weak reject (2).

A few concerns stop me from going higher, listed in descending order of severity:

  • At full budget, the evidence that agreement across several models is crucial is still limited: on the main pool, selecting by Sonnet’s answer alone matches the jury (91.1 vs 91.2), and the gains over the other baselines are modest.

  • The PRM comparison in the title and the decorrelation experiments in Section 5.4 still rely only on the truncated 4K runs.

  • The extrapolation test validates a modified version of the law rather than the original one in Section 3.2, and its accuracy gain over the simpler correlated-errors baseline is small.

  • The left panel of Figure 3 still shows the stated law, which stays nearly flat while measured accuracy rises as panel size grows. Since the figure uses a different tie rule, I suggest recreating it with the validated model and a consistent tie rule.

More generally, the paper has changed substantially since submission, and I think the revised paper should be evaluated as a new submission. With all experiments run on the full budget (30K token cap), I think it could become a solid paper.

AuthorsOpenReview ↗

Response to Reviewer KCmL (second follow-up, 1/2)

We thank the reviewer for the continued engagement and for raising the score. We deeply appreciate the reviewer's active engagement during the discussion phase, and we are committed to addressing these concerns and improving the paper regardless of the outcome. We address the remaining points in the reviewer's order.

Agreement across several models at the full budget. Sonnet-match ties the jury only on the Qwen3-235B pool (91.1 vs. 91.2), and only because Sonnet is chosen as the best single juror using the test labels. A deployment must choose the single juror without test labels, so we selected the best single juror on a labeled calibration split of 10–30% of each benchmark and evaluated both selectors on the remaining problems: over 500 random splits, the jury exceeds the calibrated single juror by +0.65 to +0.76 points on average and is ahead in every split. The advantage also holds across generators. Pooling both 30K-cap generators (Qwen3-235B and gpt-oss-120b, 7,502 problem pools over the seven benchmarks), the jury significantly outperforms every single-juror selector, even when that juror is chosen with hindsight: +0.5 [+0.2, +0.8] over Sonnet-match, +1.0 [+0.7, +1.3] over Kimi-match and +1.4 [+1.1, +1.8] over DeepSeek-match, and +0.9 [+0.4, +1.4] over Sonnet-match with the gpt-oss-120b pool alone. Over self-consistency, the jury gains +1.6 [+1.0, +2.1] with the Qwen3-235B pool and +3.3 [+2.6, +4.0] with the gpt-oss-120b pool, most where the pool leaves headroom (GPQA +4.5 and +9.6), using three single answers rather than a model that reads every candidate.

PRM comparison and the decorrelation experiments at 30K. We re-ran both at the 30K cap. All four trained verifiers (Qwen2.5-Math-PRM-7B/72B, AceMath-72B-RM, ThinkPRM-14B) now score the 30K Qwen3-235B pools with the same self-check gate and the same pool-frequency tie-break as the jury, and the revised §5.3 and its table now report these results, with the 4K table moved to Appendix B. The jury is at or above every verifier on every benchmark: on MATH-500 it significantly outperforms both discriminative PRMs (98.0 vs. 97.0, p = 0.006), while AceMath-72B and ThinkPRM, whose scores there fall below the self-check gate, revert to self-consistency (97.8; ungated 97.2 and 97.8); on GPQA it is ahead of all four (84.3 vs. 74.7–81.3), significantly so for three (p < 0.001); and on AIME-24/25 it is the strongest selector (93.3 and 86.7, vs. at most 90.0 and 80.0). Pooled over the four benchmarks, it significantly outperforms every verifier, including all three process reward models: +3.8 [+2.4, +5.4] over each Qwen2.5-Math-PRM, +2.9 [+1.6, +4.2] over AceMath-72B and +1.3 [+0.3, +2.5] over ThinkPRM (p ≤ 0.011). For §5.4, pairwise error correlation at the 30K cap is 0.75 within one model, 0.54 within a family (gpt-oss-120b and gpt-oss-20b on all seven benchmarks; 0.58 for Qwen3-235B and Qwen3.8-27B) and 0.42 across families, the same ordering as at 4K with a wider gap. The matched-accuracy isolation also holds at 30K, with exactly matched member accuracy by construction: on the same problems, a panel of k answers drawn half from Qwen3-235B and half from gpt-oss-120b beats k answers from either model alone by +1.0 [+0.8, +1.1], +1.3 [+1.1, +1.5], +1.4 [+1.1, +1.6] and +1.5 [+1.2, +1.9] pooled at k = 4, 6, 8 and 12. The combinatorial sweep also holds at 30K once member accuracy is controlled: among panels whose members are within 8 points of each other in accuracy, consensus gain over the panel's best member correlates with lower error correlation (r = +0.17 to +0.75 on the unsaturated benchmarks, pooled within-benchmark r = +0.55). The jury also improves over self-consistency at every pool size (+1.4, +1.4 and +1.6 at N = 4, 8 and 12).

Trained verifiers, 30K poolsMATH-500AIME-24AIME-25GPQA
Self-consistency97.886.780.079.8
Qwen2.5-Math-PRM-7B97.083.380.074.7
Qwen2.5-Math-PRM-72B97.086.776.774.7
AceMath-72B-RM97.8†90.080.075.3
ThinkPRM-14B97.8†90.080.081.3
Jury98.093.386.784.3

† Below the self-check gate (AUROC < 0.6); falls back to self-consistency.

Reviewer_DXtoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how to pick the correct answer from a pool of sampled reasoning chains. This is important because - with enough samples - a correct answer is usually in the pool, and so the selector therefore becomes the bottleneck.

The proposed LLM-jury works as follows: several models from different families each solve the problem once - and the method returns the pool candidate whose answer matches the most panel members, with the share of agreeing members double-hatting as a confidence score of sorts

On a pool of 12 Qwen3_235B samples across 7 benchmarks, the jury beats self-consistency and also beats Qwen scoring its own candidates. In addition, it also matches or beats 4 trained verifiers on MATH 500, AIME, and GPQA. The paper also proposes a closed-form model that predicts consensus accuracy from 3 measured panel statistics, with a mean absolute error of ~0.03. It concludes that cross-model agreement is a training-free verifier whose ceiling can be known in advance.

Critical strengths
  1. The gain over self-consistency is robust: the result holds with a different generator, an open-weight panel, a frontier panel, and pool sizes of 4 to 12 (as seen in Tables 13 to 16). On top of this, removing the generator from the vote or weighting votes by accuracy changes results by only about 1 point, so this makes it convincing that a panel of different models beats resampling a single model

  2. The paper shows clearly when agreement can be trusted: a unanimous panel on MATH-500 is right ~99.5% of the time while still answering ~85% of problems. The rate of confident shared errors is near 0 on math but reaches ~0.03 on GPQA and ~0.14 on MMLU Pro. All of this tells a practitioner where the method breaks (which is often more useful than another accuracy number).

Critical weaknesses

(1) The jury gets extra information from other models - so the comparison does not isolate the scoring signal.

The paper says all selectors score the same pool; however, the jury adds 3 new solutions from Claude, DeepSeek and Kimi, while self-consistency, the Qwen judge and the trained verifiers see only the Qwen’s 12 samples. The gain could, therefore, come from the stronger models knowledge, with agreement playing a very little role.

The paper’s own numbers support this concern: on AIME24, the panel answering directly scores 0.7 - which is above the pool’s oracle of 0.567 - so the panel solves problems that the pool never contained in the first place. Dropping Claude or Kimi from the panel also lowers AIME24 accuracy from 0.7 to 0.5. 2 missing baselines would do a lot of work to settle this question: (1) selecting the candidate that matches the strongest single member’s answer, and (2) a cross-model judge, such as Claude scoring Qwen’s candidates.

This issue is fixable - because if the jury doesn't beat the strongest member, though, then the contribution reduces to only “adding stronger models helps”

(2) The predictive law is not compared to anything simpler

The law’s inputs are measured on the same labeled problems it subsequently then predicts. Consensus accuracy mostly follows mean member accuracy, which ranges from ~0.37 to ~0.97 across benchmarks. A low error is - therefore - expected from almost any sensible model. Without a simple baseline (such as an independent-errors majority vote) it's unclear what the law adds. It's also least accurate where prediction matters the most: e.g. it predicts 0.531 against an observed 0.433 on AIME-25

Questions for authors
  1. On the same pools - how accurate is each panel member alone, and how accurate is “select the candidate matching the strongest member?" And please also clarify whether the generator votes, and explain the AIME-25 mismatch between Tables 1 and 14.
  2. How does the jury compare with a cross-model judge (for example Claude scoring each Qwen candidate)?
  3. How does the law’s error compare with a simple independent-errors majority vote that uses the same measured accuracy?
3 comments
AuthorsOpenReview ↗

Gentle reminder: author–reviewer discussion period ending in a few days

Dear Reviewer DXto,

We wanted to send a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we added the two baselines you suggested at a 30K cap across all seven benchmarks: selecting the candidate that matches the strongest member (chosen per benchmark using test labels, which never outperforms the jury) and Sonnet 4.6 judging each candidate. We also reported each member individually, clarified and redrew Figure 1 to make clear that the generator does not vote, updated the AIME-25 entry, and compared the law with independent-errors and other simpler predictors (accuracy MAE 0.015 vs. 0.036–0.081; floor MAE 0.004 vs. 0.021).

If these updates address your concerns, we would be glad to hear it, and we would be grateful if you would consider reassessing your rating accordingly. If there are any remaining questions or concerns, we would be happy to address them during the remaining discussion period.

Thank you again for the time and effort you put into reviewing our work.

Best regards,

Authors

Thanks for running the strongest-member, cross-model judge and simpler-predictor baselines. They address my questions, but the jury’s margin over them is small (~~0.5-1) - so I'm keeping my score.

AuthorsOpenReview ↗

Response to Reviewer DXto (follow-up)

We thank the reviewer for the follow-up and are glad that the new baselines address the reviewer's questions. We deeply appreciate the reviewer's engagement during the discussion phase. On the size of the margins, we would like to clarify three points: what the small margins compare, where the margins are large, and what they mean at the full budget.

The strongest-member baseline is itself cross-model agreement. The reviewer's original concern was that the jury's gain might come from the stronger models' knowledge, with agreement playing little role. Selecting the candidate that matches the strongest member's answer is agreement between the pool and one independent model, that is, a jury of one. The 30K results separate the two sources directly. Sonnet 4.6 answering alone reaches 88.4. Selecting the pool candidate that agrees with Sonnet reaches 91.1 (+2.7), and the panel answering directly without the pool reaches 90.0. Agreement between an independent model and the pool therefore adds substantially to the stronger models' own answers. The comparison between the three-member jury and Sonnet-match then measures what further jurors add to an agreement signal that is already present, which we address next.

What more jurors add. The Sonnet-match baseline is chosen with test-set hindsight, and the best single juror changes across benchmarks and generators. A deployment must choose the juror without that hindsight. When the single juror is chosen on a labeled calibration split (10–30% of each benchmark) and both selectors are evaluated on the remaining problems, the jury is ahead in all 500 random splits at every split size (+0.65 to +0.76 on average, smallest margin +0.27). Even against hindsight-selected jurors, pooled over both 30K generators (Qwen3-235B and gpt-oss-120b, 7,502 problem pools), the three-member jury exceeds every single-juror selector: +0.5 [+0.2, +0.8] over Sonnet-match, +1.0 [+0.7, +1.3] over Kimi-match and +1.4 [+1.1, +1.8] over DeepSeek-match. Larger juries also depend less on which models they contain: across all juries of one to six members, the weakest jury rises from 89.0 with one juror to 90.8 with five. Multiple jurors also give graded confidence that a single juror cannot. When all three agree with the selected candidate (80.8% of problems), it is correct 96.0% of the time, against 49.1% when at most one agrees. At 90% coverage, the agreement count reaches 95.3% accuracy, against 92.6–94.5% for single-juror match. The revised Appendix B now reports these results.

The margins are bounded by headroom, and they are large where headroom exists. At the 30K cap, self-consistency already reaches 89.6 pooled against a pool oracle of 93.6, and on MATH-500 and GSM8K every pool selector is within 1.5 points of the oracle, so no selector can improve much on these benchmarks. On GPQA, the benchmark with the most headroom, the jury reaches 84.3 against 81.3 for both Sonnet-match and the Sonnet judge. With the gpt-oss-120b pool, it reaches 85.4 against 80.8 for the best single-juror match. Pooled over all seven benchmarks, the jury closes 40% of the gap between self-consistency and the pool oracle with the Qwen3-235B pool, against 25% for the Sonnet judge. With the gpt-oss-120b pool, it closes 52%, against 38% for the best single-juror match. Against the cross-model judge, the comparison is also one of cost: the jury is ahead (+0.5 [+0.1, +1.0], on six of seven benchmarks) while using 7–11× fewer tokens, because three single answers replace a model that reads all twelve candidates. Against trained verifiers at the 30K cap, pooled over the four benchmarks it is significantly better than every one we test, by +1.3 to +3.8 points.

The law against simpler predictors. Here the margins are large. The baseline the reviewer proposed, independent-errors majority voting with the same measured member accuracies, has an accuracy MAE of 0.116 at 4K and 0.079 at 30K when extrapolating from three-member panels to larger panels, against 0.015 and 0.020 for the law (4–8× lower). Its floor MAE is 0.0144 and 0.0109, against 0.0066 and 0.0035 for the law. In panels of one generator's samples, where the law's equal-accuracy assumption holds exactly, the law's accuracy MAE is 0.0105, against 0.0197 for correlated errors without wrong-answer concentration.

Taken together, the comparison with the hindsight-selected single juror is a particularly demanding one. In the setting a practitioner faces, the three-member jury is consistently ahead, gives a graded confidence signal, and costs a fraction of a judge that reads every candidate. We thank the reviewer again for the constructive questions, which led to the baselines that now separate the agreement signal from the stronger models' knowledge.

Reviewer_RUzCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses cross-model consensus as a training-free Best-of-N selector, which it calls an "LLM-jury". Several independently trained LLMs (Qwen3-235B, DeepSeek-V3.2, Claude Sonnet 4.6, Kimi-K2.5) each solve a problem once. Each candidate in a fixed pool (N=12 samples from Qwen3-235B) is then scored by how many panel members' final answers match it. The claimed contributions are:

  1. On seven benchmarks (AIME-24/25, MATH-500, OlympiadBench, GSM8K, GPQA-Diamond, MMLU-Pro), the jury outperforms self-consistency, and also a pointwise LLM-verifier in which the generator scores its own candidates.
  2. A closed-form Beta-Binomial model that predicts consensus accuracy, the selective-prediction curve, and a "shared-error floor" (the rate at which the whole panel agrees on the same wrong answer). Its inputs are measured mean accuracy a, pairwise error correlation ρ, and wrong-answer concentration s. Prediction error on accuracy is MAE 0.028.
  3. A comparison against four trained verifiers (Qwen2.5-Math-PRM-7B/72B, AceMath-72B-RM, ThinkPRM-14B). The jury ties the strongest of them on MATH-500 and leads on GPQA.
  4. Evidence that error decorrelation is the mechanism. Error correlation is 0.68 within one model, 0.52 within one family, and 0.47 across families, and the paper adds matched-accuracy panel comparisons.

The appendices add an agreement-gated cascade, behavioral consensus on HumanEval+, consensus as a reward for rejection-sampling fine-tuning, and robustness checks: a different generator, fully open-weight and frontier panels, an ablation over N, and the sensitivity of the trained-verifier gating threshold.

Critical strengths
  1. Controlled, careful evaluation. All selectors choose from the same pool and are scored by one grading harness. Significance uses a paired bootstrap, and the main result holds under many robustness checks: a different generator (Table 13), a fully open-weight panel (Table 14), a frontier panel (Table 15), N ∈ {4, 8, 12} (Table 16), and different gating thresholds for the trained verifiers (Table 18). I therefore trust the reported numbers as measurements. My concerns are about what those numbers show, not whether they are correct.

  2. The shared-error floor characterizes when agreement fails. The paper measures the floor at about zero on competition math and 0.030 on GPQA. The GPQA case study (Appendix D) shows that the unanimous errors come from conventions all models share, such as reporting molar rather than per-gram enthalpy, not from chance agreement. The paper also separates accuracy from calibration (Appendix E): cross-model voting is more accurate, while within-model agreement is the better-calibrated confidence signal. This is the most informative part of the paper, and it is why I rate Informativeness 2.

Critical weaknesses
  1. The paper's own results contradict its "verifier" framing. Jury members never look at the candidates. The selection rule simply returns the pool candidate that matches the panel's most common answer, so the pool can only restrict the panel's answer, never improve it. Table 1 (selecting from the pool) can be compared with Tables 2 and 11 (the four-model panel answering directly). Answering directly is as good or better on 5 of 7 benchmarks:

    AIME-24AIME-25OlympiadBenchGPQAGSM8K
    Selecting from the pool (Table 1)56.740.072.635.997.2
    Panel answering directly (Tables 2, 11)70.043.373.436.997.3

    Selecting from the pool is better only on MATH-500 (96.6 vs 95.6) and MMLU-Pro (46.2 vs 45.8). The two settings do not use exactly the same panel: when answering directly, the generator's greedy answer counts as a vote, while in selection the generator only breaks ties (Appendix K). Even so, Table 17 shows that excluding the generator does not change selection accuracy.

    The headline claim that the jury "closes 100% of the oracle gap" on AIME-24 is therefore a product of the pool's ceiling: the pool's oracle is 56.7, and the panel alone exceeds it. Without the verifier framing, the method is cross-model majority voting, which the paper itself calls known (§1: "Our claim is not that 'voting helps' (known)"). This removes the main methodological contribution, and a rebuttal cannot fix it without changing the thesis and the title.

  2. The headline comparisons mix up the selection signal with panel strength and compute, and the most basic baseline is missing. The LLM-verifier baseline is the generator scoring its own candidates, while the jury is made of other models that appear to be stronger: on AIME-24 the panel answering directly beats the generator pool's oracle. The contrast between "≈0% of the oracle gap" (self-scoring) and "100%" (jury) therefore mixes "self versus other models" with "weaker versus stronger models."

    • No best-single-model baseline. The paper reports neither each member's accuracy alone nor the accuracy of the best single member. The available evidence suggests the best member may match the vote:
      • Removing Qwen3-235B or DeepSeek-V3.2 raises AIME-24 consensus from 0.70 to 0.77 (Appendix E).
      • On HumanEval+, consensus (0.909) is below the best member (0.933, Appendix H).
      • The Limitations section concedes that voting does not beat the strongest member when member strength is unequal.
    • The decorrelation evidence is thin. The matched-accuracy experiment, which is the one that isolates decorrelation, shows gains only on AIME-24 (n=30). On AIME-25 and GPQA the change ranges from −0.8 to +3.6.
    • "Outperforms PRMs" is overstated. The comparison puts M full generations from frontier models against 7B–72B scorers. On MATH-500, the trained verifiers' own domain, the jury does not significantly beat AceMath-72B-RM (96.6 vs 95.6, p=0.142), and the GPQA win is over math-specialized reward models used outside their training domain.
Questions for authors
  1. Please report, on all seven benchmarks and with the same panel members, (a) the panel's majority answer returned directly and (b) selection from the Qwen3-235B pool. Panel members already write complete solutions, so in what deployment setting is (b) preferable to (a)?

  2. Please report each panel member's single-sample accuracy on every benchmark. Please also add an LLM-verifier baseline in which the strongest panel member (e.g., Claude Sonnet 4.6) scores the Qwen3-235B pool. If the cross-model vote clearly beats both the best single member and this strong-member verifier, my view of Weakness 2 would change substantially.

  3. The law's inputs a, ρ, and s are all defined through correctness, so they need labels (Appendix B), and ρ is a direct measure of error correlation. What does the law predict that cannot be read directly from these labeled statistics, beyond the standard Beta-Binomial treatment of correlated ensemble errors? And how is it used "in advance" on a new domain that has no labels?

1 comment
AuthorsOpenReview ↗

Gentle reminder: author–reviewer discussion period ending in a few days

Dear Reviewer RUzC,

We wanted to send a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we regenerated every pool and panel answer at a 30K cap, removing the pool-ceiling confound (the AIME-24 pool oracle rises from 56.7 to 93.3), and reported results on all seven benchmarks for direct panel answering, each individual member, and Sonnet 4.6 judging each candidate. Selecting from the pool improves over direct panel answering by 1.2 points and over the strongest member by 2.8 points. We also removed the "parameter-free" and "in advance" claims, compared the law with simpler predictors given the same inputs, retitled §5.3 to distinguish significant from non-significant comparisons with trained reward models, and reframed the method as a training-free selection signal.

If these updates address your concerns, we would be glad to hear it, and we would be grateful if you would consider reassessing your rating accordingly. If there are any remaining questions or concerns, we would be happy to address them during the remaining discussion period.

Thank you again for the time and effort you put into reviewing our work.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.