acceptodds
Under review as a conference paper at ICLR 2027

Oracle Coverage Can Reverse Rankings: Finite-Budget Identification in Test-Time Scaling

Abstract

Ranking systems by pass@K (oracle coverage) can disagree with ranking them by selected accuracy. We prove that strict oracle dominance at every sampling budget can coexist with plurality regret approaching one. We then derive sharp finite-budget identification bounds on plurality accuracy given correct-answer structure, abstention mass, and the largest wrong-answer probability. With these quantities fixed, extremal allocations of wrong-answer mass determine the accuracy range. Reanalysis of public response banks on 3,600 SuperGPQA problems finds high-budget ranking reversals between reasoning-effort settings after simultaneous correction, including when invalid responses participate in voting. On a 1,800-question development split at K = 64, adding the largest wrong-answer probability narrows empirical identification widths from 30.81–44.77 to 0.49–1.39 percentage points across three effort settings. These ranges describe information loss, not statistical confidence. We also give a fixed-score stopping certificate that skips verifier queries without changing the full weighted-vote decision. On 64 problems per policy, batch-preserving execution reduces verifier-stage time by 51–70%, excluding candidate generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, large language model reasoning, model selection, finite-budget identification, inference-time selection, self-consistency, reinforcement learning with verifiable rewards

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FbAAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks whether oracle coverage identifies the best system under a deployed selector. It proves that strict oracle dominance can coexist with plurality regret approaching one, derives sharp accuracy bounds from partial categorical information, and examines ranking reversals empirically. A secondary contribution skips verifier queries while preserving the full weighted-vote decision.

Critical strengths
  • The sharp identification bounds are the strongest independent contribution. Proposition 2 characterizes the range of voting accuracies compatible with retained correct-category information, abstention mass, and the largest wrong-answer probability. This goes beyond evaluating or allocating samples under a specified distribution.

  • Proposition 1 gives a useful consequence for system selection: even complete oracle curves can support the wrong deployment choice. The empirical example is substantial—at K=80K=80, low effort has 2.86 percentage points higher oracle coverage but 7.64 points lower plurality accuracy than high effort.

  • The empirical checks are careful. The main reversal survives simultaneous correction and invalid-vote controls, while parser sensitivity and stable overall-winner cases are reported explicitly.

Critical weaknesses
  • The related work should discuss PETS (Liu et al., 2026). Its §2 and Appendix C.3 use answer distributions, population voting targets, and vote margins, sharing the foundations of this paper’s §2 and Appendix D. The novel contribution lies in cross-system identification and sharp bounds, rather than the underlying voting mechanism.

  • The practical use of the bounds remains less established than their theoretical value. The empirical ranges exclude distribution-estimation uncertainty, including unseen categories. The pilot also does not demonstrate an advantage over pass@1 or direct selector evaluation.

  • The deployment evidence has a limited scope. Including Qwen leaves the overall winner unchanged, effort settings have different generation costs, and the reported 51–70% time savings cover verification only. The query-skipping result is a useful application of decision preservation, but should remain secondary to the identification contribution.

Questions for authors
  • How sensitive are identification-based decisions to uncertainty in the largest wrong-answer mass and to unseen answer categories?

  • What practical setting benefits from partial-information bounds when direct selector evaluation is available?

  • What fraction of total pipeline time is spent in verification under the measured configuration?

5 comments
AuthorsOpenReview ↗

Response to Reviewer FbAA (Part 1/3)

Thank you for the precise review and for the clear description of what the paper contributes. As you suggested, the revision puts the identification results first, and the query-skipping certificate now appears in Section 4.4 as an application of decision preservation.

W1. PETS. Section 5 now discusses PETS (Liu et al., 2026) and the closely related Feng et al. (ICML 2026), and it makes explicit, as you put it, that our contribution lies in cross-system identification and the sharp bounds. PETS models each question's answers as a categorical distribution and defines the self-consistency rate as the probability that a finite majority vote agrees with the population majority. It allocates samples across questions with a Gaussian approximation to multinomial vote margins (its Appendix C.3). Feng et al. bound how fast self-consistency finds the mode through the margin (p1−p2)2(\sqrt{p_1}-\sqrt{p_2})^2. Both ask how a vote approaches one model's population mode under a given distribution. We ask three different questions:

  1. whether the selected answer is correct when systems are compared;
  2. which partial information identifies that accuracy, answered by sharp worst- and best-case allocations over all distributions consistent with (c,ρ,w)(c,\rho,w);
  3. how to certify these ranges from finite samples, for all budgets at once.

The two views fit together: in a PETS-style allocation, a question whose range is already decided needs no further samples.

W2/Q1. Sensitivity to uncertainty in the largest wrong mass and to unseen categories.

Unseen categories do not enter the ranges. Proposition 2 already takes the worst and best case over every way the unobserved wrong mass could be spread, including over any number of answers that were never sampled. Only the strongest wrong answer matters. Lemma 1 shows that the error in its estimate does not grow with the number of distinct answers. The new Proposition 3 bounds the competing mass using only the count of the true strongest competitor, so the bound holds even when that answer never appears in the sample.

The experiments now include this estimation uncertainty (Section 4.2, Table 2, Appendix E). We re-estimate cc and ww from nn saved responses per question and predict plurality accuracy at K=64K=64. At n=16n=16, the bias-corrected range is the most accurate rule on each bank, within 0.4, 1.6 and 1.4 points on SuperGPQA, Scorio Trace and MATH-500. A two-level bootstrap gives intervals that contain the 64-sample accuracy in every draw. Every ordering they certify is correct: on average 5.3 of 6 pairs on SuperGPQA, 122 of 190 on Scorio Trace, 15 of 21 on MATH-500, 51 of 55 on the new 11-system SuperGPQA bank from 8 of 32 responses, and 19 of 21 on MMLU-Pro. On the pairs that pass@KK orders the wrong way, they never certify the pass@KK order.

AuthorsOpenReview ↗

Response to Reviewer FbAA (Part 2/3)

Q2. Which practical setting benefits when direct selector evaluation is available?

Direct evaluation measures a selector at the budget it samples, and estimating plurality@KK directly needs at least KK responses per question for every candidate. The bounds add three things.

  1. Accuracy at budgets that were not sampled. Evaluations often use fewer samples than deployment. On MATH-500, where accuracy keeps rising with KK, plurality@16 misses the 64-sample accuracy by 4.4 points, and its interval contains it in only 47% of draws. The bias-corrected range misses by 1.4 points, and its interval always contains it. Pass@1 misses the level by 3.7 to 28.0 points, and extrapolated pass@KK overstates it by 11 to 26 points. The advantage of the range grows with the deployment budget (new Table 25). From the same 16 samples, it misses the 128- and 256-sample accuracies by 1.6 and 1.5 points, and its intervals contain them in 99% and 100% of draws. Plurality@16 misses them by 5.4 and 5.5 points, and its interval contains them in only 30% and 29% of draws. On MMLU-Pro, from 4 of 32 samples per question, the range misses the 32-sample accuracy by 0.4 points and its interval always contains it, while the interval around plurality@4 contains it in only 1% of draws.
  2. Certified comparisons. At K=64K=64 (K=32K=32 on the new banks), orderings certified by the range intervals were correct in every draw on every bank. From 16 samples, those certified at K=128K=128 and 256 were all correct as well. On the Scorio Trace pairs that pass@KK orders the wrong way, extrapolated pass@KK agrees with plurality in only 24 to 43% of draws, while the corrected range agrees in 68 to 75% (pass@1: 64 to 69%). On the new SuperGPQA bank, the intervals certify the plurality order on these pairs in 94% of draws.
  3. Screening. A system is dropped once another system is certified better than it. From 16 samples per question, this drops 3 of the 4 SuperGPQA systems, 6 of the 7 MATH-500 systems and 15 to 18 of the 20 Scorio Trace systems, and never the system that is best at K=64K=64. Even if every remaining system is then evaluated directly at K=64K=64, screening saves 56 to 64% of the generated samples (Section 4.2, new Table 22). On MMLU-Pro, 4 samples per question drop 5 of the 7 systems, never the best one, and save 62% at K=32K=32. On MATH-500 the saving grows with the deployment budget, to 75% at K=128K=128 and 80% at K=256K=256. The ranges need only per-question answer frequencies, so they can also be computed when only those are reported.

The question-disjoint pilot (Section 4.2) shows what is at stake. Development questions choose the system and held-out questions test it, and choosing by oracle coverage instead of plurality loses 2.32 points on the held-out half (simultaneous 95% CI [0.91,3.73][0.91,3.73]). Compared with pass@1 and with direct evaluation at the smaller budget, the bounds give the accurate level at unsampled budgets, certified orderings and cheaper screening, and their advantage widens at larger deployment budgets (Table 2, Appendix E.2).

AuthorsOpenReview ↗

Response to Reviewer FbAA (Part 3/3)

W3. Scope of the deployment evidence.

  • Overall winner. The census adds banks where the pass@KK winner and the plurality winner differ. On MATH-500, choosing by pass@KK at K≥64K\ge64 costs 17.6 to 18.8 points when invalid outputs count as votes. Among the seven new 3.8B to 8B SuperGPQA models, pass@32 picks Llama-3.1-8B, whose plurality accuracy is 6.3 points below Qwen2.5-7B's (simultaneous interval [3.2,9.4][3.2,9.4]; Section 4.1). This pair is itself a significant reversal, with the same signs in each half of the 3,600 questions. On 3,600 MMLU-Pro questions, pass@32 again picks Llama-3.1-8B, 9.8 points below Qwen2.5-7B ([7.2,12.4][7.2,12.4]), and this pair is again a significant reversal.
  • Generation cost of the effort settings. We repeated the effort comparison at equal generated tokens (new Appendix D.4, Table 15). A high-effort response uses on average 10,671 completion tokens and a low-effort one 355, so with the tokens of one high-effort sample, low effort draws 30. It then has 33.3 points more oracle coverage but 5.3 points lower plurality accuracy (simultaneous interval [−6.8,−3.8][-6.8,-3.8]). Of all 11 token-matched comparisons between the three settings, 9 are resolved reversals under both validity rules. Low-effort plurality accuracy stays between 39.7% and 40.3% from 30 to 80 samples, while high effort reaches 45.0% with one sample, so this ordering does not depend on how the cost of long responses is counted.
  • Verification-only savings. The certificate acts on the verifier stage. Q3 below places that stage in the whole pipeline.

Q3. Fraction of pipeline time spent in verification. We measured it on one L40S, the GPU type used for all verifier timings (Section 4.4; Appendix C.4, new Table 8). The MATH-500 pools of Base and GRPO-42 were generated with vLLM on one L40S per shard, eight questions and 256 samples per call, and the time of every call is logged. Amortized over the 500 questions, generating 64 candidates costs 6.5 s per question for Base and 5.8 s for GRPO-42. Full PRM-7B scoring of the same 64 candidates takes 5.5 to 9.8 s per question, depending on how responses are grouped into blocks. Verification is therefore 46 to 63% of end-to-end time in this configuration, and the certificate, which cuts verifier time by 51 to 70%, shortens the whole pipeline by 30 to 36%.

The code and outputs of every new analysis are in the updated supplementary material and in the anonymous repository linked in the abstract.

Thank you again. Your comments on scope and practical use made the paper clearer about when the bounds help. If the new results resolve your concerns, we would be grateful if you would take them into account in your final assessment.

Reviewer_Vm8uOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that the general practice of ranking systems by pass@K may not align with the actual accuracy of practical systems which use a selection criteria (such as plurality voting, self consistency, or a learned verifier). In particular for the case of plurality, the authors demonstrate a counterexample where pass@K success approaches 1, while plurality regret also goes to 1. (Proposition 1)

In addition, the authors show that if we know the frequency of the correct answer, and the single most frequent wrong answer, plus some other distributional properties, one can relate the plurality success of the classifier. Finally they back up their work with empiricial experiments supporting these general laws.

Critical strengths
  1. I appreciate the proofs of Proposition 1 and Proposition 2 in Appendices D and X respectively. In general, the paper is much stronger for giving theoretical backing to the heuristic observation on the shortcomings of Pass@K for evaluating practical systems.

  2. With 5 benchmarks and 10 models used, the experiments are also quite extensive and thorough.

  3. The study is well motivated - pass@K is a ubiquitous benchmark, but in real world systems without an actual oracle, is it an effective measure of the true accuracy returned by a selector?

Critical weaknesses
  1. I found the writing quite difficult to parse. The abstract, introduction and conclusions for instance contain more numbers rather than crisply conveying the major findings. For proposition 1, it is not indicated in the main body where the proof is - I had to search to find it was in Appendix D - and the statement of proposition 2 is written in a clunky manner in the main body where it took several reads to grasp exactly what was proved. This could perhaps have been mitigated, for instance, by writing Proposition 2 as equations instead of clunky prose.

  2. The writing often reads as AI generated, and the appendix is not well organized or referenced from the main body.

  3. In section 3 of the experimental protocol where the selectors are described, how exactly we measure the divergence between selector accuracy and pass@K is not clearly outlined, despite this being the most natural place one would expect such clarifications.

This is not my core area, and I accept that may have made it harder for me to follow the manuscript flow, although this is still at least partially due to clumsy presentation too.

Note: The authors have sufficiently mitigated my concerns during rebuttal on these weaknesses, so I have updated my score accordingly.

Questions for authors
  1. Can you express the statement of Proposition 2 as a set of equations? What do you mean by " the minimum accuracy at each budget K "?
  2. What are the main takeaways from this paper, both for theoreticians studying learning theory, and for practitioners trying to benchmark such systems in practice?
8 comments
AuthorsOpenReview ↗

Response to Reviewer Vm8u (Part 1/2)

Thank you for reading the paper closely and for the concrete suggestions. We followed all of them. We answer the two questions first and then describe the changes for each weakness.

Q1. Proposition 2 as equations, and the meaning of "the minimum accuracy at each budget KK".

Fix one question. Let cc be the probability of the correct answer, ρ\rho the probability of an invalid output, and q=(q1,q2,… )q=(q_1,q_2,\dots) the probabilities of the distinct wrong answers, so that W=∑aqa=1−c−ρW=\sum_a q_a=1-c-\rho. Write AK(c,q,ρ)A_K(c,q,\rho) for plurality accuracy with KK independent samples. If we know cc, ρ\rho and the largest wrong mass ww, the wrong-answer distributions that remain possible are Q(c,ρ,w)={q≥0:∑aqa=W, max⁡aqa=w}\mathcal Q(c,\rho,w)=\lbrace q\ge0:\sum_a q_a=W,\ \max_a q_a=w\rbrace, and the lowest and highest accuracies they allow are LK(c,ρ,w)=inf⁡q∈QAK(c,q,ρ),UK(c,ρ,w)=sup⁡q∈QAK(c,q,ρ).L_K(c,\rho,w)=\inf_{q\in\mathcal Q}A_K(c,q,\rho),\qquad U_K(c,\rho,w)=\sup_{q\in\mathcal Q}A_K(c,q,\rho). Proposition 2 states that, for every K≥1K\ge1, LK=AK(c,(w,…,w⏟⌊W/w⌋, W−w⌊W/w⌋),ρ),UK=lim⁡m→∞AK(c,(w,W−wm,…,W−wm),ρ).L_K=A_K\big(c,(\underset{\lfloor W/w\rfloor}{\underbrace{w,\dots,w}},\ W-w\lfloor W/w\rfloor),\rho\big),\qquad U_K=\lim_{m\to\infty}A_K\big(c,(w,\tfrac{W-w}{m},\dots,\tfrac{W-w}{m}),\rho\big). In words, the worst case packs the wrong mass into as many copies of the strongest wrong answer as possible, and the best case keeps one strongest wrong answer and spreads the rest thinly.

"The minimum accuracy at budget KK" is LKL_K: no answer distribution with these values of cc, ρ\rho and ww has lower plurality accuracy with KK samples. The two extreme distributions do not depend on KK, so one calculation gives the range at every budget. Section 2.1 now states the result in this form and adds an example. For a question with c=0.4c=0.4 and no invalid outputs, plurality accuracy at K=64K=64 can be anywhere in [0.05,1.00][0.05,1.00]; knowing w=0.3w=0.3 narrows it to [0.73,0.83][0.73,0.83], and w=0.5w=0.5 fixes it at 0.200.20.

Q2. Main takeaways (now at the end of the introduction and in Section 6).

For theory. Choosing a system for a deployed selector is an identification problem. Pass@KK identifies only the correct-answer mass. For plurality, the missing quantity is the mass of the most common wrong answer. With it, the range of possible accuracies is sharp at every budget (Proposition 2), it can be estimated without knowing the set of possible answers (Lemma 1), and finite-sample versions hold for all budgets at once (Proposition 3). The extremal argument, a majorization over how wrong answers share their mass, can serve as a template for other selectors.

For practice.

  1. Compare systems with the selector and budget that will be deployed, not with pass@KK. The two rankings drift apart as KK grows, and on MATH-500 choosing by pass@KK cost up to 19 points.
  2. If evaluation uses fewer samples than deployment, record each question's correct and largest wrong answer frequencies. They bound plurality accuracy at any budget, and from 16 samples per question they predicted 64-sample accuracy to within 1.6 points.
  3. Report how invalid outputs are treated, since that choice alone can reverse rankings at small budgets.
AuthorsOpenReview ↗

Response to Reviewer Vm8u (Part 2/2)

W1 and W2. Writing, proof locations and the appendix. Here is what changed for each point.

  • Numbers in the abstract, introduction and conclusion. These now state the findings in words. The abstract keeps only a few rounded numbers, and the concluding paragraph has none. The detailed numbers are in the results sections. The introduction now states each contribution as a finding in plain paragraphs, for example "In practice the two rankings drift apart".
  • Where the proofs are. Every formal result in the main text is followed by its proof location: Proposition 1 (Appendix C.1), Proposition 2 (C.2), Lemma 1 and the new Proposition 3 (C.3). The certificate for skipping verifier calls in Section 4.4 is Proposition 7 in C.4. When the main text cites an equation, lemma or table from the appendix, it names the appendix as well.
  • Proposition 2. It is now written as equations (Q1 above), followed by a plain reading of the two extreme cases, three consequences and a worked example.
  • Style. We rewrote the main text sentence by sentence. Sentences are shorter, internal terms are replaced by plain descriptions, and each finding in Section 4.1 has its own short paragraph with a heading. Protocol details moved to the appendix.
  • The appendix. It opens with a roadmap table (Appendix A) that maps each result in the main text to its proof or protocol. Its sections follow the order of the main text: definitions (B), proofs (C), the reversal census (D), identification experiments (E), the RLVR study (F), selectors (G), compute and reproduction (H), related work (I) and limitations (J). Every appendix with subsections now opens with a short guide to them and to the part of the main text it supports, and all headings use plain, consistent titles.

W3. How the gap between selector accuracy and pass@KK is measured. Section 3 now opens with a paragraph, "Measuring the gap", in the place you expected it. It explains how each quantity is estimated from the saved responses, when a reversal counts as statistically resolved, and which summaries we report for each bank. Section 2 defines the quantities themselves ("What we measure"), because the theory uses them. Together the two paragraphs define

  • oracle coverage Oi(K)O_i(K), the familiar pass@KK, computed exactly from the saved responses;
  • the selected accuracy Ai,s(K)A_{i,s}(K) of selector ss, averaged over random subsets of KK responses;
  • their gap Oi(K)−Ai,s(K)O_i(K)-A_{i,s}(K), the coverage the selector does not recover;
  • a ranking reversal: the oracle and selector differences between two systems have opposite signs;
  • a statistically resolved reversal: both simultaneous 95% intervals exclude zero, with one correction over all comparisons in a bank;
  • the regret of deploying the system with the highest pass@KK.

W4. We took this as a sign that the paper has to work for readers outside its immediate area. Sections 1 and 2 now carry the whole argument, and the worked example after Proposition 2 shows what the bounds mean for a single question, so the main results can be followed without the appendix.

Thank you again. If any part is still hard to follow, we would be glad to revise it further during the discussion. If the revision addresses your concerns, we would be grateful if you would take it into account in your final assessment.

Reviewer_Vm8uOpenReview ↗

Helpful for a next version.

I do genuinely appreciate the changes and updates. The more smoothly written version of this paper is one I would enjoy greatly, and the question is an important and timely one.

However, as opposed to running new experiments, it is just very difficult to improve the quality of exposition in rebuttal, where the authors are not permitted to upload a new version of the paper. And this is particularly pertinent for a mathematically motivated manuscript where exposition is critical.

So I will retain my weak rejection score. However, if the AC feels the paper can be smoothed out sufficiently between acceptance and camera-ready, I would certainly not object either. I believe this is a very promising work.

Reviewer_EusSOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the discrepancy between oracle coverage (pass@K) and practical test-time selection accuracy in LLM reasoning. It derives sharp finite-budget bounds for plurality voting and demonstrates that higher pass@K can lead to worse model rankings under practical selectors. Experiments on SuperGPQA and GRPO-trained models highlight this discrepancy, while a certified stopping method reduces verifier computation without changing selection outcomes.

Critical strengths
  • The paper derives sharp finite-budget identification bounds, providing a principled characterization of the gap between oracle coverage and practical selection accuracy.

  • The experiments demonstrate statistically significant ranking reversals on SuperGPQA, supported by controlled GRPO experiments and uncertainty analysis.

  • The work highlights the limitations of pass@K for model selection and introduces a certified stopping procedure that reduces verifier-stage computation without changing selection outcomes.

Critical weaknesses
  • While the paper demonstrates statistically significant reversals across reasoning-effort settings, these do not consistently extend to comparisons across model families. Broader evaluations would strengthen the practical significance of the findings.

  • The theoretical bounds require knowledge of answer-distribution statistics, particularly the largest incorrect-answer probability, which may be difficult to estimate reliably under limited sampling budgets. The paper does not sufficiently establish the sample efficiency or practical utility of these bounds.

Questions for authors
  • How frequently do ranking reversals occur across different model families, benchmarks, and sampling budgets, beyond comparisons between reasoning-effort settings?

  • How many samples are required to reliably estimate the largest incorrect-answer probability? Can the authors evaluate the identification bounds when these statistics are estimated from limited samples rather than assumed known?

  • Does GRPO systematically change the concentration or diversity of incorrect answers? Could the authors analyze how these distributional changes affect the gap between oracle coverage and practical selection accuracy?

9 comments
AuthorsOpenReview ↗

Response to Reviewer EusS (Part 1/3)

Thank you for the careful review and the clear questions. Your two weaknesses led to the two largest additions in the revision. We answer them below, together with Q1 to Q3.

W1/Q1. How often do ranking reversals occur across model families, benchmarks and budgets?

They occur on all four benchmarks of the census, SuperGPQA, MMLU-Pro, MATH-500 and competition math, and become more common as the budget grows. With enough questions, they are statistically significant across model families too, on two benchmarks.

What we ran (Section 4.1, Table 1, Figure 3, Appendix D). We ran a census over every multi-sample bank available to us, 44 systems from 11 model families in total:

  • Scorio Trace: 20 configurations of 18 reasoning models on AIME 2024/2025, BRUMO 2025 and HMMT 2025, 80 samples each.
  • Scorio Lite: Qwen3.6 and three effort settings of gpt-oss-20b on 3,600 SuperGPQA questions and 186 math problems.
  • Our 16 trained MATH-500 systems (Qwen2.5-Math 1.5B and 7B, Mistral-7B).
  • New for this revision: seven open instruction models of 3.8B to 8B parameters from seven developers (Meta, Qwen, Mistral, Microsoft, AI2, IBM, TII), 32 samples on each of the 3,600 SuperGPQA questions and on each of 3,600 randomly drawn MMLU-Pro questions.

For every pair of systems and every budget, we check whether pass@KK and plurality accuracy order the pair the same way. Significance uses one simultaneous correction (max-TT) over all comparisons in a bank.

How often. In each bank of Table 1, the two rankings drift apart as the budget grows. Kendall's τ\tau between them (1 means the same ranking) falls from about 0.95 at K=8K=8 to 0.79 at K=80K=80 on the reasoning models and to 0.40 at K=256K=256 on MATH-500. On the new SuperGPQA bank it falls from 0.85 at K=8K=8 to 0.60 at K=32K=32, and on MMLU-Pro from 0.90 to 0.60. At the largest budget, 10% of the Scorio Trace pairs, 28% of the MATH-500 pairs, 20% of the new SuperGPQA pairs and 19% of the MMLU-Pro pairs are ordered in opposite ways. On the new SuperGPQA bank these include 9 of the 51 cross-family pairs.

Significant cross-family reversals. Pass@KK saturates as KK grows, so pass@KK differences shrink toward zero while plurality differences stay. Showing both signs at once takes many questions. On the 3,600 SuperGPQA questions of Scorio Lite, both effort reversals are significant at every budget from 32 to 80. To make the new bank large enough, we sampled the seven models on all 3,600 SuperGPQA questions. Four cross-family reversals are then significant at K=32K=32 (simultaneous intervals in brackets):

  • OLMo-2-7B has 3.4 points higher pass@32 than Falcon3-7B ([0.3,6.6][0.3,6.6]) but 7.3 points lower plurality accuracy ([4.1,10.4][4.1,10.4]).
  • Phi-3.5-mini has 3.5 points lower pass@32 than OLMo-2-7B ([0.4,6.6][0.4,6.6]) but 3.5 points higher plurality accuracy ([0.6,6.4][0.6,6.4]).
  • Llama-3.1-8B has 3.5 points higher pass@32 than Qwen2.5-7B ([0.7,6.4][0.7,6.4]) but 6.3 points lower plurality accuracy ([3.2,9.4][3.2,9.4]).
  • Llama-3.1-8B has 3.3 points higher pass@32 than gpt-oss-20b at high effort ([0.3,6.3][0.3,6.3]) but 20.2 points lower plurality accuracy ([16.9,23.5][16.9,23.5]).

The two models on the high-coverage side, OLMo-2-7B and Llama-3.1-8B, show the mechanism. They have the highest pass@32 of the seven new models (78.0% and 79.8%): within 32 samples they reach the correct option on most questions. Yet the correct option wins the vote on only 22.4% and 27.4% of questions (Table 9). Oracle coverage rewards reaching the answer once, while plurality rewards agreeing on it.

The pattern repeats on MMLU-Pro, where Llama-3.1-8B again has the highest pass@32 (89.9%). Two cross-family reversals are significant at K=32K=32: Llama-3.1-8B has 2.6 points higher pass@32 than Qwen2.5-7B ([0.7,4.5][0.7,4.5]) and Falcon3-7B ([0.6,4.6][0.6,4.6]) but 9.8 and 9.7 points lower plurality accuracy ([7.2,12.4][7.2,12.4] and [7.0,12.4][7.0,12.4]). The Llama-Qwen reversal is significant on both benchmarks.

The result replicates. The new SuperGPQA bank was sampled in two halves of 1,800 questions, and each half shows every one of the 11 reversals at K=32K=32 with the same signs. A more lenient answer parser leaves every count unchanged, and with Benjamini-Hochberg control five cross-family reversals are significant (Appendix D). On MMLU-Pro, both significant reversals have the same signs in each half and stay significant with the lenient parser, with Benjamini-Hochberg control and when invalid outputs abstain. The programs, outputs and parsed answers of the new banks are in the updated supplementary material and in the anonymous repository linked in the abstract.

AuthorsOpenReview ↗

Response to Reviewer EusS (Part 2/3)

W1/Q1, continued. Cost of choosing by pass@KK. A reversal does not need to be significant to be costly. On MATH-500, the system with the highest pass@KK at K≥64K\ge64 is Qwen-7B Base. When invalid outputs count as votes (as in Table 1), its plurality accuracy is 17.6 to 18.8 points below the best system's, with intervals that exclude zero. Most of this comes from Base's invalid outputs, which win many votes, so the validity rule is part of what decides the ranking. Among the seven new SuperGPQA models, pass@32 picks Llama-3.1-8B, whose plurality accuracy is 6.3 points below Qwen2.5-7B's ([3.2,9.4][3.2,9.4]). On that bank, choosing by pass@32 has a significant cost in 7 of the 51 cross-family pairs. On MMLU-Pro, pass@32 again picks Llama-3.1-8B, now 9.8 points below Qwen2.5-7B ([7.2,12.4][7.2,12.4]).

Reversals are not specific to the effort settings of one model. They appear across families, benchmarks and budgets, and choosing by pass@KK has a measurable cost.

W2/Q2. How many samples are needed to estimate the largest wrong mass ww, and how do the bounds behave when it is estimated?

The number of samples needed to estimate the largest wrong mass does not depend on how many distinct answers a model can produce, and the bounds stay accurate when it is estimated. In our experiments, 16 samples per question were enough to predict plurality accuracy at a budget four times larger.

Theory (Section 2.2, Appendix C.3). Let w^\hat w be the largest frequency of a wrong answer among nn samples. Lemma 1 shows that Pr⁡[∣w^−w∣≥ε]≤(1+1/w)e−2nε2\Pr[|\hat w-w|\ge\varepsilon]\le(1+1/w)e^{-2n\varepsilon^2} for every ε≥1/n\varepsilon\ge1/n, so n≥log⁡{(1+1/w)/δ}/(2ε2)n\ge\log\lbrace(1+1/w)/\delta\rbrace/(2\varepsilon^2) samples estimate ww to within ε\varepsilon with probability 1−δ1-\delta. The number of distinct answers, seen or unseen, does not appear: the union over answers costs a factor 1/w1/w instead of the number of answers. A large or unknown answer space does not make ww hard to estimate.

Proposition 3 turns this into bounds on plurality accuracy that hold for every KK at once with probability at least 1−4δ1-4\delta. It combines four one-sided confidence bounds, on the reference-answer mass, the largest competing mass, the total correct mass and the largest wrong mass. The bound on the competing mass uses only the count of the true strongest competitor, so it remains valid when that answer never appears in the sample. A binomial term extends the guarantee to benchmark averages.

Experiments (Section 4.2, Table 2, Appendix E). We re-estimate everything from nn of the saved responses per question and predict plurality accuracy at K=64K=64, over repeated random draws.

  • On SuperGPQA, w^\hat w is biased upward by 2.1, 0.9 and 0.4 points at n=8n=8, 16 and 32, and its certified interval contains the full-pool value in every draw.
  • At n=16n=16, the bias-corrected range is the most accurate rule on every bank: within 0.4 points on SuperGPQA, 1.6 on Scorio Trace and 1.4 on MATH-500. Plurality measured at n=16n=16 misses by 4.4 points on MATH-500, where accuracy keeps rising with KK, and extrapolated pass@KK misses by 11 to 26 points.
  • A two-level bootstrap (questions, then responses within each question) gives intervals that contain the 64-sample accuracy in every draw at n=16n=16. Every ordering of two systems that they certify is correct. On average they certify 5.3 of 6 pairs on SuperGPQA, 122 of 190 on Scorio Trace and 15 of 21 on MATH-500.
  • These orderings are useful in practice for screening. From 16 samples per question they drop 3 of the 4 SuperGPQA systems, 6 of the 7 MATH-500 systems and 15 to 18 of the 20 Scorio Trace systems, and never the system that is best at K=64K=64. Even if every remaining system is then evaluated directly at K=64K=64, this saves 56 to 64% of the generated samples (Section 4.2, new Table 22).
  • On the new 11-system SuperGPQA bank, 8 of the 32 responses per question predict the 32-sample accuracy to within 0.4 points. The intervals cover it in every draw and certify 51 of the 55 pairs, all correctly. On the 11 pairs that pass@32 orders the wrong way, they certify the plurality order in 94% of draws and never the pass@32 order.
  • On MMLU-Pro, 4 of the 32 responses per question predict the 32-sample accuracy to within 0.4 points, and the intervals cover it in every draw. An interval around plurality on the same 4 responses covers it in only 1% of draws.

To answer Q2 directly: in these banks, 16 samples per question were enough to predict plurality accuracy at four times that budget. In every draw, the intervals contained the true value and every certified ordering was correct. On MATH-500, the same 16 samples also predict the accuracy at eight and sixteen times that budget, to within 1.6 and 1.5 points (new Table 25).

AuthorsOpenReview ↗

Response to Reviewer EusS (Part 3/3)

Q3. Does GRPO change how concentrated or diverse the wrong answers are, and how does that affect the gap?

Yes. GRPO makes wrong answers more concentrated. This has a small, measurable cost for plurality voting, and the gain in correct mass offsets it.

On MATH-500 we compare each GRPO model with its initialization in three families, three seeds each: Qwen2.5-Math-1.5B with full fine-tuning, and Qwen2.5-Math-7B and Mistral-7B with LoRA (Section 4.3, Table 3, Appendix F). In all nine runs, GRPO raises the probability of the most common wrong answer (by 0.9, 3.7 and 0.9 points on average) and the share of the wrong mass that this answer holds (by 1.0, 16.8 and 1.2 points). For Qwen-7B the number of distinct wrong answers per problem falls from 50 to 22. Rejection-sampled SFT does the opposite and spreads wrong mass over more answers. On OlympiadBench (Qwen-1.5B, 64 responses per problem) the most common wrong answer also gains mass in every seed (+0.6+0.6 points, 95% interval [0.2,1.1][0.2,1.1]).

To see how this moves the gap, we predict plurality accuracy at K=64K=64 from each answer histogram and split its change in two. The mass effect applies the trained correct and invalid masses but keeps the initial shape of the wrong answers. The shape effect is the rest of the change, which comes only from how the wrong mass is shared. On MATH-500 the shape effect costs 0.3, 1.6 and 1.2 points, and the mass effect adds 1.3, 3.3 and 1.0 points. The mass effect offsets the shape effect, so the oracle-plurality gap moves by at most about two points while pass@1 rises. On OlympiadBench, GRPO mainly turns invalid outputs into valid answers, and its plurality gain (+0.8+0.8 points) is a mass effect.

This matches Proposition 2: concentrating wrong mass cannot raise plurality accuracy. It also explains why answer entropy alone says little about selection. Lower entropy helps only when the mass moves to the correct answer, not when it moves to a shared wrong one. The same decomposition can be applied to any training method.

We hope these results answer your questions. If they resolve your concerns, we would be grateful if you would take them into account in your final assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion

We thank the Area Chair and the three reviewers for the discussion. Vm8u calls the question "important and timely", and FbAA calls the sharp identification bounds "the strongest independent contribution". FbAA also finds the empirical checks careful, and Vm8u finds the experiments "quite extensive and thorough".

The main points of the discussion, and where the revised PDF addresses them:

  1. Reversals across model families (EusS; Section 4.1, Table 1). We counted ranking reversals in every multi-sample bank available to us: 44 systems from 11 model families. On SuperGPQA, 9 of the 51 cross-family pairs reverse at K=32K=32, and four of these stay significant after a simultaneous correction over every comparison in the bank.
  2. Use of the bounds in practice (EusS, FbAA; Section 4.2, Appendices E.2 and E.5). From 16 responses per question, the bounds predict plurality accuracy at much larger budgets, up to K=256K=256 on MATH-500, and screening with them saves 56 to 64% of the generated responses. At the same budget, this screening chose the best system more often than pass@1, low-budget plurality and successive halving on both reasoning banks, and it never discarded the best system early. EusS wrote that our replies addressed the additional concerns and has no further questions. FbAA highlighted the new experiments on the practical utility of the bounds and the expanded deployment evaluations.
  3. Presentation (Vm8u). The abstract, introduction and conclusion state the findings in words, Proposition 2 is written as equations, the notation of Section 2 is collected in Definitions 1 to 3, and each result names the appendix that proves it. Vm8u found the revised PDF more accessible and raised the rating.
  4. Relation to PETS and Feng et al. (FbAA; Section 5). Both works study how a vote approaches one model's population mode under a given distribution. Our results concern selection across systems: an impossibility result for selection from oracle curves (Proposition 1), sharp identification bounds (Proposition 2) and their certification from a few samples (Lemma 1 and Proposition 3).

We are grateful to all three reviewers for their time and suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.